Packet transmission system and method

The network gateway system optimizes data transmission by managing multiple connections with varying characteristics, addressing the 'head of line blocking' issue in TCP, enhancing throughput and reliability.

JP2024166237AActive Publication Date: 2024-11-28DEJERO LABS
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2024153037
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2017-09-14
Filing Date
2024-09-05
Publication Date
2024-11-28
Estimated Expiration
2037-12-21

AI Technical Summary

Technical Problem

Combining data connections with significantly different latencies and characteristics, such as those found in various networks, leads to inefficiencies and reduced transmission speed due to the 'head of line blocking' problem in TCP, where acknowledgment and re-request processes hinder efficient data transmission.

Method used

A network gateway system that manages multiple network connections by monitoring transmission characteristics, routing packets based on latency, bandwidth, and packet sequence to optimize data flow, using a processor to control buffers and ensure packets are transmitted efficiently across different networks.

Benefits of technology

The system enhances transmission performance by maintaining packet sequence and reducing latency variations, thereby improving throughput and reliability across diverse network connections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166237000001_ABST
    Figure 2024166237000001_ABST
Patent Text Reader

Abstract

To provide a network gateway for routing data flows across a plurality of network connections, a method therefor, and a computer readable medium.SOLUTION: Each of multipath gateways 104, 108 is a network gateway for routing data flows across a plurality of network connections 106. The multipath gateways each include a plurality of network interfaces for transmitting data over the plurality of network connections, and at least one processor configured for: transmitting a sequential burst of packets across the network interface; on the basis of, timestamps recorded when packets in the sequential burst of packets are received at a receiving node, and a size of the packets, generating a bandwidth of the network interface; and routing a data flow of sequential packets across the plurality of network connections on the basis of, the generated bandwidth.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is a nonprovisional application and claims all benefit, including priority, of U.S. patent application Ser. No. 62 / 437,635, entitled "PACKET TRANSMISSION SYSTEM AND METHOD," filed Dec. 21, 2016, and (2) U.S. patent application Ser. No. 62 / 558,610, entitled "PACKET TRANSMISSION SYSTEM AND METHOD," filed Sep. 14, 2017. Both of these applications are incorporated by reference in their entirety.

[0002] The present disclosure relates generally to the field of electronic communications, and more specifically to improved packet transmission for protocols having competing requirements such as latency, throughput, jitter and packet sequencing, and to the combining of multiple data connections to achieve improved transmission speeds and reliability in situations where individual data connections may be lossy, unreliable, or have significantly different or variable latency, transmission rates or packet loss. [Background technology]

[0003] Transmission of electronic data over wired and wireless networks can be accomplished using a variety of packetization methods. For example, packets can be provided under a variety of protocols, such as User Datagram Protocol (UDP) and Transmission Control Protocol (TCP). For UDP packets, packets are provided on a "best effort" basis, meaning that there is no acknowledgement of receipt of packets sent at the transport layer for each packet. Thus, efficient transmission at the transport layer is simplified, leaving it up to the application to handle out-of-order or lost packets reliably. For TCP packets, error checking is performed as part of the transport protocol, and TCP packet sequence numbers and acknowledgements are utilized to ensure that applications receive all data reliably and in order. Lost packets can be explicitly retransmitted, for example.

[0004] With TCP, receipt of out-of-order packets can affect transmission efficiency due to the acknowledgement and re-request process built into TCP, since congestion control algorithms consider out-of-order packets to be caused by network transmission errors and slow down the transmission rate to compensate for these perceived errors. This issue is particularly important when simply combining data connections with significantly different latencies, which can result in a significant decrease in transmission rate (known as the "head of line blocking" problem). Summary of the Invention

[0005] It is technically difficult to bond different networks and / or connections together. A single network and / or connection is particularly prone to transmission errors, signal loss and lack of redundancy. Bonded networks and / or connections are particularly useful in various scenarios, such as news gathering and reporting, that require reliable high-bandwidth transmission through various environments where individually reliable connections may not be readily available (e.g., rural areas, high altitudes) or, if available, may be congested (e.g., sporting events) or underperforming (e.g., after natural disasters).

[0006] Applicants have developed, in some embodiments, improved data transmission management solutions that apply non-traditional approaches to utilizing technical features of network technologies in establishing hybrid network connections. As described in further detail, further technical uses and approaches require measures to establish viable connections (e.g., to avoid "head of line" blocking problems as described above) that TCP is particularly difficult to combine. These improved hybrid connections can be used, for example, to transfer high bandwidth video, voice, or other data. For example, in disaster situations, the improved hybrid connections can be utilized, among other things, to establish emergency communications, provide ground coverage, and enable emergency telemedicine.

[0007] A network gateway for controlling data flow through a network connection is described herein along with corresponding methods, apparatus, and computer-readable media. The gateway may in some embodiments be a physical hardware device with components such as a processor, an interface, a bus, a power supply, memory (ROM, RAM, Flash), etc., configured to receive and route data packets according to an instruction set representative of software and embedded firmware in the form of program code or computer logic.

[0008] The network gateway may in one example be a specialized computing device specifically optimized for bonding TCP connections, as described in some examples herein. The network gateway may in other examples be implemented with a processor and / or other computing hardware as part of a larger system. The network gateway may be a single device, or in some cases, multiple devices operating in concert. A corresponding de-bonding server may be provided to reconstruct the signal upon receipt.

[0009] In one embodiment, the network gateway is configured to route a data flow through a plurality of network connections, the network gateway comprising: a plurality of network interfaces for transmitting data through the plurality of network connections; and at least one processor configured for monitoring time-varying network transmission characteristics of the plurality of network connections; analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class defining at least one network interface requirement for the data flow; and routing packets in the data flow through the plurality of network connections based on the data flow class and the time-varying network transmission characteristics.

[0010] In another embodiment, the network gateway comprises a plurality of network interfaces including a first network interface for transmitting data through a plurality of network connections, the network gateway comprises a processor configured for transmitting sequential bursts of packets through the first network interface, the processor generates a bandwidth for the first network interface based on timestamps recorded when packets in the sequential burst of packets are received at a receiving node and sizes of the packets, and routes a data flow of the sequential packets through the plurality of network connections based on the generated bandwidth of the first network interface.

[0011] In one aspect, generating the bandwidth for the first network interface includes generating the bandwidth based on timestamps of packets in the burst that are not coalesced with a first or last packet in the burst.

[0012] In another aspect, generating the bandwidth of the first network interface includes replacing a receive timestamp of a particular packet in a burst with a transmit timestamp of a packet sent after the particular packet.

[0013] In another aspect, when the receiving node processes packets received at regular intervals, generating the bandwidth of the first network interface includes (i) generating a lower bandwidth value by using a reception timestamp of a packet in a burst selected as an end packet in the bandwidth determination, and (ii) generating an upper bandwidth value by replacing the reception timestamp of the packet in the burst selected as the end packet with a reception timestamp of a packet in the burst preceding the end packet.

[0014] In another aspect, the processor is configured to provide, for each packet of sequential packets in the data flow, the packet for routing through one of the multiple network connections based on the monitored latencies of the multiple network connections and the network connections of other packets in the data flow such that the packets in the data flow arrive at a destination node in a desired sequence.

[0015] In another aspect, the desired sequence is the original sequence of sequential packets in the data flow.

[0016] In another aspect, the desired sequence is a sequence that includes at least one misordering of packets that does not trigger a retransmission of packets within the sequence.

[0017] In another aspect, the processor is configured for receiving a packet from a source interface for routing to a destination node over a plurality of network connections, sending an acknowledgment to the source interface before routing the packet to the destination node, and storing the packet in at least one buffer before the packet is routed to the destination node.

[0018] In another aspect, the at least one processor is configured for dynamically controlling a size of the at least one buffer based on a bandwidth-delay product associated with the multiple network connections.

[0019] In another aspect, the at least one processor is configured for controlling the sending and storing of packet acknowledgments based on monitoring transmission characteristics of the plurality of network connections and unequal distribution in receiving the data flow of sequential packets.

[0020] In another aspect, the at least one processor is configured to provide the packets based on at least one of a bandwidth of the first data connection and a reduction in the number of network connections through which the data flow is routed.

[0021] In another aspect, the at least one processor is configured to route packets for a plurality of data flows, where the at least one processor is configured for providing packets of the plurality of data flows for routing through one of a plurality of network connections based on classifications of the plurality of data flows.

[0022] In another embodiment, the classification of one of the multiple data flows is changed after a threshold volume of data is routed.

[0023] According to another aspect, there is provided a data transmission management device ("gateway") suitable for controlling the routing of data packets belonging to a plurality of flows through a plurality of simultaneous data connections, each data connection carrying a plurality of packets belonging to a plurality of flows, where at least one of the plurality of flows has data packets transmitted substantially sequentially, the data transmission management device comprising: a buffer manager configured to receive flows (sequences of input data packets) from one or more input data connections, the buffer manager storing the input data packets in one or more buffers extracted for routing, each input data packet being associated with a corresponding sequence number representing its order in the flow or sequence; a connection control device configured to interface with a plurality of data connections, each of the data connections having different transmission characteristics; and a scheduler configured to control operation of the buffer manager by generating instructions to be executed by the buffer manager to control the routing of input data packets from the one or more buffers via the plurality of data connections, the routing determining for each respective input data packet (i) a corresponding data connection through which the respective input data packet is transmitted, and (ii) a connection-specific timing or sequence in which the respective input data packet is transmitted on the corresponding data connection.

[0024] In another aspect, the apparatus further comprises a network characteristic monitoring unit configured to monitor transmission characteristics over each of the plurality of data connections monitored together or individually.

[0025] In another aspect, the apparatus further comprises an operations engine configured to control operation of the scheduler by generating instructions executed by the scheduler to control routing of input data packets stored in the one or more buffers, the routing being controlled based on transmission characteristics monitored over the multiple data connections, the operations engine configured to modify the routing based on an input data set providing information regarding at least one of throughput, packet loss, latency, Explicit Congestion Notification (ECN) cost, reliability, jitter, user / administrative preferences and other network connection constraints.

[0026] In another aspect, the apparatus further comprises a sequencer configured to reconstruct input data packets received by the data transmission manager based on respective sequence numbers of the input data packets.

[0027] In another aspect, the apparatus further comprises a flow identification engine designed to identify flows based on unique characteristics and provide flow information to the action engine and the sequencer.

[0028] In another aspect, the apparatus further comprises a flow classification engine configured to classify input data flows and their requirements and, based on the characteristics and requirements of the flows, provide this information to the action engine for modification of the routing of the input data flows and to the sequencer for modification of the manner in which data packets are reassembled.

[0029] In another aspect, the buffer manager is further configured to provide excess buffering to compensate for variability in the transmission characteristics of the multiple data connections and unequal distribution of transmissions ("bursty transmissions").

[0030] In another aspect, the operation engine is further configured to determine a bandwidth-delay product for each data connection of the plurality of data connections, the bandwidth-delay product being used to modify the routing as controlled by the scheduler.

[0031] In another aspect, the action engine is further configured to perform bandwidth estimation when incomplete information is available.

[0032] In another embodiment, the one or more types of data traffic (flows) include at least one of FTP, DNS, HTTP, SSH, and SSL / TLS.

[0033] In another aspect, the operational engine is configured to determine which of the one or more data flows are latency sensitive, and for the one or more data flows that are latency sensitive, preferentially maintain routing of corresponding incoming data packets over a data connection of the multiple data connections having lower and / or similar latency characteristics. In this scenario, the operational engine may also prevent routing of non-latency sensitive flows over the multiple data connections to prevent excessive buffering / buffer bloat.

[0034] In another aspect, the operation engine is further configured to modify header information of known protocols to optimize their performance over multiple data connections. For example, if the multiple data connections include one or more high-latency terrestrial satellite connections, the optimization engine may adjust the advertised TCP window size to allow the flow to take advantage of the high bandwidth-delay product (known as TCP acceleration). These changes may further require the buffer manager to buffer additional packets on a per-flow basis. Another example is the modification of the TCP MSS header option to match the smallest MSS (maximum segment size) supported by the multiple data connections to prevent fragmentation of data packets.

[0035] In another aspect, there is provided a data transmission management method for controlling routing of data flows (comprised of data packets) through a plurality of data connections to provide improved transmission performance in which at least one of the plurality of data flows is transmitted substantially sequentially, the data transmission management method including: receiving, from one or more input data sources, a stream or sequence of input data packets associated with corresponding sequence numbers representing an order in the data flows or sequences, and storing in one or more buffers extracted for routing; interfacing with a plurality of data connections, each of the data connections having different transmission characteristics; generating routing instructions by a scheduler that controls the one or more buffers to route the input data packets through the plurality of data connections, the instructions providing for each respective input data packet (i) a corresponding data connection through which the respective input data packet is transmitted and (ii) a connection-specific timing or sequence in which the respective input data packet is transmitted on the corresponding data connection; and routing the input data packets in accordance with the routing instructions.

[0036] In another aspect, a non-transitory computer-readable medium is provided that stores instructions that, when executed, cause a process to perform steps of a data transmission management method for controlling routing of data flows (comprised of data packets) through a plurality of data connections to provide improved transmission performance in which at least one of the plurality of data flows is transmitted substantially sequentially, the method including: receiving, from one or more input data sources, a stream or sequence of input data packets associated with corresponding sequence numbers representing an order in the data flows or sequences, and storing in one or more buffers extracted for routing; interfacing with a plurality of data connections, each of the data connections having different transmission characteristics; generating routing instructions by a scheduler and controlling the one or more buffers to route the input data packets over the plurality of data connections, the instructions providing, for each respective input data packet, (i) a corresponding data connection through which the respective input data packet is transmitted, and (ii) a connection-specific timing or sequence in which the respective input data packet is transmitted on the corresponding data connection; and routing the input data packets in accordance with the routing instructions.

[0037] In various further aspects, the present disclosure provides corresponding systems and apparatus, as well as logical structures, such as sets of machine-executable coded instructions, for implementing such systems, apparatus and methods.

[0038] In this regard, before describing at least one embodiment in detail, it is to be understood that the embodiment is not limited in application to the details of construction or the arrangement of components set forth in the following description or illustrated in the drawings, and it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.

[0039] Many additional features and combinations of the embodiments described herein will be apparent to those of ordinary skill in the art after reading this disclosure.

[0040] The embodiments are shown by way of example in the drawings, and it should be expressly understood that the description and drawings are given by way of example only and are intended to aid understanding.

[0041] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0042] [Figure 1] FIG. 1 is a block schematic diagram of a system having two gateways linked by N data connections, each with a buffer manager, an action engine, a connection controller, a network characteristic monitoring unit, a flow classification engine (handling flow identification and classification), a scheduler and a sequencer, with each gateway connected to a specific endpoint, according to some embodiments. [Diagram 2] FIG. 2 is a block schematic diagram illustrating a scenario in which an Internet user connects to the Web via a combination of gateways utilizing transport links with different network characteristics, according to some embodiments. [Diagram 3] FIG. 3 is a block schematic diagram illustrating the operation of a sequencer by demonstrating the ordering of six packets according to an example based on the network of FIG. 2, in accordance with some embodiments. [Figure 4] FIG. 2 is a block schematic diagram illustrating a scenario in which a scheduler is configured to consider the latency of a data connection, according to some embodiments. [Diagram 5] FIG. 1 is a block diagram illustrating a bandwidth-delay product (BDP) according to some embodiments. [Figure 6] FIG. 1 is a block schematic diagram illustrating a system suitable for operation based on at least the dimensions (size and sensitivity to latency) of flow requests according to some embodiments. [Figure 7] 2 is a block schematic diagram illustrating the operation of a network characteristic monitoring unit according to some embodiments. [Figure 8A] 1 is an illustration of an example bandwidth estimation method illustrating an example scenario in which a transmitter transmits a burst of eight packets, according to some embodiments. [Figure 8B] 1 is an illustration of an example bandwidth estimation method that illustrates non-idealities in practical implementations, according to some embodiments. [Figure 8C] 1 is an illustration of an example bandwidth estimation method illustrating an example technique for correcting errors due at least in part to interrupt latency, according to some embodiments. [Figure 8D] 8C for correcting errors due at least in part to interrupt latency under different conditions, according to some embodiments. [Figure 8E] 1 is an illustration of an example bandwidth estimation method illustrating an example in which interrupt arbitration techniques are considered, according to some embodiments. [Figure 9] FIG. 1 is a block schematic diagram illustrating a system utilizing satellite offload (single input) according to some embodiments. [Figure 10] FIG. 2 is a block diagram illustrating an example of mixing with multiple inputs according to some embodiments. [Figure 11] FIG. 13 is a block diagram illustrating a scenario configured for non-bonded disaster recovery / load balancing use (each flow is sticky to a specific data channel with no bonding at all since no buffers are needed on the other side, i.e., there is no second gateway), according to some embodiments. [Figure 12] FIG. 1 is a block diagram illustrating a system configured for a combined disaster recovery / load balancing application in which traffic is combined and routed through a second gateway according to some embodiments. [Figure 13] 1 illustrates a series of multi-path gateways connecting two or more endpoints according to some embodiments. [Figure 14] 1 illustrates two multi-homed sites connected to the Internet in a traditional Border Gateway Protocol (BGP) transit / peering configuration according to some embodiments. [Figure 15] 1 illustrates the same two multi-homed sites connected with a multi-path gateway that allows granular control of how both outbound and inbound traffic between the two sites is routed, according to some embodiments. [Figure 16] 1 is an illustration of a computing device according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] Embodiments of the method, system and apparatus are described with reference to the drawings.

[0044] The Transmission Control Protocol (TCP) utilizes a three-way handshake to establish a connection and is therefore a reliable protocol (as opposed to the User Datagram Protocol (UDP), which is a connectionless protocol). These protocols are described, for example, in RFC 793 and RFC 7323, which are incorporated herein by reference.

[0045] TCP requires a more complex implementation and is suitable for guaranteed in-order delivery of packets. The connection must first be established between sender and receiver before any data can be transferred, and the connection is often terminated at the end of the data transmission (to free up allocated resources).

[0046] The initial handshake to establish a connection provides an initial sequence number (eg, an arbitrary sequence number) that is used during a data transmission to identify the order in which data is provided during the data transmission.

[0047] During data transmission, data generated by an application is segmented into data packets, and these data packets or segments have information prepended to them (e.g., a "header") when they are encapsulated for transmission. The headers contain information used during transmission indicating, for example, a sequence number associated with each segment of data. The sequence numbers are used by a receiving device so that the segments of data can be reordered and reconstructed even if they are received out of order, lost (e.g., a retransmission may be requested), delayed, corrupted, etc.

[0048] The TCP method employs additional processes to help reduce transmission problems; these processes include flow control, error detection, retransmission protocols, congestion control, and selective acknowledgment.

[0049] In some scenarios, improved data communication performance may be obtained through the use of multiple data connections or networks operating in concert. These data connections or networks may be considered to be "coupled" to one another such that the networks are logically joined or coupled. Two or more networks operating in concert may perform one or more communication functions, such as transmitting data, providing error correction, among others. In this context, one or more networks need not necessarily be separate networks, but may include different channels through the same network, different data connections, etc.

[0050] A combined network is desirable because there may be improved data transmission characteristics (such as increased reliability and increased speeds) that can be obtained by advantageously taking advantage of the combined characteristics of each network or by using the diverse characteristics and capabilities of the networks, such as cellular networks, Ethernet networks, WiFi, and satellite networks, among others, which have different characteristics and qualities.

[0051] For example, a particular network connection (e.g., path, channel) may be ideal for a particular type of communication (e.g., low latency but low bandwidth for error detection and control signals), while another network connection may be ideal for another type of communication (e.g., high bandwidth for high throughput data transfer). In some embodiments, network coupling may be provided such that the endpoints (e.g., transmitter, receiver) need not be aware of the communication path or technique being utilized for communicating data.

[0052] Because TCP is the protocol used for a significant number of networks and traffic, it would be particularly desirable to have a combined TCP solution with improved transfer characteristics, however, combined TCP solutions are difficult to implement.

[0053] A challenge with TCP (particularly because it requires an acknowledgment of each packet) is that TCP connections are technically difficult to "bond" together, especially when the data connections are provided over networks that have significantly different characteristics, such as different latencies. These difficulties affect the efficiency of communication systems that utilize bonding for networked TCP communication.

[0054] Data is transmitted to one or more endpoints through a combination of data connections (e.g., through different networks), and the transmission of the data is facilitated so that the data can be recombined (e.g., by a "decombining" system) and the original data can be regenerated from the transmitted data.

[0055] Such "bonding" of connections and allocations of functionality is described, for example, in Applicant's US Pat. No. 8,873,560, which is incorporated herein by reference.

[0056] There are particular technical challenges that must be overcome in the context of the use of bonding with TCP, since the bonding process typically requires segmentation of the data packets that comprise the data flow transmitted through the bonded network or data connection. Different types of traffic (e.g., voice data, control packets, video data, error control) are affected differently by ordering requirements.

[0057] Technical considerations for systems implementing bonded networks for the transfer of data include, but are not limited to, ease of binding / debinding (e.g., required computational resources and total computation time compared to available computational resources), transfer latency, packet loss, redundancy, security, maximum throughput, average throughput, and congestion management.

[0058] Simple round-robin solutions using multiple network or data connections typically only provide benefit if the connections have identical or nearly identical characteristics (e.g., latency, capacity). If one connection has significantly higher latency, all other packets will be delayed, and potentially treated as lost or delayed by the sender and receiver, thereby forcing the sender to reduce its congestion window (cwnd), resulting in reduced throughput.

[0059] If one connection has a significantly higher capacity, then when the lower capacity connection reaches its limit, all other packets sent on that connection will be lost.

[0060] Thus, simple round robin solutions typically help with load balancing and redundancy, but at the expense of throughput when connections have different characteristics.

[0061] Multipath TCP (RFC6824) is a transport layer solution commonly used to combine wireless networks such as WiFi and cellular. As described below, the system of some embodiments can provide advantageous performance compared to other solutions such as Multipath TCP (MPTCP) for the following reasons. 1) MPTCP requires both endpoints (client (C) and server (S)) to be able to process using MPTCP. Both endpoints know about the multiple paths, allowing independent congestion control algorithms to run on each of them, and the endpoints do not encounter the problems noted above with standard TCP stacks. In contrast, the system disclosed herein, in some embodiments, works when the endpoints are not configured for use with MPTCP, or the multiple paths are not directly connected to the endpoints (i.e., the multiple paths are known only to intermediate hops between the endpoints). 2) MPTCP attempts to be fair when competing with other non-MPTCP flows over congested links ("concatenated congestion control"). Without Explicit Congestion Notification (ECN), which is not widely deployed, an MPTCP system must guess whether any congestion it sees is on a link shared by all subflows. It often guesses incorrectly, resulting in lower than expected throughput.

[0062] As illustrated in Figure 1, system 100 is shown configured to utilize an improved scheduling technique on the transmitter of the system and to use a buffering system on the receiver side that provides packet ordering. The components shown are, in one embodiment, hardware components that are configured to interact with one another. In another embodiment, the components are not separate components, and two or more of the components may be implemented on a particular hardware component (e.g., a computer chip that performs the functions of two or more components).

[0063] In some embodiments, the components reside on the same platform (e.g., the same printed circuit board) and system 100 is a single device that can be moved and connected to a data center / field carry-able device (e.g., a rugged mobile transmitter), etc. In other embodiments, the components are distributed and may not all be located in close proximity, but rather communicate electronically through telecommunications (e.g., processing and control is not performed locally, but rather by components that reside in a distributed resource environment (e.g., the cloud).

[0064] In mobile scenarios, providing combined connectivity is particularly desirable when signal quality, network availability, network quality, etc. are sub-optimal (e.g., professional reporting / video production may take place in locations without strong network infrastructure).

[0065] Shown are many different data connections 106 (e.g., "paths") representing one or more networks (or network channels) labeled as Connection 1, Connection 2...Connection N. There may be multiple data connections / paths through a single network or multiple data connections that may use one or more networks.

[0066] The system 100 may be configured to communicate with various endpoints 102, 110 or applications that do not need to have any knowledge regarding the multiple paths / connections 106 used to request and receive data (e.g., the endpoints 102, 110 may function independently of the paths or connections 106). Received data may for example be reconstructed such that the original transmission can be recreated from the contributions of the different paths / connections 106 (an example usage scenario is the regeneration of video by a receiver configured to be plugged into a server rack in a data center facility and integrated with an existing broadcast infrastructure to provide improved networking capabilities).

[0067] The system 100 receives input (data flow) from a source endpoint 102 and schedules improved distribution of data packets over various connections 106, then sequences the data packets at the other end of the system 108 before transmission to a destination endpoint application 110. In doing so, the system 100 is configured to increase bandwidth to approach the sum of the maximum bandwidth of the various paths available. Compared to using a single connection, the system 100 also provides improved reliability, which can be an important consideration in time-limited and sensitive scenarios such as live event coverage when the event is taking place. In these events, there may be high signal congestion (e.g., sporting events) or unreliability along one or more paths (e.g., news coverage following a natural disaster).

[0068] In various embodiments, both the scheduler and sequencer may be provided from a cloud computing implementation, or at the endpoint (before the data is consumed by an application at the endpoint), or various combinations thereof.

[0069] The system 100 may be tuned to optimize and / or prioritize, among other things, performance, best latency, best throughput, minimum jitter (variation in latency on packet flows between two systems), connection cost, combination of connections for a particular flow (e.g., if the system 100 has information that a transmission (data flow) is of content type X, the system 100 may be configured to use only data connections with similar latency, while content type Y may allow a wider mix of data connections (or may require a larger net capacity that can only be achieved by a combination of data connections)). This tuning may be provided for the system in general, or may be specific to each flow (or set of flows based on location, start point or endpoint owner or a combination thereof, transmission time, set of available communication links, security required for transmission, etc.).

[0070] The system 100 may be generally bidirectional in that each gateway 104, 108 generally has a scheduler and sequencer for handling TCP traffic (or UDP traffic, or a combination of TCP and UDP traffic, or any kind of general IP traffic), although in some embodiments only one gateway may be needed (see, for example, FIG. 11, where simple disaster recovery is shown as a service).

[0071] The scheduling portion of the system features a novel approach to estimating the bandwidth of a given connection (see, for example, FIG. 8).

[0072] System 100 can be utilized for a variety of scenarios, for example as a failover or addition to an existing Internet connection (e.g., a VoIP telephone system, or a corporate connection to the Web), whereby additional networks (or paths) are added to seamlessly replace a dropped primary Internet connection, or where combining is used to include only more expensive networks when the primary Internet connection is saturated (as shown in Figures 11 and 12). Another use is to provide a means of maximizing the use of high cost (often sunk costs) highly reliable data connections such as satellites by allowing traffic to be offloaded to other data connections with different attributes (Figures 9 and 10).

[0073] In some embodiments, the system is a network gateway configured for routing data flows through multiple network connections.

[0074] 1 provides a schematic of a system having two gateways 104 and 108, each comprising a buffer manager 150, an action engine 152, a connection controller 154, a flow classification engine 156 (responsible for flow identification and classification), a scheduler 158, a sequencer 160 and a network characteristic monitoring unit 161, and linked by N data connections 106, with each gateway connected to a particular endpoint 102, 110. The reference letters A and B are used to distinguish the respective components of the two gateways 104 and 108.

[0075] Each gateway 104 and 108 is configured to include multiple network interfaces for transmitting data through the multiple network connections, and is a device (e.g., comprising configured hardware, software or embedded firmware) comprising a processor configured for monitoring time-varying network transmission characteristics of the multiple network connections, analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, where the data flow class defines or is otherwise associated with at least one network interface requirement of the data flow, and routing packets in the data flow through the multiple network connections based on the data flow class and the time-varying network transmission characteristics.

[0076] The buffer manager 150 is configured to configure buffers within the gateway that are configured to more efficiently manage traffic (both individual flows and combinations of multiple simultaneous flows passing through the system). In some embodiments, the buffer manager is a separate processor. In other embodiments, the buffer manager is a computational unit provided by a processor that is configured to perform, among other activities, buffer management (150).

[0077] The action engine 152 is configured to apply one or more deterministic methods and / or logical operations based on the received input data set (e.g., feedback information, network congestion information, transmission characteristics) either per user / client, destination / server, connection (e.g., latency, throughput, cost, jitter, reliability), or flow type / request (e.g., FTP vs. HTTP vs. streaming video) to inform the system about constraints that apply to the combined connections. For example, the action engine 152 may be configured to restrict a particular type of flow to a particular connection or set of data connections based on cost in one example, while for different users or flow types, reliability and low latency may be more important. For example, different conditions, triggers, methods may be utilized depending on one or more elements of the known information. The action engine 152 may be located, for example, in the same or different processor as the buffer manager 150.

[0078] The action engine 152 may be configured to generate, apply or otherwise manipulate or use one or more rule sets that determine the logical operations that control the routing through the N data connections 106 .

[0079] The flow classification engine 156 is configured to evaluate each data flow received by the multi-path gateway 104 for transmission and is configured to apply a flow classification technique to determine, if not already known, the type of traffic being transmitted and its requirements. In some embodiments, deep packet inspection techniques are configured to make the determination. In other embodiments, the evaluation is based on heuristic methods or on data flows that are marked or labeled as they are generated. In other embodiments, the evaluation is based on rules provided by a user / administrator of the system. In other embodiments, a combination of methods is used. The flow classification engine 156 is configured for use with one or more network interfaces simultaneously and may be implemented using electronic circuits or processors.

[0080] Flow identification can be performed by analyzing the information provided within the packets of the data flow, for example by examining packet header information (e.g., source / destination IP, transport protocol, transport protocol port numbers, DSCP flags, etc.).

[0081] As provided in some embodiments, different levels of identification may be performed, for example the contents of the packet body may be further inspected, for example using deep packet inspection techniques.

[0082] Once the flows have been identified, classification may involve categorizing the flows based on their requirements. Examples of classification include: 1. Low latency, low to moderate jitter, out-of-order packets, high bandwidth (live HD video broadcast); 2. Especially low latency, low to moderate jitter, out-of-order packets, medium bandwidth (Skype™, FaceTime™) (jitter is problematic for real-time communication as it can cause artifacts or degradation of communication), 3. Low latency, low to moderate jitter, out-of-order packets are OK, low bandwidth (DNS, VoIP), 4. Low latency, no jitter requirements, in-order packets preferred, low bandwidth (interactive SSH) 5. No latency requirements, no jitter requirements, in-order packets preferred, medium to high bandwidth (e.g., SCP, SFTP, FTP, HTTP), and 6. No latency requirements, no jitter requirements, in-order packets preferred, no bandwidth requirements, sustained large data transfers (e.g. file / system backups), etc. It is.

[0083] The one or more dimensions that may be classified include, but are not limited to: Latency, b. Bandwidth / Throughput, C. Jitter, d. Packet ordering, and e. Volume of data transfer Examples include:

[0084] As described further below, these classification dimensions are useful for improving efficient communication flows. Latency and bandwidth / throughput considerations are particularly important when there are flows with conflicting requirements. Example embodiments in which jitter is addressed are described further below, and the system may be configured to accommodate jitter, for example, by buffering in the scheduler or by keeping flows sticky to a particular connection. Packet ordering is described further below with an example specifically for TCP, and the volume of data is relevant when it can be used as an indicator that can reclassify flows from one type (low latency, low bandwidth) to another (non-latency sensitive, high bandwidth).

[0085] Other classification dimensions and classifications are possible, and the above are provided as classification examples. Different dimensions or classifications and / or combinations among the above may be created. For example, in media broadcasting, the system may be configured to classify video data and metadata associated with clips (e.g., GPS information, timing information, labels) or FEC data associated with a video stream.

[0086] Flow classification can be utilized to remove and / or filter out transmissions that the system is configured to prevent (e.g., peer-to-peer file sharing in some cases, or material known to be under copyright) or traffic that the system may be configured to prefer in the context of providing tiered services (e.g., other preferring particular users or software programs).

[0087] For example, in an Internet backup usage scenario, the system may be configured so that VoIP calls to / from a support organization receive a higher level of service than calls within that organization, even though the combined backup may be limited in availability (where the system can generate instructions to cause an endpoint to degrade the voice quality of some calls more than others when under constraint, or to degrade certain calls altogether when there are bandwidth constraints).

[0088] The scheduler 160 is configured to make decisions regarding which packets should be sent to which connections 106. The scheduler 160 may be considered an improved QoS engine. The scheduler 160 is implemented in some embodiments using one or more processors or standalone chips or configured circuits, such as comparator circuits or FPGAs. The scheduler 160 may comprise a series of logic gates that are checked to make the decisions.

[0089] A typical QoS engine manages a single connection, and although it may be configured to perform flow identification and classification, the end result is that the QoS engine reorders the packets and then sends them out on a single connection.

[0090] In contrast, scheduler 160 is configured to perform flow identification, classification and packet ordering, but scheduler 160 in some embodiments is further configured to make decisions about which connection to send packets to in order to provide improved transmission characteristics to the data flow and / or satisfy policies set for the flow by a user / administrator (or as set forth in various rules). Scheduler 160 may modify network interface operating characteristics, for example, by transmitting a set of control signals to the network interfaces to switch them on or off or indicate which one should be used to route data. The control signals may be instruction sets that indicate specific characteristics of the desired routing, such as timing of packets, reservation of network interfaces for particular types of traffic, etc.

[0091] For example, consider two connections with the following characteristics: 1) Connection 1: Round Trip Time (RTT) of 1 ms, estimated bandwidth of 0.5 Mbps, and 2) Connection 2: RTT of 30ms, estimated bandwidth of 10Mbps.

[0092] Scheduler 160 may attempt to reserve connection 1 exclusively for DNS traffic (small packets, low latency). In this example, there may be so much DNS traffic that it reaches the capacity of connection 1, and scheduler 160 may be configured to overflow traffic onto connection 2, although scheduler 160 may selectively do so based on other decisions or factors. For example, if scheduler 160 is configured to provide a fair decision, scheduler 160 may be configured to first overflow traffic from IP addresses that have already sent a significant amount of DNS traffic within the past X seconds.

[0093] The scheduler 160 may be configured to process decisions based on a process or methodology operating, for example, in conjunction with one or more processors or similar implementations in hardware (e.g., FPGAs). These devices may be configured for operation under the control of the operations engine 152 to split the data stream into data packets and then route the data packets into buffers (managed by a buffer manager) that send the data packets to the data connections according to rules that attempt to optimize packet delivery while taking into account the characteristics of the data connections.

[0094] The primary criteria is based on latency and bandwidth in some embodiments, but in some embodiments the path maximum transmission unit (PMTU) may also be utilized. For example, if one connection has a significantly smaller PMTU than another (e.g., 500 bytes versus 1500 bytes), it will be designated as a bad candidate for overflow (and may be avoided or deprioritized, for example) since packets sent on that connection will need to be fragmented.

[0095] Scheduler 160, in some embodiments, is not necessarily configured to communicate packets through in-order, but rather is configured to communicate packets over the various connections to meet or exceed desired QoS / QoE metrics (some of which may be defined by a network controller and others by a user / customer). In cases where packets may be communicated out of order, sequencer 162 and buffer managers may be utilized to reorder received packets.

[0096] The sequential bursts of packets are transmitted through the network interface, and an estimate of the bandwidth of the first network interface is generated based on timestamps and packet sizes recorded when packets in the sequential bursts of packets are received at the receiving node. The estimate is then utilized to route packets in the data flow of sequential packets through the set of network connections based on the generated bandwidth of the first network interface. As described below, in some embodiments, the bandwidth estimate may be generated based on timestamps of packets in a burst that are not coalesced with the first or last packet in the burst to estimate a lower bandwidth value and to estimate an upper bandwidth value (e.g., by packet substitution). The transmitted packets may be test packets, test packets "piggybacking" on data packets, or hybrid packets. When data packets are used for "piggybacking", some embodiments include flagging such data packets for increased redundancy (e.g., to increase tolerance for lost packets, especially for packets used for bandwidth testing purposes).

[0097] Sequential packets may be received in order, or within an allowed deviation from the order, such that sequencer 162 can reconstruct the packets for consumption. In some embodiments, sequencer 162 is a physical hardware device that can be incorporated into a broadcast infrastructure that receives a signal and generates an output signal that is the reconstructed signal. For example, the physical hardware device may be a rack-mounted appliance that serves as the first stage for receiving and reconstructing the signal.

[0098] Sequencer 162 is configured to sequence packets received at an endpoint and transmit them to an application in an acceptable order, which in some embodiments follows the original order, to reduce unnecessary reordering of packets for a flow or other error correction. In other embodiments, the order is within acceptable limits of order such that sequencer 162 can still reconstruct the data flow. Sequencer 162 may include buffers or other mechanisms, for example, to smooth the received flow, and in some embodiments is configured to control the sending of acknowledgments and the storing of packets based on monitoring the transmission characteristics of multiple network connections and unequal distribution in the receipt of sequential packets of the data flow.

[0099] The sequencer 162 may be implemented in hardware (e.g., a field programmable gate array) for example on a processor or under the control of an action engine 152 configured to reconstruct the data flow from received data packets extracted from a buffer.

[0100] The sequencer 162 is configured, on a per-flow basis, to hide latency differences between multiple connections that are not tolerated for each flow.

[0101] The action engine 152 can act as an aggregator of information provided by other components (including 154) and instructs the sequencer 162 via one or more control signals that indicate how the sequencer 162 should operate for a given flow.

[0102] When a system configured for a protocol such as TCP receives packets, the system is typically configured to expect (but not require) packets to arrive in order, although the system is configured to establish a time bound (usually some multiple of a round trip time or RTT) for when it expects out-of-order packets to arrive, and the system may be configured to retransmit lost packets as soon as a heuristic-based time bound is reached (e.g., initial retransmission triggered by 3 DUP ACKs).

[0103] If packets arrive at sequencer 162 on connections with significantly different latencies, sequencer 162 (on a per-flow basis) may be configured to buffer the packets until they are roughly the same age (delay) before sending them onward toward their destination. Sequencer 162 does this, for example, if the flows require matching latency and low jitter.

[0104] The sequencer 162 does not necessarily need to provide reliable, strictly in-order delivery of data packets, but in some embodiments is configured to provide what is necessary so that the system using the protocol (e.g., TCP or applications over UDP) does not prematurely determine that packets are being lost by the network.

[0105] In some embodiments, sequencer 162 is configured to monitor the latency variation (jitter) of each data connection along with packet loss (based on data maintained by operations engine 152) to predict, based on connection reliability, which data connections are likely to cause packets to be delayed by the flow more than expected (meaning that endpoints 102 and 110 will consider them lost and invoke their error correction routines).

[0106] In the case of an out-of-order situation, sequencer 162 may, for example, utilize a larger jitter buffer on connections that exhibit greater latency variation. In the case of packet retransmission, sequencer 162 may be configured to immediately request the lost packet through the "best" (most reliable, lowest latency) connection.

[0107] In one example scenario, the bandwidth-delay product estimate may not be entirely accurate and latency spikes are experienced in a connection, resulting in packets being received out of order at intermediate gateways.

[0108] In these embodiments, sequencer 162 may be configured to make predictive decisions regarding how a protocol (and / or associated application) can act with respect to out-of-order packets, and to generate instructions to reorder packets so that downstream systems are less likely to erroneously assume that the network has reached capacity (and thus throttle its transmission rate) and / or unnecessarily request retransmission of packets that were not lost.

[0109] For example, many TCP implementations use three consecutive double acknowledgments (DUP ACK) as a hint that the packet following the DUP ACK may be lost. In this example, if the receiver receives packets 1, 2, 4, 5, and 6, it will send ACK(2) three times (one for each of packets 4 / 5 / 6). The sender then recognizes that this event may hint that packet 3 may be lost in the network, and is configured to preemptively resend it before any normal retransmission timeout (RTO) timer expires.

[0110] In some embodiments, sequencer 162 may be configured to compensate for such predictive decisions. As in the example above, if sequencer 162 has buffered packets 1, 2, 4, 5, 6, 3, sequencer 162 may reorder the packets to ensure that the packets are transmitted in their proper order. However, if packets were already buffered in the order 1, 2, 4, 3, 5, 6, sequencer 162 may be configured not to bother reordering them before transmission, since in this example a predictive decision would not be triggered (given the positioning of packet 3).

[0111] The connection controller 154 is configured to perform the actual routing between the different connection paths 106, which is provided, for example, to indicate that the connections 106 to the combined links do not have to reside on the physical gateways 104, 108 (e.g. a physical gateway may have several links (Ethernet or otherwise) to physical transmit / receive equipment or satellite equipment that may be elsewhere (and may be at different locations with respect to antennas, etc.). Thus, the endpoints may be logically connected and physically separated in various ways.

[0112] In one embodiment, system 100 is configured to provide a method known as TCP acceleration, where the gateway creates a pre-buffer upon receiving a packet and provides an acknowledgment signal (e.g., an ACK flag) to the sending endpoint as if the receiving endpoint had already received the packet, allowing the sending endpoint 102 to send more packets into system 100 before the actual packet is delivered to the endpoint. In some embodiments, pre-buffering is used for TCP acceleration (opportunistic acknowledgment (ACKing) combined with buffering of available data).

[0113] This pre-buffer may reside before the first link to the transmitting endpoint 102 or elsewhere in the transmission path to the endpoint 110. The size of this pre-buffer may vary depending on feedback from the multipath network, which in some embodiments is based on an estimate or measurement of the bandwidth-delay product or a set of predetermined logical operations (where a particular application or user will receive a pre-buffer with particular characteristics such as speed, latency, throughput, etc.).

[0114] The pre-buffer may exist at various locations within an implementation, for example, the pre-buffer may exist either at the entry point to gateway 104 or along the way (before the final destination) to 110. In one embodiment, there is a series of pre-buffers, for example, as data flows from endpoint 1 to endpoint 2, there is a pre-buffer on both gateway A and gateway B.

[0115] It becomes the responsibility of the system 100 to reliably transmit data that has been accepted into the pre-buffer and opportunistically acknowledged by the endpoint 102 to the endpoint 110. The acknowledgement tells the original endpoint 102 that the endpoint 110 has received the data and therefore no longer needs to handle retransmissions via its normal TCP mechanisms.

[0116] Pre-buffering and opportunistic acknowledgments are advantageous because they remove the time limit that system 100 has available to handle loss and other non-ideal behavior of connections 106. The time limit without TCP acceleration is based on the TCP RTO calculated by endpoint 102, a value not under the control of system 100. If this time limit is exceeded, endpoint 102 a) resends data that system 100 may already have buffered, and b) decreases its cwnd, thus reducing throughput.

[0117] The size of the pre-buffer may have to be limited to limit memory usage, which requires communication of flow control information between the multipath gateways 104 and 108. For example, if the communication link between the gateway 108 and the endpoint 110 has a lower throughput than the total throughput of all connections 106, the amount of data buffered at 108 will constantly increase. If the amount buffered exceeds the limit, a flow control message is sent to 104 instructing it to temporarily stop opportunistically acknowledging data sent from the endpoint 102. If the amount buffered eventually drops below the limit, a flow control message is sent to 104 instructing it to resume opportunistic acknowledgment. The limit may be a static threshold or may be dynamically calculated, taking into account factors such as the total BDP of all connections 106 and the total number of data flows currently being handled by the system. The thresholds at which flow control start / stop messages are sent need not be the same (e.g., there may be hysteresis).

[0118] Similarly, there may be flow control signal information within the multipath gateway itself. For example, if the aggregate throughput of the connection 106 is less than the throughput between the endpoint 102 and the gateway 104, the pre-buffer in 104 will continually increase. After a limit (which may be calculated as described above) is exceeded, opportunistic acknowledgments of incoming data from the endpoint 102 may need to be temporarily stopped and then resumed when the amount of data drops below an appropriate threshold.

[0119] The above example describes the TCP acceleration method for data sent from endpoint 102 to 110. The same description applies to data sent in the opposite direction.

[0120] In another embodiment, the buffer manager is configured to provide excess buffering for ongoing transmissions per communication link to compensate for variability in the behavior of the connecting network and the potentially "bursty" nature of other activity and source transmissions to the network.

[0121] Excessive buffering may refer to, for example, intentionally accepting more packets at the ingress side than the connection's BDP at the egress side can handle. The difference between "excessive buffering" and "buffering" is that the buffer manager can buffer different amounts based on flow demand and on how the connection BDP changes in real time.

[0122] This excess buffering causes the gateway 104 to accept and buffer more data from the sending endpoint 102 than it is otherwise prepared to accommodate (e.g., more than it can "easily handle") Excess buffering can be done globally (e.g., the system is configured to utilize more than the system estimates is available in aggregate throughput), or it can be moved and managed into the connection controller on a per-connection basis, or a combination of both (e.g., multiple overbuffers per transmission) can be provided.

[0123] For example, even if system 100 only estimates that it can transmit 20 Mbps over a given set of links, system 100 may receive more than that (e.g., 30 Mbps) from transmitting endpoint 102 for the time being and buffer data that it cannot transmit immediately, perhaps based on a determination that network conditions may change based on statistical and historical knowledge of network characteristics provided by network characteristic monitoring unit 161, or a determination that there may be a time when transmitting endpoint 102 (or other incoming or outgoing transmissions) may slow down its data transmission rate.

[0124] The flow classification engine 156, in some embodiments, is configured to flag certain types of traffic, and the action engine 152, in some embodiments, may be configured to instruct the buffer manager to size and manage pre-buffering and / or over-buffering on a per-flow basis, selecting the size of the buffer based on any number of criteria (type of data, user, historical behavioral data, flow requirements).

[0125] In some embodiments, the size of these buffers is determined per transmission and also per gateway (since there may be many transmissions being routed through the gateway at one time). In one embodiment, pre-buffering and over-buffering techniques are utilized in parallel.

[0126] In some embodiments, the size of the excess buffering is determined to be substantially proportional to the bandwidth-delay product (BDP). For example, the system may be configured such that if the network has a high BDP (e.g., 10 Mbps @ 400 ms => 500 KB), the buffers should be larger so that there is always enough data available to keep the network / pipeline full of packets. Conversely, for low BDP networks, the system may be configured such that there is less buffering to prevent excessive buffer bloat.

[0127] Buffer bloat may refer to, for example, excessive buffering in a network that results in high latency and low throughput. Given the advent of cheaper and more readily available memory, many devices now utilize excessive buffers without considering the impact of such buffers. Buffer bloat is described in more detail in papers published by the Association for Computing Machinery, such as, for example, a paper entitled "Bufferbloat: What's Wrong with the Internet?" (December 7, 2011) and a paper entitled "Bufferbloat: Dark Buffers in the Internet" (November 29, 2011), both of which are incorporated herein by reference.

[0128] As an example of determining the excess buffering size with respect to bitrate and latency, a rule may be implemented requiring that the system should not add more than 50% to the base latency of the network through excess buffering. In this example, the rule indicates that the excess buffering size is "bitrate x base latency x 1.5".

[0129] In one embodiment, the action engine 152 may be included in the multi-path gateway 104, 108. In another embodiment, the action engine 152 may reside in the cloud and apply to one or more gateways 104, 108. In one embodiment, there may be multiple endpoints 102, 110 that connect to a single multi-path gateway 104, 108. In one embodiment, the endpoints 102, 110 and the multi-path gateway 104, 108 may reside on the same device.

[0130] In one embodiment, the connection controller 154 may be separate from the multi-path gateways 104, 108 (and may be physically associated with one or more connection devices (e.g., wireless interfaces or wired connections)). In another embodiment, the connection controller may be present on the gateway, but the physical connections (e.g., interfaces or wired connections) may be present on separate units, devices, or multiple devices.

[0131] Although the endpoints must be logically connected, they may be connected independent of the connections 106 (e.g., communications handled by multipath gateways 104, 108). In some embodiments, the set of connections 106 available to a given gateway may be dynamic (e.g., certain networks only available at certain times or to certain users).

[0132] In one embodiment, the traffic coming from the endpoint 102 may be controllable by the system 100 based on dynamic feedback from the system 100 (e.g., the system may be configured to change the bit rate of a video transmission originating at the endpoint). In another embodiment, the traffic coming from the endpoint 102 may not be controllable by the system 100 (e.g., web requests originating from the endpoint).

[0133] In other embodiments, there may be more than one set of multipath gateways 104, 108 in the transmission path (e.g., FIG. 13). For example, in some implementations, there may be a TCP transmission with a remote multipath gateway connecting to a gateway in the cloud, which then provides a non-multipath connection to another multipath gateway at the edge of the cloud, which then transmits to another multipath remote gateway.

[0134] Various use cases may be possible, such as a military use case where a remote field operator may need to transmit a large volume of data to another remote location. The operator's system 100 may be set up with a transmission mechanism where multiple paths are utilized to provide the data to the wider Internet. The system 100 then uses the high capacity backhaul to transmit somewhere else on the edge of the Internet where it then requires another multipath transmission to get to a second remote endpoint.

[0135] In one embodiment, gateways A 104 and B 108 may be configured to transmit control information between each other via one of the available connection paths.

[0136] FIG. 2 illustrates a scenario in which an Internet user connects to the Web through a combination of gateways that utilize transport links with different network characteristics.

[0137] Figure 2 relates to Figure 1, but shows different examples of interactions between client 202 and server 204. Although the system may ultimately be a two-way system, there may be situations where the transmission may be unidirectional or initially unidirectional (e.g., requesting data from a web page that provides the data).

[0138] A device that may be a client 202 may be any type of device capable of communicating, such as, for example, an Internet of Things (IoT) device. Another endpoint may be a server, such as an IoT hub. The illustrated connection 206 may include, for example, an LTE network, a 3G network, a satellite connection, etc. Other connection types include wired networks, WiFi networks (e.g., 802.11 standards), Bluetooth, microwave connections, optical connections, and other wireless connections, among others. The reliability and connectivity of each of these networks and / or connections may be affected by various factors, such as multipath fading, interference, noise, etc.

[0139] Bandwidth, latency, jitter, packet loss, etc. may be affected, and in some embodiments, the endpoints 202, 210 and / or multipath gateways 204, 208 may be moving, and even the number and types of available connections 206 may become dynamically available or unavailable. For example, a telecommunications-equipped VAN may move in and out of cellular coverage, satellite coverage, etc.

[0140] Figure 3 is provided to show a more detailed example of the method performed by sequencer 160 by demonstrating the ordering of six packets according to an example based on the network of Figure 2. In the example of Figure 3, the packets are transmitted in a simple round-robin fashion. The example is non-limiting and other packet transmission methods are possible.

[0141] With particular focus on latency characteristics, three types of connections between the scheduler in the multipath transmitter and the sequencer in the multipath receiver are considered: 1. LTE: 30Mbps, 50ms latency, 2.3G: 2Mbps, 120ms latency, and 3. Satellite: 10Mbps, 500ms latency Consider the following.

[0142] In this example, assume that a client machine (C) 302 connected via a multipath transmitter 304 is attempting to send TCP traffic to a server (S) 310 reachable via a multipath receiver 308, and that the system 300 is not using the TCP acceleration techniques described above.

[0143] Consider a simple algorithm that round-robins IP packets from 6 of these connections, and does nothing but be "smart". Due to the extreme difference in latency, the server (S) receiving the TCP stream will send packets in the following order: Packet 1@t=50ms (sent over LTE) Packet 4@t=50+ms (sent over LTE) Packet 2 @ t = 120ms (sent over 3G) Packet 5@t=120+ms (sent over 3G) Packet 3 @ t = 500ms (sent via satellite) Packet 6@t=500+ms (sent via satellite) Observe the packets arriving at

[0144] TCP is not well designed to handle major reordering of packets and large latencies between them. The server will send multiple double ACKs for packet 1 (to make sure the client knows the server is still waiting for the lost packet) and will cause any / all packets 2 / 3 / 5 / 6 to be resent, which is unnecessary since the packets were not lost, they are just still in transit. For this reason, many existing solutions provide very poor throughput, especially when trying to combine connections of very diverse (or different) latencies.

[0145] The ordering function of the sequencer 162 of the gateway 104, 108 provides a technical solution by buffering packets and sending them in order so that the server in question ultimately receives the packets in the order in which they were sent. The resulting order is: Packet 1@t=50ms (sent over LTE) Packet 2 @ t = 120ms (sent over 3G) Packet 3 @ t = 500ms (sent via satellite) Packet 4 @ t = 500 + ms (sent over LTE and buffered by the sequencer) Packet 5 @ t = 500 + ms (sent over 3G and buffered by the sequencer) Packet 6@t=500+ms (sent via satellite) It is.

[0146] In determining how large the buffer should be, scheduler 160 may be configured to base such determination at least in part on the bandwidth-delay product. In some embodiments, the length of the buffer may be sized based on, for example, historical data for flow demand, jitter, and performance of the set of connections (e.g., how variable was delivery for those connections, how often did the system recognize packets that arrived quickly being re-requested (e.g., the buffer may be too small), or the number of times minimal sorting was required by the sequencer (e.g., the buffer may be unnecessarily large)).

[0147] For example, if many similar networks with similar latencies are used, scheduler 160 may provide a smaller receive / sequencer buffer since there is less opportunity for mis-ordering to occur.

[0148] On the other hand, the system may be configured to choose a larger buffer on the scheduler to ensure that the network / pipeline can always remain full if either the sending endpoint or the connection is "bursty". The sizing of the buffer may depend on the side or flow requirements. If lower latency is required to endpoint 2 and the connection is otherwise good / predictable, or if the flow does not have strict requirements for packet ordering, the sizing of the buffer will tend to be a smaller buffer on the sequencer.

[0149] The buffers on scheduler 160 may be configured to ensure that connection 106 is filled by endpoint 1, and the buffers on sequencer 162 may be configured to minimize the need to re-request packets, but not to be too large and thereby cause unnecessary delays in delivery of packets to endpoint 2.

[0150] In some embodiments, scheduler 160 and sequencer 162 are not configured to provide completely guaranteed in-order reliable delivery of packets; instead, scheduler 160 and sequencer 162 only need to reorder packets sufficiently so that a protocol (e.g., TCP) or application does not erroneously treat packets that are still buffered / forwarded as lost. For example, various protocols / applications may use multiple RTTs as an indicator to determine whether a packet is lost. Thus, scheduler 160 and / or sequencer 162 may be operable to reduce extreme variability in packet buffer times.

[0151] In one embodiment, the system provides buffers on both sides of the system to maximize throughput, where the first buffer releases content to the network upon a trigger. In one embodiment, the two buffers can be coordinated together (in the scheduler 160 of gateway A and the sequencer of gateway B) to maintain a certain overall buffer length. Thus, the buffers may operate in unison to provide an overall buffer length to improve throughput while avoiding overly long buffers (buffer bloat) on either the sender or receiver. In another embodiment, the buffers may operate independently of each other.

[0152] Figure 4 illustrates when scheduler 160 is configured to be "smart" and to utilize latency in its decisions, where a simplified example of three networks having the same bandwidth but different latencies is provided for illustrative purposes.

[0153] In this example, the bandwidth is defined as "1 packet" for simplicity of calculation, although other bandwidths are possible. This assumption allows for a simplified analysis in which packet 1 arrives @t=50ms and packet 2 arrives @t=100ms. If the bandwidth of the connection were greater than 1 packet, packet 2 would arrive at t=51ms since it is pipelined after the first packet (assuming a processing / pipelining delay of, say, 1ms). In another example, if one wishes to define the analysis in terms of Mbps or Kbps, the derivation may include the assumption that all packets are 1500 bytes and that the connection has a bandwidth of 12Kbps (1500 x 8). Other assumptions are possible, for example: Connection 1-1 packet bandwidth, latency 50ms, Connection 2 - 1 packet bandwidth, 120ms latency, and Connection 3 - 1 packet bandwidth, latency 500ms It is.

[0154] Smart scheduler 160 is configured to route packets such that the potentially time-varying latencies of the various links are taken into account in real time, which results in instances where scheduler 160 only sends packets to connections 1 and 2 because all packets can be acknowledged faster (e.g., based on RTT) than it can send just one packet over connection 3.

[0155] The sequencer 162 on the multipath receiver 408 performs the following sequence: Packet 1@t=50ms (sent over connection 1) Packet 2@t=100ms (sent over connection 1) Packet 3@t=120ms (sent over connection 2) Packet 4 @ t = 150ms (sent over connection 1) Packet 5@t=200ms (sent over connection 1) Packet 6@t=240ms (sent over connection 2) Receive packets.

[0156] Assuming there are no dropped packets or congestion on the network, the packets arrive in order and sequencer 162 simply passes the packets through in order without the need for reordering or buffering. In an embodiment where multiple connections have similar characteristics, the system may select to transmit packets based on historical connection characteristics and reliability.

[0157] If packets are received out of order or lost, the sequencer 162 may be configured to request that the packets be resent over the determined most reliable connection. The multipath receiver 408 may, for example, be configured to attempt to maintain the total latency of the flow so that it has low variability. If the multipath receiver 408 expects a packet and it has not yet arrived (because the packet was lost or the connection reordered it unexpectedly), whether to make a re-request or just continue (depending on the protocol (TCP) or application for handling loss) may depend, for example, on the flow and the impact determined on the request.

[0158] 5 is provided to illustrate the use of bandwidth-delay product (BDP) and stickiness for multiple data connection routing of TCP packets. An input device 502 is shown sending packets to an output device 510 via a transmitter 504, connection 506 and receiver 508, and the system 500 is shown not employing the TCP acceleration techniques described above.

[0159] Considering the example of Figure 5, each of the three types of connections shown in 506 has a different Bandwidth Delay Product (BDP). The BDP and latency shown are provided merely as examples, and other values ​​are possible. The BDP may be determined, for example, by multiplying the bandwidth by the latency.

[0160] The 3G link (e.g., connection 1) has the smallest BDP (2Mbps x 120ms = approximately 30KB) and once it becomes full (i.e., approximately 30KB of unacknowledged packets have been sent and are in-flight in the pipeline), no other data packets should be sent over that link or they will be dropped and / or buffered, which will cause TCP flows to falsely believe they have reached channel capacity and increase the latency of the connection.

[0161] Therefore, the correct approach is to instead send the data packet over a different link (e.g., connection 2 or 3). Only when all links have reached their full BDP should the system 100 be configured to drop the packet (or, optionally, send it to any of the three connections), thereby causing the TCP flow to eventually determine that it has reached its total channel capacity, and to force the TCP flow (e.g., sending source 502) to reduce its total transmission rate in response.

[0162] A technical challenge that arises is that system 100 typically does not have information regarding the capacity of each connection. In some embodiments, bandwidth estimation techniques are applied to estimate bandwidth values ​​taking into account incomplete information so that improved routing decisions can be applied (e.g., probabilistically improving expected throughput over simple bandwidth estimation).

[0163] A bandwidth estimate is useful (even if only slightly accurate) because without it the system 100 can only determine when the connection 106 has reached channel capacity if the system 100 observes increased packet loss and / or latency. With an estimate, the system 100 can be advantageously configured to use techniques such as adding a percentage of FEC when the transmission rate begins to approach the estimate.

[0164] When the receiver 108 needs to start using FEC, the transmitter 104 may determine that this indicates that the channel capacity has been reached. This is an improvement over systems that do not use FEC because the channel capacity is determined without incurring any unrecoverable losses.

[0165] In some embodiments, a more accurate bandwidth estimation technique may be useful in determining that the system 100 should invoke an improved transmission mechanism, such as weighted round robin.

[0166] However, even with a perfect bandwidth estimation algorithm, the system 100 may not be optimal to utilize the weighted round robin mechanism. If a flow does not require the full channel capacity, all of its packets should remain "sticky" to a single connection so that no buffering / reordering is required at the receiver. That is, once a flow is assigned to a data connection, it should continue to use that connection. Only when the needs of the flow exceed the capacity of the data connection to which it is assigned should the flow "overflow" to another data connection.

[0167] The selection of the overflow connection can incorporate any number of parameters into the decision logic to provide various advantages in terms of latency, throughput, reliability, jitter, cost, etc., or any combination of these factors. For example, the LTE BDP is 30Mbps x 50ms = approximately 187.5KB, and the satellite link is 10Mbps x 500ms = approximately 625KB. If the satellite link is high cost and the goal is low latency and low cost, then the LTE connection will be "filled" with data before the satellite connection is used. Alternatively, if the satellite connection is significantly more reliable than the LTE connection and the goal is reliability, then the satellite connection will be "filled" with data before the LTE connection is utilized.

[0168] Connections 106 may also be advantageously pooled such that connections with similar characteristics (latency, throughput, jitter, and packet loss) are grouped together. This simplifies the connection management that must be done by the connection controller 154, as pooling means that fewer decisions need to be made within the connection controller. As connection characteristics change over time, connections may be combined and separated from the pool in real time as needed to ensure reliable data communication. Any connections that have characteristics that are significantly different from all other connections are not pooled and are used as individual connections.

[0169] Pooled connections also make it possible to meet the needs of flows with conflicting requirements. For example, consider a set of connections 106 that all have consistently low latency but very varying throughputs. In this scenario, significant over-buffering is required to achieve the maximum possible throughput, which can result in a significant increase in buffer bloat. If two flows are currently active, one requiring high throughput and the other requiring low latency and low throughput, by splitting the available connections 106 into two pools, one pool utilizing significant over-buffering and the other not, both flows can achieve their ideal quality of experience.

[0170] A description of the bandwidth estimation technique is provided in Figures 8A, 8B, 8C, 8D and 8E, which includes an example estimation that utilizes known information about network capabilities to estimate available bandwidth based on the number of packets to be transmitted and packet size. In practical implementations, corrections are applied to compensate for non-idealities, which may result in coalescing, etc., with respect to interrupt arbitration. An error level associated with each packet may be determined and used to improve the bandwidth estimation, and this error may depend on the bit rate (e.g., this error may become insignificant as the bit rate increases).

[0171] Using congestion control techniques well known in the art (Reno, Vegas, CUBIC and newer techniques like BBR), the throughput and latency of a connection (and thus the bandwidth-delay product or BDP of the connection) can also be estimated. It can be advantageous to dynamically use different congestion control techniques depending on the characteristics of individual (or pooled) connections and / or the requirements of the incoming data flows. Parametric combination of congestion control techniques for a single connection (or a pool of connections) can also be advantageous, since it allows techniques that can be better adapted to the needs of the connection (or pooled connections) and / or the incoming data flows.

[0172] 6 adds the dimension of flow requests (size and sensitivity to latency). There may be different input devices 602 and 603, which may be provided with different types of data as shown (602 is an FTP request and 603 is an SSH session). These flows may require classification by gateway 604 and may be provided advantageously via a connection 606 to a receiving gateway 608 based on the flow classification and via an SSH connection 612 connected to an FTP server 610 and / or input device 2.

[0173] Thus, with proper identification and classification of flows, further improved allocation of data packets to data connections (or pools of connections) can be implemented. For example, if the gateway 604 is operable to identify a flow as DNS traffic, the gateway 604 can determine that the data stream is small and latency sensitive, so that the gateway 604 keeps this data stream "sticky" to the connection or pool of connections with the lowest latency. Conversely, if the gateway 604 identifies that the data stream is a large FTP transfer, the gateway 604 should be configured to indicate a routing preference to the connection with the higher latency first, and only overflow to the lower latency connection or pool of connections if necessary and if excess capacity exists.

[0174] While the illustrated embodiment includes a traffic identifier in the first line, other solutions can use deep packet inspection, labeling of data packets or data streams to identify the type of data stream and ensure proper handling by the flow classification engine. Such an approach may aid in routing (e.g., SSH: use lower latency connection, HTTP: use higher latency connection). In one embodiment, some traffic types are always sticky (so sequencer 162 is not needed on the receiving side).

[0175] In one embodiment, packet marking by the transmitter (e.g., via Differentiated Services Code Point (DSCP)) can be used to aid in flow identification / classification to further improve the allocation of packets to connections and data flows.

[0176] In another embodiment, heuristics based on observed behavior are used to infer the nature of the flow (e.g., given the unidirectional nature back to the initiator and the high bandwidth, the gateway 604 can be configured to assume that this is an FTP flow by default and adjust connection allocations accordingly).

[0177] In another embodiment, the classification can change dynamically based on the behavior of the flow. For example, when an SSH flow first starts, it may be classified as a low-bandwidth, latency-sensitive flow. However, as the flow continues to transfer data, once it passes a certain threshold volume, its classification may change to a large data transfer that is not latency-sensitive (e.g., SCP / SFTP operation).

[0178] In FIG. 6, the flow classification engine is configured to determine (predict) that the nature of the traffic from input device 1 (602) is FTP traffic, and in anticipation of a large flow of data that is not latency sensitive, the system 100 initially routes the traffic through a high BDP satellite connection, overflowing to the LTE connection if necessary.

[0179] The flow classification engine 156 may also be configured to determine that the flow from input device 2 (603) is an SSH session, and then the gateway 604 may be configured to select low latency connection 1 (or a pool of low latency connections) to handle the flow. The flow classification engine 156, in some embodiments, monitors and / or receives statistical information about the incoming data streams that is utilized to help identify the type of stream.

[0180] In one embodiment, system 100 combines its awareness of multiple flow types in real time to dynamically adjust the mix of connections or the mix of pools of connections used.

[0181] 7, system 100 incorporates a network characteristics monitoring unit 701 in an egress multipath gateway 704, an egress multipath gateway 708, or in both multipath gateways. Various factors are monitored and / or estimated by system 100 to control how data traffic is allocated to various connections.

[0182] These factors may be established, for example, by a determination of connection reliability that is used to determine how to improve the delivery of packets (e.g., in some cases, overall reliability is more important than peak speed).

[0183] Connection reliability may be measured as a composite of one or more factors including, for example, bandwidth, latency, jitter, packet loss, and variability, among others.

[0184] In one example, the network characteristic monitoring unit 701 collects raw data such as throughput, packet loss, jitter, latency, packet ordering, etc., and calculates statistics on this collected data, typically but not limited to mean, variance, kurtosis, order statistics, rank statistics, number of times above or below pre-set / predefined thresholds, and other derived statistics.

[0185] When network characteristic monitoring units are present in both the receiver 704 and the transmitter 708, these network characteristic monitoring units may be configured to share their derived statistics (as shown by feedback 702) in real time or at predefined intervals over one or more connections 706. For improved reliability, the shared derived statistics may be repeated over multiple connections (or a pool of connections) to increase reliability.

[0186] In one embodiment, the network characteristic monitoring unit 701 at the receiver takes measurements of how many bits are corrected by forward error correction (FEC) for each packet sent on a particular connection 706. This information may be processed to provide insight into the error rate for the connection, and is provided as part of the derived statistics.

[0187] 8A, 8B, 8C, 8D, and 8E are illustrations of bandwidth estimation methods according to some embodiments. Bandwidth estimation is provided to obtain a reasonably accurate estimate of the bottleneck bandwidth (bandwidth constraint) between the transmitter and receiver without forcing transmission at any rate close to the actual bottleneck bandwidth.

[0188] The estimated bandwidth is used along with other factors to determine an overall score or fitness level for a given data connection (e.g., network "goodness"). The system may be configured for determination of the expected bandwidth of all available data connections (e.g., networks) in order to weight the transmission of packets to the various data connections.

[0189] 8A, as shown in diagram 800, the example provided includes a transmission burst of eight packets of 1400 bytes each from a transmitter 802. In an ideal model of the network, the bottleneck link would transmit these eight packets at exactly the rate at which it could effectively insert an equal delay between each packet. In this example, the network is capable of transmitting at 25 Mbps, so it inserts a delay of 448 microseconds between each packet.

[0190] At the receiver 804, packets are received and timestamps are recorded at precise 448 microsecond intervals. The bandwidth estimate is based on the sum of the packet sizes (excluding the first packet) divided by the time required to transmit all packets. The size of the first packet is excluded because the time it spends in the network is unknown, but the time it is received indicates the time when the second packet will start its transmission from the bottleneck link.

[0191] As shown in Figure 8B, in practice, the transmitter, receiver, and intermediate routing systems may introduce errors into the bandwidth estimates due to competition with other traffic, variability in processing times, network interface card (NIC) optimizations such as interrupt arbitration, etc. Error compensation may be performed, for example, through the use of various statistical methods (e.g., mean, median, etc.). Error compensation may increase the efficiency and throughput of the system, and in some embodiments, adaptive modification of the routing characteristics may be performed.

[0192] Another technique for error compensation is to adaptively vary the number of packets in each burst depending on the most recent bandwidth estimate. In general, the longer the burst, the more accurate the estimate, since only errors introduced in the inter-packet arrival times at the head and tail of the burst affect the bandwidth estimate calculation. The longer the burst, the more insignificant the head and tail errors become compared to the total burst duration. However, if the network has low bandwidth, long bursts may not be practical.

[0193] For clarity of the examples in Figures 8A, 8B, 8C, 8D and 8E, the bursts contain eight equal-sized packets, but the number of packets per burst and the size of each packet in a burst need not be fixed. Calculating the bandwidth estimate only requires knowing the total size of the packets and the total duration of the burst. However, excessively small packets and / or short bursts can magnify errors introduced at the head and tail of a burst.

[0194] In some embodiments, packets in a burst may also contain true data (essentially piggybacking metadata required for the bandwidth estimation algorithm on top of the data being transmitted). The first packet (which is discarded) may be used for this metadata. For improved reliability, subsequent packets in the burst may repeat this metadata.

[0195] If piggybacking is not possible or there is insufficient real data to transmit, then this burst may be transmitted with "dummy" data, but the burst length may be even shorter so that the burst interferes as little as possible with the actual data transmission.

[0196] Techniques may be used to partially compensate for some types of errors, such as techniques related to handling of NIC interrupts.

[0197] 8C illustrates an example timing diagram for a receiver in which NIC interrupts are modeled as being handled with significant delay, according to some embodiments. The method illustrated in FIG. 8C is utilized to compensate for latency and packet bursts resulting from NIC interrupts.

[0198] This example illustrates a decision to compensate for interrupt latency, as opposed to a simple bottleneck bandwidth estimation. These interrupt delays may be caused, for example, by a network link between the transmitter and receiver that creates a bottleneck in the received packets, resulting in packets being received in bursts as opposed to being evenly spaced as in the example of FIG. 8A.

[0199] An interrupt occurs when a packet is received, but is not executed until after a significant delay (500 microseconds in this example). During this delay, when the handler finally executes, more packets can arrive, so that it collects all packets (the first packet that triggered the interrupt + any packets that arrived during the delay) and delivers them simultaneously to the application.

[0200] An error is introduced into this model because several packets in a burst are coalesced into groups, with all packets in the group being stamped with the same receipt time. This error can be corrected by observing that if the packets being coalesced are "trimmed" from the head and tail of the burst (meaning that they are excluded from the calculation of the bandwidth estimate), the resulting packet distribution pattern is identical to the non-delayed interrupt case. The only difference is that the effective duration of the burst is made shorter by the trimming, and each of the packet receipt times is shifted in time by an equivalent amount due to the interrupt delay.

[0201] Trimming from the head of the burst requires finding the first group that contains only a single packet, while trimming from the tail requires only ignoring all but the first packet in the last group of the burst.

[0202] Thus, in some embodiments, the network characteristic monitoring unit 701 tracks packet reception characteristics (e.g., by determining an estimate of the bottleneck bandwidth). As shown in Figure 8C, the network characteristic monitoring unit 701 of some embodiments is configured to further compensate for interrupt latency, for example, by trimming coalesced packets and incorporating a time shift.

[0203] FIG. 8D illustrates a technique that can be used to compensate for interrupt latency when all groups containing two or more packets do not allow trimming from the head of the burst.

[0204] This technique may be implemented, for example, by the network characteristic monitoring unit 701 to compensate for interrupt latency incurred at a bottleneck link. Thus, the network characteristic monitoring unit 701 can adapt to interrupt latency without requiring that it be able to trim from the head of a burst.

[0205] The network characteristic monitoring unit 701 determines whether the reception time of a packet is equal to the transmission time of the subsequent packet from the bottleneck link (i.e., t(N-1) 受信 ==tN 送信 ) is configured to apply an idealized model of the bottleneck link. As can be seen from this example, this permutation (in this case, t2 送信 From t1 受信 It is clear that after performing (a) and (b), an estimate can be calculated even if the head of the burst is not trimmed.

[0206] FIG. 8E shows a receiver in which the NIC is modeled by the network characteristic monitoring unit 701 using an interrupt arbitration technique, which may mean that the interrupt handler wakes up on a fixed timer (every 1000 microseconds in this example).

[0207] Any packets received since the last execution of the handler are collected and delivered to the application.

[0208] Since the actual receipt time of the packet is no longer visible to the application, this introduces an error into the estimate, which is based on the time the interrupt occurs.

[0209] If the interrupt moderation interval is known (e.g., because it is a system constant or perhaps measured in real time based on the frequency with which groups of packets arrive at the application), then upper and lower bounds can be calculated for the bandwidth estimate. Thus, in some embodiments, the network characteristic monitoring unit 701 is configured to maintain a data structure representing the most recent estimated interrupt moderation interval as obtained from a corpus of tracked data values, and this data structure is utilized to estimate the interrupt interval including the upper and lower bounds for the bandwidth estimate.

[0210] The upper limit is calculated assuming that the last group in a burst is actually received immediately after the second final interrupt, and the lower limit is calculated assuming that the last group is received just before the final interrupt.

[0211] In one embodiment, the floor value is the threshold at which FEC begins to be generated and transmitted on the connection.

[0212] On some systems, despite interrupt coalescing, the actual receipt time of each packet can be made available to applications by the NIC or the operating system kernel. On these systems, explicit compensation for errors due to interrupt coalescing is not needed.

[0213] The network characteristic monitoring unit 701 is configured in some embodiments to utilize lower and upper bounds in determining packet reception characteristics for reconstruction.

[0214] 9 is a diagram 900 configured to show a simple example (single input) of satellite offload between Gateway A 902 and Gateway B 908. The dashed lines represent logical connections between components, while the physical connections represent the flow through the combined links.

[0215] Consider an example where there is one network (e.g., satellite or microwave) that the traffic manager 904 can control (can control who is on the network and who gets priority bandwidth, can adjust network parameters, etc.), and other networks (e.g., third party cellular networks) that the traffic manager 904 does not control. Consider further that the controlled network may have certain characteristics (e.g., bandwidth is purchased monthly in fixed "chunks" because it has already been paid for) that make it advantageous to ensure that it is "filled" with traffic (to reduce costs or maximize revenue), even if that means offloading some of the traffic to other networks.

[0216] An example is the transmission of video, where it is desired to deliver a certain level of transmission reliability that cannot be guaranteed or even provided by the cellular network. In this case, it may be desirable to send as much of the video through the satellite network, since it is required to guarantee the target level of reliability. Also, if the satellite connections are not at full capacity (assuming one or more gateways transmit simultaneously), it may be advantageous to use as many satellite connections as possible to reduce the incremental cost of using more cellular data. Note that other data combined with the video, such as FEC information or metadata or actual parts of the video stream, may also be transmitted through the combined connection. In another embodiment, it may be advantageous to offload data from satellite to cellular under the control of the network blender / traffic manager 904 such that gateways with good cellular bandwidth are instructed to use less satellite bandwidth.

[0217] The Network Blender / Traffic Manager 904 is configured to determine the optimal combination of networks (in some embodiments, more than one network is controlled, in which case the Network Blender / Traffic Manager 904 is still useful as it provides oversight for all networks and routes based on real-time measurements of throughput, reliability, packet loss, etc.). In one embodiment, one or more available network connections are "controlled" (as described above), and the Network Blender / Traffic Manager 904 is configured to receive real-time information from the gateways and control the bandwidth used by the gateways ("edge nodes") such that the entire network is managed by the Network Blender / Traffic Manager 904.

[0218] In one embodiment, the controlled network management system (CNMS) 906 may have a set of logic operations (e.g., static logic operations, dynamic logic operations) to determine admission of certain traffic under certain conditions (e.g., with respect to user or traffic type, priority based on cost, reliability, etc.) The set of logic operations may be based on what is provided in various data stores and stored in the gateway, source / endpoint, or cloud.

[0219] In some embodiments, a set of logical operations is also stored in the traffic manager 904, which may utilize logical operations from the controlled network management system 906 in combination with other logical operations to apply to the entire transmission (e.g., a user may not have to pay as much for a controlled satellite, and may be intentionally pushed away from a controlled satellite, but may still pay for the entire transmission, and thus receive priority for the remainder of the data connections in the system 100, so that there may be preferential treatment with regard to selection or allocation of higher cost / reliability non-controlled networks).

[0220] In one embodiment, the network blender / traffic manager 904 may interact with the CNMS 906 to inform the system, the multipath gateway 902, or devices connected to the multipath gateway, regarding general network conditions such that the system 100 may be configured to allow the network blender / traffic manager 904 to better utilize both the controlled network and any other transmissions that the network blender / traffic manager 904 may manage.

[0221] For example, multiple connections may be available and network conditions affecting each of the connections may vary over time, e.g., congestion, loss, and noise may be experienced by each of the connections, and these connections may be advantageously assigned to uses or functions based, for example, on their connection characteristics. Such "binding" of connections and assignment of functions is described in applicant's U.S. Patent No. 9,357,427, which is incorporated by reference.

[0222] FIG. 10 is a diagram 1000 that provides a hybrid example where there are multiple gateways connected to multiple controlled and uncontrolled networks.

[0223] The dashed lines represent logical connections between components, and the physical connections represent the flow through the coupled links.

[0224] There may be N multipath gateways (e.g., 1002, 1003), each working in concert with the network blender / traffic manager 1004 and CNMS 1006 such that flows from multiple endpoints through multiple gateways are provided with the best quality of experience considering factors such as contention for resources on the controlled networks (under control of 1004 and 1006), the service level purchased by the owners of these gateways, the quality and availability of connectivity on the uncontrolled networks attached to these gateways, and the requirements of the flows and applications generating traffic through these gateways. Transmissions may ultimately be provided to multipath gateways AA-NN (e.g., 1008, 1010). These gateways provide combined functionality with a known and controlled quality of service for each gateway on the network. In this context, quality of service means allocation of bandwidth for a particular application or class of applications through smart management of bandwidth provided by the network blender / traffic manager 1004 and CNMS 1006.

[0225] These gateways may be sold (or guaranteed) with different levels of connection reliability. If a gateway is guaranteed (or it dynamically requests) high reliability and it is not receiving high reliability, the network blender / traffic manager 1004 may be configured to provision (e.g., "steal") more reliable (e.g., satellite) bandwidth from another gateway and give this increased reliability to the gateway that requested (or was guaranteed) the higher reliability.

[0226] The Network Blender / Traffic Manager 1004 manages the allocation of traffic through controlled networks from multiple gateways and optimizes the set of transmissions based on real-time measurements of network characteristics (provided by the gateways) and configured static or dynamic logic (quality of service rules).

[0227] In one embodiment, the network blender / traffic manager 1004 may be specialized to manage the traffic of a particular user / organization. In another embodiment, the network blender / traffic manager 1004 may be configured to provide services to a wide variety of users. Where there may be multiple network blenders / traffic managers 1004, their resources may communicate with each other as needed and be provisioned as needed, where resources are provisioned / deprovisioned ("spun up or down") as needed. The network blender / traffic manager 1004 and the controlled network management system 1006 may be provided in the form of distributed network resources in some embodiments (e.g., may be run "in the cloud" in a cloud configuration).

[0228] Dynamic logic may be provided to address how to utilize the controlled network when a new transmission through it occurs (e.g., the priority given to any particular transmission may depend on other transmissions in progress).

[0229] 11 is a diagram 1100 provided to illustrate a one-to-one scenario configured for uncoupled disaster recovery / load balancing applications. In this situation, individual flows may remain sticky to a particular connection without any coupling, where there is no second gateway since the buffer is not needed at the receiver. The algorithm for allocation of flows to connections may be based on static or dynamic rules that take into account factors such as flow requirements, connection quality / reliability, transmission costs, computational limitations on the gateway, etc.

[0230] In one embodiment, the gateway 1102 is configured to add or remove connection links as needed (e.g., depending on throughput or type of traffic) that set up, rather than bind, individual sockets between the two endpoints of a communication.

[0231] In some embodiments, the connection may be coupled but not carry any data; such an implementation may provide technical advantages in failover scenarios where it may be necessary to know that the connection is up and running; activating the connection allows the system to periodically test the connection to ensure it is working, and / or move the data stream (or flow) through the new connection if the need arises.

[0232] In one embodiment, certain types of traffic from a site experiencing a disaster recovery scenario may be configured to be routed over a particular backup link (or set of links) to reduce cost, increase security, minimize latency, jitter, or any combination of such factors, etc.

[0233] For example, there may be uses of the system for disaster recovery as a service tool (or adaptive load balancer) that does not require routing through a receiver / second gateway, where embodiments of the above solution may be utilized to provide more robust connectivity.

[0234] FIG. 12 is a diagram 1200 provided to illustrate a sample combined disaster recovery / load balancing application, where traffic is combined and routed through a second gateway 1204 according to some embodiments.

[0235] As shown in Figure 12, in one embodiment, a large organization (or an organization needing security) may have two dedicated gateways 1202 and 1204 in a (relatively) closed system. In another embodiment, a customer may simply be assigned a public gateway 1204 "in the cloud." The implementation of Figure 12 functions to provide high security because the use of tiered gateways isolates the network from potentially malicious attacks from the Internet and provides a coordinated part for enforcing security policies and protocols, including quality of service, for incoming and outgoing data streams (or flows).

[0236] FIG. 13 is a diagram 1300 showing a series of multipath gateways 1304, 1306, and 1308 connecting two or more endpoints 1302 and 1310. FIG. 13 illustrates a scenario in which an operator in the field at endpoint 1 (1302) needs the benefit of having multiple paths to make a transmission (e.g., a soldier at a remote location transmits video and other data that provides situational awareness information to headquarters). A combination of multipath gateways (e.g., gateways 1304, 1306, and 1308) may be required to accomplish this. For example, the first hop of transit may require using multiple WiFi networks to send data from the field to a nearby forward operating base. At the forward operating base, the gateways may then send the data over a different set of communication links (e.g., many cellular networks) to a national headquarters, where another gateway may then use one or more fiber links to send the data to the national headquarters, which may then use a different combination of links to send the data back to a different field location.

[0237] In one embodiment, the data is available to applications at one or more stops along the transit, while in another embodiment, the data is transmitted end-to-end without being accessed by applications along the way.

[0238] In some embodiments, a system may be provided that is configured for managing multiple start / end points using overlapping pools of networks, where an intelligent engine uses knowledge of the activity of various devices and various networks to better manage the flow of transmissions. In such implementations, the system may be configured to sequence or manage network usage according to a data residency logic operation (e.g., logic rules).

[0239] 14 is a diagram 1400 illustrating two sites connected to the Internet through multiple links using a typical transit and / or peering configuration involving Border Gateway Protocol (BGP) based edge routers. In this example, site A advertises BGP routes for subnet A to its upstream peers / transit providers, and site B advertises BGP routes for subnet B.

[0240] BGP is a "coarse" distance-vector routing protocol, which means that the primary factor in how it decides to route a packet is based on the number of Autonomous Systems (AS) the packet traverses to reach its destination subnetwork. It does not consider congestion within each AS, nor does it natively support multihoming.

[0241] For example, if network path Transit A1 is deemed the shortest path for traffic from Subnet B to Subnet A, then all inbound traffic to Subnet A will arrive via Transit A1, even if it is congested. All other links (Transit A2, Peer A1, Peer A2, etc.) will be underutilized.

[0242] An approach to address this limitation is to partition subnet A into smaller subnets and selectively advertise them over a subset of the links. This is suboptimal because the traffic patterns in each of the smaller subnets are likely not to represent the desired partitioning among the given links at all times. Smaller subnets also contribute to an increase in the global BGP table size, but generally speaking, subnets smaller than / 24 are not accepted in the global BGP table.

[0243] Although the comparison with BGP routing between two sites is used as an example in this diagram, this is not a requirement. The two sites may simply be multi-homed through different providers that do not have publicly advertised subnets. In such a situation, a fully connected graph between the two multipath gateways is still established, where traffic between the two sites is routed over the connection according to any of the previously mentioned criteria (e.g. connection cost, flow demand, connection quality / reliability, etc.).

[0244] 15 includes a diagram 1500 showing the same two sites connected to two multipath gateway systems 100. The connection controller 154 forms a fully connected graph that connects transit A1 to all of (transit B1, transit B2, peer B1), transit A2 to all of (transit B1, transit B2, peer B1), and so on.

[0245] Flows generated by hosts and applications on subnets A and B are no longer routed based on advertised BGP routes, but are instead encapsulated within a protocol between the two multipath systems 104 and 108 that communicate directly via IP addresses on the transit and / or peering networks.

[0246] This allows the multipath system to route packets based on any number of the criteria listed above (eg, latency and throughput requirements of each flow, detected congestion / contention, jitter, cost, administrator policies, etc.).

[0247] In some embodiments, one or more aspects of the systems and / or methods described herein may seek to maximize the throughput of the aggregate flow of transmissions through multiple links while satisfying all latency and packet loss constraints for each transmission.

[0248] In some cases, some of these issues may be addressed by managing buffer bloat, which is latency caused by excessive buffering on a link, which can result in jitter in packet delays and increased latency.

[0249] When a connection is overloaded, the latency increases on the overloaded connection, which increases the bandwidth-delay product (BDP), but without a corresponding increase in throughput.

[0250] In some embodiments, managing buffer bloat may require selecting buffer sizes that reflect the BDP of various connection types, such as:

number

number

[0251] throughput i (t) and RtProp i For simple connections where (t) does not vary over time or varies very slowly, maximizing aggregate throughput while still satisfying the latency constraint is All RtProps i There exists a subset of connections such that (t) is <= the latency requirement of the low-latency flow, and · Pass through at least one of the subsets [k i ×bdp i (t)] is >= the volume of data generated by the low-latency flow. This can be resolved as long as the above conditions are met.

[0252] If a subset that satisfies this criterion exists, then we pass through all the remaining connections [k i ×bdp i (t)] (and including the partial BDPs of connections that are not completely consumed by low-latency flows) can be used for non-latency sensitive flows.

[0253] The maximization problem is throughput i (t) and RtProp i (t) increases rapidly and the system i This becomes evident when there are connections that vary over such a large range of values ​​that they cannot accurately follow the instantaneous value of (t). For these kinds of connections, the system uses bdp_min i (t) and bdp_max i (t) (Throughput i and RtProp i The device can be configured to measure / observe the following:

[0254] throughput i (t) and RtProp i In the case where (t) varies, the observed throughput i (t) and RtProp i Statistical methods based on the distribution of (t) can be used. In some cases, this may rely on conditional probability or iterative heuristic methods. Machine learning or artificial intelligence methods may also be used.

[0255] The system is configured to address a maximization problem when determining the optimal split of connections while satisfying latency constraints. There may be multiple subsets of connections that satisfy these constraints. Low Latency Subset ·RtProp_min i Find all connections where (t) <= the latency requirement of the low latency flow, and bdp_min iFind all subsets through these connections where the sum of (t) is >= the volume of data generated by the low-latency flows. High Latency Subset For each feasible subset calculated above on the remaining connections (connections not in the subset), bdp_max i (t), and the goal is to find the subset that maximizes this sum.

[0256] k i Decision In a simple case, k i is a constant, so we choose the buffer size for a connection as some fraction of the BDP for the connection. i A value of =1.2 has been found to work well in practice (eg, the system has been configured to buffer over 20% of the BDP of a given link).

[0257] In complex cases, k i are the types of flows that travel through connection i (and any constraints on those types of flows), bdp i Variance of (t)(k i (which may be calculated in real time and used to adjust k in real time) and the connection characteristics (latency, jitter, throughput), where we determine how much latency is needed for the types of flows and how much throughput is needed (and which combination is minimally acceptable for those types of flows), which is a function of k i This means that is a function of the flow type, BDP distribution and connection characteristics.

number

[0258] With WiFi and LTE connections, latency variations can be fairly rapid and cover a range spanning several orders of magnitude (eg, WiFi can range between <10 ms and >100 ms).

[0259] The amount of buffering chosen results in a latency of <10 ms (i.e., a small k i ), whenever the actual latency is >100ms, there is insufficient data available to keep the network full, resulting in a suboptimal average throughput.

[0260] The amount of buffering is >100ms latency (i.e., large k i ), it is possible to keep the network full or close to full at all times (which gives the optimal average throughput), but there will be additional latency due to buffering (buffer bloat) whenever the actual latency is <10ms.

[0261] In this example, if there are flows that require low latency, the system compromises and reduces the buffering on the connection used for low latency flow transmission to a level that is assumed to provide a latency of <10 ms (i.e., k i and bdp_min i (conservative / small value for (t)). In some cases, this may result in a lower average throughput overall, but it does not introduce buffer bloat that would hinder the demands of the flows. The system can be configured to talk on the remaining connections for flows that require high throughput and are not latency sensitive.

[0262] The above discussion provides many exemplary embodiments of the inventive subject matter. Although each embodiment represents a single combination of inventive elements, the inventive subject matter is considered to include all possible combinations of the elements disclosed herein. Thus, if one embodiment includes elements A, B, and C, and a second embodiment includes elements B and D, the inventive subject matter is considered to include any other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0263] The apparatus, system and method embodiments described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented in programmable computers, each of which includes at least one processor, a data storage system (including volatile or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface.

[0264] Program code is applied to input data to perform functions described herein and generate output information. The output information is applied to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments where elements may be combined, the communication interface may be a software communication interface, such as an interface for inter-process communication. In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combinations thereof.

[0265] The technical solutions of the embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which may be a compact disc read-only memory (CD-ROM), a USB flash disk or a removable hard disk. The software product includes a number of instructions that can cause a computer device (personal computer, server or network device) to execute the method provided by the embodiments.

[0266] A network gateway for controlling data flow through a network connection is described herein along with corresponding methods, apparatus, and computer-readable media. The gateway may in some embodiments be a physical hardware device comprising components such as a processor, an interface, a bus, a power supply, memory (ROM, RAM, Flash), etc., configured to receive and route data packets according to an instruction set representative of software and embedded firmware in the form of program code or computer logic.

[0267] The program code and / or computer logic may include control logic and devices, which in some embodiments may operate as controllers, routers, switches, access points, etc., and said devices may be configured to operate in a manner that is transparent to the end user (apart from possible performance improvements, especially as large volumes of data pass through said devices).

[0268] The network gateway may in one example be a specialized computing device specifically optimized for bonding TCP connections as described in some examples herein. The network gateway may in other examples be implemented using a processor and / or other computing hardware as part of a larger system. The network gateway may be a single device, or may be multiple devices possibly operating in concert.

[0269] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and specifically configured computer hardware configurations. The embodiments described herein relate to electronic machines and methods performed by electronic machines suitable for processing and transforming electromagnetic signals representing various information.

[0270] The embodiments described herein relate broadly and integrally to machines and their use, and the embodiments described herein have no meaning or practical applicability other than their use with computer hardware, machines, and various hardware components. For example, the substitution of physical hardware configured to perform various operations for non-physical hardware using mental steps may substantially affect the way the embodiments operate. Such computer hardware limitations are clearly essential elements of the embodiments described herein, and they cannot be omitted or replaced with mental means without significant impact on the operation and structure of the embodiments described herein. The computer hardware is essential to implement the various embodiments described herein, and is not merely used to perform steps quickly and efficiently.

[0271] Although the embodiments have been described in detail, it should be understood that various modifications, substitutions, and alterations can be made herein without departing from the scope thereof.

[0272] Moreover, the scope of the present application is not limited to the particular embodiments of the processes, machines, manufacture, compositions of matter, means, methods, and steps described herein. As one skilled in the art will readily appreciate from this disclosure, one may utilize any now existing or later developed processes, machines, manufacture, compositions of matter, means, methods, or steps that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein. It is therefore intended that the appended claims include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0273] As can be understood, the examples described and illustrated above are intended to be illustrative only.

Claims

1. 1. A network gateway for routing a data flow through a plurality of network connections, the network gateway comprising: a plurality of network interfaces for transmitting data over said plurality of network connections; At least one processor, monitoring time-varying network transmission characteristics of the plurality of network connections; analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class being associated with at least one network interface requirement of the data flow; routing packets within the data flows through the plurality of network connections based on the data flow classes and the time-varying network transmission characteristics; transmitting a sequential burst of packets through a first network interface of the plurality of network interfaces; generating an estimate of a bandwidth of the first network interface based on timestamps recorded as packets in the sequential burst of packets are received at a receiving node and sizes of the packets; routing packets through the plurality of network connections based on the generated bandwidth estimate of the first network interface; A processor configured for Equipped with Generating bandwidth for the first network interface when the receiving node processes packets received at regular intervals includes: generating a lower bandwidth value by using a reception timestamp of a packet in the sequential burst selected as an end packet in the bandwidth determination; generating an upper limit bandwidth value by replacing a reception timestamp of a packet in the sequential burst selected as the end packet with a reception timestamp of a packet in the sequential burst preceding the end packet; Includes a network gateway.

2. 2. The network gateway of claim 1, wherein monitoring the time-varying network transmission characteristics includes generating a bandwidth-delay product of at least one network interface of the plurality of network interfaces based on the monitored time-varying network transmission characteristics, and routing packets in the data flow is based on the bandwidth-delay product of the at least one network interface.

3. The at least one processor Analyzing at least one packet of each of a plurality of data flows of packets; and routing packets in each of the plurality of data flows based on a data flow class of each of the plurality of data flows and an available bandwidth of the network connection corresponding to the data flow class of each of the plurality of data flows; The network gateway of claim 1 , configured to:

4. 2. The network gateway of claim 1, wherein generating an estimate of the bandwidth of the first network interface comprises dividing a sum of packet sizes of packets between a first packet in the sequential burst and a second packet in the sequential burst by a time elapsed between a timestamp for the first packet and a timestamp for the second packet, where the first packet is not a first packet in the sequential burst and the second packet is after the first packet in the sequential burst.

5. 2. The network gateway of claim 1, wherein generating an estimate of the bandwidth of the first network interface includes generating the estimate of the bandwidth based on timestamps of packets in the sequential burst that have not been coalesced with a first or last packet in the sequential burst.

6. The at least one processor for each packet in the data flow, providing the packet for routing through one of the plurality of network connections based on the monitored latencies of the plurality of network connections and the network connections of other packets in the data flow such that the packets in the data flow arrive at a destination node in a desired sequence. The network gateway of claim 1 , configured for:

7. The network gateway of claim 6 , wherein the desired sequence is the original sequence of packets in the data flow.

8. 7. The network gateway of claim 6, wherein the desired sequence is a sequence that includes at least one misordering of packets that does not trigger a retransmission of a packet within the sequence.

9. The at least one processor receiving a packet from a source interface for routing to a destination node via the plurality of network connections; sending an acknowledgment to the source interface before routing the packet to the destination node; storing the packet in at least one buffer before the packet is routed to the destination node; The network gateway of claim 1 , configured for:

10. 10. The network gateway of claim 9, wherein the at least one processor is configured for dynamically controlling a size of the at least one buffer based on a bandwidth-delay product associated with the plurality of network connections.

11. 10. The network gateway of claim 9, wherein the at least one processor is configured for controlling the sending and storing of the packet acknowledgments based on monitoring transmission characteristics of the plurality of network connections and an unequal distribution in receiving data flows of sequential packets.

12. 2. The network gateway of claim 1, wherein the at least one processor is configured to route the packets based on at least one of an estimate of bandwidth of the plurality of network connections and a reduction in a number of network connections through which the data flow is routed.

13. 4. The network gateway of claim 3, wherein the at least one processor is configured to group packets of the plurality of data flows having similar data flow classes and route grouped packets through the plurality of network connections based on the classification of the plurality of data flows.

14. 2. The network gateway of claim 1, wherein the data flow class of one of the plurality of data flows changes automatically upon a threshold volume of data of the corresponding data flow being routed.

15. 2. The network gateway of claim 1, wherein the packet data flow is data packets including at least one of video and audio data.

16. 2. The network gateway of claim 1, wherein the sequential burst of packets includes both test packets utilized to make a bandwidth estimate determination and data packets including at least one of video and audio data.

17. 20. The network gateway of claim 16, wherein the data packets including at least one of the video and audio data transmitted through the sequential bursts of packets are redundant data packets configured to withstand an amount of lost data packets.

18. 1. A method for routing a data flow through a plurality of network interfaces for transmitting data over a plurality of network connections, the method comprising: monitoring time-varying network transmission characteristics of the plurality of network connections; analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class being associated with at least one network interface requirement of the data flow; routing packets within the data flows through the plurality of network connections based on the data flow classes and the time-varying network transmission characteristics; transmitting a sequential burst of packets through a first network interface of the plurality of network interfaces; generating an estimate of a bandwidth of the first network interface based on timestamps recorded as packets in the sequential burst of packets are received at a receiving node and sizes of the packets; routing packets through the plurality of network connections based on the generated bandwidth estimate of the first network interface; Including, Generating bandwidth for the first network interface when the receiving node processes packets received at regular intervals includes: generating a lower bandwidth value by using a reception timestamp of a packet in the sequential burst selected as an end packet in the bandwidth determination; generating an upper limit bandwidth value by replacing a reception timestamp of a packet in the sequential burst selected as the end packet with a reception timestamp of a packet in the sequential burst preceding the end packet; A method comprising:

19. 20. The method of claim 18, wherein monitoring the time-varying network transmission characteristics includes generating a bandwidth-delay product of at least one network interface of the plurality of network interfaces based on the monitored time-varying network transmission characteristics, and routing packets in the data flow is based on the bandwidth-delay product of the at least one network interface.

20. Analysing at least one packet of each of a plurality of data flows of packets; routing packets in each of the plurality of data flows based on a data flow class of each of the plurality of data flows and an available bandwidth of the network connection corresponding to the data flow class of each of the plurality of data flows; 20. The method of claim 18, comprising:

21. 20. The method of claim 18, wherein generating an estimate of the bandwidth of the first network interface includes dividing a sum of packet sizes of packets between a first packet in the sequential burst and a second packet in the sequential burst by a time elapsed between a timestamp for the first packet and a timestamp for the second packet, where the first packet is not a first packet in the sequential burst and the second packet is after the first packet in the sequential burst.

22. 20. The method of claim 18, wherein generating an estimate of the bandwidth of the first network interface includes generating the estimate of the bandwidth based on timestamps of packets in the sequential burst that have not been merged with a first or last packet in the sequential burst.

23. for each packet in the data flow, providing the packet for routing through one of the plurality of network connections based on the monitored latencies of the plurality of network connections and the network connections of other packets in the data flow such that the packets in the data flow arrive at a destination node in a desired sequence.

20. The method of claim 18, comprising:

24. 24. The method of claim 23, wherein the desired sequence is the original sequence of packets in the data flow.

25. 24. The method of claim 23, wherein the desired sequence is a sequence that includes at least one misordering of packets that does not trigger a retransmission of a packet within the sequence.

26. receiving a packet from a source interface for routing to a destination node via the plurality of network connections; sending an acknowledgment to the source interface before routing the packet to the destination node; storing the packet in at least one buffer before the packet is routed to the destination node; 20. The method of claim 18, comprising:

27. 27. The method of claim 26, comprising dynamically controlling a size of the at least one buffer based on a bandwidth-delay product associated with the plurality of network connections.

28. 27. The method of claim 26, comprising controlling the sending and storing of the packet acknowledgments based on monitoring transmission characteristics of the plurality of network connections and an unequal distribution in the reception of sequential packet data flows.

29. 20. The method of claim 18, comprising routing the packets based on at least one of a bandwidth of the plurality of network connections and a reduction in a number of network connections through which the data flow is routed.

30. 21. The method of claim 20, comprising grouping packets of the plurality of data flows having similar data flow classes, and routing the grouped packets through the plurality of network connections based on the classification of the plurality of data flows.

31. 20. The method of claim 18, wherein the data flow class of one of the plurality of data flows changes automatically upon a threshold volume of data of the corresponding data flow being routed.

32. 20. The method of claim 18, wherein the packet data flow is data packets containing at least one of video and audio data.

33. 20. The method of claim 18, wherein the sequential burst of packets includes both test packets utilized to make a bandwidth estimate determination and data packets including at least one of video and audio data.

34. 34. The method of claim 33, wherein the data packets including at least one of the video and audio data transmitted through the sequential bursts of packets are redundant data packets configured to withstand an amount of lost data packets.

35. A computer readable medium storing a set of machine-interpretable instructions which, when executed, cause one or more processors to perform the method of any one of claims 18 to 34.

Citation Information

Patent Citations

  • Mobile communication packet transfer control system

    JP1992010720A

  • Path controller and method

    JP2003046548A

  • Communication network and packet transfer device

    JP2004159146A

  • Method and apparatus for robust real-time estimation of bottleneck bandwidth in the Internet

    JP2004524782A

  • Communication device concurrently using a plurality of paths according to application characteristic

    JP2006060579A