System and method for transmitting data over unreliable connections

Abstract

An improved data packet communication technique adapted for communication over unreliable connections is described. Specifically, the technique can be implemented as a system and method for a networked router device configured to monitor communication characteristics and group connections into various hierarchies based on the communication reliability data. When a new packet is to be communicated, the grouped connections are aggregated and utilized to meet a target transmission reliability probability (e.g., a target value or range of values).

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application of and claims all benefit, including priority, of U.S. Patent Application No. 63 / 032,180, filed May 29, 2020, entitled "SYSTEMS AND METHODS FOR DATA TRANSMISSION ACROSS UNRELIABLE CONNECTIONS," which is incorporated herein by reference in its entirety.

[0002] This application is related to U.S. Application No. 16 / 482,972, filed December 21, 2017, entitled "PACKET TRANSMISSION SYSTEM AND METHOD," which is incorporated herein by reference in its entirety. Technical Field

[0003] TECHNICAL FIELD Embodiments of the present disclosure relate generally to the field of data networking, and more particularly, embodiments relate to apparatus, systems, and methods for communicating data over unreliable connections. [Background technology]

[0004] If the network is unreliable (expressed in terms of packet loss, latency spikes, or complete (but temporary) loss of connectivity), solutions such as ARQ (automatic repeat request) or SMPTE-2022-7 (sending duplicate packets over multiple channels) can be used to reduce the impact on the overall transmission.

[0005] For example, with ARQ, packets sent over unreliable connections are often lost / delayed and therefore retransmitted (possibly over the same connection, or over a different connection in the case of a mixed system), which incurs a latency cost, which appears to the application as excessive jitter. With SMPTE-2022-7, copies of every packet are sent over multiple channels, so the cost is in bandwidth usage (using six connections equals six times the bandwidth requirement). Summary of the Invention

[0006] A mixed-connection system can utilize more advanced techniques for handling unreliable connections because it can take advantage of the strengths and weaknesses of the connections available to the system, thereby providing a more reliable connection overall. For example, U.S. application Ser. No. 14 / 360,372 (issued as U.S. Pat. No. US9,357,427), entitled "DEVICE AND METHOD FOR CHARACTERIZATION AND OPTIMIZATION OF MULTIPLE SIMULTANEOUS REAL-TIME DATA CONNECTIONS," which is incorporated herein by reference in its entirety, describes how a high-latency, high-throughput satellite connection can be combined with a low-latency, low-throughput terrestrial connection to provide ARQ for the satellite connection using the terrestrial connection with little impact on the overall latency and reliability of the system.

[0007] Other hybrid systems may use heuristics or techniques, such as machine learning, to predict when a connection will become unreliable and then stop using the connection just before that transition occurs.

[0008] An alternative approach to potentially improving efficiency and bandwidth, which may lead to improved cost, reliability, and latency management, is described in some embodiments. The alternative approach is directed to a system configured to recognize that a connection's unreliability may be partial, rather than either completely reliable or completely unreliable. Thus, a system can be configured to efficiently obtain some benefit from unreliable connections. The system is adapted to monitor communication flows over a period of time or receive communication characteristic data from a third-party system and use this information to adjust, schedule, or control how data is communicated (e.g., how packets are sent or which network connections are used). The information is efficiently tracked in a data structure maintained over time to reduce the total number of computational steps required when routing decisions are made (each computational step at runtime impacts performance, which, when aggregated, can be significant for highly scaled systems sending large numbers of packets).

[0009] This approach is a technical and computational solution that can be implemented in the form of a physical data router or other networking device configured to control the routing of data packet communications. Other possible types of networking devices include gateways, switches, bridges, repeaters, hubs, or access points.

[0010] The apparatus includes one or more processors operating in conjunction with computer memory, which may be coupled to data storage, a corresponding method, and a non-transitory computer-readable medium (e.g., a diskette, solid-state storage, hard disk drive) storing machine-interpretable instructions (e.g., software that, when executed by a computer processor, causes the processor to perform the methods described herein in various embodiments). The non-transitory computer-readable medium may further include rule-based logic or stored routing tables that can be referenced to modify the routing path for communicating data packets or data streams.

[0011] The technique described here, termed "Probabilistic Forward Loss Correction (PFLC)," is an extension of techniques previously described for mixed connectivity systems. This extension involves exploiting the properties of the available connections to improve the latency and reliability of the application flows being served.

[0012] The evolved techniques are adapted to improve the overall efficiency of communication resource usage, which is particularly important in situations where communication resources are constrained (e.g., limited routes, e.g., rural locations), congested (e.g., many simultaneous users, e.g., sports complexes), or cost is a consideration (e.g., when there are connections available with varying levels of cost and reliability, e.g., satellite connections using radio or microwave frequencies with limited frequency range or band).

[0013] A technical advantage of the techniques described herein is that they can achieve more efficient targeting compared to other technical protocols that target packet reliability in data communications (e.g., transmission / reception).

[0014] From a reliability perspective, as described in various embodiments, maintaining a "Goldilocks" level of reliability can be important, where redundant connections are utilized to establish a reliability threshold, range, or level. The technical objective is to achieve a level of reliability without overcommitting resources, e.g., bandwidth, to unnecessarily achieve the target level of reliability desired by the application flow. Challenges with prior mixed approaches include using excessive resources to achieve the desired reliability (e.g., SMPTE-2022-7), excessive latency (e.g., simple ARQ), or excessive loss (e.g., blindly using unreliable connections and requiring applications to handle lost packets).

[0015] While resource overcommitment may result in reliable transmission for a particular communication, it may be wasteful in terms of overall resources, constraining communication or other communications for other users, or increasing overall communication costs. An example of resource overcommitment would be using network resources to repeatedly send packets over a reliable connection that is at least 99.999% reliable, when the specification only requires a 95% success rate. Another aspect of waste may relate to less reliable / unreliable connections being increasingly underutilized. For example, a connection that transmits only 65% ​​reliably may be completely ignored by other implementations, a waste of available communication resources.

[0016] The techniques described herein can be used in combination with other corrective protocols or techniques for transmitting data packets, such as best-effort transmission (e.g., user datagram protocol (UDP)) and stream-based reliable transmission protocols (e.g., transmission control protocol (TCP)). The device can control communications at different layers in the OSI protocol stack, such as the application layer, transport layer, network layer, data link layer, or physical network layer.

[0017] In some embodiments, a network controller device (e.g., a router) is described, the device including a processor coupled to computer memory and data storage, the processor configured to: receive one or more data sets indicative of monitored network communication characteristics; maintain in a data structure stored on the data storage a hierarchical representation of a plurality of connections separated into a plurality of groups, each group established based at least on a minimum probability associated with successful communication of a data packet over one or more of the plurality of connections residing within that group; and control a plurality of communications of the data packet, wherein the data packet is sent at least once over one or more of the plurality of connections such that, collectively, the plurality of communications satisfy a target probability threshold for transmission of the data packet.

[0018] In another aspect, the multiple groups are organized into multiple corresponding tiers, each tier representing the potential number of times a data packet must be transmitted over a corresponding connection in one of the tiers to achieve a target probability threshold.

[0019] In another aspect, the multiple communications include retransmissions over a connection of one of the multiple tiers, and the number of retransmissions is based on the number of times a data packet must potentially be transmitted over the corresponding connection of that tier to achieve a target probability threshold for the corresponding tier. In an alternative embodiment, the retransmissions may be performed over the same connection but at different times (i.e., in separate bursts) to prevent any retransmission losses due to temporary degradation in network reliability (i.e., burst losses due to buffer saturation on intermediate hosts in the path).

[0020] In another aspect, the retransmission is performed over a different connection in the hierarchy.

[0021] In another aspect, the plurality of communications includes retransmissions over connections at different levels of the plurality of levels.

[0022] In another aspect, the additional group is an uncertain group established for connections that do not have enough data to assess their reliability.

[0023] In another aspect, the membership of this uncertain group is periodically revised to classify connections into multiple groups.

[0024] In another aspect, connections in multiple groups are periodically monitored to transfer membership to uncertain groups where connection data is out of date or showing increasing unreliability.

[0025] In another aspect, uncertainty groups are utilized for communicating data packets using seeded reliability probabilities.

[0026] In another aspect, the seeded reliability probabilities are periodically adjusted based on monitored network communication characteristics.

[0027] In another aspect, connections that have required more than a threshold number of transmissions or retransmissions are moved to an uncertain group.

[0028] In another aspect, the processor is further configured to periodically transmit data packets over the connections in the uncertainty group, the periodic transmission incorporating a back-off timer to reduce overall system inefficiencies.

[0029] In the drawings, embodiments are shown by way of example, and it is to be expressly understood that the present specification and drawings are for illustrative purposes and as an aid to understanding only.

[0030] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0031] [Figure 1A] FIG. 1A is a data flow diagram illustrating the four components that comprise the methodology described herein. [Figure 1B] FIG. 1B is an exemplary block diagram of a communication gateway device according to some embodiments. [Figure 1C] FIG. 1C is a schematic diagram of an exemplary system describing an alternative approach that is less efficient than some of the approaches described herein in various embodiments. [Figure 2] FIG. 2 is a system-level diagram illustrating an exemplary embodiment of a scheduler mechanism that interoperates with other controller components and a flow classification engine, according to some embodiments. [Figure 3] FIG. 3 is a diagram illustrating a method for generating window sizes and ensuring sample validity, according to some embodiments. [Figure 4] FIG. 4 is a flow diagram illustrating a grouping technique of some embodiments described herein. [Figure 5] FIG. 5 illustrates an exemplary implementation, according to some embodiments. [Figure 6] FIG. 6 is a diagram illustrating state transitions between three states according to some embodiments. [Figure 7] FIG. 7 is an exemplary schematic diagram of a coordinated approach to grouping connections, according to some embodiments. [Figure 8] FIG. 8 is a block schematic diagram of an exemplary computing device, according to some embodiments. [Figure 9] FIG. 9 is a diagram illustrating a physical computer server rack, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0032] 1A is a data flow diagram 100A illustrating four components comprising the techniques described herein. These components are described as logic mechanisms and can be implemented in the form of hardware (e.g., electronic circuits, printed circuit boards, field programmable gate arrays) or software, or embedded firmware. As described in various embodiments herein, some or all of the components can be incorporated into physical network communication devices, such as routers, gateways, switches, bridges, repeaters, hubs, or access points.

[0033] In other embodiments, some or all of the components may be incorporated as corresponding network interface controllers, for example on a server that controls routing, etc., and the output of the device may be a data structure representing instructions for controlling routing, for example a routing table that is periodically or continuously updated to control communications as described herein.

[0034] Application flow identification 001 specifically deals with determining flow reliability requirements. In some embodiments, this may include techniques such as deep packet inspection (DPI), coded rules and heuristics, machine learning, or user-configured hints. For example, a Voice over Internet Protocol (VoIP) application flow may require 99% reliability for its packets, while a temperature sensor reporting periodic observations may only require 80% reliability for its packets.

[0035] The purpose of connection measurement and observation 002 is to determine and track the current and historical reliability of available connections. For example, in some embodiments, this may involve measuring and tracking historical packet loss, monitoring external metrics such as wireless signal strength, considering user / administrator hints, or machine learning techniques.

[0036] Connection grouping 003 groups connections based on their historical, current, or predicted future characteristics. For example, if there are four connections, two with 1% loss and two with 10% loss, an exemplary embodiment might put the first two connections in one group and the remaining two in another. Another exemplary embodiment might put a connection that is not similar to any other connection in its own group.

[0037] The scheduling 004 of flows to connection groups is intended to be done in an intelligent manner that meets flow requests while efficiently using connections. Using the example values ​​in the previous paragraph, one possible scheduling combination is as follows: for each VoIP packet, the system can be configured to transmit it once over any of the connections in the 1% loss group, or twice over the connections in the 10% loss group. Assuming loss rates are independent and have uniform distributions, either option would achieve or exceed the target reliability of 99%. In the above example, the system could also transmit the temperature sensor packet once over any of the available connections. Either connection on its own would meet or exceed the reliability target of 80%.

[0038] The assumption of independence and / or uniform distribution is not necessary in all embodiments: for example, there may be other approaches to overcome non-uniform distribution and correlated loss rates, such as tracking correlations or interdependencies and intentionally using low-correlated combinations of unreliable connections, or combinations of connections from different tiers.

[0039] For example, a background computational process may be instantiated (either locally or using a central system) configured to monitor statistical dependencies (e.g., track historical fluctuations) or flag types of connections that may be expected to be dependent (e.g., two modems from the same carrier, or that may be using links from the same tower and internet tower even if they operate at different frequencies, or from different carriers (e.g., Telus and Bell) that do not otherwise interact).

[0040] Connections can similarly be evaluated using, for example, header information, base station information, connection identifiers, etc. For example, there may be non-traditional connections available to be provisioned on an as-needed or priority basis, which are available to certain types of users (e.g., defense and first responder customers with dedicated channels, or priority access request capabilities).

[0041] In another embodiment, when correlations or interdependencies are known but unavoidable (e.g., there are only a few connections available, making correlations or interdependencies between enabled connections likely), the approach can include increasing the discount factor for connection reliability, or in another embodiment, simply treating a connection reliability "tier" as a lower tier. For example, if connection A (tier 3) is used for routing in combination with connection B (tier 3) and there is an interdependent correlation between them, one of connections A or B can be logically shifted during routing and treated as tier 4. Thus, a packet may be sent four times (e.g., twice on connection A and twice on connection B) instead of three times.

[0042] The hierarchy is stored in a hierarchical representation on data storage. This representation can include the use of data fields and data values ​​that link the hierarchy and its membership within a group to each specific identifier of a connection. During system operation, the data values ​​can be referenced to control specific system behavior, such as routing decisions for data packets. The hierarchical representation can be updated periodically or dynamically over time as the reliability of different connections changes or as more information becomes available. In some embodiments, multiple connections are also assigned state representation values ​​that are utilized in a state machine that transitions connections through different "states" assigned to them, automatically promoting or demoting them from trusted, unreliable, uncertain, etc. Other intermediate states are possible. These states can be transitioned through the use of automatic trigger conditions and can be logically represented through the use of associated list items or other types of data objects adapted for automatic state transitions. Both the state representation and the hierarchical representation can be utilized as part of the logical control flow when selecting a communications interface to use to communicate data packets.

[0043] As described in various embodiments herein, an important consideration is the target probability threshold. Another countervailing consideration is the importance of practical constraints on bandwidth efficiency (e.g., ensuring a "sufficient percentage of goodput" in communications). The mechanisms described herein can be used to automatically balance these considerations and can provide a technically useful mechanism for improving the overall reliability of data packet transmissions in light of limited network resources.

[0044] Furthermore, the techniques described herein are particularly useful when connection reliability is inconsistent (e.g., satellite connections where connection quality varies with cloud cover, or heavily congested networks with unpredictable usage patterns). Chunks of unreliable connections may be used together to achieve a target probability of transmission through retransmission. Layering connections into different layered representations and groups and maintaining adaptive data structures that track changes over time is useful for maintaining computational ease and reducing the overall computational overhead required to determine the reliability of individual connections.

[0045] Routing systems that utilize hierarchical connections update their hierarchies, for example, as the routing system moves through a geospatial region (e.g., when the routing system is mounted on a vehicle or carried by a person) or as conditions change for communication stations that are in stationary locations but subject to large variations in connection quality (e.g., rural communication stations in developing countries where infrastructure is not yet in place, or when connectivity is provided by a constellation of satellite orbital devices whose reliability changes over time as the orbital devices pass through a defined path overhead). The use of unreliable connections becomes feasible, albeit at the expense of transmission efficiency, when packets are repeatedly retransmitted based on automatic probabilistic modeling of the routing system through a hierarchical representation for routing control.

[0046] An elegant mechanism for reducing the likelihood of a "goodput" percentage threshold violation is to establish a limit on the number of transmissions. If the tiering number corresponds to the number of transmissions, this means that the "goodput" percentage threshold can simply be translated to the highest tier before the connection becomes unused. The "goodput" percentage threshold can be adjusted (e.g., on a user interface or where the target transmission probability is not achievable) so that the number of available tiers can be varied to modify the technical trade-off between reliability and bandwidth utilization efficiency. In some embodiments, the "goodput" percentage threshold is adjusted manually (e.g., on a slider by a network administrator); in other embodiments, the "goodput" percentage threshold is adjusted automatically by the system based on a set of stored logic rules. Different thresholds can be applied to different types of communications; for example, communications marked as highly urgent may be sent despite a very low goodput percentage, and vice versa.

[0047] To achieve a target probability of successful transmission, connections through different tiers can be mixed, for example, allowing transmissions at tier 2 and tier 3. Grouping connections into tiers in advance is useful in that connections can be easily mixed to approximately achieve the desired probability without too many calculations during routing decisions (e.g., each required calculation slows down the system's performance).

[0048] The tiered representation may also include unallocated connections awaiting tier allocation, and in some embodiments, all of these connections are associated with a state representation, e.g., undetermined, or in other embodiments, all of these connections are indicated as unreliable. In some embodiments, data packets or validation packets are periodically transmitted through the unallocated connections to automatically examine the characteristics of the connections and determine whether they can be promoted to an available tier or whether a state transition can occur.

[0049] The use of unallocated connections can be used, for example, to probe unallocated connections in overflow situations when there are not enough connections in the tier to achieve a target threshold without violating "goodput," or periodically in normal use even when there is no overflow situation. Unallocated connections can be assigned a seeded reliability probability that can be updated over time as observations of transmission characteristics are made. Each unallocated connection can be associated with a timer that is configured to reset each time the unallocated connection is probed, preventing the connection from being revalidated until a certain period of time has passed.

[0050] Specific embodiments will have different approaches for each of 001-004 and will be subsequently described in more detail.

[0051] 1B illustrates a mixed connection aggregation system 100 configured to utilize an improved scheduling technique on the transmit portion of the system and a buffering system along with packet ordering on the receive end. The components in the illustrated system are, in one embodiment, hardware components configured to interoperate with one another. In another embodiment, the hardware components are not separate components, but rather two or more of the components may be implemented on a particular hardware component (e.g., a computer chip that performs the functions of two or more of the components).

[0052] In some embodiments, the hardware components reside on the same platform (e.g., the same printed circuit board), and system 100 is a single device that can be transportable, connected to a data center / portable device in the field (e.g., a rugged mobile transmitter), etc. In other embodiments, the components are distributed and need not all be located in close proximity, but rather communicate electronically through telecommunications (e.g., processing and control is not performed locally, but rather by components resident in a distributed resource environment (e.g., the cloud)).

[0053] Providing mixed connectivity is particularly desirable in mobile scenarios where signal quality, network availability, quality network, etc. are not optimal (e.g., professional newsgathering / video production may occur in locations lacking a robust network infrastructure). Thus, in some embodiments, the devices described herein are rugged network controller devices adapted for operation in remote or mobile scenarios, and may be, for example, person-portable (e.g., worn in a backpack by a journalist-technician), cart-mobile (e.g., pushed around on a media cart), or vehicle-mobile (e.g., coupled to a media van, truck, boat, helicopter, or aircraft).

[0054] Alternatively, the devices may be located at a communications hub or other type of centralized communications facility that coordinates connections to further endpoints. For example, in this example, the devices may be located at, coupled to, or resident in communications facilities, such as communications relay sites (e.g., satellite stations, relay transmitting stations, broadcast translation stations, rebroadcast stations, relay stations, supplemental stations) that can coordinate communications over multiple different channels to various endpoints, which may be television stations, radio stations, data communications stations, personal computers, mobile devices, etc.

[0055] A number of different data connections 106 (e.g., "paths") representing one or more networks (or network channels) are shown and are labeled Connection 1, Connection 2, ..., Connection N. There may be multiple data connections / paths through a single network, or multiple data connections that may use one or more networks.

[0056] These data connections may include various types of technologies, each of which may face different communication conditions, e.g., having different types of available communication power, array architectures, polarizations, multipath propagation mechanisms, spectral interference, frequency ranges, modulations, etc., and accordingly, the connections may all have different levels of communication reliability.

[0057] The system 100 may be configured to communicate to various termination points 102, 110 or applications (e.g., termination points 102, 110 may function independently of the paths or connections 106) that do not need to have information about the multiple paths / connections 106 used to request and receive data. Received data can be reconstructed, for example, such that the original transmission can be reproduced from the contributions of the different paths / connections 106 (an exemplary usage scenario could be the playback of video by a receiver configured to plug into a server rack in a data center facility to integrate with existing broadcast infrastructure and provide improved networking capabilities).

[0058] The system 100 receives input (data flow) from a source endpoint 102, schedules improved data packet delivery over various connections 106, and then sequences the data packets at the other end 108 of the system prior to transmission to a destination endpoint application 110. In doing so, the system 100 is configured to increase bandwidth to approach the sum of the maximum bandwidths of the various available paths. Compared to using a single connection, the system 100 also provides improved reliability, which can be an important consideration in time-sensitive and highly sensitive scenarios, such as live event news gathering during an event. These events may have high signal congestion (e.g., a sporting event) or unreliability over one or more paths (e.g., reporting news after a natural disaster).

[0059] Multiple connections can be combined to act as a single connection, and different techniques can be used to coordinate routing, among other things allowing a connection to be used multiple times to send the same packet.

[0060] In various embodiments, both the scheduler 160 and the sequencer 162 can be provided from a cloud computing implementation, or at the termination point (before the data is consumed by the application at the termination point), or various combinations thereof.

[0061] System 100 may be tuned to optimize or prioritize performance, best latency, best throughput, minimum jitter (variation in latency on packet flows between two systems), cost of connections, combinations of connections for particular flows, etc. (For example, if system 100 has information that a transmission (data flow) is of type X content, system 100 may be configured to use only data connections with similar latency, while type Y content may allow a wider combination of data connections (or may require a larger net capacity that can only be achieved using a combination of data connections)).

[0062] The above adjustments may be provided to the system generally, or may be provided specifically to each flow (or set of flows based on location, ownership of either or a combination of start and / or end points, time of transmission, set of available communication links, security required for transmission, etc.).

[0063] The system 100 may generally be bidirectional in that each gateway 104, 108 generally has a scheduler 160 and a sequencer 162 to handle TCP traffic (or UDP traffic, or a combination of TCP and UDP traffic, or any type of general Internet Protocol (IP) traffic), although in some embodiments only one gateway may be required.

[0064] System 100 may be utilized in a variety of scenarios, for example, as a failover (e.g., for disaster recovery) or supplement to an existing Internet connection (e.g., a VoIP telephone system or a corporate connection to the Web), whereby an additional network (or path) is seamlessly added to either replace a dropped primary Internet connection or to supplement a saturated primary Internet connection by combining it with an even more expensive network. In a failover situation, there may be coordinated failures across many communication channels (e.g., a large-scale denial-of-service attack or a large solar / geomagnetic storm), and system 100 may then be required to efficiently use communication resources because available communication resources are in short supply.

[0065] Another use of system 100 is to provide a means of maximizing the use of high-cost (often sunk-cost), highly reliable data connections, such as satellite, by enabling the offloading of traffic to other data connections with different attributes.

[0066] In some embodiments, the system 100 is a network gateway configured to route data flows over multiple network connections.

[0067] 1B shows an overview of a system having two gateways 104 and 108 coupled by N data connections 106, each of which includes a buffer manager 150, an operations engine 152, a connection controller 154, a flow classification engine 156 (responsible for identifying and classifying flows), a scheduler 160, a sequencer 162, and a network characteristic monitoring unit 161, with each gateway connected to a specific termination point 102, 110. Reference letters A and B are used to distinguish between the respective components of the two gateways 104 and 108.

[0068] Each gateway 104, 108 is configured to include multiple network interfaces that transmit data over multiple network connections, and is an apparatus (e.g., including configured hardware, software, or embedded firmware) that includes a processor configured to: monitor time-varying network transmission characteristics of the multiple network connections; analyze at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class defining or otherwise associated with at least one network interface requirement for the data flow; and route packets of the data flow over the multiple network connections based on the data flow class and the time-varying network transmission characteristics.

[0069] Buffer manager 150 is configured to configure buffers within the gateway adapted to more efficiently manage traffic (both individual flows and combinations of multiple simultaneous flows passing through the system). In some embodiments, buffer manager 150 is a separate processor. In other embodiments, buffer manager 150 is a computational unit provided by a processor configured to perform buffer management 150, among other activities.

[0070] The operations engine 152 is configured to apply one or more deterministic methods and / or logical operations based on the received input data sets (e.g., feedback information, network congestion information, transmission characteristics) to inform the system about the constraints that will be applied to the mixed connections per user / client, destination / server, connection (e.g., latency, throughput, cost, jitter, reliability), flow type / request (e.g., FTP vs. HTTP vs. HTTP vs. streaming video).

[0071] As an illustration, the operations engine 152 may be configured to limit certain types of flows to a particular connection or set of data connections based on cost in one instance, while reliability and low latency may be more important for different users or flow types. Different conditions, triggers, and methods may be utilized depending, for example, on one or more components of known information.

[0072] The operations engine 152 may be provided, for example, on the same or a different processor as the buffer manager 150.

[0073] The operations engine 152 may be configured to generate, apply, or otherwise manipulate or use one or more sets of rules that determine the logical operations to be performed in controlling routing on the N data connections 106.

[0074] The flow classification engine 156 is configured to evaluate each data flow received by the multipath gateway 104 for transmission and to apply a flow classification technique to determine the type of traffic being transmitted and its requirements if they are not already known. In some embodiments, deep packet inspection techniques are adapted to perform the determination. In other embodiments, the evaluation is based on heuristics or on data flows that are marked or labeled at the time of creation. In other embodiments, the evaluation is based on rules provided by a user / administrator of the system. In other embodiments, a combination of methods is used.

[0075] The flow classification engine 156 is configured to interoperate with one or more network interfaces and may be implemented using electronic circuits or a processor.

[0076] Scheduler 160 is configured to make decisions regarding which packets (and their amount of redundancy) are desirable to send down which connections 106. Scheduler 160 may be thought of as an improved Quality of Service (QoS) engine. Scheduler 160 may include a series of logic gates backed by the decisions it makes.

[0077] Scheduler 160 is implemented in some embodiments using one or more processors, or a standalone chip or configured circuit, such as a comparator circuit or FPGA.

[0078] A typical QoS engine manages a single connection, and the QoS engine (or in this case, scheduler 160) may be configured to perform flow identification and classification, with the end result being that the QoS engine reorders packets before sending them on a connection.

[0079] In contrast, while scheduler 160 is configured to perform flow identification, classification, and packet reordering, scheduler 160 in some embodiments is further configured to make decisions about which connections to send packets over in order to provide improved transmission characteristics to the data flow and / or to satisfy policies set for the flow by a user / administrator (or defined in various rules). Scheduler 160 may, for example, modify network interface operating characteristics by sending a set of control signals to the network interfaces to turn them on / off or indicate which ones should be used to route data. The control signals may be a set of instructions that indicate certain characteristics of the desired routing, such as packet timing, reserving network interfaces for particular types of traffic, etc.

[0080] For example, consider two connections with the following properties:

[0081] Connection 1: 1ms round trip time (RTT), estimated bandwidth of 0.5Mbps; and

[0082] Connection 2: 30ms RTT, estimated bandwidth of 10Mbps.

[0083] Scheduler 160 may attempt to reserve connection 1 exclusively for Domain Name System (DNS) traffic (small packets, low latency). In this example, there may be so much DNS traffic that it reaches the capacity of connection 1, i.e., scheduler 160 may be configured to overflow traffic onto connection 2, but scheduler 160 may do so selectively based on other decisions or factors (e.g., if scheduler 160 is configured to provide fair decisions, scheduler 160 may be configured to first overflow traffic from IP addresses that have already sent a significant amount of DNS traffic in the past X seconds).

[0084] Scheduler 160 may be configured to process its decisions based on, for example, a process or method operating in conjunction with one or more processors or similar implementations in hardware (eg, FPGAs).

[0085] The scheduler 160 may be configured to operate under the control of the operations engine 152, breaking down the data stream into data packets and then routing the data packets to buffers (managed by the buffer manager 150) that supply the data packets to the data connection according to rules that attempt to optimize packet delivery while taking into account the characteristics of the data connection.

[0086] Scheduler 160, in one embodiment, is a data communications scheduler that receives a stratum number as a parameter when determining how many times and over which connection a packet (or portion thereof) is desired to be transmitted. Scheduler 160 is adapted to control the forwarding of data packets, for example, by adding header information or by controlling transmission over routing tables or routing policies stored in a reference data structure on computer memory or data storage.

[0087] The routing table or routing policy may be updated periodically or continuously, as described in various embodiments herein. In some embodiments, the routing table or routing policy may be stored (e.g., as an article of manufacture) as computer-interpretable instructions residing on a non-transitory computer-readable medium. The updates include adapting the repeated transmission of data packets to achieve the lowest probability of data loss using connections of questionable reliability. The routing table may be a data table in the scheduler 160 or router that lists routes to particular network destinations and may include various metrics and may be based on the sensed topology of various network connections.

[0088] As described herein, discovery techniques can also be used to adapt to new interfaces with unknown reliability or known interfaces whose reliability has changed (e.g., from trusted to unreliable or vice versa). A routing table illustratively includes network / next hop associations stored as data structures or linked data components for controlling the relaying of packets to their destinations.

[0089] Routing tables can be used to generate forwarding tables, which in some instances provide separate forwarding control and can provide a compressed or precompiled approach to optimize hardware storage and backup. Routing / forwarding tables can include intermediate network destination addresses (e.g., IPv4, IPv6), netmasks, gateways, interface addresses, and metrics.

[0090] A mixed-connection system can utilize advanced techniques for dealing with unreliable connections 106 because the system can use the strengths and weaknesses of the available connections in a way that provides a more reliable connection overall. For example, as described in U.S. patent application Ser. No. 14 / 360,372 (issued as US9,357,427), entitled "DEVICE AND METHOD FOR CHARACTERIZATION AND OPTIMIZATION OF MULTIPLE SIMULTANEOUS REAL-TIME DATA CONNECTIONS," which is incorporated herein by reference in its entirety, if a high-latency, high-throughput satellite connection is paired with a low-latency, low-throughput terrestrial connection, the terrestrial connection can be used to provide ARQ for the satellite connection with little impact on the overall latency and reliability of the system.

[0091] Other hybrid systems may use heuristics or techniques such as machine learning to predict when a connection will become unreliable and then stop using the connection just before that transition occurs.

[0092] 1C is a schematic diagram 100C of one such hybrid system that describes a less efficient approach than the approaches subsequently described herein in various embodiments. This approach is shown as an example to illustrate some of the technical challenges that lead to less efficient data transmission and / or communication.

[0093] In Figure 1C, connections below the target reliability threshold are not used to send new data: they remain in the unreliable (UNR) state and are only used to send either duplicates of packets already sent on reliable (REL) connections, or empty "dummy" [D] packets.

[0094] 1C(A) shows the initial state, where six packets are available for transmission, but both connections C1 and C2 are currently in the UNR state. As a result, they can only send empty "dummy" [D] packets; the six packets in the input queue are still not being delivered. This is the main drawback of this embodiment. Even though C1 and C2 are UNR, they are likely not dropping 100% of all packets sent; data is still in the input queue and not being delivered. Some of the dummy [D] packets are arriving, and they could potentially serve a more useful purpose if they instead contained one of the six packets in the input queue.

[0095] Figure 1C(B) shows the next state, where connection C1 has transitioned to the REL state and is now able to service the input queue. It transmits packets [1] and [2] and places them in flight towards the receiver. Connection C2 remains in the UNR state and can send only duplicates of packets 1 and 2 ([1R] and [2R] respectively), or an empty "dummy" [D] packet.

[0096] 1C(C) shows the next state, where connection C2 has also transitioned to the REL state. Connection C1 transmits packets [3] and [4], and connection C2 transmits packets [5] and [6]. Also, in this example, packet [2] previously transmitted on C1 has been lost, causing C2 to serve the ARQ request by transmitting another copy of packet 2 [2R]. This is another technical drawback of this embodiment: a conventional ARQ request handling packet 2 would result in packet 2 arriving at the receiver with high latency because an extra round trip was required.

[0097] The improved approach for handling unreliable connections in a mixed connection system described herein is referred to as PFLC and consists of four components 001, 002, 003, and 004, which are further illustrated in Figure 2. Figure 2 is a diagram 200 illustrating a scheduler 160 that interoperates with other components to control the communication and / or routing of data packets.

[0098] Flow identifier 001 identifies the flow of the reliability request. The flow classification engine 156 groups and tracks packets arriving at its input into logical flows based on techniques such as the IP5 tuple (e.g., source IP address / port, destination IP address / port, protocol). Other techniques are possible. Each of these flows contains data related to a particular application or conversation (e.g., a two-way VoIP session between two endpoints 102 or 110).

[0099] The flow identifiers also determine the reliability requirements for these flows, which in some embodiments may be accomplished by utilizing techniques such as deep packet inspection (DPI), coded rules and heuristics, machine learning, or user-configured hints.

[0100] For example, in FIG. 2, packets comprising flow 290 match a user-configured rule for Session Initiation Protocol (SIP) VoIP traffic on UDP port 5061, requesting 99% reliability.

[0101] The connection manager 002 is configured to measure and observe connection properties. The connection controller 154 is configured to track the historical, current, and predicted future reliability for each connection. For example, in some embodiments, this may include measuring packet / byte loss, monitoring external metrics such as wireless signal strength, considering user / administrator hints, or machine learning techniques.

[0102] The information may be "crowdsourced" (using nearby or "friendly" (e.g., corporate-owned) devices) based on a set of rules, machine learning, etc., or obtained through other methods. Crowdsourced data may be provided in the form of an input data set representing monitored information, which is then utilized by system 200 to modify how routing is performed for communications.

[0103] The system 200 may collect statistics (cost, loss, latency, etc.) for each combination of, but not limited to, the following: 1. Precise location (e.g. GPS coordinates) 2.Technology (2G, 3G, 4G, satellite,...) 3. Accurate time and date 4. Weather if possible 5. Carrier / Provider 6. Physical equipment (satellite antenna size, cell phone antenna form factor, etc.) 7.Other

[0104] Optimization techniques (e.g., machine learning techniques) can be applied to predict how data communications will perform. This information can be reported and accessed in a variety of ways, including peer-to-peer or through a central server that manages and weights various sources of information (e.g., by reliability or correlation). For example, if there is geographic specific fade, geolocation, or historical data may indicate the need to change processes based on location and carrier, for example. The processing of these statistics for decision making can occur on the device, on a server and pushed down, or a combination of device and server.

[0105] In some embodiments, connection controller 154 will combine its observed and predicted results into a single scalar value, p, per connection, which is the probability of independent, uniformly distributed loss. For example, in Figure 2, connections C1 and C2 are combined from the observed results as each having a p-value of 1%. Connections C3 and C4 are combined from the observed results as each having a p-value of 10%.

[0106] Some embodiments of the connection controller 154 may expose other aggregated and unaggregated observations of the underlying connections, since independent or non-uniformly distributed losses may be difficult to aggregate into a single scalar value. For example, some embodiments may include a background process (locally or with a central system) that analyzes the observations for statistical dependencies, or may pay special attention to some types of connections that may be expected to be potentially dependent based on rules / heuristics (e.g., two modems that belong to the same wireless communication carrier or from different carriers that are known to share infrastructure, or other types of connections that have some kind of interdependency in behavior due to sharing of underlying resources).

[0107] The third component 003 is a grouping of connections that have similar historical, current, or predicted future connection properties. In some embodiments, this grouping is done by the connection controller 154 based on the aggregated scalar value of p.

[0108] For example, in Figure 2, connection controller 154 places connections C1 and C2 in one group and connections C3 and C4 in a second group. Other embodiments might group based on other dimensions, using aggregated or non-aggregated observations of other underlying connection properties. For example, grouping might be based on connection cost (higher cost connections are grouped with lower priority). Another example might be grouping connections based on RTT, allowing packets with deadlines to be preferentially scheduled onto the group of connections most likely to meet their deadlines.

[0109] The fourth component 004 is to schedule packets over groups of connections in a manner that meets flow reliability requirements while simultaneously considering other constraints such as cost, efficiency, and latency.

[0110] 2, scheduler 160 scheduled packets comprising flow 290 in a manner that ensured all packets met a target reliability of 99%. Specifically, packets 291 and 292 were scheduled onto connections C1 and C2 (respectively), so that each of the connections could meet the target reliability on its own. Packet 293 was duplicated and sent once on each of C3 and C4. In this example, the combined values ​​of p for each connection were known to be uncorrelated / independent loss events, so the two transmissions of the packet resulted in a combined reliability of 99%:

number

[0111] The following paragraphs provide a more detailed description of a particular embodiment of the second component 002 of the PFLC: Measurement and observation of connection properties.

[0112] In some embodiments, the reliability of a particular connection is assessed by measuring the amount of loss experienced by that connection: the reliability of a particular connection is inversely proportional to the amount of loss experienced by that connection.

[0113] In one embodiment, the amount of loss experienced by a particular connection is measured as the percentage of packets lost on this connection. A sender divides the packets it needs to send over a particular connection into groups.

[0114] Each group is assigned a unique identifier. Packets for a particular group are tagged with that group's unique identifier. The receiver records the unique identifiers of all groups it sees and counts the number of received packets tagged with these unique identifiers. To avoid double counting, duplicate packets are discarded. The receiver periodically reports the count back to the sender. The sender can then determine the percentage of lost packets by calculating the difference between the number of packets sent and the number of packets received and dividing this difference by the number of packets sent. Each group unique identifier results in one loss percentage value.

[0115] In another embodiment, the percentage loss value is calculated using the number of bytes instead of the number of packets.

[0116] In yet another embodiment, both the number of packets and the number of bytes are used. In this approach, two loss percentage values ​​are generated for each unique identifier in the group. These two values ​​can be combined, for example, by averaging, taking the minimum, taking the maximum, or any other means, to result in a single loss percentage value that is considered representative of this connection.

[0117] Multiple loss percentage values ​​may be generated for each connection over time. In one embodiment, the most recently calculated loss percentage value for a particular connection is attributed to it. In another embodiment, the group-specific identifiers used by a particular connection are ever-increasing, and the loss percentage value from the highest group-specific identifier is attributed to this connection.

[0118] The losses experienced by a particular connection may not be uniform or consistent over time. For example, a connection may completely drop one group of packets, then completely deliver the next group of packets. Each of these two groups of packets, within a relatively short period of time, will give significantly different perspectives on the reliability of this connection.

[0119] It may be possible to overcome this technical drawback by recording and processing multiple loss percentage values ​​generated over time for a particular connection. This processing may reduce the multiple loss percentage values, for example by averaging, taking the maximum, minimum, or any other means, to a single value that is taken to be representative of the connection.

[0120] A technical approach to account for this lack of uniformity is, among other things, to use intensity-weighted measurements (where a bunch of clustered samples may be (more or less) representative than an (in time) "isolated" sample). Temporal weighted samples give snapshots of the type of loss experienced on a connection at various times, which can be observed as appearing, for example, "bursty," "uniform," etc. A historical memory of these snapshots can be tracked in data storage, which can potentially be used to weight or modify the scalar "p" value compiled for a connection.

[0121] Other approaches are possible, such as using time weighting to modify the interval, and in alternative embodiments, a sampling engine can be utilized to generate and send probe sample data packets to establish a baseline for measurements based on actual experience (e.g., some flaky links may be sampled randomly, more frequently, or with alternate types of packets / groups of packets based on experience with the link).

[0122] In some embodiments, a group of packets is sent over a connection, and only those groups of packets that saturate the connection are used to generate the loss percentage value. The reasoning behind this logic is that when a particular connection is fully utilized, only that connection may represent its loss characteristics. In some embodiments, a connection will saturate if it is not application-limited, as defined by the IETF ICCRG draft, "draft-cheng-ICHRG-delivery-rate-estimation-00," which is incorporated herein by reference in its entirety.

[0123] In one embodiment, multiple loss percentage values ​​for a particular connection are recorded for a fixed time interval, and once the age of a recorded value exceeds the fixed time interval, the value is discarded.

[0124] In another embodiment, multiple loss percentage values ​​for a particular connection are recorded up to a maximum number of values, at which point the oldest value is replaced by a newer value.

[0125] When multiple loss percentage values ​​recorded over a period of time are reduced to a single value, different connections with different loss patterns may appear to have similar loss properties. For example, assume an application sends 10 packets per second for 10 seconds, for a total of 100 packets. Consider connection C1, which has a relatively uniform loss pattern, losing one packet per second. C1 will lose 10 of the 100 packets over the 10-second period. Consider another connection, C2, which has a relatively bursty loss pattern, losing 10 packets once every 10 seconds. C2 will also lose 10 of the 100 packets over the 10-second period. Although both C1 and C2 yield a 10% loss percentage value over a fixed 10-second interval, their loss patterns are very different. The sender may benefit from using different transmission strategies for each of these connections.

[0126] According to some embodiments, by recording multiple loss percentage values ​​over several time intervals, it is possible to infer more information about the loss pattern of a particular connection: shorter intervals give a more instantaneous perspective of the loss properties of the connection; longer intervals give a more general perspective of the loss properties of the connection.

[0127] According to some embodiments, the number of time intervals is fixed. In one embodiment, three time intervals are used: short, medium, and long. If similar loss percentage values ​​are calculated (e.g., determined, observed) across all three intervals, the loss pattern can be considered relatively uniform. In such a scenario, for example, the sender may choose to calculate and send FEC to allow recovery of lost packets at the receiver, or may simply replicate enough packets to reduce the chance of the receiver detecting lost packets.

[0128] If the loss percentage value calculated over a short interval is significantly greater than the loss percentage value calculated (e.g., determined, observed) over a long interval, the loss pattern can be considered relatively bursty. In such a scenario, the sender may choose, for example, to temporarily stop sending packets until the bursty loss interval is over to avoid significant loss.

[0129] Further inferences can be made and strategies can be developed based on comparing the loss percentage values ​​calculated from various time intervals. The scenarios considered herein are merely exemplary and in no way limiting, and other variations are possible. For example, according to some embodiments, inferred information about the pattern / distribution of loss on a connection (e.g., uniform vs. bursty) obtained from analyzing loss over varying time intervals is kept in a historical record and is another dimension based on which packet scheduling decisions can be made by the PFLC component 004.

[0130] Applying the concept of intervals of different lengths to the previous example, a distinction is made between connections C1 and C2. Suppose the losses for each of C1 and C2 are recorded over two time intervals. The first time interval is relatively short, 1 second in length. The second time interval is relatively long, 10 seconds in length. For C1, both the short and long intervals always result in a loss percentage value of 10%. For C2, assume that 10 packets dropped in one shot occur on the fifth transmission event. The short interval results in a loss percentage value of 100%, while the long interval results in a loss percentage value of 20%. This difference between the short and long time intervals indicates that the loss properties of C2 are more non-uniform and bursty.

[0131] According to other embodiments, the number of time intervals is variable and can be adjusted as more information is gathered about a particular connection. For example, if a connection is determined to be of a particular type that exhibits only uniform loss, the number of intervals can be reduced to one, which should be sufficient to measure uniform loss.

[0132] According to some embodiments, the length of the time interval is fixed.

[0133] According to other embodiments, the length of the time interval is variable and can be adjusted based on information known, configured, required to be collected, or collected about a particular connection.

[0134] In one embodiment, statistical analysis can be used to determine how rapidly a connection is changing properties by determining how older loss percentage values ​​correlate with newer loss percentage values, and by selecting a cutoff lifetime (i.e., interval length) such that loss samples older than that lifetime are discarded; otherwise, stale samples may be detrimental to system performance.

[0135] In another embodiment, if the length of the loss event is to be measured, the length of the time interval may be increased if the length of the loss event is likely to exceed it.

[0136] In another embodiment, if the system is adapted to use the connection for real-time traffic, the system may shorten the time interval to allow for faster / more aggressive response.

[0137] In another embodiment, data flows may be identified as having deadlines, and those deadlines may dictate the length of the time interval used to measure loss (e.g., it is futile to use a long time interval for a deadline in the near future).

[0138] According to some embodiments, the updating of the values ​​in each of the time intervals is triggered by one or more events.

[0139] One trigger is purely the passage of time: for example, each time a new group of packets is to be transmitted, the system may discard outdated samples from each of the time intervals, determine new drop probabilities, other state information pertaining to component 004 (e.g., layer and connection state), and then determine how to transmit this new group of packets.

[0140] Another trigger could be receiving new information about loss from a peer, and the system adds this new sample to the window and recalculates as above.

[0141] Other example triggers may include events such as a connection losing or gaining physical link state (cable unplug / replug), an explicit congestion notification (ECN) from the network, a change in physical location, or a change in wireless band / frequency.

[0142] Some embodiments may weight or prioritize these triggers and choose to ignore some triggers when there is conflicting information.

[0143] Updating the loss data (which determines everything else) naturally depends on sending data and getting feedback about it, and in some embodiments, a feedback mechanism is established for sending monitored connection data including latency, throughput, and connection path (e.g., route taken).

[0144] In one embodiment, CPU and memory resource consumption can be reduced by trading off some accuracy of the measured loss percentage values. Each loss percentage value can be expressed as two values: the number of lost packets and the total number of packets. Time intervals can be further divided into smaller subintervals. Loss percentage values ​​can then be accumulated (by summing their lost packet counts and summing their total packet counts) and associated with those specific subintervals. This eliminates the need to store each loss percentage value individually and the timestamp of when it was calculated. Precision is reduced because only the subinterval timestamp is recorded. When a subinterval expires, some of its accumulated samples may have expired earlier or may not have expired yet.

[0145] To generate a single loss percentage value that is representative of the loss properties of a connection over a particular time interval, a sufficient number of loss percentage values ​​may be required to be recorded over that time interval. In one embodiment, recorded loss percentage values ​​are considered to be representative of a time interval if they span a time period that is equal to or greater than half of that time interval.

[0146] Different decisions regarding transmission strategies can be made based on the availability of a single representative loss percentage value for a particular time interval. In some embodiments, this is an aspect of PFLC component 004, where the availability of samples over configured time intervals is compiled into a per-connection state, which then influences the transmission decisions of scheduler 160. For example, a connection having many time intervals with insufficient samples can result in its state being considered INDETERMINATE, which causes scheduler 160 to only use the connection to transmit redundancy until the time interval is filled with enough samples.

[0147] 3 is a diagram 300 illustrating an embodiment that uses several time intervals, divides them into sub-intervals, and ensures that the loss percentage value generated for each interval is considered representative (i.e., valid) for the interval only if the recorded samples span more than half of the particular interval, according to some embodiments. The number of time intervals is fixed at 3, and the lengths of these intervals are fixed at 0.5 seconds, 3 seconds, and 30 seconds.

[0148] A 0.5-second time interval is considered valid if it has two subintervals, each 0.25 seconds long, and at least one subinterval has a sample in it. A 3-second time interval is considered valid if it has six subintervals, each 0.5 seconds long, and at least three subintervals have a sample in them. A 30-second time interval is considered valid if it has 10 subintervals, each 3 seconds long, and at least five subintervals have a sample in them. The same time interval is used for all connections. The number and lengths of intervals are considered appropriate for a wide variety of common connections, such as cellular and wired connections.

[0149] In other embodiments, the probability of loss for a particular connection can be assessed by measuring other properties of the connection. For example, a substantial increase in the RTT of the connection or a substantial decrease in the signal strength of a wireless connection may indicate an increased probability of a loss occurring.

[0150] According to some embodiments, measurements of connection properties other than packet loss or byte loss can be converted to loss percentage values ​​by custom mapping. The resulting loss percentage values ​​can then be recorded and utilized in the same manner as actual loss percentage values.

[0151] In one embodiment, the RTT of a group of packets is measured as the time between transmitting this group of packets on a particular connection and receiving an acknowledgment from the receiver about how many packets were delivered. If the measured RTT for this group of packets exceeds a certain threshold, the loss percentage value is recorded as 100%. If the measured RTT for this group of packets does not exceed the threshold, the loss percentage value is recorded as the actual percentage of packets in this group that were delivered to the receiver. If, after a certain waiting period, no acknowledgment was received (either because all packets were lost or because the acknowledgment itself was lost), the loss percentage value is recorded as 100%. This provides a mapping from RTT to loss percentage values.

[0152] In some embodiments, a maximum threshold for RTT is configured per flow as a function of each flow's latency requirements. For example, it is intuitively desirable for a VoIP flow requiring an RTT of <300 ms to treat connections with an RTT of >= 300 ms as effectively 100% lost.

[0153] The reliability of a single connection can be quantified by relating it inversely to the probability of dropping data transmitted on that connection. In one embodiment, the probability of dropping data transmitted on a connection (i.e., the drop probability) is considered to be equal to the loss percentage value measured on that connection, and the reliability can be quantified as a percentage value equal to 1 minus the drop probability.

[0154] The following paragraphs provide a detailed description of a specific embodiment of PFLC component 003: grouping connections with similar historical, current, or predicted future reliability.

[0155] In some embodiments, the drop probability p for each aggregated connection (calculated by component 002) is used in conjunction with the reliability target T (calculated by component 001) to obtain the relationship:

number

[0156] For each connection, solving for n in this relationship and mapping it to the next integer greater than or equal to n (call it N) results in a value that represents the number of times a packet must be sent on that connection in order to probabilistically meet or exceed the reliability target T.

[0157] This number N refers to the hierarchy of connections, and connections are grouped as a function of their hierarchy number.

[0158] In some embodiments, a constant value for the reliability target T is used for all flows, meaning that each connection is only assigned to one tier (varying only based on each connection's value for p), resulting in only one grouping of connections based on the computed set of N values.

[0159] In some embodiments, each flow has its own reliability target (e.g., a VoIP flow wanting T=99% vs. a temperature sensor wanting T=80%). This results in each connection being assigned a different stratum N per flow, meaning that each flow may see a different grouping of connections as a function of both p and T.

[0160] In some embodiments, a limit is placed on the value of N in order to prevent excessive bandwidth consumption: combinations of p and T that result in a value of N greater than the upper bound M imply that these connections are grouped into tier M.

[0161] In some embodiments, tier N refers to tier 0 and is treated differently for packet scheduling / transmission purposes. For example, connections at this tier may be considered unreliable (too many transmissions required to achieve T) and are therefore only used to duplicate packets already transmitted on other connections (best effort / preemptive ARQ).

[0162] FIG. 4 is a flow diagram 400 illustrating the grouping technique of some embodiments described herein.

[0163] Flow classification engine 156 identifies flow 401 as having a target reliability T=99% and flow 402 as having a target reliability T=80%.

[0164] The connection controller 154 calculated the combined p-values ​​(packet loss) for connections C1, C2, C3, C4, and C5 as (respectively): 1%, 5%, 10%, 35%, and 40%.

[0165] The scheduler 160 is configured so that the tier upper limit M=3.

[0166] For flow 401 (T=99%), the calculated stratum N for each connection is:

[0167]

number

[0168]

number

[0169]

number

[0170]

number

[0171]

number

[0172] When scheduler 160 schedules packets from flow 401 for transmission, the connection groupings it uses are: tier 1 {C1}, tier 2 {C2, C3}, tier 0 {C4, C5}.

[0173] For flow 402 (T=80%), the calculated stratum N for each connection is:

[0174]

number

[0175]

number

[0176]

number

[0177]

number

[0178]

number

[0179] When scheduler 160 schedules packets from flow 402 for transmission, the connection grouping it uses is: tier 1 {C1, C2, C3}, tier 2 {C4, C5}.

[0180] In another embodiment, connection groupings (tiers) and the associated metrics for generating them (e.g., information inferred about T, p, and loss distributions) can be individually optimized for each data flow or data type (e.g., by flow identifier 001). In one embodiment, specific statistics are generated for each connection per flow and provided to scheduler 160 so that more intelligent decisions about packet transmissions can be made (component 004 of PFLC). The statistics collected may be different for each connection per flow.

[0181] The following paragraphs provide a detailed description of a specific embodiment of PFLC component 004: efficiently scheduling packets on a group of connections in a manner that meets flow reliability requirements.

[0182] In a simplified exemplary embodiment, the monitored statistics are packet and byte loss (and some RTT / retransmission timeout (RTO) conversion) over a fixed, predetermined interval, and this is done only on a per-connection (not per-flow) basis. These loss statistics are converted into stratum and state values ​​that are fed to scheduler 160 to help determine which packets to transmit, on which connections, how many times, and how frequently.

[0183] In another embodiment, connection hierarchies may be established by a common set of rules, and scheduler 160 relaxes the restrictions placed on which tier a given data flow can use. In yet another embodiment, a combination of the above two approaches may be used.

[0184] The approach to determining how to utilize connections for particular types of traffic can also vary. In one embodiment, given customer or application indications (e.g., provided in control signals, overhead data) regarding the nature of the transmission, the system is configured to use lower tier connections to benefit both the transmission (using cheaper but less reliable connections) and the rest of the network (saving space on more reliable connections for flows requiring them).

[0185] In some scenarios, using a lower tier connection may consume more data by forcing more applications onto a less reliable network, which may be more expensive due to the multiple retransmissions required, or it may be cheaper if the less reliable network is less costly.

[0186] As an example, for transmissions deemed or directed as urgent, a system may be willing to incur the extra expense of sending duplicate information over an unreliable network, or for non-urgent transmissions, a system may be willing to incur some packet loss (especially if data is cheap over unreliable connections) in order to maintain capacity on reliable networks. These additional costs may be charged by the routing device / system and tracked for later verification. There may be various amounts of added redundancy (e.g., a large amount resulting in a significant increase in reliability, or a small amount resulting in a lesser increase). A data packet communicator may modify the desired reliability based on cost (e.g., 98%, 99%, 99.9%, or 99.99% reliability may result in significant costs).

[0187] Per-connection adjustments (as it is the connection that changes) are possible in some embodiments, and the traffic being sent over a connection may dictate how aggressively the system manages transmissions.

[0188] According to some embodiments, the per-flow, per-connection tier allocation calculated by the PFLC component 003 of the present invention nominally corresponds to the number of times the scheduler 160 schedules transmission of packets from a particular flow onto connections belonging to that tier.

[0189] In another embodiment, scheduler 160 is configured to implement different sorting orders for the preferred order of transmission using a connection, with granularity as low as per packet, where one dimension of sorting may be cost among others. In this variation, rather than having connections essentially "pull" packets from scheduler 160 in bursts (where implementing multi-dimensional prioritization would be prohibitively complex), scheduler 160 is instead configured to push packets onto the connection.

[0190] Another variation on cost management proposes a simpler alternative to real-time sorting that still achieves some of the benefits of cost optimization. For example, scheduler 160 may establish a simple "highest stratum / highest iteration" value for each connection (rather than the entire system) based on per-packet cost. In this example, for expensive links, the system may be configured to impose an upper limit at stratum X, such that the connection is marked as unreliable even if it is evaluated as stratum X+1, where there may be many other connections X+1, X+2, etc. that are satisfied for transmission by the system, until it qualifies for a stratum requiring fewer iterations. In terms of overall cost, the difference between one and two iterations is much higher than between nine and ten iterations, so in practice, X is likely to be a low number.

[0191] For example, if an expensive connection is capped at tier X but actually belongs to tier X+M, it need not be considered unreliable. The connection may still be used at tier X+M as long as there are other connections there, in such a way that lost transmission attempts on the expensive connection (M, suppose the system has sent a packet on it X times) can be handled by other connections at that tier.

[0192] In some embodiments, the caps imposed on iterations may be static, and they may be independent of what other connections are available and where the traffic levels are. In other embodiments, the caps may be dynamic and depend on other monitored information (e.g., if there are many other links available or traffic is low, a satellite may be capped at Tier 1; otherwise, the satellite may be assigned to Tier 2 (and place an absolute cap there), or the highest tier may be adjusted based on the relative price difference between multiple connections). In this variation, cost optimization is decoupled on the tiering mechanism (and avoids flow classification or system-wide optimization of costs across all links).

[0193] This approach may not prevent a system from making heavy use of an expensive connection if it qualifies for a higher tier (if it were the only tier 1 connection and there was capacity, the system might still use it, even if a full-cost analysis would lead the system to select a much cheaper connection measured at tier 3), but it may increase the likelihood that the system is getting significant financial / economic value from each packet sent on that connection (compared to the connection itself) if the link is used.

[0194] On the unreliable side, this approach can be added to exponential backoff from sending on unreliable connections to further reduce costs. For example, this can involve controlling routing to backoff more aggressively on expensive connections, by creating an "unreliable tier" with a similar upper bound as above, except related to how often and with how many packets the system revalidates.

[0195] Other variations are possible, for example, there is a continuum between a simple approach and one towards full cost optimization with hyper-dynamic tiering via real-time connection sorting, access to real-time connection pricing, hints from the application, or predicted results from a flow classification engine, etc. All of these approaches are contemplated in various embodiments herein. As noted above, the idea of ​​a basic per-link tier cap works even better as a "poor man's cost optimization" in the context of connection tiering that can be used in simpler implementations.

[0196] FIG. 5 illustrates such an embodiment, where T=99%, p C1 =1%, p C2 =5%, p C3 =10%, p C4 =35%, p C5 This is the case for example flow 500, which has a per-flow p value for each of the =40% of connections and a per-flow tier limit M=6.

[0197] If scheduler 160 schedules a packet from this flow on layer 1 (which contains only C1), the packet is transmitted exactly once on C1 until either a positive acknowledgment (ACK) or negative acknowledgment (NACK) is received or an RTO occurs.

[0198] If scheduler 160 schedules packets from this flow on layer 2 (including C2 and C3), the packet is transmitted nominally twice until either an ACK or NACK is received or an RTO occurs. C1 , and p C2 Based on the value of , the packet is transmitted twice preemptively (before NACK or RTO) according to the relation

number

[0199] Preemptive sending is where the name PFLC comes from: the scheduler 160 can reduce latency for real-time applications because there are no round trips required for ARQ, provided that the assessment of real-time flows and connections is correct.

[0200] Scheduler 160 also reduces bandwidth requirements compared to SMPTE-2022-7: flow and connection evaluation allows duplicate packets to be transmitted only on C2 and C3 in this example, and not on all of C1, C2, and C3.

[0201] In some embodiments, the duplicate transmissions are preferably split, one on C2 and the other on C3, although they can be both on C2 or both on C3 if desired.

[0202] In some embodiments, overlapping transmissions on the same connection are separated in time depending on the loss profile / distribution, e.g., a bursty loss distribution benefits from further separating the transmissions in time, while a uniform loss does not.

[0203] In some embodiments, if an ACK is received before the second transmission occurs, the second transmission is skipped. This can occur if there is a time separation between the transmissions combined with low latency on the connection of the first transmission.

[0204] This approach is the same for higher-numbered tiers. For example, if scheduler 160 schedules a packet from this flow on tier 5 (which contains only C4), the packet will be transmitted nominally five times. Transmissions are preemptive (prior to a NACK or RTO), may be separated in time, and may be truncated if an ACK is received first.

[0205] In some embodiments, once scheduler 160 selects a tier for a packet, subsequent preemptive transmissions may occur only on connections within the same tier, a process referred to as "pinning" the packet to a tier.

[0206] In other embodiments, subsequent preemptive transmissions may be performed on any combination of higher numbered hierarchies according to the relationship

number

[0207] For example, a packet might be sent once over a connection at tier 2 and twice over a connection at tier 4, using a combination of tiers 2 and 4. This is useful for avoiding the issue of reliability concentrating on one particular tier, and allows for a more flexible approach to using available bandwidth or connectivity (e.g., for cost or load balancing reasons).

[0208] Transmitting packets using a combination of layers is also useful to avoid the issues of reliability concentrating on one particular connection (e.g., connection 7 may be the only connection in that layer and may be down for a period of time, the entire time the packet is being retransmitted, so the reliability from multiple transmissions is actually lower than if multiple transmissions of the packet were sent over independent connections).

[0209] Another potential benefit of transmitting packets using a combination of tiers is when performance degradation (eg, heat) occurs from over-utilization of a connection for a short time frame.

[0210] In some embodiments, the above scheduling rules and hierarchical allocations are applied to bytes and byte ranges belonging to flows, rather than packets.

[0211] An optional feedback channel (e.g., a high reliability / low latency channel) can be provided, which can also be a connection used to monitor the performance of the solution. Feedback channels are useful for perturbing connection selection, especially when probabilities have correlations between them, and attempt to encourage the use of as independent probabilities as possible to establish goals. Feedback channels can interoperate with timestamped packets as well as timestamped acknowledgments.

[0212] Thus, scheduler 160 can utilize multiple methods of transmission, e.g., unicast or broadcast. Connections to be utilized can be identified for tracking in a broadcast domain, and a broadcast domain can be established for a particular hierarchical connection or for all connections selected to transmit packets.

[0213] Other models of forwarding are possible. Further description of potential variations and approaches is provided further herein. Various approaches are possible for determining the probability of success for a particular connection, for use in tiering the connections.

[0214] An aspect of the PFLC component 004 in some embodiments is the concept of connection states: reliable, unreliable, and indeterminate. These states and the transitions between them are described below. Other states are possible, and these states are shown by way of example.

[0215] When a packet is transmitted by a group of connections belonging to a particular tier, a certain number of copies of this packet are also transmitted. The higher the tier, the more copies are transmitted. This implies that the "goodput" (the actual useful throughput experienced by an application flow) will be lower at higher tiers because more bytes (i.e., more duplication, or more redundancy) are transmitted overall to deliver identical information to the receiver. There may be a "tipping point" where the benefits of delivering information to the receiver with a selected target probability outweigh some drawbacks, including but not limited to increased cost of transmission. For example, users of a system may state an upper limit they are willing to pay for the system to operate. This limit may translate into a byte or bit rate threshold based on how much the underlying connections charge for usage. For example: ● Your connectivity provider uses a burst pricing model, charging you based on the 95th percentile of your throughput usage over a month (based on a 5-minute average sample). ● Customers are only willing to pay for up to 100Mbps of throughput under this billing model. ● The goodput requirement of an application flow is 50Mbps, but due to the unreliability of the connection, the retransmission technique may require every packet to be replicated twice (each packet will be transmitted a total of three times). The goal of achieving 50Mbps goodput may result in a total throughput usage of 150Mbps, exceeding the customer's limit. Therefore, it is advisable not to use this connection.

[0216] In some embodiments, the byte or bit rate threshold is maintained as a data value in a data field of a data structure that is referenced to determine whether a connection should be used by the system. This data value may be used for comparison to toggle the connection's usage state. In the above example, the threshold data field stores a data value of 100 Mbps, and the throughput usage rate can be determined to be 3 x 50 = 150 Mbps. Because 150 > 100, the connection will not be used (due to excessive bandwidth usage) even if it is possible to achieve the desired probability of successful transmission. This is useful when a trade-off needs to be made between efficient bandwidth usage and reliable data communication. A corresponding threshold data field can be tracked for each connection or group of connections.

[0217] As described below, thresholds can also be used to establish a maximum stratum. The maximum stratum can be established periodically to reduce the number of calculations required at runtime and thus reduce computational load. According to some embodiments, a maximum stratum is defined for a particular connection to prevent the system from using that connection beyond its tipping point. If a transmission event with a given delivery target probability requires the use of this connection (based on its drop probability) at a stratum higher than the maximum stratum, the connection will not participate in this transmission event. For example, one stratum may require seven retransmissions to achieve the required probability of communication success, leading to inefficient use of bandwidth resources.

[0218] In one embodiment, the same highest stratum is assigned to all connections to simplify and increase predictability of operation.

[0219] Measuring the network properties of a particular connection is only possible if this connection participates in transmission events. A particular connection must participate in enough transmission events to keep the measurements up to date.

[0220] A connection may not be participating in a transmission event due to a combination of its highest stratum and its drop probability. Scheduler 160 or connection manager (e.g., connection controller) 154 may decide to force this connection to participate in some transmission events in order to make new measurements of the network properties of this connection. When the drop probability of this connection is updated, the decision on whether or not this connection should participate in a transmission event can be reevaluated.

[0221] A connection may not participate in a transmission event because the sender does not have enough data to transmit to maintain management of all or any of the connections. If sufficient data becomes available again and it is desirable to resume transmission on this connection, scheduler 160 or connection manager 154 should take into account that the most recent measurements of the network properties of this connection are not available when making new transmission decisions.

[0222] Similarly, a connection may not be participating in a transmission event due to a configured priority routing preference, which may cause a lower priority connection to be overridden because the higher priority connection has sufficient bandwidth to handle the priority data flow. In the absence of other traffic, this limits the information the system has about lower priority connections, and the scheduler 160 or connection manager 154 must consider stale observations of connection properties.

[0223] In some embodiments, a connection that is not participating in a transmission event does not have an assigned tier (in code, this may be implemented, for example, by setting the tier to 0, although this contradicts the mathematical definition of tier). Once enough statistics have been collected that indicate this connection can participate in a transmission event, the connection can be assigned to a tier.

[0224] In one embodiment, each connection is assigned a state that reflects its current condition or operating state. A connection that is participating in a transmission event is said to be in a RELIABLE state. A connection that is not participating in a transmission event due to a combination of its drop probability and highest stratum is said to be in an UNRELIABLE state. A connection that does not have a record of recent measurements is said to be in an INDETERMINATE state. As the transmission system continues to run, each connection can move between three states:

[0225] In some embodiments, these connection states may vary from flow to flow due to p, T, and latency requirements may also vary from flow to flow. For example, a flow with a large T value and / or lower latency requirements may consider a particular connection to be UNRELIABLE, while this same connection may be considered RELIABLE by another flow with a small T value and / or no latency requirements.

[0226] However, some connection state may be shared by all flows, e.g., a connection with stale observations for that time interval may be considered INDETERMINATE for all flows, regardless of their T values.

[0227] In some embodiments, a connection in the RELIABLE state is a connection that is assigned to a tier greater than or equal to tier 1 and less than tier M. A connection in the UNRELIABLE or INDETERMINATE state is assigned to tier 0 or tier M and is used accordingly (primarily to carry duplicates of packets transmitted on the reliable tier).

[0228] The distinction between the UNRELIABLE and INDETERMINATE states is that an UNRELIABLE connection utilizes a back-off timer to reduce the frequency of packet transmissions. The rationale is that a connection that has been measured conclusively as unreliable over all available measurement periods is likely still unreliable, so the back-off timer prevents excessive use of resources just to frequently re-measure the connection. The back-off timer still allows this periodic re-measurement, it's just that the intervals are larger.

[0229] 6 is a diagram 600 describing the state transitions of a particular connection according to one embodiment. Alternative approaches to state transitions are possible, and there may be more, fewer, or different states.

[0230] As described above, the network properties of the connection are converted into loss percentage values ​​(or other values ​​indicative of communication or transmission reliability) and then recorded over multiple (e.g., three) time intervals and reduced to one or more values ​​deemed representative of one or more time intervals, or one or more time intervals.

[0231] The transmission system is assumed to have a predetermined target reliability expressed as a percentage value. To compare a representative loss percentage value p for a time interval with the target reliability of the system, the reliability value for that time interval is

number

[0232] 6, the state transitions 600 depend on the number of valid reliability values, how they compare to the system's target reliability, the previous state of the connection, and the current state of the connection. An example is shown, which comes from the number of valid reliability values ​​being 0, 1, 2, and 3.

[0233] If the reliability values ​​for all three time intervals are invalid, all observations about the connection are stale. The resulting state transition is intended to be notable and transitions the connection out of the reliability hierarchy. Connections in UNRELIABLE or INDETERMINATE state remain in those states. Connections in RELIABLE state transition to INDETERMINATE state.

[0234] If the reliability value for only one time interval is valid, the following logic applies.

[0235] If the reliability value is greater than or equal to the target reliability, a state transition is intended to be cautiously optimistic because there is a new observation indicating that the connection may be reliable. A connection in UNRELIABLE state transitions to INDETERMINATE state, disabling the backoff timer with the goal of gaining more observations and confirming reliability as quickly as possible. A connection in RELIABLE state remains in the same state. A connection in INDETERMINATE state transitions to RELIABLE if it was previously RELIABLE (cautiously optimistic, as previous reliability has now been reconfirmed), otherwise it remains in the same state (it was previously unreliable, so this single new observation alone is insufficient to cause a transition).

[0236] If the value is less than the target reliability, the new observation indicates that the connection is unreliable. A connection in UNRELIABLE state remains in that state (the new observation reconfirms the previous state). A connection in RELIABLE state transitions to INDETERMINATE state (attempting to obtain an observation as soon as possible, with no backoff timer, as it is cautionary but optimistic that the unreliability is temporary). A connection in INDETERMINATE state transitions to UNRELIABLE if its previous state was UNRELIABLE (confirming that the previous unreliability is now reconfirmed), otherwise it remains in the same state.

[0237] If reliability values ​​for only two time intervals are available, the following logic applies.

[0238] If the reliability values ​​for both time intervals are greater than or equal to the target reliability, then the majority of time intervals agree that the connection is reliable, and the connection transitions to the RELIABLE state.

[0239] If one time interval has a reliability value that is greater than or equal to the target reliability and the other has a reliability value that is less than the target reliability, then there is a mismatch between the two intervals (e.g., the shorter interval may see bursty loss that exceeds the reliability target, while the longer interval may see uniform loss that is below the reliability target). This mismatch causes the connection to transition to the INDETERMINATE state.

[0240] If the reliability values ​​for both time intervals are less than the target reliability, the connection transitions to the UNRELIABLE state, since the majority of time intervals agree that the connection is unreliable.

[0241] If the reliability values ​​for all three time intervals are valid, the following logic applies.

[0242] If the reliability values ​​for all three time intervals are greater than or equal to the target reliability, the time intervals are all three consistent in their reliability agreement and the connection transitions to the RELIABLE state.

[0243] If the reliability values ​​for two time intervals are greater than or equal to the target reliability and the reliability value for the third time interval is less than the target reliability, the connection transitions to the INDETERMINATE state. There is a discrepancy between the three time intervals. Even if the majority believes the connection is trustworthy, one clearly believes the connection is untrustworthy. More observations are requested as soon as possible (without a backoff timer) to resolve this discrepancy.

[0244] If one or zero of the time intervals are above the target reliability and the other time intervals have reliability values ​​below the target reliability, the connection transitions to the UNRELIABLE state (the majority of the time intervals consider the connection unreliable).

[0245] In other embodiments, state transitions may have different rules than those described above. State transitions may also take other information into account, such as:

[0246] Historical information, including previous combinations of losses over various time windows, possibly beyond those described herein, based on previous measurements and states. Data regarding "rate of state change" that can be used to predict the stability of a given connection in a given state.

[0247] In an alternative embodiment, this historical trend data is utilized to determine the rate at which traffic is transmitted over a particular connection and changes in rate.

[0248] In further embodiments, there is technical coordination between downstream endpoints and the router system described herein. For example, every packet sent keeps track of what tier it was sent on and what the expected probability of arrival is. There may be metadata associated with packets handled within scheduler 160. In another embodiment, the data may be included / embedded in the packet itself.

[0249] This association information can be used to assess the end-to-end reliability of a connection and can be particularly useful when reliability data is out of date and needs to be updated, or when the data is simply missing.

[0250] In an exemplary embodiment, an unreliable stratum (e.g., stratum 0) is sometimes utilized to transmit redundant packets to verify authenticity while potentially increasing the probability of success of the overall data communication.

[0251] The following paragraphs describe in more detail how some embodiments of the PFLC component 004 use the concepts of connection hierarchy and connection state together to intelligently transmit packets over available connections that can meet the flow's target reliability and latency constraints.

[0252] In some embodiments, scheduler 160 performs the task of providing a set of packets (bursts) to be immediately transmitted over a specified connection, subject to constraints imposed on the total size (in bytes) of the burst or the number of packets in the burst, and on the requirements of the packets' associated flow and the determined tier of the connection. In other embodiments, scheduler 160 performs the task of determining the set of packets to be transmitted next and optimally allocating those packets to connections that have available capacity.

[0253] In some embodiments, scheduler 160 will track a set of metadata about each packet that it contributed to a connection until such time as the packet is acknowledged as received by the peer or until the packet is deemed too old to be used, even if it is subsequently retransmitted.

[0254] In some embodiments, such metadata includes the cumulative probability that a packet was lost in transmission, the current in-flight count, and the current tier that transmitted the packet. These statistics allow scheduler 160 to determine whether enough of a given packet has been transmitted to meet reliable delivery requirements.

[0255] Other embodiments may instead choose to use metadata including the number of times a packet has been sent on each connection, and use that information to calculate the estimated cumulative drop probability of the packet.

[0256] When scheduler 160 is tasked with generating bursts of traffic for a given connection, some embodiments may use the following algorithm: Step 1: Subject to tier-based transmission constraints, retransmit any previously transmitted packets that do not meet the target reliability criteria (e.g., packets assigned to tier 2 and transmitted only once by a connection in that tier). Step 2: Subject to tier-based transmission constraints, transmit new packets. Step 3: Attempt to meet the minimum burst size by retransmitting previously transmitted packets. Step 4: Attempt to meet the minimum burst size by transmitting packets with dummy payloads.

[0257] In step 1, some embodiments iteratively scan the list of transmitted packet metadata and find the highest cumulative drop probability p (i.e.,

number

[0258] Such a packet is eligible if its assigned stratum matches the stratum of the current connection, or if the packet does not currently have an assigned stratum. If the packet is eligible, its metadata is updated as follows: the packet's cumulative drop probability is updated to a new cumulative drop probability, which is the old cumulative drop probability multiplied by the drop probability of the connection carrying the scheduled packet; the packet's assigned stratum is updated to the same stratum as the stratum of the connection carrying the scheduled packet; and the inflight counter is incremented.

[0259] Some embodiments may choose to limit the number of times a given packet will appear in the same burst (time separation of packets), with the goal of reducing the risk that loss of a burst will affect the overall delivery of the packet, and rather choose to allow packets to still be scheduled in separate bursts on the same connection or on different connections within the same tier.

[0260] In step 2, provided that the burst size request has not yet been met, scheduler 160 may take a new packet from the input queue such that the request of its associated flow allows its transmission over the current connection. For example, a flow with an RTT upper limit can only be transmitted over connections with an RTT below this threshold.

[0261] Such a packet is assigned to that burst and is assigned an initial cumulative drop probability equal to the drop probability of the connection. The packet's tier is assigned to the tier of the current connection if the connection is reliable; otherwise, the packet's tier remains unassigned. The packet's inflight counter is also set to 1.

[0262] Some embodiments may choose to include a packet multiple times in this same burst (adjusting the cumulative drop probability as in step 1), subject to the target reliability T and burst size constraints, while other embodiments may choose to limit the number of times a given packet can appear in the same burst (time separation of packets) in order to reduce the risk of the packet not being delivered due to loss of the entire burst.

[0263] In step 3, even if there are no more packets available for scheduling after the first two steps, the scheduler 160 may be configured to pad the burst down to a minimum number of packets or bytes (i.e., to allow the network characteristic monitoring unit 161 to better determine the connection characteristics). In these cases, some embodiments of the system are adapted to repeatedly include in-flight packets that have already met the target cumulative drop probability. The selection of packets may be ordered by some criteria, for example, sorted by descending cumulative drop probability, which may help to equalize the probability of packets arriving at their destination.

[0264] As in step 1, the cumulative drop probability of such a packet is updated to include the drop probability of the current connection. Unlike step 1, the assigned stratum is not updated, and unlike step 1, neither the target reliability T nor whether the assigned stratum matches the stratum of the connection is considered when choosing to include a packet for padding.

[0265] Some embodiments may choose to limit the number of times a packet can appear in the same burst (time separation of packets), so a packet may be ineligible for inclusion as padding if it has already appeared in a burst through steps 1 or 2. Because packets that appear as padding contribute to the cumulative drop probability of the packet, packets scheduled in this manner may not strictly follow the general principle of sending a packet n times over tier N, as sending a packet one or more times as padding may be equivalent, from the perspective of drop probability, to sending that packet one or more times over the connection of the packet's assigned tier.

[0266] In step 4, if the minimum burst size requirement cannot be met in step 3, packets with meaningless data in their payloads ("dummy" packets) are generated in order to meet the burst size requirement. Such packets are discarded after receipt by the peer and are not subject to metadata tracking within scheduler 160.

[0267] When scheduler 160 is notified that a peer has received an instance of a packet (over any connection), the packet's metadata is removed from tracking. When scheduler 160 is notified that a packet instance has been lost, the in-flight counter for that packet is decremented. If the new value of the in-flight counter is zero, and the cumulative drop probability p for the packet is

number

[0268] As part of its flow request, a packet may include an upper bound (deadline) imposed on its lifetime, beyond which the flow may not request retransmission.

[0269] Scheduler 160 is also notified when the count of connections at a given tier transitions to zero. When this happens, all packets in the metadata queue assigned to that tier are marked as unassigned because there are no longer any connections in that tier that satisfy the retransmission request from step 1. Packets marked as unassigned maintain the same drop probability they had while in the queue, and this drop probability is used in future transmission decisions. For example, if a packet was initially assigned to tier 2 and transmitted only once, when tier 2 becomes empty and the packet is subsequently reassigned to tier 3, it may not require three transmissions to meet the target drop probability. The partial drop probability from the initial tier 2 transmission is preserved.

[0270] As a concrete example of some of the embodiments described in steps 1-4 above, consider Figures 7(A)-7(E). For the purposes of this example, there is a single flow with target reliability set to T=98%, no latency constraint, and highest stratum set to M=3 (meaning only stratums <3 are treated as RELIABLE, the rest are stratum 0 and are either UNRELIABLE or INDETERMINATE).

[0271] 7(A) shows three connections C1, C2, and C3, each in an INDETERMINATE state with a 27.1% assigned drop probability. The input queue initially has packets 1 through 6 waiting in the queue, and scheduler 160 is not currently tracking any packets. In this embodiment, scheduler 160 tracks packet data through the ordered list with packets with the highest cumulative drop probability appearing on the right, and packets further to the right in the list are considered first for steps 1 and 3 in the algorithm above.

[0272] Connection C1 requests a burst of three packets. At this point, no packets have been sent previously, so step 1 yields no packets. In step 2, scheduler 160 dequeues packet 1 from the input queue and assigns it a cumulative drop probability (CDP) of 27.1%, matching C1's p. Its inflight count is set to 1, but because C1 is not in the RELIABLE state, its tier remains unassigned.

[0273] So the metadata queue looks like this:

[0274] [Packets=1, CDP=27.1%, Inflight Count=1, Layer=Unassigned]

[0275] Proceed with step 2 until the burst request of three packets is satisfied by dequeuing packets 2 and 3 from the input queue and assigning them to the metadata queue in the same way that the first packet was processed. Because the burst request has been satisfied, steps 3 and 4 are not performed. This leaves the metadata queue in the following state:

[0276] [Packets=3, CDP=27.1%, Inflight Count=1, Layer=Unassigned]

[0277] Packets=2, CDP=27.1%, Inflight Count=1, Hierarchy=Unassigned]

[0278] [Packets=1, CDP=27.1%, Inflight Count=1, Layer=Unassigned]

[0279] Next, connection C2 requests a burst of three packets. In step 1, packet 1 appears at the end of the queue.

number

[0280] [Packets=1, CDP=7.34%, Inflight Count=2, Layer=Unassigned]

[0281] Packets=3, CDP=27.1%, Inflight Count=1, Hierarchy=Unassigned]

[0282] [Packets=2, CDP=27.1%, Inflight Count=1, Layer=Unassigned]

[0283] Step 1 continues with packets 2 and 3, then steps 2-4 are skipped since the burst request is satisfied. The metadata queue at this point is:

[0284] [Packets=3, CDP=7.34%, Inflight Count=2, Layer=Unassigned]

[0285] [Packets=2, CDP=7.34%, Inflight Count=2, Layer=Unassigned]

[0286] [Packets=1, CDP=7.34%, Inflight Count=2, Layer=Unassigned]

[0287] Next, connection C3 requests a burst of three packets. Each of the three packets on the metadata queue is eligible, so they are all added to the burst and their metadata is updated. Again, because C3 is not in the RELIABLE state, the tier is left unallocated. The metadata queue now looks like this:

[0288] [Packets=3, CDP=1.99%, Inflight Count=3, Layer=Unassigned]

[0289] [Packets=2, CDP=1.99%, Inflight Count=3, Layer=Unassigned]

[0290] [Packets=1, CDP=1.99%, Inflight Count=3, Layer=Unassigned]

[0291] This is the state illustrated in Figure 7(A). Note that even though no packets were sent over the connections that were RELIABLE, subsequent requests to schedule packets on any of those connections will not result in packets from the metadata queue in step 1, because the CDP for each packet meets the target reliability criterion. Also note that packets 1, 2, and 3 are in-flight on connections C1, C2, and C3, respectively.

[0292] Scheduler 160 then receives acknowledgments that packets 1, 2, and 3 were received by the peer, so their metadata is removed from the queue. The connection characteristics have also been updated by connection controller 154, so that C1 now has p=1%, is RELIABLE, and is in tier 1, C2 has p=3%, is RELIABLE, and is in tier 2, and C3 has p=40%, is UNRELIABLE. Finally, packets 7, 8, 9, 10, and 11 arrive at the input queue and are queued behind packets 4, 5, and 6.

[0293] Connection C1 then requests a burst of two packets. Since the metadata queue is empty, step 1 is skipped. Step 2 dequeues packets 4 and 5 from the input queue, assigns them to the burst, and updates the metadata queue with this information. Notably, since C1's tier is reliable, all of these packets are assigned to C1's tier. Since these packets satisfy the burst request, steps 3 and 4 are skipped. The metadata queue now looks like this:

[0294] [Packets=5, CDP=1%, Inflight Count=1, Layer=1]

[0295] [Packets=4, CDP=1%, Inflight Count=1, Layer=1]

[0296] Connection C2 then requests a burst of two packets. In step 1, there were no packets in the metadata queue that were eligible to be included in this burst, which means that they all meet the target reliability requirements.

number

[0297] Step 2 is then applied to dequeue packets 6 and 7 from the input queue. These packets have appropriately updated metadata and, since they satisfy the burst requirement, steps 3 and 4 are skipped. The metadata queue now looks like this:

[0298] [Packets=5, CDP=1%, Inflight Count=1, Layer=1]

[0299] [Packets=4, CDP=1%, Inflight Count=1, Layer=1]

[0300] [Packets=7, CDP=3%, Inflight Count=1, Layer=2]

[0301] [Packets=6, CDP=3%, Inflight Count=1, Layer=2]

[0302] Connection C3 then requests a burst of size 1. In step 1, no packets are eligible to be sent on C3. Packets 4 and 5 are ineligible because they already meet the reliability criteria. While packets 4 and 5 do not meet the reliability criteria, they are assigned to tier 2 and are therefore ineligible to be sent over C3, which does not have an assigned tier. Step 2 then dequeues packet 8 and updates the metadata accordingly, which satisfies the burst request, and steps 4 and 5 are skipped. The metadata queue now looks like this:

[0303] [Packets=5, CDP=1%, Inflight Count=1, Layer=1]

[0304] [Packets=4, CDP=1%, Inflight Count=1, Layer=1]

[0305] [Packets=7, CDP=3%, Inflight Count=1, Layer=2]

[0306] [Packets=6, CDP=3%, Inflight Count=1, Layer=2]

[0307] [Packets=8, CDP=40%, Inflight Count=1, Layer=Unassigned]

[0308] This results in the situation illustrated in FIG. 7(B), with packets 4 and 5 in flight on C1, packets 5 and 6 in flight on C2, and packet 8 in flight on C3.

[0309] Next, packets 4 and 5 receive an acknowledgement, and the stratum and p values ​​for each connection remain the same. The input queue still contains packets 9, 10, and 11, and the metadata queue is as follows:

[0310] [Packets=7, CDP=3%, Inflight Count=1, Layer=2]

[0311] [Packets=6, CDP=3%, Inflight Count=1, Layer=2]

[0312] [Packets=8, CDP=40%, Inflight Count=1, Layer=Unassigned]

[0313] Next, C1 requests a burst of three packets. In step 1, the first packet is packet 8, which qualifies because it does not meet the reliability requirements and its tier is unassigned. Other packets in the queue do not meet the reliability requirements, but do not qualify for this burst because their assigned tiers do in fact match C1's tier. Packet 8 is added to the burst and its metadata is updated so that its assigned tier is 1, resulting in the following metadata queue:

[0314] [Packets=8, CDP=0.4%, Inflight Count=2, Layer=1]

[0315] [Packets=7, CDP=3%, Inflight Count=1, Layer=2]

[0316] [Packets=6, CDP=3%, Inflight Count=1, Layer=2]

[0317] Step 2: The system then dequeues packets 9 and 10 from the input queue and adds them to the burst. This satisfies the burst request, and steps 3 and 4 are skipped. The metadata queue now looks like this:

[0318] [Packets=8, CDP=0.4%, Inflight Count=2, Layer=1]

[0319] [Packets=10, CDP=1%, Inflight Count=1, Layer=1]

[0320] [Packets=9, CDP=1%, Inflight Count=1, Layer=1]

[0321] [Packets=7, CDP=3%, Inflight Count=1, Layer=2]

[0322] [Packets=6, CDP=3%, Inflight Count=1, Layer=2]

[0323] Next, connection C2 requests a burst of size 3. In step 1, packets 6 and 7 are eligible and are added to the burst, while packets 8, 9, and 10 are ineligible. In step 2, the burst request is satisfied by dequeuing packet 11 from the input queue and adding it to the burst. The input queue is now empty, and the metadata queue is observed as follows:

[0324] [Packets=7, CDP=0.1%, Inflight Count=2, Layer=2]

[0325] [Packets=6, CDP=0.1%, Inflight Count=2, Layer=2]

[0326] [Packets=8, CDP=0.4%, Inflight Count=2, Layer=1]

[0327] [Packets=10, CDP=1%, Inflight Count=1, Layer=1]

[0328] [Packets=9, CDP=1%, Inflight Count=1, Layer=1]

[0329] [Packets=11, CDP=3%, Inflight Count=1, Layer=2]

[0330] Next, C3 requests a burst of size 3, minimum size 0. There are no eligible packets from step 1 because all packets meet the reliability requirements or have already been assigned to a tier. In step 2, the input queue is empty. Steps 3 and 4 are not required because there is no minimum size associated with the burst. This situation is illustrated in Figure 7(C), where packets 8, 9, and 10 are in flight on C1, 6, 7, and 11 are in flight on C2, and packet 8 is also in flight on C3.

[0331] Next, scheduler 160 is notified that packets 7, 8, 9, and 10 have been acknowledged by the peer, so their metadata is deleted. The metadata queue is as follows:

[0332] [Packets=6, CDP=0.1%, Inflight Count=2, Layer=2]

[0333] [Packets=11, CDP=3%, Inflight Count=1, Layer=2]

[0334] One instance of packet 6 is marked as lost. The metadata queue looks like this:

[0335] [Packets=6, CDP=0.1%, Inflight Count=1, Layer=2]

[0336] [Packets=11, CDP=3%, Inflight Count=1, Layer=2]

[0337] Another instance of packet 6 is marked as lost, which causes the packet's metadata to be reset (assuming the packet's flow criteria allows for retransmission of the packet). The metadata queue now looks like this:

[0338] [Packets=11, CDP=3%, Inflight Count=1, Layer=2]

[0339] [Packets=6, CDP=100%, Inflight Count=0, Layer=Unassigned]

[0340] As a result of these losses, the characteristics of connection C2 are adjusted to have p=27.1%, which is now INDETERMINATE. The transition to INDETERMINATE removes C2 from tier 2, so the number of connections in tier 2 becomes zero, and scheduler 160 is so notified. As a result, scheduler 160 adjusts the metadata of any packets that were assigned to tier 2, which are now unassigned.

[0341] [Packets=11, CDP=3%, Inflight Count=1, Layer=Unassigned]

[0342] [Packets=6, CDP=100%, Inflight Count=0, Layer=Unassigned]

[0343] Connection C1 now requests a burst of three packets. In step 1, it finds that packets 6 and 11 are eligible and adds them to the burst. Step 2 finds that there are no packets on the input queue, so steps 3 and 4 are not required because padding was not requested. The metadata queue now looks like this:

[0344] [Packets=11, CDP=0.03%, Inflight Count=2, Layer=1]

[0345] [Packets=6, CDP=1%, Inflight Count=1, Layer=1]

[0346] These last three queue states are illustrated in Figure 7(D).

[0347] Next, C2 requests a burst of exactly three packets. Step 1 results in no packets because all packets in the metadata queue meet the reliability requirements. Step 2 results in no packets because the input queue is empty. Step 3 results in packets 6 and 11, whose metadata is updated. Step 4 results in a dummy packet that is not recorded in the metadata queue, which pads the burst to the requested three packets. The metadata queue now looks like this:

[0348] [Packets=11, CDP=0.01%, Inflight Count=3, Layer=1]

[0349] [Packets=6, CDP=0.427%, Inflight Count=2, Layer=1]

[0350] The example ends with both remaining packets being acknowledged and the metadata queue being empty.

[0351] FIG. 8 is a block schematic diagram of an exemplary computing device 800 according to some embodiments. The exemplary computing device 800 can be utilized to implement part or all of system 100 and is directed to a network computing device (e.g., a router, gateway, network switch, or hub) according to various embodiments that controls the routing and / or use of various network connections and / or communication paths. The exemplary computing device 800 can include a processor 802 (e.g., a hardware processor, microprocessor, reduced instruction set processor, or central processing unit) interoperating with computer memory 804 (e.g., read-only memory, embedded memory, or random access memory), and may have one or more input / output interfaces 806 for receiving commands or a display component, such as an associated display having graphics rendered thereon or a graphical user interface. The computer memory 804 can store data, such as datasets enabling computation, command instructions, data structures, such as routing tables and / or routing rules, observed network communication characteristics, and the like. One or more network interfaces 808 are electrically coupled to computing device 800 to enable electronic communication through the one or more network interfaces. As mentioned herein, network interface 808 may be utilized to provide one or more network connections over which transmission or communication of data packets may be coordinated in a manner that allows for more efficient use of communication resources.

[0352] Figure 9 is a diagram 900 illustrating a physical computing server rack, according to some embodiments. Within the computing server rack shown in Figure 9, there are multiple networked computing components, including rack-mounted computing equipment, e.g., networked routers, switches, hubs, and gateways. In this example, the components interoperate with each other to control routing, e.g., by establishing and / or periodically updating routing tables and / or routing rules. In another embodiment, the components interoperate to control routing, e.g., by controlling network connections, routing data packets, etc., in accordance with the routing tables and / or routing rules.

[0353] As described herein, the techniques are technical and computational solutions that can be implemented in the form of a physical data router or other networking device configured to control the routing of data packet communications. The device can include one or more processors operating in conjunction with computer memory, which may be coupled to data storage. Corresponding methods and non-transitory computer-readable media (e.g., diskettes, solid-state storage, hard disk drives) storing machine-interpretable instructions (e.g., software that, when executed by a computer processor, causes the processor to perform the methods described herein in various embodiments) are also included.

[0354] The terms "connected" or "coupled" may include both direct coupling (where two components coupled together are in contact with each other) and indirect coupling (where at least one additional element is located between the two elements).

[0355] While the embodiments have been described in detail, it is to be understood that various changes, substitutions, and alterations can be made herein without departing from the scope. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods, and steps described in the specification.

[0356] As will be readily apparent to those skilled in the art from this disclosure, any now existing or later developed processes, machines, manufacture, compositions of matter, means, methods, or steps may be utilized which perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein, and it is therefore intended that the appended claims include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0357] As will be understood, the examples illustrated above are intended to be illustrative only.

Claims

1. A processor coupled to computer memory and data storage, comprising: receiving one or more data sets indicative of monitored network communication characteristics; maintaining in a data structure stored on a data storage a hierarchical representation of the plurality of connections separated into a plurality of groups, each group established based at least on a minimum probability associated with successful communication of data packets over one or more of the plurality of connections residing within the group; controlling a plurality of communications of a data packet such that a data packet is transmitted at least once over one or more of a plurality of connections such that, collectively, the plurality of communications cause transmission of the data packet to satisfy a target probability of success threshold associated with successful communication of the data packet over the one or more of the plurality of connections; a processor comprising: Including, the plurality of groups are organized into a plurality of corresponding tiers, each tier representing a number of times the data packet must be transmitted over a corresponding connection of any of the tiers to achieve the target probability threshold; a network router computing device, wherein the plurality of communications includes retransmissions over a connection of one of the plurality of corresponding hierarchies, and the number of retransmissions is based on the number of times a data packet may have to be transmitted over the corresponding connection of the hierarchical layer to achieve a target probability threshold for the corresponding hierarchical layer.

2. 2. The network router computing device of claim 1, wherein the retransmissions are performed over different connections in the hierarchy.

3. 2. The network router computing device of claim 1, wherein the plurality of communications includes retransmissions over connections of different hierarchies of the plurality of corresponding hierarchies.

4. 2. The network router computing device of claim 1, wherein an additional group of the plurality of groups is an uncertain group established for connections that do not have enough data to assess reliability, or an untrusted group established for connections that are indicated to be below a threshold reliability for data communications.

5. 5. The network router computing device of claim 4, wherein membership of the uncertain or unreliable group is periodically revised to categorize connections into multiple groups.

6. 5. The network router computing device of claim 4, wherein connectivity of the plurality of groups is periodically monitored and membership is moved to the uncertain group or the unreliable group when connectivity data becomes stale or indicates increased reliability.

7. 5. The network router computing device of claim 4, wherein the uncertainty group is utilized for communicating data packets using assigned reliability probabilities.

8. 8. The network router computing device of claim 7, wherein the assigned reliability probability is periodically adjusted based on monitored network communication characteristics.

9. 5. The network router computing device of claim 4, wherein the connections for which a number of transmissions or retransmissions has been requested is greater than a threshold are moved to the uncertain group or the unreliable group.

10. 5. The network router computing device of claim 4, wherein the processor is further configured to periodically transmit data packets over connections of the uncertain group or the unreliable group, the periodic transmission of the data packets incorporating a back-off timer that reduces overall system inefficiencies.

11. The network router computing device of claim 1 , wherein the processor is configured to maintain one or more state machines that classify each connection of the plurality of connections.

12. 12. The network router computing device of claim 11, wherein the one or more state machines are adapted to control the classification of one of the corresponding connections of the hierarchy as trusted or untrusted.

13. 13. The network router computing device of claim 12, wherein the one or more state machines are adapted to control the classification of the corresponding connection as indeterminate in addition to trusted or untrusted.

14. 13. The network router computing device of claim 12, wherein the one or more state machines are periodically updated to modify the classification of the corresponding connections as trusted or untrusted.

15. 13. The network router computing device of claim 12, wherein the one or more state machines control group transitions of the one or more connections.

16. 10. The network router computing device of claim 1, wherein the network router computing device is a portable communication controller that can be carried by a person.

17. 10. The network router computing device of claim 1, wherein the network router computing device is a portable communication device coupled to a vehicle.

18. 10. The network router computing device of claim 1, wherein the network router computing device is a portable communications device that resides within a communications relay station.

19. receiving one or more data sets indicative of monitored network communication characteristics; maintaining in a data structure stored on a data storage a hierarchical representation of the plurality of connections separated into a plurality of groups, each group established based at least on a minimum probability associated with successful communication of data packets over one or more of the plurality of connections residing within the group; controlling a plurality of communications of a data packet such that a data packet is transmitted at least once over one or more of a plurality of connections such that, collectively, the plurality of communications cause transmission of the data packet to satisfy a target probability of success threshold associated with successful communication of the data packet over the one or more of the plurality of connections; Including, the plurality of groups are organized into a plurality of corresponding tiers, each tier representing a number of times the data packet must be transmitted over a corresponding connection of any of the tiers to achieve the target probability threshold; A method of performing network communications, wherein the plurality of communications includes retransmissions over a connection of one of the plurality of corresponding layers, and the number of retransmissions is based on the number of times a data packet may have to be transmitted over the corresponding connection of the layer to achieve a target probability threshold for the corresponding layer.

20. 20. The method of claim 19, wherein the retransmission is performed over a different connection of the layer.

21. 20. The method of claim 19, wherein the plurality of communications comprises retransmissions over connections of different tiers of the plurality of corresponding tiers.

22. 20. The method of claim 19, wherein an additional group of the plurality of groups is an uncertain group established for connections that do not have sufficient data to assess reliability, or an unreliable group established for connections that are indicated to be below a threshold reliability for data communication.

23. 23. The method of claim 22, wherein the membership of the uncertain or unreliable group is periodically revised to classify connections into multiple groups.

24. 23. The method of claim 22, wherein connectivity of the plurality of groups is periodically monitored and membership is moved to the uncertain group or the unreliable group when connectivity data becomes stale or indicates increased reliability.

25. 23. The method of claim 22, wherein the uncertain group is utilized for communicating data packets using assigned reliability probabilities.

26. 26. The method of claim 25, wherein the assigned reliability probability is periodically adjusted based on monitored network communication characteristics.

27. 23. The method of claim 22, wherein the connections for which a number of transmissions or retransmissions have been requested is greater than a threshold are moved to the uncertain group or the unreliable group.

28. 23. The method of claim 22, comprising periodically transmitting data packets over a connection of the uncertain group or the unreliable group, the periodic transmission of the data packets incorporating a back-off timer that reduces overall system inefficiencies.

29. 20. The method of claim 19, comprising maintaining one or more state machines that classify each connection of the plurality of connections.

30. 30. The method of claim 29, wherein the one or more state machines are adapted to control the classification of one of the corresponding connections of the hierarchy as trusted or untrusted.

31. 31. The method of claim 30, wherein the one or more state machines are adapted to control classification of the corresponding connection as indeterminate in addition to trusted or untrusted.

32. 31. The method of claim 30, wherein the one or more state machines are periodically updated to modify the classification of the corresponding connections as trusted or untrusted.

33. 31. The method of claim 30, wherein the one or more state machines control group transitions of the one or more connections.

34. 20. The method of claim 19, wherein the method is performed on a portable communication controller that is portable with a person.

35. 20. The method of claim 19, wherein the method is performed on a portable communication device coupled to the vehicle.

36. 20. The method of claim 19, wherein the method is performed on a portable communication device resident within a communication relay station.

37. A non-transitory computer readable medium storing non-transitory instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 19 to 36.

Images