Transport Control Protocol Trip Time Estimation

Abstract

To generate estimated performance characteristics of a TCP connection between a source endpoint and a destination endpoint communicating through a middle-box.SOLUTION: A method includes sparsely sampling, at a middle-box, a plurality of packets from a transport control protocol (TCP) connection between a source endpoint and a destination endpoint communicating through the middle-box, generating a time stamp, and recording a sequence number and an acknowledgement number for each packet. The method also includes generating an estimated performance characteristic for the TCP connection between the source endpoint and the destination endpoint communicating through the middle-box over a predetermined period of time based on at least two of the time stamp, the sequence number, or the acknowledgement number.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] Technical Field TECHNICAL FIELD This disclosure relates to transport control protocol trip time estimation. [Background technology]

[0002] background Traditionally, processes running on nodes can help understand the performance of Transport Control Protocol (TCP) connections between nodes (or endpoints). For example, a system may be arranged to obtain the round trip time (RTT) of a packet by measuring the start time when the node transmits the packet and the acknowledgement time when the node receives the acknowledgement. However, as more intermediate nodes are implemented between more traditional endpoints by using middleboxes, the measurement of TCP performance at the endpoints must consider new performance measurement techniques to facilitate TCP connection optimization. Summary of the Invention [Means for solving the problem]

[0003] overview One aspect of the present disclosure provides a method for estimating performance characteristics of a transport control protocol. The method includes data processing hardware of a middlebox culling and sampling a plurality of packets from a Transport Control Protocol (TCP) connection between a source endpoint and a destination endpoint communicating through the middlebox over a predetermined time period. For each packet of the sampled plurality of packets, the method includes the data processing hardware generating a timestamp at which the packet was sampled, and the data processing hardware recording a sequence number and an acknowledgment number of the packet. The method also includes the data processing hardware generating an estimated performance characteristic of the TCP connection between the source endpoint and the destination endpoint communicating through the middlebox over the predetermined time period based on at least two of the timestamp, the sequence number, or the acknowledgment number corresponding to one or more packets of the sampled plurality of packets.

[0004] In some implementations, the method includes the data processing hardware determining a first sequence number recorded in a first packet of the sampled plurality of packets transferred between the middlebox and the destination endpoint. The method also includes the data processing hardware determining whether the first sequence number recorded in the first packet matches any of the acknowledgment numbers recorded in the sampled plurality of packets transferred between the destination endpoint and the middlebox. In this implementation, if the first sequence number matches an acknowledgment number recorded in a second packet of the sampled plurality of packets, the method includes the data processing hardware determining a first trip time between the middlebox and the destination endpoint based on a difference between a timestamp generated for the first packet and a timestamp generated for the second packet. In this implementation, if the first sequence number does not match any of the acknowledgment numbers recorded in the sampled packets, the method includes the data processing hardware estimating a timestamp of a first unsampled packet from the TCP connection by interpolating the acknowledgment numbers recorded in the sampled packets, and the first unsampled packet is associated with the acknowledgment number that matches the first sequence number. The implementation further includes the data processing hardware determining a first trip time between the middlebox and the destination endpoint based on a difference between a first timestamp of the generated first packet and the estimated timestamp for the first unsampled packet.

[0005] In some examples, the method includes the data processing hardware identifying a second sequence number recorded in a third packet of the sampled plurality of packets, the third packet being forwarded between the middlebox and the source endpoint. The method also includes the data processing hardware determining whether the second sequence number recorded in the third packet matches any of the acknowledgment numbers recorded in the sampled plurality of packets forwarded between the source endpoint and the middlebox. In this example, if the second sequence number matches the acknowledgment number recorded in a fourth packet of the sampled plurality of packets, the method includes the data processing hardware determining a second trip time between the middlebox and the source endpoint based on a difference between a timestamp generated for the third packet and a timestamp generated for the fourth packet. In this example, if the second sequence number does not match any of the acknowledgment numbers recorded in the sampled plurality of packets, the method further includes the data processing hardware estimating a timestamp of a second unsampled packet from the TCP connection by interpolating the acknowledgment numbers recorded in the sampled plurality of packets, the second unsampled packet being associated with an acknowledgment number that matches the second sequence number, and the data processing hardware determining a second trip time between the middlebox and the source endpoint based on a difference between a third timestamp generated for the third packet and the estimated timestamp for the second unsampled packet. Generating the estimated performance characteristic of the TCP connection includes calculating a round trip time between the source endpoint and the destination endpoint communicating through the middlebox by summing the first trip time between the middlebox and the destination endpoint and the second trip time between the middlebox and the source endpoint.

[0006] In another configuration, the method includes estimating timestamps of unsampled packets from the TCP connection by interpolating acknowledgment numbers recorded in the sampled packets. The method may also include estimating timestamps of unsampled packets from the TCP connection by interpolating sequence numbers recorded in the sampled packets.

[0007] In some implementations, generating an estimated performance characteristic of the TCP connection includes data processing hardware determining a slope of the acknowledgment numbers over a predetermined time period based on the acknowledgment numbers recorded in the sampled packets forwarded from the destination endpoint to the source endpoint through the middlebox, and generating a throughput based on the slope of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the source endpoint to the destination endpoint through the middlebox. In other implementations, generating an estimated performance characteristic of the TCP connection includes determining a slope of the acknowledgment numbers over time based on the acknowledgment numbers recorded in the sampled packets forwarded from the source endpoint to the destination endpoint through the middlebox, and generating a throughput based on the slope of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the destination endpoint to the source endpoint through the middlebox.

[0008] In some examples, the method further comprises: at a particular point in time, the data processing hardware The method includes receiving a first packet communicated from a source endpoint over a TCP connection, the first packet including a first sequence number and a first acknowledgment number, and the data processing hardware receiving a second packet communicated from a destination endpoint over a TCP connection, the second packet including a second sequence number and a second acknowledgment number. The method also includes the data processing hardware estimating a window size of the source endpoint by determining a difference between the first sequence number associated with the first packet and the second acknowledgment number associated with the second packet. In these examples, the method may further include the data processing hardware estimating a window size of the destination endpoint by determining a difference between the second sequence number associated with the second packet and the first acknowledgment number associated with the first packet.

[0009] In some implementations, sparsely sampling the plurality of packets includes sampling fewer than all packets from a TCP connection between a source endpoint and a destination endpoint. Sparsely sampling the plurality of packets may include uniformly sampling packets from the TCP connection in a stateless and systematic manner. The source endpoint may be a user equipment (UE) that communicates with a remote distributed network, and the destination endpoint corresponds to a server that communicates with the remote distributed network.

[0010] Another aspect of the present disclosure provides a system for estimating performance characteristics of a transport control protocol. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform the following operations: decimating and sampling, at the middlebox, a plurality of packets from a Transport Control Protocol (TCP) connection between a source endpoint and a destination endpoint communicating through the middlebox over a predetermined time period; generating, for each packet of the sampled plurality of packets, a timestamp when the packet was sampled, and recording a sequence number and an acknowledgment number of the packet; generating an estimated performance characteristic of the TCP connection between the source endpoint and the destination endpoint communicating through the middlebox over a predetermined time period based on at least two of the timestamp, the sequence number, or the acknowledgment number corresponding to one or more packets of the sampled plurality of packets.

[0011] In some implementations, the operations include identifying a first sequence number recorded in a first packet of the sampled packets forwarded between the middlebox and the destination endpoint. The operations also include determining whether the first sequence number recorded in the first packet matches any of the acknowledgment numbers recorded in the sampled packets forwarded between the destination endpoint and the middlebox. In this implementation, if the first sequence number matches the acknowledgment number recorded in a second packet of the sampled packets, the operations include determining a first trip time between the middlebox and the destination endpoint based on a difference between a timestamp generated for the first packet and a timestamp generated for the second packet. In this implementation, if the first sequence number does not match any of the acknowledgment numbers recorded in the sampled packets, the operations estimate a timestamp of a first unsampled packet from the TCP connection by interpolating the acknowledgment numbers recorded in the sampled packets, the first unsampled packet being associated with an acknowledgment number that matches the first sequence number. In this implementation, the operations include determining a first trip time between the middlebox and the destination endpoint based on a difference between a timestamp generated for the first packet and a timestamp generated for the second packet. and determining a first trip time between the middlebox and the destination endpoint based on a difference between the first timestamp of the one packet and the estimated timestamp for the first unsampled packet.

[0012] In some examples, the operations include identifying a second sequence number recorded in a third packet of the sampled plurality of packets, the third packet forwarded between the middlebox and the source endpoint. The operations also include determining whether the second sequence number recorded in the third packet matches any of the acknowledgment numbers recorded in the sampled plurality of packets forwarded between the source endpoint and the middlebox. In this example, if the second sequence number matches the acknowledgment number recorded in a fourth packet of the sampled plurality of packets, the operations include determining a second trip time between the middlebox and the source endpoint based on a difference between a timestamp generated for the third packet and a timestamp of the fourth packet. In this example, if the second sequence number does not match any of the acknowledgment numbers recorded in the sampled plurality of packets, the operations include estimating a timestamp of a second unsampled packet from the TCP connection by interpolating the acknowledgment numbers recorded in the sampled plurality of packets, the second unsampled packet being associated with an acknowledgment number that matches the second sequence number, and determining a second trip time between the middlebox and the source endpoint based on a difference between a third timestamp generated for the third packet and the estimated timestamp for the second unsampled packet. Generating the estimated performance characteristic of the TCP connection may include calculating a round trip time between the source endpoint and the destination endpoint communicating through the middlebox by summing the first trip time between the middlebox and the destination endpoint and the second trip time between the middlebox and the source endpoint.

[0013] In another configuration, the operations include estimating a time of the unrecorded acknowledgment number by interpolating acknowledgment numbers recorded in the sampled packets. The operations may include estimating a time of the unrecorded sequence number by interpolating sequence numbers recorded in the sampled packets.

[0014] In some implementations, generating an estimated performance characteristic of the TCP connection includes determining a slope of the acknowledgment numbers over a predetermined time period based on the acknowledgment numbers recorded in the sampled packets forwarded from the destination endpoint to the source endpoint through the middlebox, and generating a throughput based on the slope of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the source endpoint to the destination endpoint through the middlebox. In other implementations, generating an estimated performance characteristic of the TCP connection includes determining a slope of the acknowledgment numbers over time based on the acknowledgment numbers recorded in the sampled packets forwarded from the source endpoint to the destination endpoint through the middlebox, and generating a throughput based on the slope of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the destination endpoint to the source endpoint through the middlebox.

[0015] In some examples, the operations include receiving, at a particular point in time, a first packet communicated from a source endpoint over a TCP connection, the first packet including a first sequence number and a first acknowledgment number, and receiving a second packet communicated from a destination endpoint over the TCP connection, the second packet including a second sequence number and a second acknowledgment number. The operations also include receiving a second packet associated with the first packet. and estimating a window size for the source endpoint by determining a difference between the associated first sequence number and a second acknowledgment number associated with the second packet. In these examples, the operations may further include estimating a window size for the destination endpoint by determining a difference between the associated second sequence number and the first acknowledgment number associated with the first packet.

[0016] In some implementations, sparsely sampling the plurality of packets includes sampling fewer than all packets from a TCP connection between a source endpoint and a destination endpoint. Additionally or alternatively, sparsely sampling the plurality of packets may include uniformly sampling packets from the TCP connection in a stateless and systematic manner. The source endpoint may be a user equipment (UE) that communicates with a remote distributed network, and the destination endpoint corresponds to a server that communicates with the remote distributed network.

[0017] Another aspect of the present disclosure provides a method for estimating a transport control protocol trip time. The method includes data processing hardware of a middlebox culling and sampling a plurality of packets from a Transport Control Protocol (TCP) connection between endpoints communicating through the middlebox over a predetermined time period. For each packet of the sampled plurality of packets, the method includes the data processing hardware generating a timestamp at which the packet was sampled. The method includes the data processing hardware determining that a first sequence number of a first sampled packet of the sampled plurality of packets forwarded between the middlebox and the corresponding endpoint does not match any of the acknowledgment numbers of the sampled plurality of packets. The method also includes the data processing hardware estimating a time of an acknowledgment number matching the first sequence number by interpolating the acknowledgment numbers of the sampled plurality of packets. Furthermore, the method includes the data processing hardware determining a trip time between the middlebox and the corresponding endpoint based on a difference between a first timestamp associated with the first packet and the estimated time.

[0018] This aspect may include one or more of the following optional features: In some examples, throttling and sampling the plurality of packets includes sampling fewer than all packets from a TCP connection between the endpoints communicating through the middlebox. Optionally, it may include uniformly sampling packets from the TCP connection in a stateless and systematic manner. The endpoints may include a user equipment (UE) communicating with a remote distributed network as a source endpoint and a server communicating with the remote distributed network as a destination endpoint.

[0019] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0020] [Figure 1A] FIG. 1 is a schematic diagram illustrating an example communication network implementing Transport Control Protocol (TCP) connections. [Figure 1B] FIG. 1 is a schematic diagram illustrating an example communication network implementing Transport Control Protocol (TCP) connections. [Figure 1C] FIG. 1 is a schematic diagram illustrating an example communication network implementing Transport Control Protocol (TCP) connections. [Figure 1D] FIG. 1 is a schematic diagram illustrating an exemplary TCP connection between endpoints of a communication network. [Figure 2A] FIG. 1 is a schematic diagram illustrating an example performance evaluation apparatus for a middlebox in a communication network. [Figure 2B] FIG. 1 is a schematic diagram illustrating an example performance evaluation apparatus for a middlebox in a communication network. [Figure 2C] FIG. 1 is a schematic diagram illustrating an example performance evaluation apparatus for a middlebox in a communication network. [Figure 2D] FIG. 1 is a schematic diagram illustrating an example performance evaluation apparatus for a middlebox in a communication network. [Figure 2E] FIG. 1 is a schematic diagram illustrating an example performance evaluation apparatus for a middlebox in a communication network. [Diagram 3] 1 is a flow chart illustrating an exemplary arrangement of operations of a method for estimating TCP performance. [Figure 4] 1 is a flow chart illustrating an exemplary arrangement of operations of a method for estimating TCP performance. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example computing device that may be used to implement the system and method for migrating virtual network functions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description Like reference symbols in the various drawings indicate like elements.

[0022] The Transport Control Protocol / Internet Protocol (TCP / IP) suite is primarily used for network communication. The TCP / IP protocol suite was originally designed based on end-to-end communication. For example, a source endpoint, such as a user, communicates with a destination endpoint, such as a server. This end-to-end design generally assumes that a gateway node (i.e., a router) configured to communicate between two separate networks forwards a TCP / IP packet stream of data packets to the destination endpoint (e.g., directly or via a hop between intermediate destinations). Based on this assumption, the router does not need to modify packets (e.g., by modifying the header or payload) during the transmission of the packets. However, currently, with the development of network communication, the nodes connected within the network and / or between networks are becoming more complex. In other words, the network deploys hardware and / or software as nodes in the network environment to perform network functions (i.e., network services) other than the function of routing packets. For example, these network functions include firewalls for network protection, network address translation (NAT), load balancing, and virtual private network (VPN) tunneling, etc. To perform these network functions, network administrators and / or network providers implement middleboxes.

[0023] Unfortunately, middleboxes can interfere with generic end-to-end designs that transmit packets using TCP (i.e., over TCP connections). For example, some middleboxes are configured to inspect and / or modify packets to perform network functions. Also, middleboxes often receive packets from many TCP connections that are multiplexed through the middlebox, and typically operate above the Internet Protocol (IP) layer. Because middleboxes operate above the IP layer, they do not use TCP flow markers such as start or end markers or acknowledgments of a flow. In other words, As such, the operation of a middlebox may not provide the same or similar indicators of packet flow performance as a TCP flow (e.g., an end-to-end TCP connection). Without indicators of packet flow performance, a network administrator or network service provider may have difficulty evaluating and / or improving network communications.

[0024] Determining connection characteristics (e.g., round trip time, throughput, window size, etc.) of a TCP connection is important and useful in helping to configure, improve, and / or optimize communications in a network environment. In other words, the performance of a TCP connection can affect how effectively entities communicate with each other and whether they are actually providing a particular service (e.g., streaming audio / video at a target quality level) over the TCP connection. In some scenarios, the connection characteristics are determined or estimated at one or both ends of the TCP connection (i.e., one or both flow endpoints, e.g., source endpoint and destination endpoint). However, this often requires custom-made functionality. If the endpoints have minimal processing capabilities, it may not be possible to give custom-made functionality to such flow endpoints. For example, some Internet of Things (IOT) devices (e.g., consumer electronics, etc.) have low power or low processing capabilities. In some instances, determining and / or estimating the performance of a connection (e.g., a connection between two endpoints communicating through a middlebox) places additional strain on the connection itself. In other words, it is important for the system to determine the connection characteristics without adversely affecting the connection. Otherwise, the determined / estimated connection characteristics may not represent or may not accurately measure the actual connection characteristics to be achieved (thereby failing to accurately represent the system configuration) and may strain the connection as the system determines the connection characteristics, potentially reducing throughput and / or causing disadvantage to all parties involved in the communication.

[0025] To address these issues, implementations of the present disclosure provide a system that generates TCP connection characteristics, or estimates of TCP connection characteristics, in a middlebox associated with a TCP connection in a manner that minimizes the impact on the connection itself and on the entities involved in the connection. Moreover, because the middlebox samples only TCP packets / segments (e.g., a small portion of the TCP packets / segments communicated over the TCP connection) to generate the estimates, the middlebox imposes relatively little processing overhead and has minimal (or unnoticeable) impact on the performance of the TCP connection. Thus, the performance of other network connections managed by the middlebox is not affected.

[0026] 1A and 1B, in some implementations, the networking environment 100 is a distributed system (e.g., a remote network such as a cloud environment) that includes loosely coupled computing resources 110, 110a-n. The computing resources 110 may be referred to as servers 110. These computing resources 110 may include data processing hardware 112 (e.g., one or more central processing units (CPUs) shown in FIGS. 1B and 1C) and / or memory hardware 114 (e.g., flash memory, random access memory (RAM), phase change memory (PCM), and / or disks shown in FIGS. 1B and 1C) that are accessible to one or more clients 120, 120a-n. For example, the computing resources 110 are shown as multiple servers. The clients 120 can communicate with the computing resources 110 hosted on the physical network layer 102 via a network 130. For example, the clients 120 and the servers 110 represent a group of machines (e.g., hosts) that form endpoints of a physical network.

[0027] Network 130 includes many different types of networks ranging from private and enterprise networks to public networks. As some more specific examples, networks include personal area networks (PANs), local area networks (LANs), wireless local area networks (WLANs), wide area networks (WANs), storage / system area networks (SANs), passive optical networks (PONs), enterprise private networks (EPNs), virtual private networks (VPNs), radio access networks (RANs), packet core networks, etc. In some configurations, clients 120 and end hosts (i.e., hosts of computing resources 110) can access (i.e., connect to) network 130 through hardware such as a network interface controller (NIC) (also known as a network interface card, network adapter, or LAN adapter) or, in the case of optical networks, through a synchronous transport module (STM). For example, FIG. 1A shows each client 120 including at least one NIC 122, 122a-n and each server 110 including at least one NIC 116, 116a-n.

[0028] With reference to FIG. 1A, in some examples, the network 130 includes a physical network layer 102 and a virtual network layer 104. The virtual network layer 104 can be overlaid on the physical network layer 102 to form a virtual network (also known as an overlay network) with its own Internet Protocol (IP) address space. The virtual network layer 104 allows (e.g., by a network administrator) to manage, design, and / or deploy the network 130 to separate network functions from hardware equipment (e.g., proprietary hardware equipment). In other words, the virtual network layer 104 enables virtual network functions (also called network function virtualization), which allows network services to be provided in software, allowing more adaptability and scalability than traditional networks that include dedicated hardware. For example, software-defined networking (SDN) of virtual networks removes the control plane from the network hardware and implements the control plane in software. In some implementations, the virtual network layer 104 of the network 130 performs network functions in a hybrid virtualization approach, using software to virtually perform some network functions and dedicated hardware to perform other network functions. The virtual and / or hybrid approach can provide network providers or network administrators with flexibility and / or optimization regarding network space, power, size, scalability, deployment time, and / or repair and maintenance. For example, the virtual network layer 104 is dynamically configured to allow fundamental changes to the physical network layer 102 without interrupting network services.

[0029] The virtual network layer 104 may be configured to perform various network functions and / or mirror traditional network functions of a physical network. As some examples, the network functions include routing, network address translation (NAT), load balancing (LB), security (e.g., firewalls and intrusion detection / prevention systems), protocol translation, wide area network optimization, proxying, and caching. In other words, the virtual network functions are applicable to any data plane processing or control plane functions in the network 130. Also, FIG. 1A illustrates the physical network layer 102 and the virtual network layer 104 with dotted lines. This means that some or all of the functions of these layers 102, 104 may be physical and / or virtual. For example, the components in the illustrated virtual network layer 104, such as the client virtual machine 140, the middlebox 160, and / or the backend virtual machine 150, may be managed on the virtual network layer 104. Instead of operating as a dedicated physical hardware (i.e., a physical layer), depending on the network 130 and / or the services offered by the network 130, different types of physical layer and virtual layer configurations have various advantages.

[0030] 1A and 1B, the network environment 100 includes various endpoints. These endpoints may be physical endpoints (e.g., disposed on the physical network layer 102) or virtual endpoints (e.g., disposed on the virtual network layer 104). For example, as shown in FIGS. 1A-1C, the client 110 and / or the server 120 may be physical endpoints that communicate with each other via network addresses. In other words, the client 110 may be a source endpoint and the server 120 may be a destination endpoint (or vice versa). Data 170 in the form of packets (also referred to as data packets 170 or packets 170) may be communicated between the endpoints based on the assigned network addresses.

[0031] In some examples, the network environment 100 includes virtual network endpoints. FIG. 1A illustrates the virtual network endpoints as virtual machines (VMs), such as client virtual machines (Client VMs) 140, 140a-n and back-end virtual machines (Back-end VMs) 150, 150a-n. Generally, a virtual machine (VM) refers to an emulation or image of one or more host computer systems that allows a user to have the functionality of the host computer system without interfering with the underlying operating system of the corresponding host computer system. VMs may be created to perform dedicated tasks in one or more host environments. In some examples, the virtual network layer 104 allows multiple VMs to be created simultaneously in a host computer system. To allow multiple VMs to operate simultaneously, VMs are often isolated from critical operating system resources of the host computer system in the form of a sandbox. VMs are advantageous to the virtual network layer 104 because they can be designed to map (e.g., access) one or more computing resources 110 of the physical network layer 102 through the back-end VMs 150. For example, one or more client VMs 140 can be mapped to one or more back-end VMs 150. Each backend VM 150 is a VM associated with a computing resource 110 of the physical network layer 102 (e.g., a distributed system). By mapping to a particular computing resource 110, a client VM 140, along with a backend VM 150, may be designed for a particular network service (e.g., a network application). In some configurations, the backend VM 150 operates as a host-based VM whose data is stored on a server of the physical network layer 102 and may be utilized by or assigned to various remote clients 120. For example, the virtual network layer 104 including VMs enables a centrally managed network structure.Although network virtualization has been described generally with respect to VMs, network virtualization (e.g., with virtual network layer 104) may be generalized to connect other types of network endpoints, including virtual private networks (VPNs), bare metal servers, proxies, etc. For example, networking environment 100 supports VPN gateways as clients that forward requests from customer premises. Thus, in some examples, VMs are more generally network endpoints to accommodate other types of network connections.

[0032] In some implementations, to perform network functions, the network environment 100 includes at least one middlebox 160. As shown in FIG. 1A, the middlebox 160 may be configured to perform operations in conjunction with data processing hardware 502 (e.g., 3 and 4, and memory hardware 504 for storing instructions executable on the data processing hardware 502 to perform the methods 300, 400. The middlebox 160 may be physical hardware (e.g., a dedicated device on the physical network layer 102) or software configured to perform network functions (e.g., as shown in FIG. 1A). In general, the middlebox 160 refers to an intermediate device for performing functions other than the usual or standard functions of an Internet Protocol (IP) router on a datagram path between a source endpoint / host (e.g., a client via the client VM 140) and a destination endpoint / host (e.g., a server on the physical network layer 102 via the backend VM 150). In other words, the middlebox 160 is typically responsible for network functions other than packet routing (e.g., network address translation (NAT), network address port translation (NAPT), load balancing (LB), firewalls, intrusion detection / prevention systems, protocol translation, and proxies). The middlebox 160 is typically responsible for network functions other than routing, but may be part of the device that handles routing. Networking environment 100 thereby includes a single device that supports both routing and middlebox functions.

[0033] In the networking environment 100, the middlebox 160 receives network flows from one or more clients 120 to perform network functions. A network connection with the middlebox 160 may be established based on a connection request from the client 120. In other words, the client 120 and the backend (i.e., the server 110) may negotiate a connection. As a result, the middlebox 160 sits in the middle and processes packets belonging to the connection, modifying the packets as necessary. Once a connection is established with the middlebox 160 (i.e., a virtual connection is formed between the client VM 140 and the backend VM 150), the middlebox 160 receives network traffic (e.g., data packets 170) based on the configured network functions of the middlebox 160.

[0034] In some examples, the middlebox 160 performs NAT or NAPT on packets between the client 120 and the backend (e.g., backend VMs). In the case of NAPT, the middlebox 160 tracks each connection and the port mapping assigned to each connection. In another example, the middlebox 160 is a layer 4 load balancer middlebox that forwards packets between the client 120 and the backend, and potentially tracks the connection to ensure that future packets belonging to a connection continue to be sent to the same backend. Alternatively, the middlebox 160 may be a layer 7 (i.e., application layer) load balancer in which the client 120 establishes a Transmission Control Protocol (TCP) connection with the middlebox 160, and the middlebox 160 establishes a separate connection with the backend (i.e., on behalf of the client 120). In the case of a layer 7 load balancer, the transport protocol state (e.g., TCP sequence number and window size) is forwarded from the middlebox 160 to each end host.

[0035] As mentioned above, in some implementations, an overlay network, such as the virtual network layer 104, need not perform middlebox functions. For example, in a network such as a physical Ethernet network, software running on an end host can direct a connection to the middlebox 160 by setting the destination media access control (MAC) address to a MAC address that corresponds to the middlebox 160. In this case, to load balance the connection, the host sets the MAC address to the address of the destination (or the address of an appropriate gateway to reach the destination). In other words, the middlebox 160 communicates with a physical machine, not a VM. For example, in the case of load balancing, the client 120 and the server 110 do not need to be VMs on the virtual network layer 104, but rather connections between physical machines.

[0036] Middlebox 160 differs from routers in that it often performs stateful network functions. More specifically, stateful network functions refer to network functions that track the operational state and / or characteristics of network connections associated with the network function. For example, middlebox 160 tracks connections to recognize connections, inspect packet context (e.g., periodically inspect payloads), and / or associate new connections with existing connections. By performing stateful functions, middlebox 160 includes greater data granularity to identify and / or recall connection events or connection data associated with previous sessions of client 120 or VM. These stateful functions can provide layers 102 and 104 with security and performance that can help stabilize network services.

[0037] With reference to FIG. 1B, in some implementations, the network environment 100 includes two or more types of networks 130, 130a-c that communicate with each other. In FIG. 1B, the clients 120 are shown as user equipment (UE) associated with the RAN 130, 130a. The RAN 130a includes three UEs 120a-c that communicate with a base station (e.g., an evolved node B (eNB)). In this example, the eNB connects via an interface to an evolved packet core (EPC) network 130, 130b that includes a middlebox 160. Similarly, the EPC 130b can communicate with an external network, such as a remote network 130, 130c (e.g., a distributed system or cloud environment). The remote network 130c includes servers 110a-n configured to communicate packets 170 of data to and from the clients 120a-c via the networks 130a-c and store the packets 170.

[0038] 1C illustrates a simplified network environment 100 showing a network connection 180 between two endpoints 190. The network connection 180 passes through a middlebox 160, which causes a packet 170 (e.g., packet 170, 170 sx1-6 ) source endpoint 190 s 1 to the middlebox 160 (e.g., packets 170, 170 xd1-6 ) from the middlebox 160 to the destination endpoint 190 d The packet 170 is also transmitted in the reverse direction via network connection 180 to a destination endpoint 190 (shown as, for example, server 110). d from a source endpoint 190 (e.g., shown as client 120) s For example, FIG. 1C illustrates a packet dx1-6 ) destination endpoint 190 d 1 to the middlebox 160 (e.g., packets 170, 170 xs1-6 From the middlebox 160 to the source endpoint 190s 1C, the subscripts on the packet 170 identify the direction in which the packet 170 is transmitted. The "D" indicates the direction of the packet 170. d "X" refers to the middlebox 160. "S" refers to the source endpoint 190. s That is, "DX" refers to the destination endpoint 190 d 160 to the source endpoint 190, and “XS” corresponds to a packet 170 transmitted from the middlebox 160 to the source endpoint 190. s 1 corresponds to a packet 170 transmitted to

[0039] Source endpoint 190 s and destination endpoint 190 d To transmit packets 170 between the host and the Internet Protocol (IP), the connection 180 uses the Transmission Control Protocol (TCP). For example, the connection 180 may be referred to as a TCP connection 180. TCP is a host-to-host protocol for packet-switched communication networks 130. In general, TCP forms the interface between users or application processes and the Internet Protocol (IP). TCP segments a fixed number of bytes (known as packets 170). The TCP packet packaging is configured to transmit a continuous stream of bytes (i.e., octets) between endpoints 190 (e.g., source and destination). When a TCP connection 180 exists, TCP transmits two byte streams (or packet streams) in each direction (e.g., (1) from the source endpoint 190 to the destination endpoint 190). s From destination endpoint 190 d , and (2) the destination endpoint 190 d From source endpoint 190 s The IEEE 802.11a / b / g / n protocol is configured to send one packet at a time (in the direction to the

[0040] TCP also includes a specific structure for certain packets 170 (e.g., as shown in FIG. 1C). The structure of the packet 170 can ensure reliability of the data (i.e., the payload of the packet 170) as well as communicate characteristics about the TCP connection 180. To ensure reliability of data that is damaged, lost, duplicated, or delivered out of order, TCP assigns a sequence number 172 to each byte sent and requests a positive acknowledgement (ACK) from the destination port (e.g., one of the endpoints 190 depending on the direction in which the packet 170 is sent). In some examples, the ACK is the packet 170 itself, while in other examples, the ACK is an acknowledgement number 174. In still other examples, the ACK endpoint (e.g., designated by the destination port) sends the data to the source endpoint 190 (e.g., designated by the source port). s When sending to a destination, the ACK includes both the ACK packet 170 itself and an acknowledgement number 174. If the ACK is not received within a timeout interval, TCP retransmits the data (e.g., packet 170). The destination (e.g., one of the endpoints 190 depending on the direction in which the packet 170 is sent) can use the sequence number 172 to reorder or eliminate duplicates of the packets 170, if necessary. TCP allows the destination to determine the amount of flow it will receive in addition to reliability. In some examples, after the receiver successfully receives the last packet 170 it wants to receive, the destination communicates a window 178, including the ACK, to indicate the range of acceptable sequence numbers 172. The window 178 refers to the allowable bandwidth of data (i.e., bytes) transferred to the destination at any one time.

[0041] Source endpoint 190 s and destination endpoint 190 d To establish a TCP connection 180 between the source endpoint 190 and the destination endpoint 120, the endpoints 190 first perform a three-way handshake. In the first step of the three-way handshake, the source endpoint 190 (shown as, for example, client 120) sDestination endpoint 190 (shown, for example, as server 110) d Thus, the source endpoint 190 s The source endpoint 190 is notified by sending a packet 170 containing a synchronization sequence number (SYN). s destination endpoint 190 d and the source endpoint 190 s Set the sequence number where you want to start to the destination endpoint 190 d In the second step of the three-way handshake, the destination endpoint 190 d (e.g., server 110) responds with a SYN and an ACK. d The source endpoint 190 sends an ACK s The source endpoint 190 acknowledges the packet 170 sent from the d In the third step, the source endpoint 190 s destination endpoint 190 d This allows the TCP connection 180 to be established with the source endpoint 190 s and destination endpoint 190 d is established between

[0042] Although the three-way handshake establishes a TCP connection 180 between endpoints 190, middleboxes 160 acting as intermediate devices between the endpoints 190 thwart traditional means of monitoring TCP performance between the connected endpoints 190. As previously mentioned, some middleboxes 160 inspect and / or modify packets 170 in order to perform network functions. If packets 170 are modified, then they may be compromised in order to monitor TCP performance. Conventional techniques for detecting packet flow performance can no longer rely on fields / attributes of standard packet structures. Additionally or alternatively, middleboxes 160 often receive packets from many TCP connections 180 that are multiplexed through the middlebox and typically operate above the Internet Protocol (IP) layer. Because middleboxes operate above the IP layer, they do not use start or end markers of flows or confirmatory TCP flow markers. In other words, the operation of a middlebox cannot provide the same or similar indicators of packet flow performance as a TCP flow (e.g., an end-to-end TCP connection). Without indicators of packet flow performance, network administrators or network service providers have difficulty evaluating and / or improving network communications.

[0043] 1A-1C, the network environment 100 further includes a performance evaluator 200 disposed in or coupled to the middlebox 160. The performance evaluator 200 is configured to estimate performance characteristics 202 (FIG. 1C) of the TCP connection 180 between the endpoints 190. For example, FIG. 1C illustrates the performance evaluator 200 for estimating one or more performance characteristics 202, 202a-c, of the TCP connection 180 between the client 120 and the server 110. In some examples, the performance characteristics 202 include a trip time 202a (e.g., a round trip time, RTT), a throughput 202b, and an estimated window size 202c. As shown in FIG. 1C, the performance evaluator 200 estimates the performance characteristics 202 by sampling packets 170 from the TCP connection 180. For example, in FIG. 1C, the performance evaluator 200 estimates the performance characteristics 202 by sampling packets 170 from the TCP connection 180. s Two packets moving to 170 xs2,6 and source endpoint 190 s Two packets 170 travelling from the sx3,4 and destination endpoint 190 d A packet 170 travels from the dx1and from the middlebox 160 to the destination endpoint 190 d One packet moving to 170 xd4 This indicates that performance evaluator 200 is disposed in middlebox 160 and configured to monitor packets 170 and / or packet activity, such that performance evaluator 200 is able to monitor each segment of bidirectional TCP connection 180 between endpoints 190.

[0044] In some implementations, once the TCP connection 180 is established, each endpoint 190 (e.g., source endpoint 190 s or destination endpoint 190 d ) transfers bytes of data as packets 170. Each endpoint 190 can transfer data over a TCP connection 180 and therefore has its own timeline (e.g., shown below each endpoint in FIG. 1D ) for the sequence numbers 172 of the packets 170 that are sent. In other words, the source endpoint 190 s starts at initial sequence number (ISN) 172 and reaches source endpoint 190 s From destination endpoint 190 d The destination endpoint 190 increments the initial sequence number (ISN) 172 for each byte of data sent to the d (e.g. source endpoint 190 s Starting with an initial sequence number (ISN) of 172 (which is likely to be different from the ISN of the d From source endpoint 190 s Each sequence number 172 is incremented by one for each byte of data sent to the source endpoint 190. s Bytes transferred from destination endpoint 190 d The sequence numbers of packets 170 forwarded from are managed independently of each other so that the sequence numbers of packets 170 forwarded from are not incremented (or vice versa).

[0045] Referring to FIG. 1D, source endpoint 190 s The ISN 172 of packet 170 sd1 (shown in Figure 1) starts with sequence number 172, which is equal to 1024, and d The ISN 172 of packet 170 sd1 The source endpoint 190 starts with sequence number 172, which is equal to 4000 (shown in s is a 32-byte data Destination endpoint 190 d , it increments its sequence number of 1024 to 1056 by 32 bytes (e.g., as shown in the first packet 170 sd1 2nd packet 170 from sd2 On the other hand, even if the source endpoint is 190 over TCP connection 180, s 32 bytes are sent from destination endpoint 190 d will wait for the destination endpoint 190 until it has sent its own data (i.e., bytes). d For example, the destination endpoint 190 maintains the sequence number 172 at 4000. d sends 328 bytes to source endpoint 190 s (e.g., as shown in the first packet 170, ds1 2nd packet 170 from ds2 Meanwhile, the source endpoint 190 s maintains its sequence number 172 at 1056. The incremented sequence number 172 is taken as the sequence number 172 of subsequent packets 170 sent by the endpoint 190. For example, the source endpoint 190 s A first packet 170 containing 32 bytes of data sent by sd1 has ISN 172 equal to 1024 and source endpoint 190 s The second packet 170 sent by sd2 The source endpoint is 190 sThe first packet 170 previously sent by sd1 The destination endpoint 190 has a sequence number 172 incremented to 1056 by 32 bytes of data in d similarly increments the sequence number 172 of each packet 170 (e.g., the first packet 170 ds1 and the second packet 170 ds2 ). The payload 176 of a packet 170 generally refers to the bytes contained in the packet 170. For example, s The first packet 170 sent by sd1 contains a 32-byte payload 176 and is forwarded to a destination endpoint 190 d The first packet 170 sent by ds1 contains a 328 byte payload 186. In some instances, the TCP packet 170 has a maximum number of bytes that can be packaged for TCP. In other instances, the connection between the endpoints 190 itself further limits the maximum number of bytes of the payload 176 due to limitations / constraints imposed by the connection 180.

[0046] 1C and 1D, in addition to the sequence number 172, the packet 170 may include an acknowledgement (ACK) number 174. The ACK number 174 in the packet 170 indicates that the sender of the packet 170 has received the sequence number 172 (e.g., from the opposite endpoint 190) prior to the ACK number 174 and is ready to receive the next sequence number 172. For example, although there may be some delay between the endpoints 190 (e.g., delays in other intermediate hops or in the TCP connection 180), the sender of the packet 170 may receive the next sequence number 172 prior to the ACK number 174. d Source endpoint 190 s First packet of 32 bytes from 170 sd1 If a message is received, the destination endpoint is 190. d From source endpoint 190 s Next packet sent to 170 ds1 destination endpoint 190 d Source endpoint 190s to indicate the next expected byte you want to receive from the source endpoint 190 s First packet 170 from sd1 The destination endpoint 190 acknowledges receipt of the 32 bytes by incrementing the ACK number 174 by one byte and including the ACK number 174 of the next sequence number 172 (i.e., sequence number 172 of 1024). d The first packet 170 contains a sequence number 172 equal to 1024. sd1 When receiving 32 bytes from destination endpoint 190 d The next packet sent from 170 ds1 The ACK number 174 of the destination endpoint 190 is equal to 1025. d has received sequence number 1024 and wants to receive the next one, 1025).

[0047] 2A to 2D, the performance evaluation device 200 includes a sampling device 210 and an estimation device 220. The sampling device 210 is connected to an endpoint 190 (e.g., a source endpoint 190 s and destination endpoint 190 d 1C is configured to sample a number of packets 170 from a TCP connection 180 between the , a packet 170 sampled by the sampling device 210 and provided to the performance evaluator 200 is shown in a dotted box. By sampling packets from the packet stream of a TCP connection 180, the performance evaluator 200 can reduce the burden (e.g., processor and / or resource costs) on the middlebox 160 or the TCP connection 180. For example, the performance evaluator 200 can evaluate each packet 170, all packets 170, or a majority of packets 170 that pass through the middlebox 160 for a particular TCP connection 180. This approach is accurate, but places a heavy burden on the resources of the middlebox 160 (e.g., by building a complete TCP proxy to measure each flow (i.e., connection 180) in the middlebox 160). In other words, this approach recreates the TCP endpoints of each flow in the middlebox 160. In addition to the burden on the resources of the middlebox 160, this approach also introduces additional latency to the client 120, depending on the window size of the proxy. The additional delay results in a diminished user experience, especially if the TCP connection 180 carries media transmissions (e.g., audio, video, or both in real-time communication). Such sampling becomes increasingly important as the number of clients and / or hosts increases and the middlebox 160 correspondingly increases network flows and connections that require more programming resources to program and / or manage the operation of the middlebox 160, especially as the middlebox 160 performs more functions with stateful tracking (e.g., connection tables). For example, the number of packets passing through the middlebox 160 may reach millions. In this case, the middlebox 160, as an intermediate device, may add delays, bottlenecks, and additional hops to the network functions between the endpoints 190.

[0048] In some examples, the sampling device 210 generates, for each sampled packet 170, a timestamp 212 of when it samples each packet 170 from the middlebox 160. In other words, the timestamp 212 corresponds to the time t at which the packet 170 arrives at the middlebox 160. In some implementations, the sampling device 210 records the sequence number 172 and / or acknowledgment number 174 of each packet 170 along with the timestamp 212 of each packet 170. The sampling device 210 may also be configured to record and / or store other fields or attributes of each packet 170 (e.g., packet header information, payload size, flags, and checksum). If the sampled packet 170 does not include an attribute recorded by the sampling device 210, the sampling device 210 may flag the packet 170 and / or continue to process the packet 170 (e.g., record other packet attributes and / or generate a timestamp 212). For example, the packet 170 does not include the acknowledgment number 174. The sampling device 210 may be configured to record and / or store the sequence numbers 172, acknowledgment numbers 174, timestamps 212, and any other packet attributes of the sampled packets 170. Additionally or alternatively, the sampling device 210 generates a plot 214 or trend line of each sequence number 172 and / or each acknowledgment number 174 recorded with a corresponding timestamp 212. In other words, the sampling device 210 may plot these numbers 172, 174 against time. By generating the plot 214 or trend line, the sampling device 210 may streamline the estimation of the performance characteristics 202 by the performance evaluation device 200 or allow various network entities (e.g., network users and network administrators) to view the performance of the TCP connection 180 over time. As shown in FIG. 2A, the sampling device 210 may plot the recorded numbers 172, 174 on the same plot 214 or on separate plots (not shown).FIG. 2A illustrates that the performance evaluator 200 generates a plot 214 to estimate one or more performance characteristics 202 . 2 shows a sampling device 210 having a plot 214 of numbers 172, 174 versus time t, which illustrates the generating and / or recording functions of the sampling device 210 without requiring the sampling device 210 to do so.

[0049] With particular reference to FIG. 2A, the sampling device 210 is receiving (i.e., sampling) two packets 1703 and 1709. At this time, the sampling device 210 generates a first timestamp 2123 for the first sampled packet 1703 and a second timestamp 2129 for the second sampled packet 1709. In FIG. 2A, the sampling device 210 generates a plot 214 as one possible means for recording and / or storing the numbers 172, 174 associated with the two packets 1703 and 1709. For example, the sequence number 172 and the acknowledgement number 174 are shown as points on respective trend lines. In some examples, the trend lines represent an interpolation of known data points (e.g., the recorded numbers 172, 174 and the timestamp 212). Interpolation is often used in data sampling to generate a function (e.g., the slope of the trend line in FIG. 2A) that represents the sampled data points. This function can then be used to approximate or estimate the value of unknown data points (e.g., unsampled data points). In some examples, based on interpolation of information (e.g., packet attributes) from sampled packets 170, performance evaluator 200 (e.g., estimator 220) estimates performance characteristics 202. As an example, performance evaluator 200 uses trend lines of sequence numbers 174 to estimate that packet 1706 between two packets 1703 and 1709 is likely sampled at a timestamp 212 associated with the midpoint between sequence numbers 172 of the two packets 1703 and 1709.

[0050] In some implementations, the sampler 210 sparsely samples a number of packets 170 from the TCP connection 180. Sparsely sampling, rather than evaluating all packets 170, ensures minimal resources of the middlebox 160. In a general sense, sparsely sampling packets 170 from the TCP connection 180 means sampling fewer than all packets 170 in a packet stream communicated over the TCP connection 180. In some examples, sparsely sampling means not sampling a majority of packets 170 in a packet stream communicated over the TCP connection 180. In some implementations, the sampler 210 is configured to sparsely sample packets 170 of the TCP connection 180 by sampling between one and ten percent (1%-10%) of the packets 170 of the TCP connection 180. In other implementations, the sampler 210 is configured to decimate and sample the packets 170 of the TCP connection 180 by sampling less than 1 percent (e.g., 0.1%) of the packets 170 of the TCP connection 180. Regardless of the sparseness of the sampling, the sampling may be performed with a systematic sampling frequency. The sampling may be performed randomly or uniformly (e.g., systematically at several different intervals). The sampling by the sampler 210 may be performed in a stateless manner. For example, the sampler 210 does not consider the operating state and / or characteristics of the TCP connection 180 during sampling.

[0051] In some configurations, the estimator 220 estimates the number of packets that are sent to the source endpoint 190 communicating through the middlebox 160 over a predetermined period of time based on at least two of the timestamp 212, the sequence number 172, or the acknowledgment number 174 corresponding to one or more packets of the sampled plurality of packets 170. s and destination endpoint 190 dThe function of the estimator 220 is to generate an estimated performance characteristic 202 of the TCP connection 180 between the The time may vary depending on the type of performance characteristic 202. FIG. 2B shows an estimator 220 for determining a performance characteristic 202 corresponding to an RTT 202a (or trip time 222). FIG. 2C shows an estimator 2220 for determining a performance characteristic 202 corresponding to a throughput 202b. FIGS. 2D and 2E show the time for determining the performance characteristic 202 corresponding to the destination endpoint 190. d and source endpoint 190 s 2 shows an estimator 220 for determining a performance characteristic 202 corresponding to each of the estimated window sizes 202c.

[0052] 2B, in some examples, the estimator 220 analyzes segments of a TCP connection 180 between endpoints 190 to estimate a round trip time (RTT) 202a. In other words, the RTT 202a is determined by: (i) the source endpoint 190 s and (ii) the first trip time 222, 222a (e.g., the device / client side trip time) of the TCP connection segment between the endpoint 190 and the middlebox 160; and d and the middlebox 160. To accurately estimate the RTT 202a, the estimator 220 can calculate the sum of the first trip time 222a and the second trip time 222b (e.g., the server / service-side trip time). While possibly not precise, the estimator 220 can generate an estimate of the RTT 202a between the endpoints 190 communicating through the middlebox 160 by estimating the RTT 202a based on the trip time 222 of either TCP connection segment and doubling this trip time 222.

[0053] In some examples, for a trip time 222 (e.g., a first trip time 222a or a second trip time 222b), the estimator 220 identifies sequence numbers 172 from sampled packets 170 forwarded by a TCP connection 180 in a particular TCP connection segment. For example, FIG. 2B illustrates the sequence numbers 172 from the source endpoint 190 with the subscript “sx / xs.” s At least one packet 170 sampled from a TCP connection segment between the sx / xs The subscript "dx / xd" indicates the destination endpoint 190 d At least one packet 170 sampled from a TCP connection segment between the dx / xd The time stamp 212 generated by the sampling device 210 is shown. sx / xs , timestamp 212 dx / xd and associated numbers 172, 174, the estimation device 220 may determine (e.g., s The performance evaluation device 200 determines the time at which it receives an acknowledgment number 174 matching the sequence number 172 from a TCP connection segment (between the performance evaluation device 200 and the middlebox 160). In other words, the time difference between the time at which the sequence number 172 is communicated to the TCP connection segment (e.g., the timestamp 212 when the performance evaluation device 200 records the sequence number 172) and the time at which it confirms an acknowledgment number 174 identifying the sequence number 172 in the same TCP connection segment (e.g., the timestamp 212 when the performance evaluation device records the acknowledgment number 174 matching the sequence number 172) corresponds to the trip time 222 of the TCP connection segment. This holds true for all TCP connection segments.

[0054] The estimator 220 may match either the known sequence number 172 or the estimated sequence number 172 recorded by the sampler 210 based on an interpolation of two or more known sequence numbers 172 recorded by the sampler 210. Because the sampler 210 has not sampled all packets 170 communicated over the TCP connection 180 between the endpoints 190, the estimator 220 may not receive a recorded acknowledgement number 174 that matches the sequence number 172. If the recorded acknowledgement number 174 does not match the sequence number 172 identified by the estimator 220 and used in determining the trip time 222, the estimator 220 may match the sequence number 172 recorded in the packet 170 sampled by the sampler 210. The estimator 220 is configured to estimate the time at which a corresponding acknowledgment number 174 matches an identified sequence number 172 based on an interpolation of the matched acknowledgment numbers 174. For example, FIG. 2B uses white dots on the trend line, rather than black dots on the trend line, to indicate that the acknowledgment numbers 174 that match the recorded sequence numbers 172 from the sampling device 210 are unknown. The estimator 220 determines the time at which each of the acknowledgment numbers 174 matches the sequence number 172 to determine both a trip time 222a and a trip time 222b. The first trip time 222a is shown as the difference between the estimated matching confirmation time t3 and the timestamp 212, t1 corresponding to the recorded sequence number 172. Similarly, the second trip time 222a is shown as the difference between the estimated matching confirmation time t4 and the timestamp 212, t2 corresponding to the recorded sequence number 172. In some configurations, the estimator 220 assumes that the sampling unit 210 (e.g., due to infrequent or sparse sampling) has not recorded a matching acknowledgment number 174. In other words, the estimator 220 may not be configured to attempt to identify a recorded acknowledgment number 174 that matches the sequence number 172.

[0055] Referring to FIG. 2C, the estimation device 220 may estimate the number of packets in either direction of the TCP connection 180 between the endpoints 190 (e.g., the source endpoint 190 s From destination endpoint 190 d To, or destination endpoint 190 d From source endpoint 190 s In either direction, the estimator 220 determines a slope "m" of the acknowledgment number 174 over a given period of time t based on the recorded acknowledgment numbers 174. Because the acknowledgment numbers 174 use sequence numbers 172 to verify the data sequence received from the packets 170, the slope m of the acknowledgment number 174 corresponds to the rate at which the packets 170 are being or were forwarded to the endpoint 190 communicating the acknowledgment number 174. In other words, the slope m of the acknowledgment number 174 corresponds to the rate at which the packets 170 are being or were forwarded to the endpoint 190 communicating the acknowledgment number 174. d When the source endpoint 190 communicates the acknowledgement number 174, the gradient m is s sends packet 170 to destination endpoint 190 d The rate at which the source endpoint 190 is transmitting data to the s When the acknowledgment numbers 174 are being communicated, the gradient m of these acknowledgment numbers 174 is d sends packet 170 to source endpoint 190 s This corresponds to the speed at which data is being transferred.

[0056] In some examples, as shown in FIG. 2D, each endpoint 190 of a TCP connection 180 sends a set of packets 170 (called a window) to the other endpoint 190 instead of sending one packet 170 at a time and waiting for a response (e.g., an acknowledgment packet 170). Each side has its own window. Each endpoint 190 can minimize acknowledgment delays by acknowledging a packet 170 or a window of packets 170. Because the size of the window can affect the TCP connection 180, the performance evaluation device 200 is configured to generate an estimated window size 202c for each endpoint 190 to optimize communication between the endpoints 190. In some examples, the estimation device 220 generates the estimated window size 202c by selecting packets 170 sampled by the sampling device 210. The estimation device 220 determines the difference between the sequence number 172 and the acknowledgment number 174 (e.g., the recorded acknowledgment number 174 or the estimated acknowledgment number 174) recorded at the same timestamp 212 based on the sampled packets 170. For example, in FIG. 2D (e.g., destination endpoint 190 d The acknowledgment number 174 associated with the packet 170 sent by the destination endpoint 190 d When communicated by the estimator 220, the window size 202c estimated by the estimator 220 is d Conversely, in FIG. 2E, for example, s The acknowledgment number 174 associated with the packet 170 sent by the source endpoint 190 s The window size estimated by the estimation unit 220 when 202c is the source endpoint 190 s2D shows a block around the numbers 172, 174 at the same time as the estimated window size 202c. In some implementations, the sampling device 210 is configured to sample packets 170 of each TCP flow direction (e.g., a pair of packets at the same time stamp 212) each time the sampling device 210 samples a TCP packet flow. Thus, the estimation device 220 generates the estimated window size 202c more efficiently. In other implementations, the sampling device 210 samples only one packet 170 in any TCP flow direction. Therefore, the estimator 220 generates the estimated window size 202 c by interpolating the sampled packets 170 .

[0057] 2D and 2E show different plots 214 that the sampler 210 and / or estimator 220 can generate to determine the estimated window size 202c. In FIG. 2D, the sampler 210 and / or estimator 220 may generate a plot 214 (e.g., the plot 214 shown at 172 on the trend line in FIG. 2D) based on the estimated window size 202c. s (shown as source endpoint 190) s (e.g., reference number 174 on the trend line of FIG. 2D ) d Destination endpoint 190 d In FIG. 2E , the sampling device 210 and / or the estimator 220 generate a plot 214 showing the acknowledgement numbers 174 from the source endpoints 190 (e.g., shown as reference number 174s on the trend line in FIG. 2E ). s2E) together with a receipt confirmation number 174 from the destination endpoint 190 d In both plots 214, the estimation window size 202c is equal to the distance between a point on the trend line for sequence number 172 and a point on the trend line for acknowledgment number 174 at the same time t.

[0058] 3 is a flow chart illustrating an example arrangement of operations for a method 300 for estimating TCP performance characteristics 202. At operation 302, the method 300 begins by estimating a source endpoint 190 communicating through a middlebox 160 over a predetermined period of time t. s and destination endpoint 190 d At operation 304, the method 400 performs operations 304a and 304b for each packet 170 of the sampled plurality of packets 170. At operation 304a, for each sampled packet 170, the method 300 generates a timestamp 212 for when each packet 170 was sampled. At operation 304b, for each sampled packet 170, the method 300 records a sequence number 172 and an acknowledgment number 174 for each packet 170. Each packet 170 may not have an acknowledgment packet 174. At operation 306, the method 300 determines whether a source endpoint 190 communicating through the middlebox 160 for a predetermined period of time t is associated with a packet 170 that is not associated with a source endpoint 190 communicating through the middlebox 160 for a predetermined period of time t based on at least two of the timestamp 212, the sequence number 172, or the acknowledgment number 174 of one or more packets of the sampled plurality of packets 170. s and destination endpoint 190 d The method generates estimated performance characteristics 202 of the TCP connection 180 between the two.

[0059] 4 is a flow chart illustrating an example arrangement of operations for a method 400 for estimating the trip time of a TCP connection 180. At operation 402, the method 400 culls and samples a plurality of packets 170 at a middlebox 160 from a TCP connection 180 between endpoints 190 communicating through the middlebox 160 over a predetermined period of time t. At operation 404, each sampled packet 170 of the sampled plurality of packets 170 is For the sampled packet 170, the method 400 generates a timestamp 212 when the packet 170 was sampled. At operation 406, the method 400 determines that a first sequence number 172 of a first sampled packet 170 of the plurality of sampled packets 170 forwarded between the middlebox 160 and the corresponding endpoint 190 does not match any of the acknowledgment numbers 174 of the sampled packets 170. At operation 408, the method 400 estimates a time t corresponding to the acknowledgment number 174 that matches the first sequence number 172 based on an interpolation of the plurality of acknowledgment numbers 174 of the sampled packets 170. At operation 410, the method 400 determines a trip time between the middlebox 160 and the corresponding endpoint 190 based on the difference between the first timestamp 212 associated with the first packet 170 and the estimated time.

[0060] 5 is a schematic diagram illustrating an exemplary computing device 500 that may be used to implement the systems (e.g., performance evaluation device 200) and methods (e.g., methods 300 and 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants (PDAs), servers, blade servers, mainframes, and other suitable computers. The illustrated elements, their connections and relationships, and their functions are merely exemplary and are not intended to limit the practice of the inventions described and / or claimed herein.

[0061] Computing device 500 includes a processor (e.g., data processing hardware) 510, a memory (e.g., memory hardware) 520, a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 550 and storage device 530. Elements 510, 520, 530, 540, 550, and 560 are interconnected via various buses, may be implemented on a common motherboard, or may be implemented in any other manner as appropriate. Processor 510 can display graphical information on a graphical user interface (GUI) of an external input / output device, for example, on a display 580 connected to high-speed interface 540, by processing instructions executed within computing device 500, including instructions stored in memory 520 or storage device 530. In other embodiments, multiple processors and / or multiple buses can be used, along with multiple memories and multiple types of memories, as needed. Also, multiple computing devices 500 may be connected together such that each device performs a portion of the required operations (eg, as a server bank, a collection of blade servers, or a multi-processor system).

[0062] The memory 520 stores information non-transiently in the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device for temporarily or permanently storing a program (e.g., a sequence of instructions) or data (e.g., program state information) used by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (usually used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0063] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a series of devices including a floppy disk drive, a hard disk drive, an optical disk drive, a tape disk drive, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In further implementations, the computer program product can be tangibly embodied in an information carrier. Also, the computer program product can include instructions. These instructions, when executed, can perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as, for example, the memory 520, the storage device 530, or a memory on the processor 510.

[0064] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the low-speed bandwidth-intensive operations. Such an allocation of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 550 into which various expansion cards (not shown) can be inserted. In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as, for example, a keyboard, a pointing device, a scanner, or may be coupled to a networking device, such as a switch or a router, via a network adapter.

[0065] As shown, computing device 500 may be implemented in a number of different forms. For example, computing device 500 may be implemented as a standard server 500a or multiple times in a cluster of standard servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0066] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system. The programmable system includes at least one programmable special purpose or general purpose processor coupled to a storage system and configured to receive data and instructions from the storage system and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0067] A computer program (also known as a program, software, software application or code) includes machine instructions for a programmable processor and may be implemented in a high-level procedural programming language and / or a high-level object-oriented programming language and / or may be implemented in an assembly / machine language. The terms "machine-readable medium" and "computer-readable medium" as used herein refer to any computer program product, non-transitory computer-readable medium, machine and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) including a machine-readable medium that receives machine instructions as a machine-readable signal used to provide machine instructions and / or data to a programmable processor. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0068] The processes and logic flows described herein may be executed by one or more programmable processors executing one or more computer programs to perform functions by processing input data and generating output. The processes and logic flows may also be executed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, general purpose microprocessors, special purpose microprocessors, and one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The essential elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operably coupled to one or more mass storage devices to receive data from the one or more mass storage devices, transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. Computer readable media suitable for the storage of computer program instructions and data include all non-volatile memories, media, and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0069] To provide for interaction with a user, one or more aspects of the present disclosure may be implemented on a computer that includes a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user. The feedback provided to the user may be, for example, any sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. The input received from the user may include acoustic input, speech input, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device being used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0070] Although several implementations have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. 1. A computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform the following operations: Sampling a number of packets from a Transport Control Protocol (TCP) connection between a source endpoint and a destination endpoint communicating through a middlebox over a predetermined period of time; recording an acknowledgment number for each packet of the sampled plurality of packets; determining a slope of the trend line of the acknowledgment number; generating estimated performance characteristics of the TCP connection between the source endpoint and the destination endpoint communicating through the middlebox over the predetermined period of time, the estimated performance characteristics including a throughput of the TCP connection based on a slope of the acknowledgment numbers; The method, wherein the operations further include estimating timestamps of unsampled packets from the TCP connection by interpolating the acknowledgment numbers recorded for the sampled packets.

2. The operations further include determining a trip time between the middlebox and the destination endpoint based on a difference between a timestamp generated for a first packet of the sampled plurality of packets and a timestamp generated for a second packet of the sampled plurality of packets; the first packet having a first sequence number; The method of claim 1 , wherein the acknowledgment number of the second packet matches the first sequence number.

3. The unsampled packet is associated with an acknowledgment number that matches a first sequence number of a first packet of the sampled plurality of packets; 2. The method of claim 1 , wherein the operations further include determining a trip time between the middlebox and the destination endpoint based on a difference between a first timestamp generated for the first packet and the timestamp estimated for the unsampled packet.

4. Generating the estimated performance characteristic of the TCP connection comprises:

4. The method of claim 1, further comprising calculating a round-trip time between the source endpoint and the destination endpoint through the middlebox by summing a trip time between the middlebox and the destination endpoint and a trip time between the middlebox and the source endpoint.

5. 2. The method of claim 1, wherein the operations further comprise estimating timestamps of unsampled packets from the TCP connection by interpolating sequence numbers recorded for the sampled packets.

6. Determining the slope of the trend line of the acknowledgment number over time includes: determining the slope of the trend line of acknowledgment numbers based on the acknowledgment numbers recorded for the sampled packets forwarded from the destination endpoint to the source endpoint through the middlebox; and generating the throughput based on the slope of the trend line of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the source endpoint to the destination endpoint through the middlebox.

7. Determining the slope of the trend line of the acknowledgment number over time includes: determining the slope of the trend line of acknowledgment numbers based on the acknowledgment numbers recorded for the sampled packets forwarded from the source endpoint to the destination endpoint through the middlebox; and and generating the throughput based on the slope of the trend line of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the destination endpoint to the source endpoint through the middlebox.

8. The operation includes, at a given time, receiving a first packet communicated from the source endpoint over the TCP connection, the first packet including a first sequence number and a first acknowledgment number; receiving a second packet communicated from the destination endpoint via the TCP connection, the second packet including a second sequence number and a second acknowledgment number; 8. The method of claim 1, further comprising estimating a window size of the estimated performance characteristic of the TCP connection of the source endpoint by determining a difference between the first sequence number associated with the first packet and the second acknowledgment number associated with the second packet.

9. 9. The method of claim 8, estimating the window size of the estimated performance characteristic of the TCP connection at the destination endpoint by determining a difference between the second sequence number associated with the second packet and the first acknowledgment number associated with the first packet.

10. 1. A system comprising: Data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform the following operations: The operation includes: Sampling a number of packets from a Transport Control Protocol (TCP) connection between a source endpoint and a destination endpoint communicating through a middlebox over a predetermined period of time; recording an acknowledgment number for each packet of the sampled plurality of packets; determining a slope of the trend line of the acknowledgment number; generating estimated performance characteristics of the TCP connection between the source endpoint and the destination endpoint communicating through the middlebox over the predetermined period of time, the estimated performance characteristics including a throughput of the TCP connection based on a slope of the acknowledgment numbers; The operations further include estimating timestamps of unsampled packets from the TCP connection by interpolating the acknowledgment numbers recorded for the sampled packets.

11. The operations further include determining a trip time between the middlebox and the destination endpoint based on a difference between a timestamp generated for a first packet of the sampled plurality of packets and a timestamp generated for a second packet of the sampled plurality of packets; the first packet having a first sequence number; The system of claim 10 , wherein the acknowledgment number of the second packet matches the first sequence number.

12. The method of claim 11, wherein the unsampled packet is associated with an acknowledgment number that matches a first sequence number of a first packet of the sampled plurality of packets; 11. The system of claim 10, wherein the operations further include determining a trip time between the middlebox and the destination endpoint based on a difference between a first timestamp generated for the first packet and the timestamp estimated for the unsampled packet.

13. Generating the estimated performance characteristic of the TCP connection comprises:

13. The system of claim 10, further comprising: calculating a round-trip time between the source endpoint and the destination endpoint through the middlebox by summing a trip time between the middlebox and the destination endpoint and a trip time between the middlebox and the source endpoint.

14. 11. The system of claim 10, wherein the operations further comprise estimating timestamps of unsampled packets from the TCP connection by interpolating sequence numbers recorded for the sampled packets.

15. Determining the slope of the trend line of the acknowledgment number over time includes: determining the slope of the trend line of acknowledgment numbers based on the acknowledgment numbers recorded for the sampled packets forwarded from the destination endpoint to the source endpoint through the middlebox; and generating the throughput based on the slope of the trend line of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the source endpoint to the destination endpoint through the middlebox.

16. Determining the slope of the trend line of the acknowledgment number over time includes: determining the slope of the trend line of acknowledgment numbers based on the acknowledgment numbers recorded for the sampled packets forwarded from the destination endpoint to the source endpoint through the middlebox; and generating the throughput based on the slope of the trend line of the acknowledgment numbers, the throughput corresponding to a rate of forwarding packets from the source endpoint to the destination endpoint through the middlebox.

17. The operation includes, at a given time, receiving a first packet communicated from the source endpoint over the TCP connection, the first packet including a first sequence number and a first acknowledgment number; receiving a second packet communicated from the destination endpoint via the TCP connection, the second packet including a second sequence number and a second acknowledgment number; 17. The system of claim 10, further comprising estimating a window size of the estimated performance characteristic of the TCP connection of the source endpoint by determining a difference between the first sequence number associated with the first packet and the second acknowledgment number associated with the second packet.

18. 20. The system of claim 17, further comprising: estimating the window size of the estimated performance characteristic of the TCP connection at the destination endpoint by determining a difference between the second sequence number associated with the second packet and the first acknowledgment number associated with the first packet.

Images