Network latency estimation in distributed computing systems

Abstract

Techniques for network delay estimation in a computer network are disclosed herein. One example technique includes instructing a first node in the computer network to perform trace route operations with a second node along a first round-trip route and a second round-trip route between the first node and the second node, respectively. The first round-trip route includes an inbound network path and an outbound network path opposite the inbound network path of an existing round-trip route between the first node and the second node. The second round-trip route has the outbound network path of the existing round-trip route and an inbound network path opposite the outbound network path. The example technique also includes determining, upon receiving trace route information from additional trace route operations, a delay difference between the inbound network path and the outbound network path of the existing round-trip route based on the received additional trace route information.

Description

Background Technology

[0001] Distributed computing systems typically include routers, switches, bridges, and other types of network devices that interconnect a large number of servers, network storage devices, or other computing devices. Each server can host one or more virtual machines (“VMs”), containers, virtual switches, or other virtual functions. VMs or containers can facilitate the execution of appropriate applications for individual users to provide them with the desired cloud services or other suitable computing services. Summary of the Invention

[0002] This synopsis is provided to introduce a set of concepts in a simplified form, which will be further described in the detailed embodiments below. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] Data centers or other large-scale distributed computing systems can provide computing resources, such as virtual machines, containers, or applications, as computing services accessible to users via computer networks. These computing services can also communicate with each other via computer networks to exchange data. For example, a source virtual machine in an online marketplace (e.g., a stock exchange, currency exchange, commercial exchange, online auction platform, etc.) can run applications to provide computing services that monitor online transactions / bids and distribute price data to multiple users subscribed to the information source. Thus, the source virtual machine can be configured to transmit price data via computer networks to various other virtual machines or containers within the distributed computing system. These other virtual machines, containers, or applications can then present the price data transmitted from the information source to users for consumption and action.

[0004] Latency estimation can be used to facilitate various aspects of operation in such data centers or large-scale distributed computing systems. For example, clocks running on various virtual machines, containers, or servers can be periodically synchronized to allow them to coordinate their operation. During clock synchronization in a data center, packets of synchronization messages can be transmitted from the master node (e.g., the master clock) to the secondary nodes, with timestamps offset by an estimated delay between the master and secondary nodes.

[0005] Typically, the delay offset can be estimated by dividing the round-trip time (RTT) between the primary and secondary nodes by 2. RTT is usually measured as follows:

[0006] RTT=(t4-t1)-(t3-t2)

[0007] Where t1 is the timestamp of the request packet transmission from the master node; t2 is the timestamp of the request packet reception at the secondary node; t3 is the timestamp of the response packet transmission from the secondary node; and t4 is the timestamp of the response packet reception at the master node. Therefore, assuming there is no clock rate difference between the master and secondary nodes (i.e., the time counting intervals on the two clocks are the same or substantially similar), the clock at the secondary node can be set as follows:

[0008]

[0009] The aforementioned clock synchronization techniques assume that the one-way path delay between the master and slave nodes is half the measured RTT. However, this assumption may not be accurate in some cases. For example, when a request packet is transmitted from the master node to the slave node, the request packet may traverse the computer network along an outbound network path different from the inbound network path through which the response packet is transmitted. Therefore, the inbound network path may include many more network hops than the outbound network path, making the inbound network delay potentially twice that of the outbound network delay. Thus, assuming that the inbound and outbound network delays are equal will lead to inaccurate clock synchronization between the master and slave nodes.

[0010] Several embodiments of the disclosed technology address several of the aforementioned difficulties by implementing tracing routing techniques to allocate single-path network latency to network paths in computer networks within data centers or other suitable types of distributed computing systems. In some implementations, when measuring the RTT between a master node and a secondary node, a latency controller can be configured to cause the master and secondary nodes to trace a specific connection between them that corresponds to the measured RTT of the network path. In one example, the latency controller can be configured to cause tracing routing packets to be transmitted between the master and secondary nodes, for example, by executing a tracing routing command on the master and / or secondary node. Tracing routing commands can be implemented to identify and store inbound and outbound network paths for round-trip routes between the master and secondary nodes. The inbound and outbound paths can be identified (e.g., IP addresses, MAC addresses, etc.) by one or more network identifiers of network devices (e.g., switches, routers, etc.) in the underlying network between the master and secondary nodes. In other implementations, the master and secondary nodes can be programmed to perform tracing routing operations automatically, periodically, or in other suitable ways. In other examples, the delay controller can be configured to cause the primary and secondary nodes to perform operations similar to tracing routes without executing external commands such as tracing route commands.

[0011] In some embodiments, the master and / or secondary nodes may be virtual machines, containers, or other suitable types of virtualization components or other suitable types of hosts on a server. To facilitate tracing routing operations, the host may be configured to expose such functionality to the virtual components via software components in the operating system, hypervisor, or network device driver. For example, the hypervisor may be configured to provide an application programming interface (API) that allows virtual machines or containers to request tracing routing operations, as described below. In other examples, the operating system may provide such functionality to the virtual machine or container as part of a virtual network interface card (NIC).

[0012] Generally, the trace route command is an example of a network diagnostic function that can be used to identify the network path that packets take from a source to a destination in a computer network. The trace route function can list, for example, intermediate network devices or "hops" between a source (e.g., a primary node) and a destination (e.g., a secondary node), which packets pass through before reaching their destination or failing to reach it and being dropped at the intermediate network devices.

[0013] Tracing routes typically operates by having each network device along the network path return a message (such as an Internet Control Message Protocol (ICMP) message) to the source (e.g., the master node) that invoked the tracing route function. Packets typically contain a Time-to-Live (TTL) value, which specifies the number of hops a packet can traverse in the computer network before it expires. Each time a packet traverses a network device, its TTL value is decremented by a preset amount (e.g., 1). When a packet's TTL expires at an intermediate network device (e.g., the TTL value reaches 0), the network device discards the packet and transmits a "TTL Timeout" message to the source that invoked the tracing route function.

[0014] Tracing routing typically operates over Internet Protocol (IP) networks using IP datagrams. In some embodiments, a source can use tracing routing to transmit a series of ICMP echo request packets with different TTL values ​​to identify intermediate network devices. For example, a source can invoke tracing routing to initially send a first packet with a TTL value of 1 from the source to the destination. A first network device along the network path decrements the TTL of the first packet to 0 and discards it because the TTL has expired. The first network device then returns a "TTL Timeout" message to the source. Based on the returned "TTL Timeout" message, the source can identify the first network device as the first hop on the network path to the destination. The source can then send a second packet with a TTL value of 2 to the destination. Upon receiving the second packet, the first network device along the network path decrements the TTL of the second packet to 1 and forwards the second packet to a second network device along the network path. Upon receiving the second packet, the second network device along the network path decrements the TTL of the second packet to 0 and discards it because the TTL has expired. The second network device then returns a "TTL Timeout" message to the source. Based on the "TTL timeout" message returned from the second network device, the source can identify the second network device as the second hop on the network path from the source to the destination.

[0015] The source can then send additional packets with TTL values ​​of 3, 4, 5, etc., to the destination, causing each subsequent network device along the network path to return a "TTL Timeout" message. Therefore, based on the "TTL Timeout" messages from subsequent network devices, the source can discover intermediate hops in the network path (referred to as the "outbound network path") until it reaches the destination or reaches the maximum TTL value and the tracing route operation ends. The destination can also invoke the same tracing route function to discover intermediate network devices from the destination to the source (referred to as the "inbound network path"). As discussed in more detail below, packets with the same or substantially similar 5-tuples (e.g., source IP address, destination IP address, source port, destination port, and transport protocol) can be used to traverse the same intermediate network devices or connections as the original packet flow along the network path. The source and destination can then report the network route with both the inbound and outbound network paths between the source and destination to the latency controller.

[0016] When obtaining trace routing data for network routing, the delay controller can be configured to determine whether the inbound and outbound network paths are generally symmetrical or approximately symmetrical. As used herein, the term "symmetrical network path" generally refers to a network path having the same intermediate network devices arranged in the same sequence as each other. Therefore, inbound packets will traverse the computer network along the same network devices, in the opposite direction to outbound packets. In response to determining that the inbound and outbound network paths are symmetrical, the delay controller can be configured to accordingly indicate and set the delay offset to half of the measured RTT between the source and destination (e.g., primary and secondary nodes).

[0017] On the other hand, in response to the asymmetry in determining the inbound and outbound network paths, the delay controller can be configured to indicate that the assumption that inbound and outbound delays are equal is inaccurate. In some implementations, the delay controller can also be configured to cause the primary and secondary nodes to perform tracing routing operations and RTT measurements for symmetric round-trip routes constructed based on the inbound and outbound network paths of the asymmetric network paths. For example, a first symmetric round-trip route can be constructed using an outbound network path and a new inbound network path that is the opposite of the outbound network path. A second symmetric round-trip route can be constructed using an inbound network path and a new outbound network path that is the opposite of the inbound network path.

[0018] By manipulating the 5-tuple via source routing, packets can be guided to follow two constructed symmetric round-trip routes. Based on this source route, while measuring the RTT, the source specifies the network route the packet takes through the computer network (or via other appropriate means). By comparing the measured RTTs of the two symmetric round-trip routes, the delay controller can be configured to subtract the delay difference (θ) between the inbound and outbound network paths of the original network route between the source and destination, as follows:

[0019]

[0020] Where RTT1 is the RTT of the first symmetric round-trip route, and RTT2 is the RTT of the second symmetric round-trip route. In some embodiments, with a delay difference (θ) between the asymmetric inbound and outbound network paths, the delay controller can then adjust the delay offset of the clock synchronization packets as follows:

[0021]

[0022] Where RTT is the round-trip time between the master node and the secondary node; t1 is the clock time on the master node; and t2 is the clock time on the secondary node. In other embodiments, the delay controller can be configured to indicate that one or more symmetric network paths are available in the computer network between the master and secondary nodes by changing the 5-tuple. The delay controller can then utilize the discovered symmetric network routes to perform delay estimation using the additional RTT measured with the changed 5-tuple.

[0023] In other implementations, instead of constructing and routing packets following two constructed first and second symmetric round-trip routes, the delay controller can be configured to continuously monitor trace routing data from primary, secondary, or other nodes in the computer network to find network routes (or portions of routes) that at least approximately match (e.g., >90%, >95%, >99%, 100%, or other suitable thresholds) the first and second symmetric round-trip routes. Such trace routing data can then be used to estimate the delay difference between the asymmetric inbound and outbound network paths, as described above. In a further implementation, the delay difference can be estimated using trace routing data from only one of the first or second symmetric round-trip routes by subtracting the RTT of either the first or second symmetric round-trip route from the RTT of the asymmetric round-trip route between the primary and secondary nodes, or vice versa.

[0024] Therefore, several embodiments of the disclosed technology can accommodate the differences between the inbound and outbound network paths of the round-trip routes between a master node and a secondary node. By constructing and measuring the RTT of two new round-trip routes using asymmetric inbound and outbound network paths, the delay difference between the inbound and outbound network paths of the original asymmetric round-trip routes can be estimated or measured. Therefore, the delay estimation between inbound and outbound network traffic between a master node and a secondary node may be more accurate than assuming that the delays of both inbound and outbound network traffic are the same.

[0025] Several embodiments of the disclosed techniques may be useful when using datasets from a large number of network devices to determine the latency of links between pairs of network devices in a computer network. For example, the tracing routing results of a symmetric path and the corresponding RTT can be a projection of the latency in the computer network. By obtaining many such projections, the latency of each link can be determined using tomographic reconstruction or other suitable techniques.

[0026] Additional embodiments of the disclosed technology also involve presenting the operating temperature and / or ambient temperature of one or more hosts hosting master and secondary nodes to a delay controller. Without being bound by theory, it is assumed that the timing of the digital clock in a computer server (e.g., clock skew, timing interval, etc.) is affected by temperature variations at the computer server. The correlation between the timing of the clock on the host and the operating / ambient temperature can be developed using historical measurements or other suitable methods. Therefore, using this correlation, the delay controller (or other suitable component in the computing system) can empirically or otherwise determine one or more clock correction factors (e.g., skew, frequency, timing, etc.) based on the current temperature of the computer server hosting master and secondary nodes and / or one or more of their components. This can further improve the accuracy of clock synchronization. The information required for such clock correction can be obtained using dedicated API calls to master and / or secondary nodes and other nodes in a distributed computing system, similar to the methods described above. The delay controller can also be configured to determine the temperature of one or more components (e.g., CPU) on the node to further improve the estimation of drift experienced by the clock source due to thermal factors. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating a distributed computing system for network latency estimation according to an embodiment of the disclosed technology.

[0028] Figure 2 This illustrates an embodiment based on the disclosed technology. Figure 1 A schematic diagram of some hardware / software components of a distributed computing system.

[0029] Figure 3A and Figure 3B This is a schematic diagram illustrating some example operations of network latency estimation in a distributed computing system according to embodiments of the disclosed technology.

[0030] Figure 4A and Figure 4B This is a schematic diagram illustrating some example operations of implementing network latency estimation according to embodiments of the disclosed technology.

[0031] Figure 5A and Figure 5B This is a schematic diagram illustrating some example operations of network latency estimation in a distributed computing system according to embodiments of the disclosed technology.

[0032] Figure 6A and Figure 6B This illustrates an embodiment based on the disclosed technology. Figure 1 A schematic block diagram of an example component of the delay controller.

[0033] Figure 7Aand Figure 7B This is a flowchart illustrating certain processes for network latency estimation according to embodiments of the disclosed technology.

[0034] Figure 8 Is suitable Figure 1 The computing devices of certain components in a distributed computing system. Detailed Implementation

[0035] The following describes certain embodiments of systems, devices, components, modules, routines, data structures, and processes for network latency estimation in data centers or other suitable distributed computing systems. Specific details of the components are included in the following description to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the art will also understand that the technology may have additional embodiments. The technology may also be practiced without the following references. Figure 1 Several details of the embodiment described in Figure 6.

[0036] As used herein, the term "distributed computing system" generally refers to an interconnected computer system with multiple network nodes that interconnect multiple servers or hosts to each other and / or to an external network (e.g., the Internet). The term "network node" generally refers to a physical network device. Example network nodes include routers, switches, hubs, bridges, load balancers, security gateways, or firewalls. "Host" generally refers to a physical computing device. In some embodiments, a host may be configured to implement, for example, one or more virtual machines, virtual switches, or other suitable virtualization components. For example, a host may include a server having a hypervisor configured to support one or more virtual machines, virtual switches, or other suitable types of virtual components. In other embodiments, a host may be configured to execute appropriate applications directly on top of an operating system.

[0037] In some implementations, computer networks can be conceptually divided into overlay networks implemented on an underlying network. An "overlay network" generally refers to an abstract network implemented and operating on top of the underlying network. The underlying network may include multiple physical network devices interconnected with each other. An overlay network may include one or more virtual networks. A "virtual network" generally refers to an abstraction of a portion of the underlying network within the overlay network. A virtual network may include one or more virtual endpoints, referred to as "tenant sites," which are used by users or "tenants" to access the virtual network and associated computing, storage, or other suitable resources. A tenant site may host one or more tenant endpoints ("TEPs"), such as virtual machines. Virtual networks may interconnect multiple TEPs on different hosts.

[0038] Virtual network nodes in an overlay network can be connected to each other via virtual links that correspond to one or more network routes, each corresponding to one or more physical network devices in the underlying network. In other implementations, a computer network may consist only of the underlying network. As used herein, a “network route” or “network path” generally refers to a sequence of one or more network nodes through which a packet travels from a source (e.g., a first host) to a destination (e.g., a second host). A “round-trip” network path generally refers to a pair of inbound and outbound network paths between a source and a destination. In some examples, the inbound and outbound network paths may be symmetrical, for example, having the same sequence of intermediate network nodes in opposite directions. In other examples, the inbound and outbound network paths may be asymmetrical, for example, having different sequences and / or intermediate network nodes in opposite directions.

[0039] As used herein, a “packet” generally refers to a formatted unit of data carried by a packet-switched network. Packets typically include both user data and control data. Control data provides information for transmitting user data. For example, control data may include source and destination network addresses / ports, error check codes, sorting information, hop count, priority information, security information, or other appropriate information about the user data. Typically, control data may be included in the packet header and / or trailer. The header and trailer may include one or more data fields containing appropriate information.

[0040] Figure 1 This is a schematic diagram illustrating a distributed computing system 100 that implements network latency estimation according to an embodiment of the disclosed technology. Figure 1 As shown, the distributed computing system 100 may include an underlying network 108 that interconnects multiple hosts 106, multiple client devices 102 associated with corresponding users 101, and a delay controller 125 operatively coupled to each other. Although Figure 1 The diagram illustrates specific components of the distributed computing system 100, but in other embodiments, the distributed computing system 100 may also include additional and / or different components or arrangements. For example, in some embodiments, the distributed computing system 100 may also include network storage devices, servers, and / or other suitable components in appropriate configurations.

[0041] like Figure 1As shown, the underlying network 108 may include one or more network nodes 112 interconnecting multiple hosts 106 and client devices 102 of user 101. In some embodiments, hosts 106 may be organized into racks, action zones, groups, sets, or other suitable partitions. For example, in the illustrated embodiment, hosts 106 are grouped into three clusters identified as first, second, and third clusters 107a to 107c, respectively. Each cluster 107a to 107c is operatively coupled to a corresponding network node 112a to 112c, which are collectively referred to as “top-of-rack” network nodes or “TORs”. TORs 112a to 112c may then be operatively coupled to additional network nodes 112 to form a computer network in a hierarchical, flat, mesh, or other suitable type of topology. The underlying network may allow communication between hosts 106, delay controller 125, and client devices 102 of user 101. In other embodiments, multiple clusters 107a to 107c may share a single network node 112, or may have other suitable arrangements.

[0042] Host 106 can be configured to provide computing, storage, and / or other cloud or other suitable types of computing services to user 101. For example, see the following reference... Figure 2 In more detail, one of the hosts 106 can start and maintain one or more virtual machines 144 upon request from user 101. Figure 2 (shown in the diagram) or a container (not shown). User 101 can then utilize the provided virtual machine 144 or container to perform database, computation, communication, and / or other suitable tasks. In some embodiments, one of the hosts 106 can provide virtual machines 144 to multiple users 101. For example, host 106a can host three virtual machines 144 corresponding to each of users 101a-101c, respectively. In other embodiments, multiple hosts 106 can host virtual machines 144 for users 101a through 101c.

[0043] Client devices 102 may each include a computing device that facilitates user 101's access to computing services provided by host 106 via underlying network 108. In the illustrated embodiment, client devices 102 each include a desktop computer. In other embodiments, client devices 102 may also include laptop computers, tablet computers, smartphones, or other suitable computing devices. Although for illustrative purposes... Figure 1 The illustration shows three users 101a to 101c, but in other embodiments, the distributed computing system 100 can facilitate access to cloud or other suitable types of computing services provided by the distributed computing system 100 by any appropriate number of users 101.

[0044] The delay controller 125 can be configured to manage and control network latency estimation between endpoints (e.g., virtual machines) in the distributed computing system 100. For example, the delay controller 125 can be configured to cause various endpoints in the distributed computing system 100 to perform a tracing route operation and identify network paths with packets traversing one or more network nodes 112 in the underlying network 100. The delay controller 125 can also be configured to determine whether the tracing route operation indicates that the round-trip route between a pair of endpoints includes a symmetric inbound and outbound network path or an asymmetric inbound and outbound network path. In response to determining that the round-trip route includes an asymmetric inbound and outbound network path, the delay controller 125 can also be configured to construct a new symmetric round-trip route pair using each of the inbound and outbound network paths from the asymmetric inbound and outbound network paths. By measuring the RTT of the constructed symmetric round-trip route, the delay controller can be configured to determine the latency difference between the inbound and outbound network paths. The delay controller can then adjust the delay offset between the pairs of endpoints accordingly to improve the consistency of clock synchronization in the distributed computing system 100, as shown in the following reference. Figures 3A to 5B A more detailed description.

[0045] Despite delay controller 125 in Figure 1 While represented as a single entity, in some implementations, the delay controller 125 can be implemented in a distributed manner. For example, one or more portions of the logic of the delay controller 125 can be executed distributed across one or more hosts in the host 106. For instance, each host 106 may include instructions whose execution causes a first host 106a to independently trace a network path routed to a second host 106b. The first host 106a can then report the discovered network path to the second host 106b. The first host 106a and the second host 106b can each determine whether the discovered path is symmetric and adjust the delay estimate accordingly.

[0046] Figure 2 This is a schematic diagram illustrating certain hardware / software components of a distributed computing system 100 according to an embodiment of the disclosed technology. Figure 2 It shows that it can be used Figure 1 The overlay network 108' is implemented on the underlying network 108. Although Figure 2 The diagram illustrates a specific configuration of overlay network 108', but in other embodiments, overlay network 108' may also be configured in other suitable ways. Figure 2 For clarity, only [the text] is shown. Figure 1 Some components of the underlying network 108.

[0047] exist Figure 2As illustrated in the other accompanying figures, the various software components, objects, classes, modules, and routines can be computer programs, programs, or processes written in source code in C, C++, C#, Java, and / or other suitable programming languages. Components can include, but are not limited to, one or more modules, objects, classes, routines, properties, procedures, threads, executables, libraries, or other components. Components can be in source code form or binary form. Components can include aspects of pre-compiled source code (e.g., classes, properties, processes, routines), compiled binary units (e.g., libraries, executables), or artifacts instantiated and used at runtime (e.g., objects, processes, threads).

[0048] Components within a system can take different forms. As an example, a system comprising a first component, a second component, and a third component could include a system where the first component is an attribute in the source code, the second component is a binary compiler library, and the third component is a thread created at runtime, without limitation. Computer programs, programs, or processes can be compiled into object code, intermediate code, or machine code and rendered for execution by one or more processors of a personal computer, network server, laptop computer, smartphone, and / or other suitable computing device.

[0049] Similarly, components may include hardware circuitry. Those skilled in the art will recognize that hardware can be considered as rigid software, and software as fluidized hardware. By way of example only, software instructions in a component may be programmed into a programmable logic array circuit or may be designed as hardware circuitry with appropriate integrated circuits. Likewise, hardware can be emulated by software. Various implementations of source, intermediate, and / or object code and associated data may be stored in computer memory, including read-only memory, random access memory, disk storage media, optical storage media, flash memory devices, and / or other suitable computer-readable storage media other than those for propagating signals.

[0050] like Figure 2 As shown, source host 106a and destination hosts 106b and 106b' (destination host 106b is shown only in detailed components) may each include a processor 132, a memory 134, a network interface card 136, and a packet processor 138 operatively coupled to each other. In other embodiments, host 106 may also include an input / output device configured to accept input from and provide output to an operator and / or automation software controller (not shown) or other suitable type of hardware component.

[0051] Processor 132 may include a microprocessor, cache, and / or other suitable logic devices. Memory 134 may include volatile and / or non-volatile media (e.g., ROM; RAM, disk storage media; optical storage media; flash memory devices, and / or other suitable storage media) and / or be configured to store data received from processor 132 and instructions for processor 132 (e.g., instructions for executing the following references) Figures 3A to 5B Other types of computer-readable storage media (the instructions for the methods discussed). Although in Figure 2 Only one processor 132 and one memory 134 are shown in each host 106, but in other embodiments, each host 106 may include two, six, eight or any other suitable number of processors 132 and / or memory 134.

[0052] Source host 106a and destination host 106b may each contain instructions executable by processor 132 in memory 134 to enable each processor 132 to provide a hypervisor 140 (identified as first hypervisor 140a and second hypervisor 140b, respectively) and an operating system 141 (identified as first operating system 141a and second operating system 141b, respectively). Even though hypervisor 140 and operating system 141 are shown as separate components, in other embodiments, hypervisor 140 may operate on operating system 141 executed on host 106 or on firmware components of host 106.

[0053] The hypervisor 140 can be configured to create, monitor, terminate, and / or otherwise manage one or more virtual machines 144 organized into tenant site 142. For example, as Figure 2 As shown, source host 106a can provide a first management program 140a to manage the first and second tenant sites 142a and 142b respectively. Destination host 106b can provide a second management program 140b to manage the first and second tenant sites 142a' and 142b' respectively. Management program 140b... Figure 2 These are shown as software components. However, in other embodiments, the hypervisor 140 may be firmware and / or hardware components. Tenant sites 142 may each include multiple virtual machines 144 for a specific tenant (not shown). For example, both source host 106a and destination host 106b may host tenant sites 142a and 142a' of the first tenant 101a. Figure 1 Both source host 106a and destination host 106b can host tenant sites 142b and 142b' of the second tenant 101b. Figure 1 Each virtual machine 144 can execute the corresponding operating system, middleware, and / or application.

[0054] Also Figure 2 As shown, the distributed computing system 100 may include an overlay network 108' having one or more virtual networks 146 interconnecting tenant sites 142a and 142b across multiple hosts 106. For example, a first virtual network 142a interconnects first tenant sites 142a and 142a' at source host 106a with destination host 106b. A second virtual network 146b interconnects second tenant sites 142b and 142b' at source host 106a with destination host 106b. Even though a single virtual network 146 is shown to correspond to one tenant site 142, in other embodiments, multiple virtual networks 146 (not shown) may be configured to correspond to a single tenant site 146.

[0055] Virtual machine 144 can be configured to execute one or more applications 147 to provide appropriate cloud or other suitable type of computing services to user 101. Figure 1 For example, source host 106a can execute application 147 configured to provide a computing service that monitors online transactions and distributes price data to multiple users 101 who have subscribed to the computing service. Even if virtual machine 144 resides on a different host 106, virtual machine 144 on virtual network 146 can also be accessed via the underlying network 108. Figure 1 They communicate with each other.

[0056] Communication within each virtual network 146 can be isolated from other virtual networks 146. In some embodiments, communication may be allowed to traverse from one virtual network 146 to another via a security gateway or in other controlled manner. A virtual network address may correspond to a virtual machine 144 within a specific virtual network 146. Therefore, different virtual networks 146 may use the same one or more virtual network addresses. Example virtual network addresses may include IP addresses, MAC addresses, and / or other appropriate addresses. To facilitate communication between virtual machines 144, a virtual switch (not shown) may be configured to switch or filter packets 114 directed to different virtual machines 144 via network interface card 136 and facilitated by packet processor 138.

[0057] like Figure 2 As shown, to facilitate communication between themselves or with external devices, each host 106 may also include features for communicating with a computer network (e.g., Figure 1The underlying network 108) interfaces with a network interface card (NIC) 136. NIC 136 may include a network adapter, LAN adapter, physical network interface, or other suitable hardware circuitry and / or firmware to enable communication between hosts 106 via a network medium (e.g., fiber optic) that transmits / receives data (e.g., as packets) according to Ethernet, Fibre Channel, Wi-Fi, or other suitable physical and / or data link layer standards. During operation, NIC 136 may facilitate communication to / from suitable software components executing on host 106. Example software components may include a virtual switch 141, a virtual machine 144, an application 147 executing on virtual machine 144, a hypervisor 140, or other suitable types of components.

[0058] In some implementations, the packet processor 138 may be interconnected to and / or integrated with the NIC 136 to facilitate network traffic operations for implementing communication security, performing network virtualization, translating network addresses, maintaining / limiting communication flow states, or performing other appropriate functions. In some implementations, the packet processor 138 may include a field-programmable gate array (“FPGA”) integrated with the NIC 136.

[0059] An FPGA may include an array of logic circuits and a hierarchical structure of reconfigurable interconnects, allowing a user to “connect” the logic circuits together like logic gates after manufacturing. Thus, user 101 can configure logic blocks in the FPGA to perform complex combinational functions, or simply simple logic operations to synthesize equivalent functions that can be executed in hardware at a much faster speed than in software. In the illustrated embodiment, packet processor 138 has one interface communicatively coupled to NIC 136, and another interface coupled to a network switch (e.g., a top-of-rack switch or “TOR” switch). In other embodiments, packet processor 138 may also include application-specific integrated circuits (“ASICs”), microprocessors, or other suitable hardware circuitry. In any of the foregoing embodiments, packet processor 138 may be programmed by processor 132 (or suitable software components associated therewith) to route packets within packet processor 138 to implement various aspects of time-sensitive data delivery, as referenced below. Figures 3A to 5B A more detailed description.

[0060] During operation, processor 132 and / or user 101 ( Figure 1The logic circuitry in packet processor 138 can be configured to perform complex combinational functions or simple logical operations to synthesize equivalent functions that can be executed in hardware at a much faster speed than in software. For example, packet processor 138 can be configured to process inbound / outbound packets of individual flows according to configuration policies or rules contained in flow tables (e.g., MAT). Flow tables can contain data representing processing actions corresponding to each flow to enable private virtual networks with customer-provided address spaces, scalable load balancers, security groups and access control lists (“ACLs”), virtual routing tables, bandwidth metering, quality of service (“QoS”), etc.

[0061] Therefore, once packet processor 138 identifies an inbound / outbound packet as belonging to a specific flow, packet processor 138 can apply one or more corresponding policies from the flow table before forwarding the processed packet to NIC 136 or TOR 112. For example, as Figure 2 As shown, application 147, virtual machine 144, and / or other suitable software components on source host 106a can generate outbound packets 114 destined for other applications 147 at destination hosts 106b and 106b'. NIC 136 at source host 106a can forward the generated packets 114 to packet processor 138 for processing according to certain policies in the flow table. Once processed, packet processor 138 can forward the outbound packets 114 to a first TOR 112a, which in turn forwards the packets to a second TOR 112b via overlay / underlying networks 108 and 108'.

[0062] Then, the second ToR 112b can forward packet 114 to packet processor 138 at destination hosts 106b and 106b' for processing according to other policies in another flow table at destination hosts 106b and 106b'. If packet processor 138 cannot identify the packet as belonging to any flow, packet processor 138 can forward the packet to processor 132 via NIC 136 for exception handling. In another example, when the first TOR 112a receives inbound packet 115 from destination host 106b, for example via the second TOR 112b, the first TOR 112a can forward packet 115 to packet processor 138 for processing according to the policy associated with the flow of packet 115. Packet processor 138 can then forward the processed packet 115 to NIC 136 for forwarding to, for example, application 147 or virtual machine 144.

[0063] The transmission of inbound packet 114 and outbound packet 115 may experience delays between the transmission and reception of these packets 114 and 115. This delay generally refers to the delay within the distributed computing system 100. Delay estimation can be used to facilitate various aspects of operation within the distributed computing system 100. For example, during a clock synchronization process, packets of synchronization messages can be transmitted from source host 106a to destination host 106b, with the timestamp offset based on the estimated delay between source host 106a and destination host 106b.

[0064] Figure 3A This is an example timing diagram illustrating source host 106a transmitting outbound packet 114 to destination host 106b along a symmetrical network path and receiving a response as inbound packet 115 from destination host 106b. (See example...) Figure 3A As shown, at time T1, source host 106a can transmit outbound packet 114 to destination host 106b. At time T2, destination host 106b receives outbound packet 114. After processing outbound packet 114 for a period of time, destination host 106b transmits a response as inbound packet 115 to source host 106a at time T3. At time T4, source host 106a receives inbound packet 115 from destination host 106b. Therefore, in the example shown, the outbound delay is equal to the difference between T2 and T1, while the inbound delay is equal to the difference between T4 and T3. The sum of the outbound delay and the inbound delay equals the RTT between source and destination hosts 106a and 106b.

[0065] exist Figure 3A In the example timing diagram shown, the inbound delay is generally equal to the outbound delay. Figure 3B This is a schematic diagram illustrating a scenario where the arrival and departure delays are roughly equal. For example... Figure 3B As shown, the underlying network 108 includes a computer network with network nodes, such as switches 1 to 4 (112a to 112d), interconnecting the source host 106a and the destination host 106b respectively. During operation, outbound packets 114 can traverse the underlying network 108 via switches 1 (112a) and 4 (112d) before reaching the destination host 106b. Therefore, the outbound network path 117a is as follows:

[0066] Source host 106a → Switch 1 → Switch 4 → Destination host 106b. Similarly, inbound packet 114 can traverse the underlying network 108 via Switch 4 112d and Switch 1 112a before reaching source host 106a to form the following inbound network path 117b:

[0067] Destination host 106b → Switch 4 → Switch 1 → Source host 106a. Therefore, the inbound and outbound network paths 117a and 117b include the same sequence of identical network nodes 112 (i.e., switches 1 and 4) in opposite directions. The example inbound and outbound network paths 117a and 117b can be described as symmetrical.

[0068] Typically, delay controller 125 ( Figure 1 The delay offset can be estimated by dividing the round-trip time (RTT) between the source host 106a and the destination host 106b by 2. RTT is typically measured as follows:

[0069] RTT=(t4-t1)-(t3-t2)

[0070] Where t1 is the timestamp of the request packet transmission from source host 106a; t2 is the timestamp of the request packet reception at destination host 106b; t3 is the timestamp of the response packet transmission from destination host 106b; and t4 is the timestamp of the response packet reception at source host 106a. Therefore, assuming there is no clock rate offset or "drift" between source 106a and destination host 106b, the clock at destination host 106b can be set as follows:

[0071]

[0072] The aforementioned clock synchronization technique assumes that the one-way path delay between source host 106a and destination host 106b is half of the measured RTT. However, this assumption may not be accurate in some situations. For example, when outbound packet 114 is transmitted from source host 106a to destination host 106b, outbound packet 114 may traverse the underlying network 108 along an outbound network path different from the inbound network path used to transmit inbound packet 115. Figure 4A This is an example timing diagram illustrating source host 106a transmitting outbound packet 114 to destination host 106b along an asymmetric network path and receiving a response as inbound packet 115 from destination host 106b. (See example...) Figure 4A As shown, the inbound delay (i.e., the difference between T4 and T3) may be much greater than the inbound delay (i.e., the difference between T2 and T1).

[0073] This difference may be due to different network nodes 112 ( Figure 1 This is caused by the inbound network path of the sequence of network node 112 and / or network node 112. Figure 4B This is a schematic diagram illustrating a scenario where the inbound delay may be greater than the outbound delay. For example... Figure 4B As shown, outbound network path 117a and Figure 3B The same as shown, but the inbound network path 117b' is... Figure 3B The above is different. Instead, inbound network path 117b' includes the following:

[0074] Destination host 106b → Switch 4 → Switch 3 → Switch 2 → Switch 1 → Source host 106a

[0075] Therefore, the inbound network path 117b' is much longer than the outbound network path 117a, which may result in a much larger inbound latency than outbound latency. Therefore, the assumption that inbound and outbound network latency are equal will lead to inaccurate clock synchronization between the source host 106a and the destination 106b.

[0076] Several embodiments of the disclosed technology can address several of the aforementioned difficulties by implementing tracing routing techniques to allocate single-path network latency to either inbound network path 117a or outbound network path 117b'. In some implementations, when measuring the RTT between source host 106a and destination host 106b, the latency controller 125 ( Figure 1 The source host 106a and destination host 106b can be configured to issue tracing route commands between the source host 106a and destination host 106b, respectively. The tracing route commands can identify and store the inbound and outbound network paths 117a and 117b' of the round-trip route between the source host 106a and destination host 106b. The inbound and outbound paths 117a and 117b' can be defined by the underlying network 108 between the source host 106a and destination host 106b. Figure 1 The source host 106a and destination host 106b are identified by one or more network identifiers (e.g., IP address, MAC address, etc.) of network devices (e.g., switches, routers, etc.). In other implementations, the source host 106a and destination host 106b may be programmed to perform tracing routing operations automatically, periodically, or in other appropriate ways.

[0077] Generally, the trace route command is an example of a network diagnostic function that can be used to identify the network path taken by a packet from its source to its destination. The trace route function typically lists, for example, the intermediate network nodes 112 ( ) that the packet traverses between the source and destination. Figure 1The tracing route function operates by having each network node 112 along the network path return a message such as an ICMP message to the source (e.g., source host 106a) that invoked the tracing route function. Packets typically contain a Time-to-Live (TTL) value, which specifies how many hops a packet can traverse before expiring. Each time a packet traverses a network device, its TTL value is decremented by a preset amount (e.g., 1). When a packet's TTL value expires at an intermediate network device (e.g., the TTL value reaches 0), the network device discards the packet and transmits a "TTL Timeout" message to the source that invoked the tracing route function.

[0078] Tracing routing typically operates over Internet Protocol (IP) networks using IP datagrams. In some embodiments, a source can use tracing routing to send a series of ICMP echo request packets with different TTL values ​​to identify intermediate network nodes. For example, a source can invoke tracing routing to initially send a first packet with a TTL value of 1 from the source to the destination. A first network device along the network path decrements the TTL of the first packet to 0 and discards it because the TTL has expired. The first network device then returns a "TTL Timeout" message to the source. Based on the returned "TTL Timeout" message, the source can identify the first network device as the first hop on the network path to the destination. The source can then send a second packet with a TTL value of 2 to the destination. Upon receiving the second packet, the first network device along the network path decrements the TTL of the second packet to 1 and forwards the second packet to a second network device in the network path. Upon receiving the second packet, the second network device along the network path decrements the TTL of the second packet to 0 and discards it because the TTL has expired. The second network device then returns a "TTL Timeout" message to the source. Based on the "TTL timeout" message returned from the second network device, the source can identify the second network device as the second hop on the network path from the source to the destination.

[0079] The source can then send additional packets with TTL values ​​of 3, 4, 5, etc., to the destination, causing each subsequent network device along the network path to return a "TTL Timeout" message. Therefore, based on the "TTL Timeout" messages from subsequent network devices, the source can discover intermediate hops in the network path (referred to as the "outbound network path") until it reaches the destination or reaches the maximum TTL value and the tracing route operation ends. The destination can also invoke the same tracing route function to discover intermediate network devices from the destination to the source (referred to as the "inbound network path"). As discussed in more detail below, packets with the same or substantially similar 5-tuples (e.g., source IP address, destination IP address, source port, destination port, and transport protocol) can be used to traverse the same intermediate network devices or connections as the original packet flow along the network path. The source and destination can then report the network route with both the inbound and outbound network paths between the source and destination to the delay controller.

[0080] When obtaining network routing data, delay controller 125 can be configured to determine whether inbound network path 117a and outbound network path 117b are symmetrical. In response to determining that inbound network path 117a and outbound network path 117b are symmetrical (e.g., ...), the delay controller 125 can be configured to determine whether inbound network path 117a and outbound network path 117b are symmetrical. Figure 3B As shown), the delay controller 125 can be configured to indicate the delay offset and set it to half of the RTT measured between the source host 106a and the destination host 106b. The determination of the inbound network path 117a and the outbound network path 117b' is asymmetric (e.g., Figure 4B As shown, the delay controller 125 can be configured to indicate that the assumption that inbound and outbound delays are equal is inaccurate.

[0081] The delay controller 125 can also be configured to enable the source host 106a and the destination host 106b to interact with each other based on... Figure 4B The inbound network path 117a and outbound network path 117b' of the asymmetric network path shown are used to construct additional symmetric round-trip routes to perform additional tracing route operations and RTT measurements. For example, as Figure 5A As shown, a first symmetric round-trip route 121a can be constructed using an outbound network path 117a and a new inbound network path 117b, which is the opposite of the outbound network path 117a. Figure 5B As shown, a second symmetric round-trip route 121b can be constructed using an inbound network path 117b' and a new outbound network path 117a' that is the opposite of the inbound network path 117b'.

[0082] When measuring RTT, packets can be guided to follow the first and second symmetric round-trip routes by manipulating the packet's 5-tuple or through other appropriate means. This can be achieved by comparing the measured RTT (in...) Figure 5A and5B (As shown in the diagram as "RTT path AA" and "RTT path B'-B'"), the delay controller can be configured to subtract the delay difference (θ) between the inbound network path 117a and the outbound network path 117b' as follows:

[0083]

[0084] Wherein, RTT1 is the RTT of the first symmetric round-trip route RTT path AA, and RTT2 is the RTT of the second symmetric round-trip route RTT path B'-B'. Given the delay difference between the inbound network path and the outbound network paths 117a and 117b', the delay controller 125 can then adjust the delay offset of the synchronization packets as follows:

[0085]

[0086] Where RTT is the round-trip time between source host 106a and destination host 106b; t1 is the clock time at source host 106a; and t2 is the clock time at destination host 106b.

[0087] In other implementations, instead of constructing and routing packets following two constructed first and second symmetric round-trip routes 121a and 121b, the delay controller 125 can be configured to continue monitoring source host 106a, destination host 106b, or other host 106 in the computer network 108. Figure 1 and 2 Tracing route data (as shown) is used to find network routes (or partial routes) that at least approximately match (e.g., >90%, >95%, >99%, 100%, or other suitable thresholds) the first symmetric round-trip route 121a and the second symmetric round-trip route 121b. Such tracing route data can then be used to estimate the delay difference between the asymmetric inbound network path 117a and the outbound network path 117b', as described above. In other implementations, the delay difference can be estimated using tracing route data from only one of the symmetric round-trip routes 121a or 121b by subtracting the RTT of either the first symmetric round-trip route 121a or the second symmetric round-trip route 121b from the RTT of the asymmetric round-trip route between the source host 106a and the destination host 106b, or vice versa.

[0088] Therefore, multiple embodiments of the disclosed latency estimation technique can accommodate differences between the inbound and outbound network paths 117a and 117b' of the round-trip route between source host 106a and destination host 106b. This is achieved by constructing and measuring the asymmetric inbound and outbound network paths 117a and 117b' respectively. Figure 5A and 5BThe RTT of the first symmetric round-trip route 121a and the second symmetric round-trip route 121b shown can be used to estimate or measure the delay difference between the inbound network path 117a and the outbound network path 117b'. Thus, the delay estimate between inbound and outbound network services between the source host 106a and the destination host 106b can be more accurate than assuming that the delays of both inbound and outbound network services are the same.

[0089] Figure 6A and 6B This illustrates an embodiment based on the disclosed technology. Figure 1 A schematic block diagram of an example component of the delay controller 125. (See attached diagram.) Figure 6A As shown, the delay controller 125 may include an interface component 122, an analysis component 124, and a control component 126 that are operatively coupled to each other. Although Figure 6A Specific components are shown, but in other embodiments, the delay controller 125 may include additional and / or different components.

[0090] like Figure 6A As shown, in some embodiments, interface component 122 may be configured to periodically, on demand, or otherwise appropriately transmit tracing route instructions 150 to source host 106a and destination host 106b. The tracing route instructions 150 can, for example, execute tracing route commands to cause source host 106a and destination host 106b to perform tracing route type operations. Therefore, source host 106a and destination host 106b may each transmit a series of packets 114 and 115 to list intermediate network nodes 112 in the underlying network 108, as referenced above. Figures 3A to 4B As described. Upon completion of the route tracing operation, source host 106a and destination host 106b can respectively transmit outbound path data 152a and inbound path data 152b to delay controller 125 for analysis. In other embodiments, source host 106a and destination host 106b can be configured to... Figure 1 The aforementioned tracing route operation may be automatically performed by instructions from another entity (not shown) in the ) or by other appropriate means.

[0091] Upon receiving inbound and outbound path data 152a and 152b, the analysis component 124 of the delay controller 125 can be configured to determine whether the round-trip routes between the source and destination hosts 106a and 106b are symmetrical, as referenced above. Figures 3A to 4B In more detail, when determining whether the round-trip route is symmetric or asymmetric, the analysis component 124 may instruct the control component 126 to issue a notification 154 regarding the routing symmetry of the round-trip route to the source host 106a and the destination host 106b or other appropriate entities in the distributed computing system 100.

[0092] In response to the determination that the round-trip routes between source host 106a and destination host 106b are asymmetrical, the analysis component 124 can be configured to construct two symmetrical round-trip routes using the inbound and outbound network paths from the original round-trip routes, as referenced above. Figure 5A and Figure 5B In more detail, the analysis component 124 can be configured to instruct the control component 126 to issue additional tracing route instructions 150' to the source host 106a and the destination host 106b to perform additional tracing route operations. The additional tracing route instructions 150' may also include data representing 5-tuples corresponding to each of the inbound and outbound network paths, such that the new round-trip routes are symmetric.

[0093] In response, source host 106a and destination host 106b can perform the aforementioned additional tracing routing operation by, for example, transmitting additional packets 114' and 115' to each other. Upon completion of the additional tracing routing operation, source host 106a and destination host 106b can be configured to transmit additional inbound path data 152a' and outbound path data 152b' of the new symmetric round-trip route to delay controller 125. Analysis component 124 can then be configured to analyze the additional inbound path data 152a' and outbound path data 152b' to determine the delay difference 156 between the inbound and outbound network paths, as referenced above. Figure 5A and 5B For more detailed description, interface component 125 can then be configured to provide the determined latency difference 156 to source host 106a and destination host 106b, or other appropriate entities in distributed computing system 100.

[0094] In other embodiments, the analysis component 124 may be configured to use datasets from multiple network nodes 112 to determine the latency of links between network node pairs 112 in the underlying network 108. For example, the tracing results of symmetric routes and the corresponding RTT can be a projection of the latency in the underlying network 108. By obtaining many such projections, the latency of each link can be determined using tomographic reconstruction or other suitable techniques.

[0095] In another embodiment, source host 106a and destination host 106b may also be configured to present operating temperature and / or ambient temperature to delay controller 125. Without being theoretically constrained, it is assumed that the timing of the digital clock in the computer server (e.g., clock skew, clock intervals, etc.) will be affected by temperature variations at the computer server. The correlation between the timing of the clock on the host and the operating / ambient temperature can be developed using historical measurements or other suitable methods. Therefore, using this correlation, the analysis component 124 of delay controller 125 (or other suitable components in distributed computing system 100) can empirically or otherwise determine one or more clock correction factors (e.g., skew, frequency, timing, etc.) based on the current temperature of the source or destination host 106a or 106b and / or one or more of its components. This can further improve the accuracy of clock synchronization. The information required for such clock correction can be obtained using API calls to source and destination hosts 106a and 106b, as well as other hosts 106 in distributed computing system 100. The delay controller 125 can also be configured to determine the temperature of one or more components (e.g., CPU) on the host 106 to further improve the estimation of the drift experienced by the clock source due to thermal factors.

[0096] Figure 7A and Figure 7B This is a flowchart illustrating certain processes for network latency estimation according to embodiments of the disclosed technology. Although the following... Figure 1 The process is described in the context of the distributed computing system 100, but in other embodiments, the process or aspects thereof may be implemented in a computing system with additional and / or different components.

[0097] like Figure 7A As shown, process 200 may include receiving tracing routing information for the round-trip route between the source node and the destination node at stage 202. (Refer to the above...) Figures 3A to 5B This describes various techniques for performing tracing routing operations between the source and destination nodes. Tracing routing information can identify a sequence of network nodes from the source node to the destination node.112 Figure 1 The outbound network path and the inbound network path with a network node sequence 112 from the destination node to the source node.

[0098] Then, process 200 may include a determination phase 204 to determine whether the round-trip route is symmetrical. In some embodiments, the round-trip route is symmetrical when the network node sequence of the inbound network path is the same as or substantially the same as the network node sequence of the reverse outbound network path (e.g., greater than about 95% match). In other embodiments, the round-trip route may be indicated as symmetrical when the inbound and outbound network paths match by more than 80%, 85%, 90%, or other suitable thresholds.

[0099] In response to determining that the round-trip routes are symmetrical, process 200 may include, at stage 206, assigning a single-path delay of RTT / 2 to the inbound and outbound network paths of the round-trip routes. Otherwise, process 200 may continue performing additional tracing route operations at stage 208 to infer or estimate the delay difference between the inbound and outbound network paths. See below for reference. Figure 7B Describe an example operation that performs additional tracing routing operations.

[0100] like Figure 7B As shown, an example operation could include constructing a new round-trip route pair at stage 210 using the asymmetric inbound and outbound network paths of the original round-trip route. (See above reference...) Figure 5A and Figure 5B An illustrative example of constructing such a new round-trip route is described. Operations may then include instructing the source and destination nodes to perform additional tracing routing operations along the new round-trip route in phase 212. During the execution of the additional tracing routing operations, 5-tuple packets may be manipulated to follow the new round-trip route between the source and destination nodes. Upon completion of the additional tracing routing operations, these operations may include calculating, in phase 214, the delay difference between the inbound and outbound network paths, for example, as half the difference between the RTTs of the new round-trip routes.

[0101] Figure 8 It is suitable for Figure 1 The computing device 300 is a component of a distributed computing system 100. For example, the computing device 300 may be adapted to... Figure 1 The host device 106, client device 102, or delay controller 125 are used. In a very basic configuration 302, the computing device 300 may include one or more processors 304 and system memory 306. A memory bus 308 is used for communication between the processors 304 and the system memory 306.

[0102] Depending on the desired configuration, processor 304 can be of any type, including but not limited to microprocessors (μP), microcontrollers (μC), digital signal processors (DSPs), or any combination thereof. Processor 304 may include multiple L1 caches, such as L1 cache 310 and L2 cache 312, processor core 314, and registers 316. Example processor core 314 may include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. Example memory controller 318 may also be used with processor 304, or in some implementations, memory controller 318 may be an internal part of processor 304.

[0103] Depending on the desired configuration, system memory 306 can be of any type, including but not limited to volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, flash memory, etc.), or any combination thereof. System memory 306 may include operating system 320, one or more applications 322, and program data 324. Figure 8 As shown, the operating system 320 may include a hypervisor 140 for managing one or more virtual machines 144. The basic configuration 302 described herein... Figure 8 The components shown are those within the inner dashed lines.

[0104] Computing device 300 may have additional features or functions, as well as additional interfaces that facilitate communication between basic configuration 302 and any other devices and interfaces. For example, a bus / interface controller 330 may be used to facilitate communication between basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. Data storage device 332 may be a removable storage device 336, a non-removable storage device 338, or a combination thereof. Examples of removable and non-removable storage devices include disk devices such as floppy disk drives and hard disk drives (HDDs), optical disk drives such as optical disc drives (CDs) or digital versatile disc (DVD) drives, solid-state drives (SSDs), and magnetic tape drives. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The terms “computer-readable storage medium” or “computer-readable storage device” do not include signal propagation and communication media.

[0105] System memory 306, removable storage device 336, and non-removable storage device 338 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 300. Any such computer-readable storage medium may be part of computing device 300. The term "computer-readable storage medium" does not include signal propagation and communication media.

[0106] The computing device 300 may also include an interface bus 340 for facilitating communication from various interface devices (e.g., output devices 342, peripheral interfaces 344, and communication devices 346) via a bus / interface controller 330 to the basic configuration 302. An example output device 342 includes a graphics processing unit 348 and an audio processing unit 350, which may be configured to communicate with various external devices such as displays or speakers via one or more A / V ports 352. An example peripheral interface 344 includes a serial interface controller 354 or a parallel interface controller 356, which may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 358. An example communication device 346 includes a network controller 360, which may be arranged to facilitate communication with one or more other computing devices 362 via a network communication link through one or more communication ports 364.

[0107] A network communication link can be an example of a communication medium. A communication medium can typically be implemented by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A “modulated data signal” can be a signal whose one or more characteristics are set or altered in a way that encodes information in the signal. As an example and not a limitation, a communication medium can include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, radio frequency (RF), microwave, infrared (IR), and other wireless media. The term “computer-readable medium” as used herein can include both storage media and communication media.

[0108] The computing device 300 can be implemented as part of a small portable (or mobile) electronic device that includes any of the functions described above, such as a cellular phone, a personal digital assistant (PDA), a personal media player device, a wireless network viewing device, a personal headset device, a dedicated device, or a hybrid device. The computing device 300 can also be implemented as a personal computer that includes both laptop and non-laptop computer configurations.

[0109] As will be understood from the foregoing, specific embodiments of this disclosure have been described herein for illustrative purposes, but various modifications may be made without departing from this disclosure. Furthermore, many elements of one embodiment may be combined with elements of other embodiments, supplementing or replacing elements of those embodiments. Therefore, this technology is not limited except as provided in the appended claims.

Claims

1. A method for network latency estimation in a distributed computing system, the distributed computing system having a first node and a second node, the first node and the second node being interconnected via a computer network having multiple network devices, comprising: Receive data representing tracing routing information of existing round-trip routes in the computer network between the first node and the second node, the existing round-trip routes including an outbound network path from the first node to the second node and an inbound network path from the second node to the first node; Based on the tracing route information received from the existing round-trip routes, determine whether the existing round-trip routes are symmetric or asymmetric; In response to determining that the existing round-trip route is symmetric, a delay value is assigned to both the inbound network path and the outbound network path, wherein the assigned delay value is equal to half the round-trip time between the first node and the second node; as well as In response to determining that the existing round-trip route between the first node and the second node is asymmetric Instruct the first node and the second node to perform additional tracing route operations along the first round-trip route and the second round-trip route between the first node and the second node, respectively, wherein: The first round-trip route includes the inbound network path of the existing round-trip route and an outbound network path that is opposite to the inbound network path of the existing round-trip route; and The second round-trip route has the outbound network path of the existing round-trip route and an inbound network path that is opposite to the outbound network path of the existing round-trip route; Receive additional tracing routing information from the additional tracing routing operations performed by the first node and the second node; and The delay difference between the inbound network path and the outbound network path of the existing round-trip route is determined based on the received additional tracing routing information or an approximately matching symmetric round-trip route.

2. The method according to claim 1, wherein: The additional tracing route information includes the first round-trip time RTT1 of the first round-trip route and the second round-trip time RTT2 of the second round-trip route; as well as Determining the latency difference includes: calculating the latency difference between the inbound network path and the outbound network path of the existing round-trip route between the first node and the second node as follows. θ : 。 3. The method according to claim 1, wherein: The additional tracing route information includes the first round-trip time RTT1 of the first round-trip route and the second round-trip time RTT2 of the second round-trip route; as well as Determining the latency difference includes: calculating the latency difference between the inbound network path and the outbound network path of the existing round-trip route between the first node and the second node as follows. θ : ;and The method further includes adjusting the delay offset between the clock on the first node and another clock on the second node as follows: in, RTT It is the round-trip time between the first node and the second node along the existing round-trip route; t1 It is the clock time at the first node; and t2 It is the clock time at the second node.

4. The method of claim 1, wherein indicating the first node and the second node comprises: The first node and the second node are instructed to manipulate the 5-tuple of the packet to be transmitted between the first node and the second node to follow the first round-trip route and the second round-trip route.

5. The method according to claim 1, further comprising: Receive further tracing routing information for other symmetric round-trip routes in the computer network; as well as The tomographic reconstruction is applied to the received tracing routing information to derive the latency of each link between network device pairs in the computer network.

6. The method according to claim 1, further comprising: Receive the first operating temperature and the second operating temperature at the first node and the second node, respectively; as well as The timing interval of the clock on the first node or the second node is adjusted based on the correlation between the received first operating temperature and the second operating temperature and the clock drift.

7. A computing device in a distributed computing system, the distributed computing system having a first node and a second node, the first node and the second node being interconnected via a computer network having multiple network devices, the computing device comprising: processor; as well as A memory, operatively coupled to the processor, contains instructions executable by the processor to cause the computing device to: Receive data representing tracing routing information in the computer network indicating existing round-trip routes between the first node and the second node. Based on the tracing route information received from the existing round-trip routes, determine whether the existing round-trip routes are symmetric or asymmetric; In response to determining that the existing round-trip route is symmetric, a delay value is assigned to both the inbound network path and the outbound network path, wherein the assigned delay value is equal to half the round-trip time between the first node and the second node; as well as In response to determining that the existing round-trip route is asymmetric based on the tracing route information received from the existing round-trip route, Instruct the first node and the second node to perform additional tracing route operations along the first round-trip route and the second round-trip route between the first node and the second node, respectively, wherein: The first round-trip route includes the inbound network path of the existing round-trip route between the first node and the second node and the outbound network path that is opposite to the inbound network path of the existing round-trip route; and The second round-trip route has an outbound network path of the existing round-trip route and an inbound network path that is opposite to the outbound network path of the existing round-trip route; and When receiving additional tracing routing information from the additional tracing routing operation performed by the first node and the second node, the delay difference between the inbound network path and the outbound network path of the existing round-trip route is determined based on the received additional tracing routing information or an approximately matching symmetric round-trip route.

8. The computing device according to claim 7, wherein: The additional tracing route information includes the first round-trip time RTT1 of the first round-trip route and the second round-trip time RTT2 of the second round-trip route; as well as Determining the latency difference includes: calculating the latency difference between the inbound network path and the outbound network path of the existing round-trip route between the first node and the second node as follows. θ : 。 9. The computing device according to claim 7, wherein: The additional tracing route information includes the first round-trip time RTT1 of the first round-trip route and the second round-trip time RTT2 of the second round-trip route; as well as Determining the latency difference includes: calculating the latency difference between the inbound network path and the outbound network path of the existing round-trip route between the first node and the second node as follows. θ : ;and The memory also includes additional instructions executable by the processor to cause the computing device to adjust the delay offset between the clock on the first node and another clock on the second node as follows: in, RTT It is the round-trip time between the first node and the second node along the existing round-trip route; t1 It is the clock time at the first node; and t2 It is the clock time at the second node.

10. The computing device of claim 7, wherein indicating the first node and the second node comprises: The first node and the second node are instructed to manipulate the 5-tuple of the packet to be transmitted between the first node and the second node to follow the first round-trip route and the second round-trip route.

11. A method for estimating network latency in a distributed computing system, the distributed computing system having a first node and a second node interconnected by a computer network having multiple network devices, comprising: Receive data representing tracing routing information of existing round-trip routes in the computer network between the first node and the second node, the existing round-trip routes including an outbound network path from the first node to the second node and an inbound network path from the second node to the first node; Determine whether the inbound network path is the opposite of the outbound network path of the existing round-trip route; as well as In response to determining that the inbound network path is opposite to the outbound network path of the existing round-trip route, a delay value is assigned to both the inbound network path and the outbound network path, wherein the assigned delay value is equal to half the round-trip time between the first node and the second node; as well as In response to determining that the inbound network path is not the opposite of the outbound network path of the existing round-trip route, Instruct the first node and the second node to perform additional tracing route operations along the first round-trip route and the second round-trip route between the first node and the second node, respectively, wherein: The first round-trip route includes the inbound network path of the existing round-trip route and an outbound network path that is opposite to the inbound network path of the existing round-trip route; and The second round-trip route has the outbound network path of the existing round-trip route and an inbound network path that is opposite to the outbound network path of the existing round-trip route; Receive additional tracing routing information from the additional tracing routing operations performed by the first node and the second node; and The delay difference between the inbound network path and the outbound network path of the existing round-trip route is determined based on the received additional tracing routing information or an approximately matching symmetric round-trip route.

12. The method of claim 11, wherein: Additional tracing route information includes the first round-trip time (RTT1) of the first round-trip route and the second round-trip time (RTT2) of the second round-trip route; as well as Determining the latency difference includes: calculating the latency difference between the inbound network path and the outbound network path of the existing round-trip route between the first node and the second node as follows. θ : 。 13. The method of claim 11, wherein: The additional tracing route information includes the first round-trip time RTT1 of the first round-trip route and the second round-trip time RTT2 of the second round-trip route; as well as Determining the latency difference includes: calculating the latency difference between the inbound network path and the outbound network path of the existing round-trip route between the first node and the second node as follows. θ : ;and The method further includes adjusting the delay offset between the clock on the first node and another clock on the second node as follows: in, RTT It is the round-trip time between the first node and the second node along the existing round-trip route; t1 It is the clock time at the first node; and t2 It is the clock time at the second node.

14. The method of claim 11, wherein indicating the first node and the second node comprises: The first node and the second node are instructed to manipulate the 5-tuple of the packet to be transmitted between the first node and the second node to follow the first round-trip route and the second round-trip route.

Images