Mechanism for precise time synchronization in data center networks

Abstract

The present disclosure relates to mechanisms for precise time synchronization in data center networks. Aspects of the present disclosure relate to supporting time synchronization across data center networks with higher accuracy. Time synchronization includes both software-based and hardware-based time synchronization mechanisms to provide more precise time synchronization across various nodes in a data center network. The software-based mechanisms can provide an initial coarse time synchronization, while the hardware-based mechanisms can provide a subsequent more fine-grained time synchronization.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to mechanisms for precise time synchronization in data center networks. BACKGROUND

[0002] A time synchronization protocol can involve building a tree of nodes, typically with the highest precision clock at the root of the tree. A master node can send a synchronization message at a first time T1. A local node can receive the synchronization message at a second time T1'. The local node can send a delay request at a third time T2'. The master node can receive the delay request at a fourth time T2. Assuming that the delay is a one-way delay between the nodes and the offset is the current offset between the master clock and the local clock, then T1' - T1 = delay + offset, and T2 - T2' = delay - offset. With the four timestamps T1, T1', T2, and T2', the local node can solve for the delay and the offset and adjust its local clock using the offset.

[0003] Devices that share a single clock, such as network routers, typically have a single local time domain and run a time synchronization protocol at all ports of the device. One port can be configured to synchronize the internal domain to an external master device, while the remaining ports can run as masters for downstream devices / domains. A firmware-controlled digital phase-locked loop (DPLL) can keep the time of the device, speeding up or slowing down in response to synchronization messages that arrive periodically, typically about once per second. One instance of the DPLL can run for each clock domain.

[0004] For good performance of time synchronization, it is assumed that the delay is symmetric in both directions and that delay variations, such as jitter, should be minimized. To achieve this, any type of queuing is avoided between taking a transmit timestamp, such as the first time T1, and taking a receive timestamp, such as the second time T1'. The timestamps are captured near the point of transmission / reception, such as performed in the physical layer.

[0005] Software overhead in a time synchronization protocol can result in a significant increase in delay, thereby reducing the accuracy of time synchronization. SUMMARY

[0006] Aspects of the present disclosure relate to supporting time synchronization across data center networks with higher accuracy. Time synchronization includes both software-based and hardware-based time synchronization mechanisms to provide more precise time synchronization across various nodes in a data center network. The software-based mechanism can provide an initial coarse time synchronization, while the hardware-based mechanism can provide a subsequent more fine-grained time synchronization.

[0007] One aspect of the present disclosure provides a method for synchronizing time between a master hardware clock and one or more client hardware clocks. The method includes receiving, by one or more processors of a client hardware clock, a time synchronization message; performing, by the one or more processors, a validity check on the time synchronization message; updating, by the one or more processors, a time of the client hardware clock in response to the received time synchronization message passing the validity check; incrementing, by the one or more processors, a validity update count in response to the received time synchronization message passing the validity check; and, locking, by the one or more processors, the time of the client hardware clock in response to the validity update count increasing beyond a threshold value.

[0008] In an example, the time synchronization message is sent from one or more processors of the master hardware clock or one of the upstream client hardware clocks. In another example, the method further includes sending, with the one or more processors, the time synchronization message to a downstream client hardware clock. In yet another example, updating the time of the client hardware clock further includes at least one of: adjusting the time forward or backward by a number of steps; or, adjusting a frequency of the client hardware clock. In yet another example, locking the client hardware clock further includes determining that the client hardware clock is within a threshold range from the master hardware clock.

[0009] In yet another example, the method further includes sending, by the one or more processors, a delay request message; receiving, by the one or more processors, a delay response message; and, determining, by the one or more processors, a hardware delay time based on a time difference from when the delay request message was sent to when the delay response message was received. In yet another example, updating the time of the client hardware clock further includes removing the hardware delay time from the time of the client hardware clock. In yet another example, the hardware delay time is fixed.

[0010] In yet another example, performing the validity check further includes determining that a time in the time synchronization message is within a threshold value. In yet another example, the method further includes skipping, by the one or more processors, an update to the time of the client hardware clock in response to the received time synchronization message failing the validity check. In yet another example, the method further includes incrementing, by the one or more processors, a skip update count in response to the received time synchronization message failing the validity check. In yet another example, the method further includes resetting, by the one or more processors, the client hardware clock in response to the skip update count increasing beyond a threshold value.

[0011] Another aspect of the present disclosure provides a system comprising: one or more processors; and, one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for synchronizing time between a master hardware clock and one or more client hardware clocks, the operations comprising: receiving a time synchronization message; performing a validity check on the time synchronization message; updating a time of a client hardware clock in response to the received time synchronization message passing the validity check; incrementing a valid update count in response to the received time synchronization message passing the validity check; and, locking the time of the client hardware clock in response to the valid update count increasing beyond a threshold value.

[0012] In an example, updating the time of the client hardware clock further comprises at least one of: adjusting the time forward or backward by a number of steps; or, adjusting a frequency of the client hardware clock. In another example, locking the client hardware clock further comprises determining that the client hardware clock is within a threshold range from the master hardware clock.

[0013] In yet another example, the operations further comprise: sending a delay request message; receiving a delay response message; and, determining a hardware delay time based on a time difference from when the delay request message was sent to when the delay response message was received; wherein updating the time of the client hardware clock further comprises removing the hardware delay time from the time of the client hardware clock.

[0014] In yet another example, the operations further comprise: skipping an update of the time of the client hardware clock in response to the received time synchronization message failing the validity check; incrementing a skip update count in response to the received time synchronization message failing the validity check; and, resetting the client hardware clock in response to the skip update count increasing beyond a threshold value.

[0015] Yet another aspect of the present disclosure provides a non-transitory computer- readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for synchronizing time between a master hardware clock and one or more client hardware clocks. The operations comprise: receiving a time synchronization message; performing a validity check on the time synchronization message; updating a time of a client hardware clock in response to the received time synchronization message passing the validity check; incrementing a valid update count in response to the received time synchronization message passing the validity check; and, locking the time of the client hardware clock in response to the valid update count increasing beyond a threshold value.

[0016] In an example, the operations further include: sending a delay request message; receiving a delay response message; and determining a hardware delay time based on a time difference from when the delay request message was sent to when the delay response message was received; wherein updating the time of the client hardware clock further includes removing the hardware delay time from the time of the client hardware clock.

[0017] In another example, the operations further include: in response to the received time synchronization message failing the validity check, skipping updating the time of the client hardware clock; in response to the received time synchronization message failing the validity check, incrementing a skip update count; and in response to the skip update count incrementing past a threshold, resetting the client hardware clock. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A block diagram depicting an example network environment with network devices is depicted in accordance with aspects of the present disclosure.

[0019] Figure 2 A block diagram depicting an example time synchronization of network devices is depicted in accordance with aspects of the present disclosure.

[0020] Figure 3 A block diagram depicting an example configuration for time adjustment is depicted in accordance with aspects of the present disclosure.

[0021] Figure 4 An example time synchronization message configured as a time-ordered set is depicted in accordance with aspects of the present disclosure.

[0022] Figure 5 A flow diagram of an example process for synchronizing time between a master hardware clock and one or more client hardware clocks is depicted in accordance with aspects of the present disclosure.

[0023] Figure 6 An example computing system is depicted in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0024] Generally disclosed herein are implementations for supporting precise time synchronization across a data center network. Time synchronization includes both software-based and hardware-based time synchronization mechanisms to provide more accurate time synchronization across various nodes in a data center network. Initial coarse time synchronization across a data center network can be performed by software, while more fine-grained and more accurate time synchronization can be performed by hardware circuitry.

[0025] Each network interface controller (NIC) includes a hardware clock that will be used as the global time within the NIC and its corresponding host. Each hardware clock can be set by writing a new value to a current time control status register (CSR), writing to a time adjustment CSR either forward or backward by a desired step, or receiving a valid time synchronization message that requires the clock to be adjusted forward or backward by one step. Time adjustment or time synchronization clock adjustment can be performed by running the hardware clock slower or faster by as many cycles as the step desired. Hardware clock speed adjustment can be performed by adjusting a frequency that is incremented or decremented by a configurable value right shifted by itself. The configurable value can be set actively at boot time and slowed down to smooth response when acquiring a lock.

[0026] Each hardware clock includes fine frequency adjustment, which can be implemented via the least significant bits used for fractional clock counting. By changing the configurable value, the clock frequency can be set faster or slower. An average of the previous time synchronization differences can be continuously calculated. This can enable the NIC to use fine frequency adjustment to attempt to match the frequency of the master NIC, thereby minimizing the difference seen with each time synchronization message.

[0027] Each hardware clock can maintain a status bit indicating the quality of the local current time, which corresponds to several parameters used to indicate the accuracy of each hardware clock. The time quality parameters can include a time locked parameter, a skip update parameter, and a software bit parameter. The time locked parameter can determine whether the time of the hardware clock is considered to be locked, which corresponds to being within a threshold range of the master clock. The skip update parameter can correspond to the number of skipped time synchronization messages reflected in the current time. The software bit parameter can be opaque to the hardware, such that the software can use the bits to pass additional information between nodes without involving the hardware.

[0028] The current time is distributed from the master hardware clock to the client hardware clock (and from there to a second hop client hardware clock, etc.) via time synchronization messages. The time ordered set can include a subtype, a time locked parameter, a skip update parameter, a software bit parameter, a time, and an error check.

[0029] Each NIC can be master, client, or free running. Master NICs can only transmit time synchronization messages. Master NICs can include software-based time synchronization to synchronize clocks using time adjustment and fine frequency adjustment. Client NICs can receive time synchronization messages from upstream NICs (which can be master NICs or other client NICs closer to master NICs) or send time synchronization messages to downstream NICs. Client NICs can include software-based time synchronization for initial setup of coarse time. Free running NICs are independent of master and client NICs. Free running NICs can include software-based time synchronization to synchronize clocks using time adjustment and fine frequency adjustment.

[0030] Each NIC can send time synchronization messages on any number of output links. Time synchronization messages are sent at a configured interval, which can be set by a time update timer register. On a client hardware clock, an incoming time synchronization message can cause an update to be transmitted downstream. A time synchronization message can indicate the actual time of transmission + / - a delay. The delay is accounted for by incrementing the time field in the time synchronization message for each period the time synchronization message waits to be sent.

[0031] Each NIC can receive time synchronization messages via a CSR, which can control which port, if any, will enjoy the time synchronization message. When an enabled port receives a time synchronization message, the time in the time synchronization message can be adjusted for any on-chip delays to account for the time synchronization message reaching the hardware clock. The delays can include the time to leave the far end NIC, the time through the channel, the time to enter the local NIC, and the time of flight across the NIC to the hardware clock.

[0032] Each NIC includes link delay measurement capability, taking into account variable length cables, channels, etc. A NIC can transmit a delay request message to the switch. The delay request can be generated upon chip reset or loss of time lock. The switch latches the delay request message and generates a delay response back to the NIC. As with the time synchronization message, the time field of the delay message is incremented for each period the delay response waits to be sent. The NIC can receive the delay response and subtract the received time field from the current time.

[0033] Each NIC can be initialized with the actual global time. The initial global time can be written to all NICs, and then the sending and receiving of time synchronization messages can be enabled. Time limit registers can be set for an acceptable threshold, so different NICs can lock to the master. Once all NICs are locked, the time limit registers can be made tighter.

[0034] For validity checks, time synchronization messages received at a receiving NIC can be received on a port enabled for time synchronization reception, perform validity error checks, indicate a lock threshold period of time with a local time, and have an incoming time locked. The hardware clock can internally maintain two CSR-readable counts: a skipped update count and a valid update count. Any message that fails a validity check involving the time lock can cause an increment of the skipped update count and a reset of the valid update count. When the number of skipped updates increases beyond a time-limited skipped message threshold, the master time can have changed, causing the time lock parameter to be cleared and the NIC to be interrupted. When a valid update is received, the hardware clock can move to the received time and the valid update count is incremented. The skipped update count can be reset to the incoming skipped update value. When the number of valid updates increases beyond a time-limited lock message threshold, the hardware clock can be locked. The time lock parameter is updated according to the internal skipped update and valid update counts, while the time itself is updated according to the validity of incoming time synchronization messages.

[0035] Figure 1 A block diagram depicting an example network environment 100 with a network device 110 is depicted. The network environment 100 includes a network 120 of interconnected network nodes 125. The network nodes 125 participate in the network 120 as data sources, data destinations or sinks, and / or intermediate nodes, such as switches, routers, and / or gateways, that propagate data from sources to destinations through the network 120. The network 120 includes the network device 110 having links 127 to various other participating network nodes 125.

[0036] The network 120 can facilitate interaction among the participating devices. Example networks include the Internet, a local network, a network structure, or any other local or wide area network. The network 120 can be composed of multiple connected sub-networks or autonomous networks. The network 120 can be a local area network (LAN), such as a corporate intranet, a metropolitan area network (MAN), a wide area network (WAN), the Internet, such as the Internet, or a peer-to-peer network. Any type and / or form of data and / or computer networks can be used for the network 120. The network can be public, private, or a combination of public and private. Generally, the network 120 can be used to communicate information among computing devices; for example, the network nodes 125 and the network device 110.

[0037] The network device 110 can be a host device or a server executing one or more virtual machines 130 on one or more processors of the network device 110. The network device 110 can include a network interface driver 140, a memory 150, a network interface card 160, a real operating system (OS) 170, a hypervisor 180, and virtual machines 130. The network device 110 can communicate with other network devices over a network 120. The network device 110 and other network devices can create or participate in one or more virtual networks to facilitate communication between virtual machines 130. The one or more virtual networks can overlay the physical network that makes up the network 120. A system of data packet encapsulation can be used to route data packets within one of the virtual networks that overlays the physical network. The network device 110 and network nodes 125 can route data packets according to virtual internet protocol addresses (VIPs) and MAC addresses. In some examples, the network device 110 can host one or more virtual machines 130 that send or receive data packets. In other examples, the network device 110 can be an intermediary node for a path between two virtual machines executing on two separate network devices. Thus, the network device 110 can act as a source, destination, or intermediary switch in one or more virtual networks.

[0038] The hypervisor 180 can manage the operation of the virtual machines 130, including initiating and terminating virtual machines 130, managing the allocation of memory 150 to virtual machines 130, and live migrating virtual machines 130 to other network devices.

[0039] The memory 150 can store data and / or computer-executable instructions related to the operation and use of the network interface driver 140. The memory 150 can include, by way of example, high bandwidth memory (HBM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), static random access memory (SRAM), and / or flash memory. The memory 150 can store computer-executable instructions of a transport protocol module 175, such as a transmission control protocol (TCP) module or a TCP layer of a network stack, to be executed on a processor of the network device 110. While the transport protocol module 175 can also be contained in the real OS as shown in the network interface card 160. The functionality that arises within the TCP layer of the network stack can additionally or alternatively be executed in another network protocol module within a transport layer, a network layer, or a combined transport / network layer of the network stack. For example, the functionality can be implemented in a user datagram protocol (UDP) module, a reliable datagram protocol (RDP) module, a reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module. The network layer, the transport layer, or the combined transport / network layer can generally be referred to as a packet layer of the network stack.

[0040] The network interface driver 140 can include a network interface driver software module running on the real OS 170. As an example, the network interface driver 140 can be a set of computer-executable instructions stored in the memory 150 that, when executed by the processor, facilitate facilitating network communications. As another example, the network interface driver 140 can be implemented as logic in a hardware processor or other integrated circuit, or as a combination of hardware and software logic. The network interface driver 140 can communicate with one of the software virtual machines 130 directly or via the hypervisor 180 and / or a guest OS of the virtual machine 130.

[0041] The network interface driver 140 can be included in a network stack layer of the real OS 170 of the network device 110. In an example, the network interface driver 140 can be included in a data link layer of the network stack and can communicate with software modules or applications included in an application layer of the network stack. In another example, the network interface driver 140 can be included within a link layer of the network stack and can communicate with TCP / IP modules included in an internet / transport layer of the network stack. In other examples, functionality occurring within a network stack layer can additionally or alternatively be configured to receive data packets from another network or transport layer protocol module, such as a UDP module, a RDP module, a RUDP module, or a DCCP module. Although not shown, the network interface driver 140 can be included as part of the network interface card 160.

[0042] The network interface card 160 can be configured to communicate data packets over a network protocol, a layer 2 protocol. The network interface card 160 can include a packet forwarder 162, a time synchronizer 164, and a memory 166.

[0043] The memory 166 can store data and / or computer-executable instructions related to the operation and use of the network interface card 160. For example, the memory 166 can store time synchronization messages and instructions related to validity checks. As an example, the memory 166 can include SRAM, HBM, DRAM, SDRAM, and / or flash memory.

[0044] The packet forwarder 162 can be implemented in an application-specific integrated circuit (ASIC) or as software executing on a microprocessor. The packet forwarder 162 can receive data packets for transmission from the virtual machines 130 and route them to one or more destination virtual machines executing on a remote network device. The packet forwarder 162 can also perform functions that allow the network interface card 160 to act as a destination for data packets received over the links 127. For example, the packet forwarder 162 can receive a data packet over one of the links 127, decapsulate the data packet, and forward the data packet to one or more destination virtual machines 130 executing on the network device 110. The packet forwarder 162 can send and receive data packets based on connection entries stored in the cache 166. Each transmission or reception of a data packet can correspond to a deallocation or allocation, respectively, of memory 150.

[0045] The time synchronizer 164 can be implemented in an application-specific integrated circuit (ASIC) or as a combination of hardware and software executing on a microprocessor. The time synchronizer 164 can support precise time synchronization via both software-based and hardware-based time synchronization mechanisms. Initial coarse time synchronization can be performed by software, while subsequent finer time synchronization can be performed by hardware.

[0046] Figure 2 A block diagram of an example time synchronization 200 of a network device 202 is depicted. The network device 202 can correspond to the network device 110 as depicted in Figure 1 The network device 202 includes a network interface controller (NIC) 204, which can include time synchronization software 206, time synchronization hardware 208, and a hardware clock 210. The time synchronization software 206, time synchronization hardware 208, and hardware clock 210 can correspond to the time synchronizer 164 as depicted in Figure 1 The time synchronization 200 with both time synchronization software 206 and time synchronization hardware 208 can allow for more accurate time synchronization, resulting in more efficient database replicas and saving computational resources to maintain database replicas.

[0047] The time synchronization software 206 can include one or more processors in communication with memory. The processors can be any logic circuit that executes instructions fetched from memory, including one or more central processing units (CPUs) and / or graphics processing units (GPUs). The time synchronization hardware 208 can be implemented as a hardware logic circuit, such as a combination of logic gates, registers, and / or flip-flops, using a field-programmable gate array and / or an application-specific integrated circuit (ASIC), such as a tensor processing unit (TPU).

[0048] Hardware clock 210 can be used as a global time within NIC 204 and its corresponding network device 202. Hardware clock 210 can count in core clock cycles, such as 900 MHz, 1.111 ns per cycle. Hardware clock 210 can correspond to a 62-bit clock that rolls over in about 180 years.

[0049] NIC 204 can set hardware clock 210 by writing a new value to a current time control status register (CSR), writing to a time adjustment CSR in a forward or backward desired step, or receiving a valid time synchronization message that requires a step forward or backward. NIC 204 can perform a time adjustment or time synchronization clock adjustment by running hardware clock 210 slower or faster by as many cycles as the step desired. For example, to step backward 50 cycles, hardware clock 210 can increment 0.5 steps (instead of 1.0 steps) for 100 cycles, resulting in a loss of 50 cycles.

[0050] NIC 204 can perform a speed adjustment of hardware clock 210 by adjusting the frequency via time synchronization software 206. To speed up, the frequency can be incremented by shifting right by a configurable value N. As an example, the configurable value N can range from 1 to 7. To slow down, the frequency can be decremented by shifting right by a configurable value N. This can result in the frequency being multiplied or divided by the configurable value N. At startup, time synchronization software 206 can aggressively set the configurable value, such as setting N = 1 to 2. Over time, time synchronization software 206 can slow down the configurable value, such as setting N = 2 to 7, to smooth the response when acquiring lock.

[0051] Hardware clock 210 can include a fine frequency adjustment performed by time synchronization hardware 208. As an example, the fine frequency adjustment can have a resolution of 0.5 parts per billion. The fine frequency adjustment can be implemented via additional bits for the least significant bits of the fractional clock count. Figure 3 A block diagram of an example configuration 300 for time adjustment is depicted. Configuration 300 includes a time register 302, a time fraction adjustment register 304, and a fine frequency adjustment register 306. As an example, the time register can include 62 bits, the time fraction adjustment register can include 31 bits, and the fine frequency adjustment register 306 can include 32 bits. It should be noted that the bits shown in each register are exemplary and not intended to be limiting.

[0052] The fine frequency adjustment register 306 can indicate how much to increment the time register 302 per cycle. The fine frequency adjustment register 306 can default to incrementing the time register 302 by 1 bit per cycle, such as by being set to 0x80000000. The frequency of the hardware clock 210 can be set faster or slower by changing the configurable value N. For example, to set the hardware clock frequency faster, the fine frequency adjustment register 306 can be set to (0x80000000 + ((N « 31) / 1,000,000,000)).

[0053] The time fraction adjustment register 304 can be used for high resolution time adjustment. For example, the time fraction adjustment register 304 can adjust time to 0.5 parts per billion.

[0054] The average of the previous time synchronization difference between the NIC 204 and the upstream NIC can be continuously calculated by the NIC 204. This can enable the NIC 204 to attempt to match the frequency of the master NIC via the time synchronization hardware 208 using fine frequency adjustment, thereby minimizing the difference seen using each synchronization message. The average can effectively low pass filter the time synchronization difference. Since individual time synchronization messages can have varying differences due to, for example, network effects, the average can cancel out the effects of these network effects, which allows for faster convergence to the master clock frequency.

[0055] Referring back to Figure 2 , the time synchronization hardware 208 can maintain a status bit indicating the quality of the local current time. The quality can correspond to several time quality parameters indicating the accuracy of the hardware clock 210. The time quality parameters can include a time locked parameter, a skip update parameter, and a software bit parameter.

[0056] The time locked parameter can determine whether the time of the hardware clock 210 is considered to be locked, which corresponds to being within a threshold range, such as + / - 100 ns, from the master clock. The range whether the time is considered to be locked can be configurable. On the master clock, the time locked parameter should always be locked, such as indicated by a “1” bit. On other hardware clocks, if the time is less than the lock threshold after a configurable number of time synchronization updates, such as 3, the time synchronization message can be updated with a step count less than the lock threshold. If the time is greater than the lock threshold after the configurable number of time synchronization updates, the time synchronization message can be updated with a step count greater than the lock threshold.

[0057] The skip-update parameter can correspond to multiple skip time synchronization messages reflected in the current time. On a master clock, no time synchronization messages should be skipped, such as indicated by a "0" bit. As other hardware clocks, the skip-update parameter is incremented, such as by "1" bit, after a time synchronization message timeout or if an incoming time synchronization message is not accepted, such as based on a validity check. The skip-update parameter can be copied from an incoming time synchronization message. For example, a hardware clock that is two hops away from a master clock can have missed an update, while an upstream hardware clock can have incremented its skipped update by 2, which would result in a local skipped update parameter equal to 3. This can also be communicated in any outgoing time synchronization message. The skip-update parameter can saturate when it reaches its maximum value, such as a value of 7. Once the skip-update parameter saturates, it no longer increments. Once the skip-update parameter saturates, the time synchronization hardware 208 can issue an interrupt to the NIC 204 so that the NIC 204 can take further action to identify why so many messages were skipped.

[0058] The software bit parameter can be opaque to the time synchronization hardware 208 so that the NIC 204 can use these bits to pass additional information between network devices without involving the time synchronization hardware 208 or the hardware clocks 210. At a master clock, the software bit parameter can be copied from the CSR into an outgoing message. At other hardware clocks, the software bit parameter can be copied from an incoming message and passed downstream. Any changes in the software bit parameter on an incoming message to a hardware clock can result in an interrupt being communicated to the NIC 204.

[0059] The current time is distributed from the master hardware clock to the client hardware clocks (and from there to second-hop client hardware clocks, etc.) via time synchronization messages. Figure 4 An example time synchronization message 400 configured as an ordered time set is depicted. The message can include a type indicating that the message is a time synchronization message 400, such as indicated by "10" in Figure 4 The time synchronization message 400 can include a number of bits that are always 0 for a time synchronization message, such as indicated by "0x00" in Figure 4 The time synchronization message 400 can also include a subtype 402. The subtype 402 can correspond to the type of time synchronization message, such as a time update, a delay request, or a delay response.

[0060] The time synchronization message 400 can also include a time lock parameter 404, a skip-update parameter 406, a software bit parameter 408, a time 410, and an error check 412. The time synchronization message 400 can also include unused bits reserved for future use, such as indicated by "0x00" in Figure 4The error check 412 can be a cyclic redundancy check (CRR) as an example. The time 410 included in the time synchronization message 400 can correspond to the least significant bits of time. For example, the time bits can be the least significant 38 bits of time, which gives a rollover period of about 5 minutes. Since the time synchronization message 400 can contain the least significant bits of time, each NIC can calculate the time synchronization by comparing its local time to the incoming time to determine +0, +1, +2, etc.

[0061] Each NIC can be a host, a client, or free running. Referring back to Figure 2 , the NIC 204 can correspond to a client NIC of a client device. Figure 2 The master NIC 212 and the other client NICs 214, 216 of the master device 218 and the other client devices 220, 222 are also depicted, respectively. Although only two other client devices are shown for simplicity, any number of client devices can be included for time synchronization.

[0062] A master NIC, such as the master NIC 212, can only send time synchronization messages. The master NIC can include a software-based time synchronization 224 and a hardware-based time synchronization 226 to synchronize a master hardware clock 228 using time adjustment and fine frequency adjustment. The software-based time synchronization 224 can include a network time protocol or a precision time protocol. The hardware-based time synchronization 226 can generate time synchronization messages to transmit to downstream client NICs, such as the NIC 204.

[0063] A client NIC, such as the client NIC 204, 214, or 216, can receive time synchronization messages from one or more upstream NICs and / or send time synchronization messages to one or more downstream NICs. An upstream NIC can be a master NIC or another client NIC closer in a time distribution tree to the master NIC, such as the client NIC 204. A downstream NIC can be another client NIC further in the time distribution tree from the master NIC, such as the other client NICs 214, 216. Each client NIC can include a software-based time synchronization for initial setting of coarse time of a hardware clock and a hardware-based time synchronization for subsequent fine tuning of time of the hardware clock. As Figure 2 shown, the other client NICs 214, 216 include time synchronization software 230, 232, time synchronization hardware 234, 236, and hardware clocks 238, 240, respectively.

[0064] Free running NICs (not shown) are independent of the master NIC and client NICs. In free running NICs, hardware-based time synchronization can be disabled and only software-based time synchronization can be enabled. Free running NICs can include software-based time synchronization to synchronize clocks using time adjustment and fine frequency adjustment. Free running allows clocks to count in time units such as ns instead of cycles by setting the fine frequency adjustment in time units, such as 1.25 ns for an 800 MHz clock.

[0065] Each NIC can send time synchronization messages on any number of outgoing links via one or more ports in one or more physical layers. For example, master NIC 212 can send time synchronization messages through physical layer 242, client NIC 204 can send time synchronization messages through physical layer 244, and other client NICs 214, 216 can send time synchronization messages through physical layers 246, 248, respectively. Time synchronization messages are sent at configured intervals such as approximately 1 us, 10, or 100 us. The configured intervals can be set by a time update timer register in the time synchronization hardware. On a client hardware clock, an incoming time synchronization message can cause an update to be passed downstream even if the time update timer has not expired. This can ensure that even the farthest node in the time distribution tree does not deviate too much from the master clock. The time synchronization messages can indicate the actual time of transmission + / - a delay, which can be + / - 1 ns, 2.5 ns, 5 ns, or 10 ns as examples. The delay is accounted for by incrementing a time field in the time synchronization message for each cycle the time synchronization message is waiting to be sent, which includes the time of flight across the network device from the hardware clock and the arbitration time to be sent.

[0066] Each NIC can receive time synchronization messages via the CSR, which can control which port, if any, can accept time synchronization messages. While each NIC can receive time synchronization messages from multiple upstream NICs, each NIC should accept and update its time from only one upstream NIC. For example, if accepted, client NIC 204 can receive a time synchronization message through physical layer 244, and other client NICs 214, 216 can receive time synchronization messages through physical layers 246, 248, respectively. If none of its ports are programmed to accept time synchronization messages, a NIC can only have software-based synchronization enabled, thus corresponding to a free-running NIC. A single port in a NIC can be enabled to receive time synchronization messages to prevent a NIC from receiving time synchronization messages from multiple upstream NICs, which can corrupt time calculations. When an enabled port receives a time synchronization message, the time in the time synchronization message can be adjusted for any delay to bring the time synchronization message to the hardware clock. The delay can include the time to get off the sending NIC, the time across the links in the network, the time to get into the receiving NIC, and the time of flight across network devices to the hardware clock.

[0067] Each port of a NIC can latch—e.g., extract and save—the quality parameters of an incoming time synchronization message regardless of whether the port is configured to accept time synchronization messages on that port. This can enable a NIC to check the quality of the clock received at each port, if any, to help decide which port, if any, should be enabled for time synchronization message reception.

[0068] A NIC can include link delay measurement capability to account for variable length cables, channels, etc. when sending and / or receiving time synchronization messages. Time synchronization software, such as time synchronization software 206 in NIC 204, can transmit a delay request message to switch 250. Time synchronization software 206 can generate a delay request based on a reset of network device 202 or upon a time lock loss. Switch 250 latches the delay request message and generates a delay response back to time synchronization software 206. Just as with time synchronization messages, the time field of the delay message is incremented every cycle the delay response is waiting to be sent. Time synchronization software 206 can receive the delay response and subtract the received time field from the current time. The link delay can indicate the round trip delay from one hardware clock to another hardware clock through physical channels, serializer / deserializers (SERDES), on-chip routing, etc. Because the clocks are hardware-based, the link delay measurement can be fixed, allowing for more accurate synchronization than purely software-based clocks.

[0069] The NICs can be initialized with actual global time. For example, the time synchronization software 206 can initialize the NIC 204 with a global time as a random value, as synchronization involves relative time. The initial time can be relatively inaccurate in the client NICs, e.g., + / - 10 seconds, as long as the threshold value in the time limit register is also large. The time limit register can be included as part of the time synchronization hardware, such as the time synchronization hardware 208. Once the initial global time is written to all the NICs in the network, the respective time synchronization software can implement the sending and receiving of time synchronization messages. Each time limit register in the respective NICs can be set for an acceptable threshold value, so different NICs can be locked to a master NIC. Once all the NICs are time-locked, the time limit register can be made tighter, such as approximately + / - 1-100 ns.

[0070] The time offset between the host and any clients should not exceed the configured threshold value of the time limit register, such as initially + / - 10 seconds but later tightened to + / - 1-100 ns. Otherwise, the time synchronization software in the NICs can reset the time. During the reset, the time synchronization software can unlock the time lock, and can ensure that no time synchronization messages are accepted or sent.

[0071] The time synchronization messages can be checked for validity to be accepted at a received NIC, such as the NIC 204. The validity check can be performed by the time synchronization hardware, such as the time synchronization hardware 208. The validity check can include determining that the time synchronization message is received at a port enabled for time synchronization reception, the time synchronization message has a valid error check, the time synchronization message indicates a time within a lock threshold period of a local time, and / or has a locked incoming time.

[0072] A hardware clock, such as the hardware clock 210, can internally maintain two CSR-readable counts: a skip update count and a valid update count. The skip update count can correspond to an amount, such as an integer amount, used to determine whether the hardware clock should be reset. The valid update count can correspond to an amount, such as an integer amount, used to determine whether the hardware clock should be time-locked. Any time synchronization message that does not pass the validity check involving time-locking can result in an increment of the skip update count and a reset of the valid update count. The increment of the skip update count and the reset of the valid update count can be performed by the time synchronization hardware, such as the time synchronization hardware 208. When the number of skip updates increases beyond a time limit skip message threshold, such as 7, the master time can have changed, resulting in clearing the time-locked parameter and interrupting the NIC. The clearing of the time-locked parameter and the interrupting of the NIC can also be performed by the time synchronization hardware, such as the time synchronization hardware 208.

[0073] When a valid update is received, the hardware clock can move to the received time and the valid update count is incremented. The skip update count can be reset to the incoming skip update value. The valid update count increment and the skip update count reset can be performed by time synchronization hardware, such as time synchronization hardware 208. The incoming skip update is copied into the local skip update to provide information about how much the current client clock can drift from the master clock. When the number of valid updates increases beyond a time-limited lock message threshold, such as 7, the hardware clock can lock. The lock can also be performed by time synchronization hardware, such as time synchronization hardware 208. The time lock parameter is updated based on the internal skip update and valid update counts, while the time itself is updated based on the validity of the incoming time synchronization messages.

[0074] Figure 5 A flowchart depicting an example process 500 for synchronizing time between a master hardware clock and one or more client hardware clocks is shown. The example process 500 can be performed on a system of one or more processors in one or more locations, such as the time synchronizer 164 of the network interface card 160 as depicted in FIG. 1. Figure 1

[0075] As shown in block 510, the time synchronizer 164 of a client hardware clock can receive a time synchronization message. The time synchronization message can be sent from a time synchronizer of a master hardware clock or an upstream client hardware clock.

[0076] As shown in block 520, the time synchronizer 164 can perform a validity check on the time synchronization message. For the validity check, the time synchronizer can determine that the time synchronization message was received at a port of the network interface card 160 that is enabled for time synchronization message reception, that the time synchronization message has a valid error check, that the time synchronization message indicates a time that is within a lock threshold period of the client hardware clock, and / or that the incoming time is locked.

[0077] As shown in block 530, in response to the received time synchronization message passing the validity check, the time synchronizer 164 can update the time of the client hardware clock. Updating the time can include running the client hardware clock slower or faster by as many periods as required based on the step size. Updating the time can also include adjusting the frequency of the client hardware clock.

[0078] As shown in block 540, in response to the received time synchronization message passing the validity check, the time synchronizer 164 can increment a valid update count. The time synchronizer 164 can also reset a skip update count to an incoming skip update value. The valid update count and the skip update count can be maintained in the client hardware clock.

[0079] ​As shown in block 550, in response to the valid update count incrementing to exceed a threshold, time synchronizer 164 can lock the client hardware clock's time. Locking the client hardware clock can indicate that the client hardware clock is within a desired threshold range from the master hardware clock.

[0080] Figure 6 An example computing system 600 is depicted, which can be configured as follows: Figure 1 The network device 110. The computing system 600 may include one or more processors 610 communicating with memory 620, at least one network interface controller 650 having a network interface port 660 for connecting to a network, input / output (I / O) components 670, and other components 680. As an example, the processor 610 may communicate with memory 620, network interface controller 650, I / O components 670, and other components 680 via bus 640. The processor 610 may be incorporated into or connected to cache 630. In some instances, instructions are read from memory 620 into cache 630 and executed by processor 610 from cache 630.

[0081] Processor 610 can be any logic circuit that executes instructions fetched from memory 620 or cache 630. Processor 610 can be a microprocessor unit or a dedicated processor. Computing device 600 can be based on any processor or processor group capable of operating as described herein. Processor 610 can be a single-core or multi-core processor. Processor 610 can be multiple different processors. Processor 610 can be implemented as one or more on-chip circuits.

[0082] Memory 620 can be any device suitable for storing computer-readable data. Memory 620 can be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media and storage devices, semiconductor storage devices—such as EPROM, EEPROM, SDRAM and flash memory devices, magnetic disks, magneto-optical disks and optical disks—such as CDROM, DVD-ROM, or... Optical disc. The computing system 600 may have any number of memory devices 620.

[0083] The cache 630 may be a form of computer memory placed adjacent to the processor 610 for fast access times. The cache 630 may be part of the processor 610 or on the same chip as the processor 610. There may be multiple levels of cache 630, such as two-level and three-level cache layers.

[0084] The network interface controller 650 can manage the exchange of data via the network interface 660. The network interface controller 650 can handle the physical and data link layers of the open systems interconnection (OSI) model for network communications. Some of the tasks of the network interface controller 650 can be handled by one or more of the processors 610. The network interface controller 650 can be incorporated into the processor 610, such as circuitry on the same chip.

[0085] The computing system 600 can have multiple network interfaces 660 controlled by a single controller 650 or multiple network interface controllers 650. Each network interface 660 can be a connection point for a physical network link, such as a category 5 Ethernet link. The network interface controller 650 can support wireless network connections and the interface port 660 can be a wireless, e.g., radio, receiver / transmitter. The network interface controller 650 can implement one or more network protocols, such as Ethernet. The computing system 600 can exchange data with other computing systems via physical or wireless links through the network interface 660. The network interface 660 can link to another device directly or through intermediate devices, e.g., network devices such as hubs, bridges, switches, or routers, that connect the computing device 600 to a data network, such as the Internet.

[0086] The computing system 600 can include or provide an interface for one or more input or output (I / O) components 670. Input devices can include, by way of example, keyboards, microphones, touch screens, sensors, and pointing devices, such as mice or trackpads. Output devices can include, by way of example, video displays, speakers, and printers.

[0087] Other components 680 can include I / O interfaces, external serial device ports, and any additional coprocessors. For example, the computing system 600 can include an interface, such as a universal serial bus (USB) interface, for connecting input, output, or additional storage devices, such as portable flash drives or external media drives. The computing device 600 can include additional coprocessors, such as a math coprocessor, to assist the processor 610 with high precision or complex calculations.

[0088] The foregoing alternative examples are not mutually exclusive unless otherwise specified, but can be implemented in various combinations to achieve unique advantages. As such variations and combinations of features can be utilized in accordance with the subject matter disclosed herein without departing from the scope of the claims, the foregoing description of the embodiments should not be taken as limiting the scope of the subject matter as defined by the claims. Rather, the intent is to cover all modifications and alternatives that fall within the scope of the claims. Furthermore, the examples described herein and elsewhere, as well as the provisions of the articles "a," "an," and "the," should be construed to be inclusive, rather than exclusive, unless otherwise specified.

Claims

1. A method for synchronizing time between a master hardware clock and one or more client hardware clocks, the method comprising: receiving, by one or more processors of a client hardware clock of the one or more client hardware clocks, a time synchronization message; performing, by the one or more processors, a validity check on the time synchronization message; in response to the received time synchronization message passing the validity check, updating, by the one or more processors, a time of the client hardware clock; in response to the received time synchronization message passing the validity check, incrementing, by the one or more processors, a valid update count; and in response to the valid update count increasing beyond a valid update threshold, locking, by the one or more processors, the time of the client hardware clock. the time synchronization message is received from the master hardware clock or one or more processors of an upstream client hardware clock.

2. The method of claim 1, wherein, sending, with the one or more processors, the time synchronization message to a downstream client hardware clock.

3. The method of claim 1, further comprising: updating the time of the client hardware clock further comprises at least one of:

4. The method of claim 1, wherein, adjusting the time forward or backward by a number of steps; and adjusting a frequency of the client hardware clock. locking the client hardware clock further comprises determining that the client hardware clock is within a threshold range from the master hardware clock.

5. The method of claim 1, wherein, 6. The method of claim 1, further comprising: sending, by the one or more processors, a delay request message; receiving, by the one or more processors, a delay response message; and determining, by the one or more processors, a hardware delay time based on a time difference between a time at which the delay request message was sent and a time at which the delay response message was received. updating the time of the client hardware clock further comprises removing the hardware delay time from the time of the client hardware clock. the hardware delay time is fixed.

7. The method of claim 6, wherein, performing the validity check comprises determining that a time in the time synchronization message is within a threshold range.

8. The method of claim 6, wherein, 10. The method of claim 1, further comprising skipping, by the one or more processors, an update to the time of the client hardware clock in response to the received time synchronization message failing the validity check.

9. The method of claim 1, wherein, 11. The method of claim 10, further comprising incrementing, by the one or more processors, a skip update count in response to the received time synchronization message failing the validity check.

12. The method of claim 11, further comprising resetting, by the one or more processors, the client hardware clock in response to the skip update count increasing beyond a skip update threshold.

13. A system comprising: one or more processors; and ​ ​ ​ one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for synchronizing time between a master hardware clock and one or more client hardware clocks, the operations comprising: receiving a time synchronization message; performing a validity check on the time synchronization message; updating a time of a client hardware clock of the one or more client hardware clocks in response to the received time synchronization message passing the validity check; incrementing a valid update count in response to the received time synchronization message passing the validity check; and locking the time of the client hardware clock in response to the valid update count increasing beyond a valid update threshold.

14. The system of claim 13, wherein, updating the time of the client hardware clock further comprises at least one of: adjusting the time forward or backward by a number of steps; and adjusting a frequency of the client hardware clock.

15. The system of claim 13, wherein, locking the client hardware clock further comprises determining that the client hardware clock is within a threshold range from the master hardware clock.

16. The system of claim 13, wherein, the operations further comprise: sending a latency request message; receiving a latency response message; and determining a hardware latency time based on a time difference between a time at which the latency request message was sent and a time at which the latency response message was received; wherein updating the time of the client hardware clock further comprises removing the hardware latency time from the time of the client hardware clock.

17. The system of claim 13, wherein, the operations further comprise: skipping an update of the time of the client hardware clock in response to the received time synchronization message failing the validity check; incrementing a skip update count in response to the received time synchronization message failing the validity check; and resetting the client hardware clock in response to the skip update count increasing beyond a skip update threshold.

18. A non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for synchronizing time between a master hardware clock and one or more client hardware clocks, the operations comprising: receiving a time synchronization message; performing a validity check on the time synchronization message; updating a time of a client hardware clock of the one or more client hardware clocks in response to the received time synchronization message passing the validity check; incrementing a valid update count in response to the received time synchronization message passing the validity check; and locking the time of the client hardware clock in response to the valid update count increasing beyond a valid update threshold. the operations further comprise:

19. The non-transitory computer-readable medium of claim 18, wherein, sending a latency request message; receiving a latency response message; and determining a hardware latency time based on a time difference between a time at which the latency request message was sent and a time at which the latency response message was received; ​ wherein updating the time of the client hardware clock further comprises removing the hardware delay time from the time of the client hardware clock.

20. The non-transitory computer-readable medium of claim 18, wherein, The operations further comprise: in response to the received time synchronization message failing the validity check, skipping updating of the time of the client hardware clock; in response to the received time synchronization message failing the validity check, incrementing a skip update count; and in response to the skip update count incrementing past a skip update threshold, resetting the client hardware clock.

Images