Hardware Clock with Built-in Accuracy Check

Abstract

A network device includes one or more ports for connecting to a communication network, packet processing circuitry, and clock circuitry. The packet processing circuitry is configured to transfer packets on the communication network via the ports. The clock circuitry includes: a hardware clock configured to indicate network time for synchronizing network devices in the communication network; and a built-in accuracy test circuit configured to check the accuracy of the hardware clock.

Description

Technical Field

[0001] The present invention generally relates to network devices, and more particularly to built-in accuracy checking of hardware clocks in network devices. Background Art

[0002] Computers and communication networks can use various schemes and protocols to synchronize network nodes to a common time base. A common example of such a protocol is the Precision Time Protocol (PTP) defined in IEEE Standard 1588-2002 and later versions. PTP is used to synchronize clocks across a computer network and can achieve accuracies in the sub-microsecond range.

[0003] U.S. Patent 8,370,675 describes techniques for precision clock synchronization in network nodes, for example. In some embodiments, a device includes a real-time clock circuit configured to output a local clock time and including: a register coupled to receive an offset value; an adder coupled to add the local clock time to the offset value in the register to give an adjusted value of the local clock time; and a main processor configured to calculate an offset value between the local clock time and a reference clock time and load the offset value into the register.

[0004] U.S. Patent Application Publication 2016 / 0315756 describes a system for testing the quality of a recovered clock and includes a test device that acts as a timing synchronization protocol master for communicating with a device under test that acts as a timing synchronization protocol slave or a timing synchronization protocol boundary clock to synchronize the clock of the device under test with the clock of the test device. Summary of the Invention

[0005] Embodiments of the present invention described herein provide a network device that includes one or more ports for connecting to a communication network, packet processing circuitry, and clock circuitry. The packet processing circuitry is configured to transfer packets over the communication network via the ports. The clock circuitry includes: a hardware clock configured to indicate a network time for synchronizing network devices in the communication network; and a built-in accuracy test circuit configured to check the accuracy of the hardware clock.

[0006] In some embodiments, the hardware clock is configured to track the network time according to the Precision Time Protocol (PTP). In some embodiments, the built-in accuracy test circuit is configured to receive an external reference signal from outside the network device and sample the network time output from the hardware clock at a timing derived from the external reference signal.

[0007] In an example embodiment, the built-in accuracy test circuit is configured to transmit sampled network time from the network device. Additionally or alternatively, the built-in accuracy test circuit may be configured to provide the sampled network time to a controller in the network device. In the disclosed embodiments, the external reference signal includes a Pulse-Per-Second (PPS) signal. In some embodiments, the accuracy of the hardware clock is indicated by the deviation between the sampled network time and the external reference signal.

[0008] In the disclosed embodiments, the external reference signal includes a dedicated test signal that is different from the Pulse-Per-Second (PPS) signal and includes a predetermined pattern, and the built-in accuracy test circuit is configured to identify the dedicated test signal and sample the network time at a timing derived from the predetermined pattern. In an embodiment, the external reference signal includes a 10 MHz signal. In an embodiment, the hardware clock is configured to indicate the network time on a parallel output interface, and the built-in accuracy test circuit includes a set of flip-flops (FFs) that are configured to sample the parallel output interface at a timing derived from the external reference signal.

[0009] According to an embodiment of the present invention, additionally provided is a method that includes transmitting packets on a communication network in a network device. A hardware clock in the network device is used to indicate network time for synchronizing network devices in the communication network. A built-in accuracy test circuit in the network device is used to check the accuracy of the hardware clock. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present invention will be more fully understood from the following detailed description of embodiments of the invention in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a block diagram schematically illustrating a network device employing built-in hardware clock accuracy checking according to an embodiment of the present invention; and

[0012] Figure 2 is a flowchart schematically illustrating a method for built-in hardware clock accuracy checking according to an embodiment of the present invention. DETAILED DESCRIPTION

[0013] OVERVIEW

[0014] Embodiments of the present invention described herein provide techniques for built-in accuracy testing of a hardware clock in a network device. For example, the disclosed techniques can be used to test the accuracy of a Precision Time Protocol (PTP) hardware clock (PHC) integrated into a network adapter and a network switch.

[0015] In some embodiments, a network device is configured to transfer packets over a communication network. Among other components, the network device includes a hardware clock configured to track and indicate network time for synchronizing network devices in the communication network. The network device also includes a built-in accuracy test circuit configured to check the accuracy of the hardware clock.

[0016] Typically, the built-in accuracy test circuit tests the accuracy of the hardware clock relative to an external reference signal (e.g., a Pulse-Per-Second (PPS) signal received from outside the network device). In an embodiment, the hardware clock outputs a digital word indicating the current network time, and the built-in accuracy test circuit samples this digital word at a timing derived from the external reference signal. For example, when using a PPS signal, the built-in accuracy test circuit may sample (take a "snapshot") the digital word generated by the hardware clock at the rising or falling edge of the PPS signal.

[0017] Accordingly, the sampled network time indicates an estimated network time as tracked and indicated by the hardware clock when the external signal arrives in the network device. When the external reference signal is derived from a certain standard timebase (e.g., from the master clock of the network), the sampled network time indicates the accuracy of the hardware clock relative to this standard timebase.

[0018] In various embodiments, the built-in accuracy test circuit can act on the sampled network time in various ways, such as sending the sampled network time to various destinations for processing. The processes of sampling and reporting can be performed, for example, periodically, in response to certain events, or on demand.

[0019] The disclosed techniques provide a scalable and cost-effective means for testing the accuracy of a hardware clock in a network device. In principle, the output of the hardware clock can be sampled and its accuracy tested using commercial test equipment. However, such measurements are not suitable for anything other than sporadic measurements on a small number of network devices. In contrast, the disclosed techniques can be applied on an ongoing basis to a large number of network devices, such as an entire data center, without the need for any external test equipment.

[0020] System Description

[0021] Figure 1 is a block diagram schematically illustrating a network device 20 employing built-in hardware clock accuracy checking according to an embodiment of the present invention. For example, the network device 20 may include a network adapter (such as an Ethernet network interface controller (network card) or an InfinibandTM host channel adapter (HCA)), a network switch or router, a network-enabled graphics processing unit (GPU), or any other suitable type of device capable of network communication.

[0022] The network device 20 includes one or more network ports 24 for receiving packets from a network (not shown) and transmitting packets to the network. The network may include, for example, an Ethernet or an InfiniBand network, or any other suitable network type.

[0023] The network device 20 further includes packet processing circuitry for transmitting (transmitting and receiving) packets on the network via port 24. In this example, the packet processing circuitry includes a data path 28. The data path 28 receives packets from the network via port 24, processes the packets, and sends the packets to the network via the port. Such data paths are more typically depicted by way of example as a switch or a router. For example, in a network adapter, the data path may receive packets from a host and send the packets to the network, and vice versa. Further alternatively, any other suitable packet processing circuitry with any other suitable functionality may be used.

[0024] The network device 20 further includes a hardware clock, which is a PTP hardware clock (PHC) 32 in this example. The PHC 32 is configured to track the current network time, i.e., a common time base for synchronizing various network devices in the network. To assist in accurate tracking, the PHC 32 may be adjusted (“trained”) in various ways. In some embodiments, the network device 20 includes a PPS-IN input interface 44 for receiving a PPS input signal for training the PHC 32. Additionally or alternatively, the PHC 32 may receive an adjustment (represented as “clock adjustment” in the figure) from a local host via a suitable local interface.

[0025] Generally, the PHC 32 outputs a multi-bit digital word (indicated as “current time” in the figure), which indicates the current network time tracked by the PHC at any given time. In some embodiments, the PHC 32 also generates a PPS output signal, which may be output from the network device via the PPS-OUT interface 48.

[0026] The current time output (“current time”) may be used for various purposes in the network device 20. For example, to support PTP, the data path 28 may include an ingress timestamp circuit 36 and an egress timestamp circuit 40. The ingress timestamp circuit 36 is configured to timestamp an incoming PTP packet with the current time when it enters the network device, and the egress timestamp circuit 40 is configured to timestamp an outgoing PTP packet with the current time when it leaves the network device.

[0027] As another example, data path 28 or network device 20 as a whole can perform various packet processing operations according to the current time. For example, such techniques are described in U.S. Patent Application 16 / 782,075 entitled "Network Adapter with Time-Aware Packet-Processing Pipeline", U.S. Patent Application 16 / 910,193 entitled "Packet Scheduling System with Desired Physical Transmission Time for Packets", U.S. Patent Application 16 / 921,993 entitled "TDMA Networking using Commodity NIC / Switch", and U.S. Patent Application 17 / 067,690 entitled "Packet Transmission Using Scheduled Prefetching", the disclosures of which are incorporated herein by reference.

[0028] Regardless of the specific use of network time in network device 20, it is highly desirable to evaluate the accuracy of PHC 32 in tracking and indicating network time. To this end, network device 20 includes a built-in accuracy check (test) circuit 52. The operation of circuit 52 will be described in further detail below. Briefly, circuit 52 receives an external reference signal (denoted as "external-reference") via the PPS-IN interface 44. The external reference signal can include the same PPS input signal used to train PHC 32, or a different signal, such as a dedicated signal for accuracy testing. Circuit 52 samples the "current time" output of PHC 32 at a timing derived from the external reference signal. The sampling time (denoted as "@current time sampled at external-reference" in the figure) is output as an analysis.

[0029] Figure 1 The configuration of network device 20 shown is an example configuration described purely for conceptual clarity. Any other suitable configuration may be used in alternative embodiments. The various elements of network device 20 can be implemented using suitable hardware (such as in one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs)). The various elements of network device 20 can be implemented using hardware, software, or a combination of hardware and software elements.

[0030] Built-in PHC accuracy check

[0031] Figure 2 FIG. 1 schematically shows a flowchart of a method for checking the accuracy of an internal hardware clock in a network device 20 according to an embodiment of the present invention. The method starts with the PHC 32 tracking the network time in a tracking step 60. In a reference input step 64, the built-in accuracy test circuit 52 receives an external reference signal (“external-reference”) via the PPS-IN interface 44. At a sampling step 68, the circuit 52 samples the “current time” output of the PHC 32 at a timing derived from the external reference signal.

[0032] In some embodiments, the external reference signal includes a PPS signal (e.g., a PPS input signal for training the PHC 32, or another PPS signal). In other embodiments, the external reference signal includes a dedicated test signal that is different from a conventional PPS signal and is intended for checking the accuracy of the PHC 32. For example, the external test signal may include a predetermined pattern of rising and / or falling edges (e.g., three consecutive edges), the timing of which is derived from the actual network time. The circuit 52 may monitor the PPS-IN input interface 44, detect the dedicated test signal, and sample the PHC output at the timing defined by the predetermined pattern of edges. The external reference signal may have any suitable frequency—in one embodiment, the signal is a 10 MHz signal.

[0033] In a non-limiting embodiment, the PHC 32 has a parallel output interface that outputs the current network time and is always valid for readout by a client. The test circuit 52 may include, for example, a set of flip-flops (FF) that sample the parallel output interface of the PHC 32 when triggered by the external reference signal. In this context, the circuit 52 may be regarded as an additional client of the PHC. In alternative embodiments, any other suitable configuration may be used.

[0034] In an accuracy estimation step 72, the accuracy of the PHC 32 is evaluated. In some embodiments, the accuracy of the PHC 32 is estimated within the network device 20. For example, when the network device 20 is a network switch, the circuit 52 may send the sampled network time to a controller of the network switch, and the controller of the network switch runs software for estimating the PHC accuracy.

[0035] In other embodiments, circuit 52 may send the sampled network time to a destination external to network device 20. Such destinations may include, for example, an analyzer or other suitable collector node. For example, when network device 20 is a network switch, circuit 52 may send the sampled network time over the network, e.g., in a communication packet, or output the sampled network time on a local output interface. For example, when network device 20 is a network adapter, circuit 52 may send the sampled network time to a local host, such as a computing node in which the network adapter is installed. Additionally or alternatively, the sampled network time may be sent to any other suitable destination or destinations for analysis.

[0036] Assuming that the external reference signal tracks the actual network time with high accuracy, the sampled time generated by circuit 52 indicates the accuracy with which PHC 32 tracks the network time.

[0037] In various embodiments, the sampled network time generated by circuit 52 (“@external-reference sampled current time”) may be used in various ways to estimate the deviation between the network time indicated by PHC 32 and the actual network time. In this context, the term “deviation between the network time indicated by PHC 32 and the actual network time” may refer to the single absolute difference between the network time indicated by PHC 32 and the actual network time, some statistical measure of the difference between the network time indicated by PHC 32 and the actual network time, or any other suitable form of deviation.

[0038] In some embodiments, the accuracy of PHC 32 is estimated based on a single measurement by circuit 52, i.e., based on a single sampled network time. Multiple such measurements may be performed at different times. In other embodiments, the accuracy of PHC 32 is estimated based on multiple measurements (e.g., by averaging over multiple sampled network times or applying any other suitable statistical calculation).

[0039] For example, in some embodiments, the sampling operation in circuit 52 has a constant delay relative to an external reference signal. In some embodiments, the magnitude of this constant delay is known. In these embodiments, the absolute difference between the network time indicated by PHC 32 and the actual network time can be derived. In other embodiments, the delay is constant but unknown. In these embodiments, it is only possible to estimate the statistical deviation (between the network time indicated by PHC 32 and the actual network time) over multiple measurements. For example, such measurements can be used to estimate the root mean square (RMS) error or standard deviation of the network time indicated by PHC over a certain time interval. Additionally, alternatively, the sampled network time (“@external-reference sampled current time”) can be used in any other suitable way to estimate any other suitable metric of the accuracy of PHC 32.

[0040] The accuracy testing techniques described herein can be used in a variety of systems and applications, e.g., for testing the multiple hardware clocks of multiple network devices in data centers, telecommunication systems, automotive and industrial networks, robotic factories, and many other places.

[0041] Accordingly, it should be understood that the above-described embodiments are cited by way of example and that the invention is not limited to what is specifically shown and described above. Rather, the scope of the invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications thereof that would occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art. In the case of documents incorporated by reference in this patent application, those documents are considered to be part of this application, except to the extent that any terms are defined in those incorporated documents in a manner that conflicts with the definitions expressly or implicitly set forth in this specification, in which case only the definitions in this specification shall be considered.

Claims

1. A network device, comprising: one or more ports configured to connect to a communication network; packet processing circuitry configured to transfer packets over the communication network via the ports; and a clock circuit comprising: a hardware clock configured to output a number indicative of a network time for synchronizing the network device to one or more other network devices in the communication network; and an in-built accuracy test circuit configured to receive an external reference signal having a predetermined pattern of rising and / or falling edges and to sample the number output by the hardware clock at a timing derived from the external reference signal.

2. The network device according to claim 1, wherein the hardware clock is configured to track the network time according to the Precision Time Protocol (PTP).

3. The network device according to claim 1, wherein the in-built accuracy test circuit is configured to transmit a sampled digital output from the network device.

4. The network device according to claim 1, wherein the in-built accuracy test circuit is configured to provide a sampled digital output to a controller in the network device.

5. The network device according to claim 1, wherein the external reference signal comprises a Pulse Per Second (PPS) signal.

6. The network device according to claim 1, wherein the accuracy of the hardware clock is indicated by a deviation between the sampled network time and the external reference signal.

7. The network device according to claim 1, wherein the external reference signal comprises a dedicated test signal different from the Pulse Per Second (PPS) signal and comprising the predetermined pattern, and wherein the in-built accuracy test circuit is configured to identify the dedicated test signal and sample the network time at the timing derived from the predetermined pattern.

8. The network device according to claim 1, wherein the external reference signal comprises a 10 MHz signal.

9. The network device according to claim 1, wherein the hardware clock is configured to indicate the network time on a parallel output interface, and wherein the in-built accuracy test circuit comprises a set of flip-flops (FFs) configured to sample the parallel output interface at the timing derived from the external reference signal.

10. A method for checking the accuracy of a hardware clock, comprising: transferring packets over a communication network in a network device; using a hardware clock in the network device that outputs a number indicative of the network time to indicate a network time for synchronizing network devices in the communication network; and checking the accuracy of the hardware clock using an in-built accuracy test circuit in the network device, wherein the in-built accuracy test circuit receives an external reference signal having a predetermined pattern of rising and / or falling edges and samples the number output by the hardware clock at a timing derived from the external reference signal.

11. The method according to claim 10, wherein indicating the network time includes tracking the network time according to the Precision Time Protocol (PTP).

12. The method according to claim 10, wherein checking the accuracy of the hardware clock further includes transmitting the sampled network time from the network device.

13. The method according to claim 10, wherein checking the accuracy of the hardware clock further includes providing the sampled network time to a controller in the network device.

14. The method according to claim 10, wherein the external reference signal includes a Pulse-Per-Second (PPS) signal.

15. The method according to claim 10, wherein the accuracy of the hardware clock is indicated by a deviation between the sampled network time and the external reference signal.

16. The method according to claim 10, wherein the external reference signal includes a dedicated test signal that is different from the Pulse-Per-Second (PPS) signal and includes the predetermined pattern, and wherein sampling the network time includes identifying the dedicated test signal and sampling the network time at the timing derived from the predetermined pattern.

17. The method according to claim 10, wherein the external reference signal includes a 10 MHz signal.

18. The method according to claim 10, wherein indicating the network time includes outputting the network time on a parallel output interface, and wherein sampling the network time includes sampling the parallel output interface at the timing derived from the external reference signal using a set of flip-flops (FF).

Images