Software-Controlled Clock Synchronization of Network Devices

Abstract

A synchronous communication system includes a plurality of network communication devices, one of which is designated as a root device and the others are designated as slave devices. Each network communication device includes one or more ports and communication circuitry that processes communication signals received by the one or more ports to recover a corresponding remote clock from each signal. A synchronization circuit is integrated in the root device and provides a root clock signal that is transmitted to the slave devices via a clock link. A host processor selects one of the ports of one of the network communication devices as a master port, finds a clock difference between the root clock signal and the corresponding remote clock recovered from the master port, and outputs a control signal in response to the clock difference that causes the synchronization circuit to adjust the root clock signal.

Description

Technical Field

[0001] The present invention generally relates to systems and methods for network communication, and more particularly to clock synchronization of network devices. Background Art

[0002] Clock synchronization between network devices is used in many network applications, such as applications where a group of nodes in a network need to maintain consistent, accurate, and stable clocks within the group. Another application of using synchronized clock values is to measure the delay between devices. If the clocks are not synchronized, the resulting delay measurements will be inaccurate. The synchronization requirements in many applications are extremely strict. For example, ITU-T Recommendation G.8273.2 allows a maximum synchronization error of 30 ns between telecommunication boundary clocks.

[0003] Synchronous Ethernet (SyncE) is a computer network standard of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T), which helps to transmit clock signals through the Ethernet physical layer. In particular, SyncE synchronizes the clocks inside the network with the master clock. Each network device, such as a switch, network interface card (NIC), or router, needs to recover the master clock from the high-speed data received from the clock source and use the recovered master clock for its own data transmission. In this way, the master clock propagates throughout the network.

[0004] U.S. Patent 10,778,406 describes a technique for precisely synthesizing clock synchronization between network devices, the disclosure of which is incorporated herein by reference. In this patent, a network device includes a frequency generation circuit configured to generate a clock signal, a phase-locked loop configured to generate a local clock based on the clock signal, and a plurality of receivers configured to receive respective data streams from respective remote clocks. Each receiver of the plurality of receivers is configured to recover the remote clock from the respective data stream. A controller is configured to identify the remote clock recovered by one of the plurality of receivers as the master clock, find the clock difference between the identified remote clock and the local clock, and provide a control signal to the frequency generation circuit in response to the clock difference, which causes the frequency generation circuit to adjust the clock signal to iteratively reduce the absolute value of the clock difference. Summary of the Invention

[0005] The embodiments of the present invention described below provide improved methods and devices for clock synchronization.

[0006] Accordingly, according to an embodiment of the present invention, a synchronous communication system is provided, including a plurality of network communication devices, including one network communication device designated as a root device and one or more other network communication devices designated as slave devices. Each network communication device includes one or more ports configured to transmit and receive respective communication signals through respective network links, a communication circuit configured to process the communication signals received by the one or more ports to recover a respective remote clock from each of the communication signals. A synchronization circuit is integrated in the root device and configured to provide a root clock signal to the communication circuit in the root device. A clock link is coupled to transmit the root clock signal from the root device to the slave devices to be used as a local clock signal for the communication circuit in the slave devices. A host processor is coupled to the network communication devices through a control link and is driven by software to select one of the ports of one of the network communication devices as a master port to find a clock difference between the root clock signal and the respective remote clock recovered from the master port, and output a control signal in response to the clock difference to cause the synchronization circuit to adjust the root clock signal.

[0007] In the disclosed embodiment, the host processor is configured to select the master port from the ports of both the root device and the slave devices.

[0008] In one embodiment, the clock link includes a cable interconnecting the network communication devices independently of the network links. In some embodiments, the clock link is configured to transmit the root clock signal from the root device to a first of the slave devices and serially transmit from the first of the slave devices to a second of the slave devices. Additionally or alternatively, the clock link is configured to transmit the root clock signal from the root device in parallel to at least a first and a second of the slave devices.

[0009] In the disclosed embodiment, the network communication device includes a network interface controller (NIC) and / or a network switch. Additionally or alternatively, the system includes a peripheral component bus connecting the host processor to the network communication devices, where the control link is implemented on the peripheral component bus.

[0010] In some embodiments, the slave device includes a respective synchronization circuit configured to receive the root clock signal from the synchronization circuit of the root device and generate the local clock signal in response to the root clock signal. In the disclosed embodiment, the synchronization circuit includes a voltage controlled oscillator (VCO) and is configured to adjust the voltage applied to the VCO in response to the control signal.

[0011] According to an embodiment of the present invention, there is also provided a synchronization method, including connecting one or more ports in each of a plurality of network communication devices to send and receive corresponding communication signals through corresponding network links. Processing the communication signals received by the one or more ports to recover corresponding remote clocks from each of the communication signals. Designating one of the network communication devices as a root device and designating one or more other network communication devices as slave devices. Providing a root clock signal from a synchronization circuit in the root device to a communication circuit in the root device. Transmitting the root clock signal from the root device to the slave devices via a clock link to be used as a local clock signal for the communication circuits in the slave devices. Coupling a host processor to the network communication devices via a control link. Selecting one of the ports of one of the network communication devices as a master port by the host processor. Finding a clock difference between the root clock signal and the corresponding remote clock recovered from the master port. Responsive to the clock difference, outputting a control signal that causes the synchronization circuit to adjust the root clock signal. The present invention will be more fully understood from the following detailed description of its embodiments in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram schematically showing a synchronization network communication system according to an embodiment of the present invention;

[0013] Figure 2 is a block diagram schematically showing details of communication and synchronization circuits in a network device according to an embodiment of the present invention;

[0014] Figure 3 is a block diagram schematically showing details of software and circuits for clock synchronization in a network communication system according to an embodiment of the present invention; and

[0015] Figure 4 is a flowchart schematically showing a method for clock synchronization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] OVERVIEW

[0017] In a SyncE - compliant switch, all switch ports are precisely locked to a master clock. The above - mentioned U.S. Patent 10,778,406 describes effective methods and circuits for identifying a port serving as a master clock source and adjusting and distributing the master clock to other ports.

[0018] To increase the number of synchronization ports in the current SyncE network, the master clock is transmitted to other switches through the Ethernet physical layer, which means the same path is used for the clock and network data. Each additional hop along this clock / data path adds noise and latency. For this reason, the SyncE solution is not easily scalable to larger networks that include a large number of devices and Ethernet links.

[0019] Embodiments of the present invention described herein address these limitations by enabling multiple independent network devices to share the same precise root clock, which is generated by a root device under the control of synchronization software running on a host processor. The root clock is distributed over dedicated clock links, such as cable links between network devices, rather than relying on the data network for clock distribution as in traditional SyncE installations. The clock links can be arranged in any desired topology, including serial and / or parallel clock links, in order to serve a large number of network devices located in close proximity to each other. Such a solution is particularly suitable, for example, for sharing a precise root clock between multiple network interface controllers (NICs) serving the same host computer or between multiple NICs or switches on different racks of the same rack.

[0020] The disclosed embodiments thus pertain to a synchronous communication system that includes multiple network communication devices (e.g., NICs or switches). Each network communication device includes one or more ports through which it sends and receives communication signals via respective network links, as well as communication circuitry that processes the communication signals received by the device's ports in order to recover a corresponding remote clock from each received signal. One of the network communication devices is designated as the root device, while the other network communication devices are designated as slave devices. Synchronization circuitry integrated in the root device provides a precise root clock signal to the communication circuitry in the root device. This root clock signal is transmitted from the root device to the slave devices via the clock link to be used as the local clock signal for the communication circuitry in the slave devices. Thus, all network communication devices are precisely synchronized with each other via the root clock.

[0021] The host processor running the synchronization software is coupled to the network communication devices via a control link. To synchronize the root clock with a reference clock received from the network, the host processor selects one of the ports of one of the network communication devices as the master port. (For example, the host processor can make the selection autonomously or in response to a network management command.) The master port can be selected from the ports on any network communication device and does not need to be on the root device. The host processor finds the clock difference between the root clock signal and the corresponding remote clock recovered from the master port. Based on this clock difference, the host processor outputs a control signal to the synchronization circuitry in the root device, which causes the synchronization circuitry to adjust the root clock signal to reduce the difference between the root clock signal and the remote clock recovered from the master port.

[0022] This clock adjustment process is typically repeated, so that the clock difference gradually decreases, and the root clock remains within a small range around the reference clock. All slave devices receive and use this precise reference clock as the source for their own local clocks. Methods and circuit elements for accurately adjusting the root clock in a synchronization circuit are further described in U.S. Patent No. 10,778,406 and U.S. Patent Application No. 16 / 920,772, filed July 6, 2020, which are assigned to the assignee of the present patent application, and the disclosures of which are incorporated herein by reference.

[0023] System Description

[0024] Figure 1 FIG. 1 schematically shows a block diagram of a synchronization communication system 20 according to an embodiment of the present invention. The system 20 in this example includes a distribution unit (DU) 22 that communicates with a plurality of radio units (RUs) 24 in a radio telecommunications network (e.g., a cellular network) via a network communication link 23 (e.g., a high-speed Ethernet link). Such networks are subject to strict synchronization requirements and are thus good examples of the kind of environment in which embodiments of the present invention can be advantageously deployed. Alternatively, the principles of the present invention can be applied to other types of network communication systems that require precise clock control (especially when scalability is important). After reading this specification, such alternative embodiments and applications will be readily understood by those skilled in the art and are considered to be within the scope of the present invention.

[0025] The DU 22 includes a host processor 26, such as a server including a suitable central processing unit (CPU) and memory, which is connected to a plurality of network communication devices via a control link 27, in this case NICs 28, 30, 32, 34,.... Each NIC typically includes an integrated circuit chip or chipset located on a corresponding circuit board. The control link 27 is implemented in such units on a peripheral component bus (e.g., a bus) that connects the host processor to the NICs. Alternatively or additionally, the control link can include a network link, such as an Ethernet link. Each of the NICs 28, 30, 32, 34 includes two ports 38 (labeled port 0 and port 1) that send and receive corresponding communication signals via corresponding network links 23 under the control of a communication circuit 36. Alternatively, each NIC or other network communication device in the DU 22 can include a single port or three or more ports.

[0026] The communication circuit 36 performs network interface and control functions known in the art, including an Ethernet physical layer (PHY) and media access control (MAC) functions in this example. The PHY function includes processing communication signals received by port 38 from network link 23 to recover the corresponding remote clock from each received communication signal. The synchronization circuit 40 in each NIC 28 provides a local clock signal that is applied by the communication circuit 36 to modulate communication signals transmitted to network link 23 through the corresponding port 38. The interface and control functions of the communication circuit 36 are typically implemented in digital logic circuitry, which can be hardwired or programmable. Alternatively or additionally, at least some functions of the communication circuit can be performed by an embedded microprocessor or microcontroller under the control of suitable software or firmware.

[0027] For clock synchronization purposes, one of the network communication devices—in this example, NIC 28—is designated as the root device. The other NICs 30, 32, 34 are designated as slave devices. The synchronization circuit 40 in the root device generates a root clock signal under the control of synchronization control software 44 running on host processor 26. (Details of the circuitry and processing used to generate the root clock are shown in the figure below and will be described below with reference thereto.) The clock link 42 conveys the root clock signal from the root device to the slave devices for use as the local clock signal for the communication circuit 36 in the slave devices.

[0028] Typically, the synchronization circuit 40 in the slave devices (e.g., the synchronization circuits in NICs 30, 32, 34,...) receives the root clock signal from the synchronization circuit of NIC 28 (the root device) and generates a corresponding local clock signal at the same frequency as the root clock signal. Since the slave devices rely on the root clock signal in this way, the synchronization circuits 40 in the slave devices can be simple in their construction and capabilities, for example, including a phase-locked loop (PLL) with appropriate input and output connections. Alternatively, for greater generality and robustness in the configuration and operation of system 20, the slave devices can include more complex synchronization circuits, similar to the synchronization circuit in the root device, as described below.

[0029] The clock link 42 can include, for example, a dedicated cable or printed circuit trace, the length and transmission characteristics of which are selected to ensure that the synchronization circuits 40 in the slave devices can precisely synchronize their respective local clocks with the root clock signal received through link 42. In the illustrated example, the clock link 42 serially connects NIC 28 to NICs 30, 32, and 34. Alternatively or additionally, the clock link can be arranged such that at least some of the slave devices are connected in parallel to the root device (e.g., in a fan-out configuration where each of NICs 30, 32, 34 is directly connected to the synchronization circuit 40 of NIC 28 through a corresponding clock link).

[0030] The host processor 26 is driven by the synchronization control software 44 to interact with the synchronization circuit 40 and control the synchronization of the NICs 28, 30, 32, 34,.... The software 44 can be downloaded to the host processor 26 electronically, for example, via a network. Alternatively or additionally, the software can be stored on a tangible, non-transitory computer-readable medium, such as an optical, magnetic, or electronic storage medium.

[0031] As further described below, the software 44 causes the host processor 26 to select one of the ports 38 of one of the NICs as the master port, for example, port 1 on the NIC 30. (As previously mentioned, the master port can be located on any port of the NIC, including the root device and the slave devices, and the software 44 can even cause the host processor 26 to change the master port from time to time.) The communication circuit 36 in the NIC 30 recovers the remote clock from the signals received on the corresponding network link 23 at the master port and reports the remote clock frequency to the host processor 26. The host processor calculates the clock difference between the frequency of the root clock signal and the remote clock frequency. Alternatively, the communication circuit 36 in the NIC 30 can calculate the clock difference and report it to the host processor. In either case, based on the clock difference, the host processor 26 outputs a control signal to the synchronization circuit 40 on the NIC 28 (root device), instructing the synchronization circuit to adjust the root clock signal to reduce the clock difference. This process is repeated during the operation of the system 20.

[0032] Thus, the root clock signal generated by the synchronization circuit 40 in the NIC 28 is precisely locked to the remote clock recovered from the communication signal received at the master port with low jitter and low drift. When synchronization is achieved with sufficient accuracy, for example, as described below, the frequency of the root clock will converge within a few parts per billion (PPB) of the received signal clock. Therefore, in accordance with SyncE requirements, the NICs 28, 30, 32, 34,... are also able to precisely lock their transmit symbol rate to the receive symbol rate within a few PPB.

[0033] Synchronization Circuit and Process

[0034] Figure 2 is a block diagram schematically showing the details of the communication circuit 36 and the synchronization circuit 40 in the NIC 28 according to an embodiment of the present invention. Figure 2 The elements of the communication circuit 36 shown in are those used in clock recovery and are thus common to all NICs 28, 30, 32, 34,.... Since the NIC 28 is the root device in this example, the synchronization circuit 40 is configured for precise clock adjustment and control. As previously mentioned, the slave devices (e.g., NICs 30, 32, 34) can have similar synchronization circuits, or alternatively, their synchronization circuits can be simpler and can even be limited to the PLL 52 (shown as Figure 2a part of the communication circuit 36 therein).

[0035] The communication circuit 36 includes two receivers 50, 51 that receive input communication signals from respective network links 23 through port 0 and port 1. The receivers 50 and 51 demodulate and buffer the data carried by the signals in respective buffers 54. Clock and data recovery (CDR) circuits 56 operating in each of the receivers 50, 51 recover the remote clock from the received signals, for example, based on transitions in the signal level. The clock recovery can be implemented based on any suitable process known in the art, such as a delay-locked loop or digital oversampling of the input signal. The receivers 50, 51 forward the data received in the buffers to the memory of the host processor 26 via the control link 27 for further processing and forwarding.

[0036] The synchronization circuit 40 generates a clock signal (in this case, the root clock signal), which is transmitted to the PLL 52 and to the other NICs 30, 32, 34 via the clock link 42. Based on this clock signal, the PLL 52 generates a local clock, for example, in the GHz range. The CDR circuits 56 of each of the receivers 50, 51 output the recovered frequency value and / or clock difference, which is the difference between the remote clock frequency recovered from the respective network link 23 and the local clock frequency generated by the PLL 52. The communication circuit 36 transmits the recovered frequency or clock difference to the synchronization control software 44 running on the host processor 26 via the control link 27. Similar clock frequencies or differences are calculated by the receivers in the other NICs and transmitted to the host processor similarly.

[0037] The synchronization control software 44 selects one of the ports 38 on one of the NICs as the master port and generates a control signal based on the clock differences of the respective receivers. The selection of the master port can be controlled by an external network management function. Alternatively or additionally, the software 44 itself can select the master port based on suitable criteria and, if appropriate, can change the selection of the master port during system operation. For example, the master port can be selected in response to a message indicating the clock quality or based on knowledge of the quality of the crystal oscillator driving the clock signal.

[0038] The synchronization circuit 40 in this embodiment includes a drift cleaner 58 that adjusts the frequency of the root clock signal in response to a control signal generated by synchronization control software 44 running on the host processor 26. The drift cleaner 58 can advantageously include a voltage-controlled oscillator (VCO) having a voltage controller that adjusts the voltage applied to the VCO in response to a control signal from the software 44. Depending on the sign of the clock difference calculated at the primary port, the voltage adjustment can be positive or negative and can be applied in very fine steps, such as on the order of parts per billion (PPB) of the clock frequency, or even lower. In one embodiment, the drift cleaner 58 includes an ultra-low jitter network synchronizer clock LMK05318 available from Texas Instruments Incorporated (Dallas, Texas), which uses VCO technology to generate an accurate and stable output clock. This component has inputs for two different clocks, including a crystal oscillator (XO) 60 as its primary frequency source and a temperature-controlled crystal oscillator (TCXO) 62 as the reference frequency source. It is capable of locking the root clock signal to the reference frequency received from the primary port within a few PPB.

[0039] Alternatively, the synchronization circuit 40 can include other types of frequency generation circuits known in the art. For example, the synchronization circuit 40 can include an oscillator having a clock switching circuit and mixers and a PLL. As another example, the synchronization circuit 40 can include an analog or digital frequency synthesizer based on a VCO or other frequency synthesis components. Further details of these types of frequency generation circuits are presented in the above U.S. Patent 10,778,406 and U.S. Patent Application 16 / 920,772.

[0040] Now refer Figure 3 and Figure 4 , which schematically illustrate a clock synchronization method in the system 20 according to an embodiment of the present invention. Figure 3 is a block diagram showing details of the synchronization control software 44 and its interaction with the NICs 28, 30, and 32 during the clock synchronization process, while Figure 4 is a flowchart showing the steps in the method. In this example, the NIC 28 is the root device and is connected to the NICs 30 and 32 via the parallel clock link 42. Port 1 of the NIC 30 has been selected as the primary port.

[0041] As Figure 3 shown, the synchronization control software 44 assigns all the NICs 28, 30, 32 sharing the same root clock to the synchronization group 70. The software 44 running on the host processor 26 can assign all the NICs (or other network devices) connected to the host processor to the same synchronization group, or it can alternately manage multiple groups, each with its own root clock.

[0042] Within each such group 70, the NICs 28, 30, 32 register their respective ports 38 as synchronization providers 72 in a registration step 80( Figure 4 ). The NIC 28 is registered as the root clock source for the group. The software 44 selects a primary synchronization provider, i.e., the primary port (port 1 of NIC 30), in a primary selection step 82. As described above, the NIC 30 measures the clock frequency of the communication signal received on port 1 and transmits the frequency (or the frequency difference relative to the root clock) to the software 44.

[0043] Based on the frequency difference, the software 44 calculates a frequency adjustment to be applied to the root clock in a frequency calculation step 84. As previously described, the adjustment can be positive or negative, and it can be gradually applied in iterative steps, as described in U.S. Patent Application 16 / 920,772. In a clock synchronization step 86, the software 44 drives the host processor 26 to generate a frequency control signal and transmits it to the NIC 28. The frequency control signal can take the form of, for example, a clock synchronization command 74, which is sent to the NIC 28 via the control link 27. The synchronization circuit 40 in the NIC 28 adjusts the root clock accordingly. In a clock transmission step 88, the root clock is continuously transmitted to the NICs 30 and 32 via the clock link 42 through physical transmission.

[0044] Although, for the sake of concreteness and clarity, the above-described embodiments relate to a specific type of network and system configuration, the principles of the present invention can be similarly implemented in other hardware and software environments so as to synchronize not only the NIC communications precisely but also switches and other network devices. All such alternative applications and implementations are considered to be within the scope of the present invention. It should thus be understood that the above-described embodiments are cited by way of example, and the present invention is not limited to what has been specifically shown and described above. On the contrary, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications that would occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

Claims

1. A synchronous communication system, comprising: A plurality of network communication devices, including one network communication device designated as a root device and one or more other network communication devices designated as slave devices, each network communication device comprising: One or more ports configured to transmit and receive respective communication signals via respective network links; and Communication circuitry configured to process the communication signals received by the one or more ports to recover a respective remote clock from each of the communication signals; Synchronization circuitry integrated in the root device and configured to provide a root clock signal to the communication circuitry in the root device; A clock link coupled to transmit the root clock signal from the root device to the slave devices to be used as a local clock signal for the communication circuitry in the slave devices; and A host processor coupled to the network communication devices via a control link and driven by software to select one of the ports of one of the network communication devices as a master port to find a clock difference between the root clock signal and the respective remote clock recovered from the master port, and output a control signal in response to the clock difference to cause the synchronization circuitry to adjust the root clock signal.

2. The system according to claim 1, wherein the host processor is configured to select the master port from the ports of both the root device and the slave devices.

3. The system according to claim 1, wherein the clock link comprises a cable interconnecting the network communication devices independently of the network links.

4. The system according to claim 1, wherein the clock link is configured to transmit the root clock signal from the root device to a first of the slave devices and serially from the first of the slave devices to a second of the slave devices.

5. The system according to claim 1, wherein the clock link is configured to transmit the root clock signal from the root device in parallel to at least a first and a second of the slave devices.

6. The system according to claim 1, wherein the network communication devices include network interface controllers.

7. The system according to claim 1, wherein the network communication devices include network switches.

8. The system according to claim 1, including a peripheral component bus connecting the host processor to the network communication devices, wherein the control link is implemented on the peripheral component bus.

9. The system according to claim 1, wherein the slave devices include respective synchronization circuitry configured to receive the root clock signal from the synchronization circuitry of the root device and generate the local clock signal in response to the root clock signal.

10. The system according to claim 1, wherein the synchronization circuitry includes a voltage controlled oscillator (VCO) and is configured to adjust a voltage applied to the VCO in response to the control signal.

11. A synchronization method, comprising: Connect one or more ports in each of a plurality of network communication devices to transmit and receive respective communication signals via respective network links; Process the communication signals received by the one or more ports to recover a respective remote clock from each of the communication signals; Designate one of the network communication devices as a root device and one or more other network communication devices as slave devices; Provide a root clock signal from a synchronization circuit in the root device to a communication circuit in the root device; Transmit the root clock signal from the root device to the slave devices via a clock link to serve as a local clock signal for the communication circuits in the slave devices; Couple a host processor to the network communication devices via a control link; Select, by the host processor, one of the ports of one of the network communication devices as a master port; Find a clock difference between the root clock signal and the respective remote clock recovered from the master port; and In response to the clock difference, output a control signal that causes the synchronization circuit to adjust the root clock signal.

12. The method according to claim 11, wherein the host processor is configured to select the master port from the ports of both the root device and the slave devices.

13. The method according to claim 11, wherein the clock link includes a cable that interconnects the network communication devices independently of the network links.

14. The method according to claim 11, wherein transmitting the root clock signal includes transmitting the root clock signal from the root device to a first of the slave devices and serially transmitting from the first of the slave devices to a second of the slave devices.

15. The method according to claim 11, wherein transmitting the root clock signal includes transmitting the root clock signal from the root device in parallel to at least a first and a second of the slave devices.

16. The method according to claim 11, wherein the network communication device includes a network interface controller.

17. The method according to claim 11, wherein the network communication device includes a network switch.

18. The method according to claim 11, wherein the control link is implemented on a peripheral component bus.

19. The method according to claim 11, wherein the slave devices include respective synchronization circuits, and the method includes receiving, in the slave devices, the root clock signal from the synchronization circuit of the root device and generating, in response to the root clock signal, the local clock signal for the respective synchronization circuits of the slave devices.

20. The method according to claim 11, wherein the synchronization circuit includes a voltage controlled oscillator (VCO), and providing the root clock signal includes adjusting a voltage applied to the voltage controlled oscillator (VCO) in response to the control signal.

Images