Synchronization in distributed communication systems

Abstract

A distributed routing system for use in a communications network is provided, the distributed routing system including at least one cluster having a plurality of cluster elements, wherein the cluster elements used to forward communications traffic among the plurality of cluster elements are synchronized to a single clock among themselves and thereafter synchronized to an external communications element, and optionally, all the cluster elements used to forward communications traffic are configured to implement the IEEE 1588 standard and / or Synchronous Ethernet (Sync-E).

Description

[Technical Field]

[0001] This disclosure relates generally to the field of distributed computing, and more particularly to the operation of distributed routers.

[0002] ASIC: Application-Specific Integrated Circuit eNB: Evolved Node B L2: Layer 2 OOB: Out of Band PCI: Peripheral Component Interconnect PTP: Precision Time Protocol RAN: Radio Access Network SDH: Synchronous Digital Hierarchy SoC: System on a Chip Sync-E: Synchronous Ethernet TOD: Time of Day NCM: Network Cluster Management Control Plane: A logical layer that contains all applications related to all functions and processes that determine the routes used to manage the data plane, etc. This definition includes, but is not limited to, configuration engines, routing stacks, routing protocols, spanning trees, IdPs (identity providers), and user-facing services. PCIe: PCI Express is a term for a high-speed serial expansion bus standard and is a common motherboard interface for various connections, such as Ethernet hardware connections. PCI Express includes high maximum system bus throughput, low I / O pin count and small physical footprint, good performance scaling of bus devices, more detailed error detection and reporting mechanisms (Advanced Error Reporting, AER), hardware support for I / O virtualization, and native hot-swap capabilities. White Box: A commodity that is open or industry standards-based hardware for switches and / or routers in the forwarding plane. White boxes provide users with the fundamental hardware elements of their network. [Background technology]

[0003] Cellular systems have always required strict synchronization. Early implementations relied on transmission systems (such as SDH) as the clock source for the synchronization process. However, as SDH networks in RAN architectures are replaced by packetized networks, SDH systems are no longer applicable. Meanwhile, advances in radio technology and the trend toward using disaggregated radio systems including eNB units have created a need for more precise clock distribution systems that can be implemented in such disaggregated radio systems.

[0004] Industry standard techniques for clock distribution are based on two main protocols. a) The Precision Time Protocol (PTP) is a protocol used to synchronize clocks across computer networks. It achieves clock accuracy in the sub-microsecond range over local area networks, making it suitable for measurement and control systems. This approach provides options for distributing phase and time-of-day (TOD) information. PTP selects a master source of time for the IEEE 1588 domain and each network segment within that domain. Clocks determine the offset between themselves and their master. To synchronize precisely to their masters, clocks must individually determine the network transit time of synchronization (Sync) messages. Transit time is determined indirectly by measuring the round-trip time from each clock to its master, and clocks initiate exchanges with their masters designed to measure transit time. b) Synchronous Ethernet (Sync-E) is an ITU-T standard for computer networks that facilitates the transport of clock signals over the Ethernet physical layer. Such signals are then made traceable to an external clock, providing an option for the distribution of phase information via a dedicated channel over the Ethernet physical layer.

[0005] However, systems such as 5G cellular systems require synchronization accuracy of the order of nanoseconds (ns). Similarly, to reach such high accuracy levels over packetized networks, all network nodes belonging to the system must be subject to clock correction.

[0006] Packet synchronization in prior art solutions was performed at the forwarding ASIC level. Devices called "system-on-chip" (SoC) devices work very well when addressing synchronization issues. However, a single chip is limited in the amount of traffic it can forward, so multi-chip systems were built. The current approach to building multi-chip systems is by using a chassis cage that contains several line cards, each containing a small number of forwarding chips that interconnect with a dedicated fabric card.

[0007] Since all chips need to update their associated clocks, all chips contained within such a chassis cage arrangement need to have a very good level of mutual synchronization between them. The chassis cage is a proprietary device manufactured using a special design. Intra-chassis synchronization is achieved by implementing dedicated synchronization lanes that are designed as part of the chassis cage during the design phase.

[0008] A partitioned and distributed system is a system whose components are located on different network nodes and communicate and coordinate their operations by forwarding messages between each other. Implementing this concept relies on the use of commodity hardware such as white-box network devices and commercially available, off-the-shelf servers.

[0009] White-box network devices are capable of time synchronization when operating as standalone devices, but when they are grouped together to form a distributed cluster, synchronizing the internal clocks among the various cluster components becomes a significant challenge. Clusters are typically deployed to improve performance and availability over single devices (e.g., computers) and are typically more cost-effective than individual devices of comparable speed and availability. However, cluster computing technology poses several challenges. Two of these challenges stand out: the first is application complexity, and the second is synchronization of cluster elements.

[0010] Application complexity arises from the distributed nature of cluster computing: for example, the solution architecture must be able to solve the problem of how to use network elements when the task at hand is divided among them, yet ensure that from the customer application's perspective it appears to be communicating with a single logical unit.

[0011] On the other hand, synchronization of cluster elements is about system consistency: all data units shared between elements must be synchronized to ensure consistency of behavior across the cluster.

[0012] Therefore, a solution is needed, and the solution will ensure that: a. Cluster components are synchronized to the level necessary to meet the requirements of the cellular network. b. The solution should rely on standard components available as off-the-shelf components by commodity hardware vendors. c. The system must appear externally as a single node with respect to the clock mechanism. Summary of the Invention [Problem to be solved by the invention]

[0013] The present disclosure can be summarized by reference to the appended claims.

[0014] An object of the present disclosure is to provide a novel solution for use in a distributed routing system for synchronizing cluster elements configured to forward communication traffic.

[0015] Other objects of the present disclosure will become apparent from the following description. [Means for solving the problem]

[0016] According to a first embodiment of the present disclosure, there is provided a distributed routing system for use in a communications network, the distributed routing system including at least one cluster having a plurality of cluster elements, wherein the cluster elements used to forward communications traffic among the plurality of cluster elements are synchronized to a single clock among the cluster elements, and then all of the cluster elements are synchronized to an external communications element (e.g., a customer clock).

[0017] As used herein and in the claims, the term "cluster" is used to refer to a set of loosely or tightly connected computing entities that operate together so that in many respects the cluster appears as a single system. A computer cluster has each set of nodes performing the same tasks, controlled and scheduled by software.

[0018] According to another embodiment of the invention, all the cluster elements used to transfer communication traffic are configured to implement the IEEE 1588 standard and / or Synchronous Ethernet (Sync-E).

[0019] According to another embodiment, all the cluster elements used to forward communication traffic are configured to be synchronized by using their out-of-band network as an intra-cluster synchronization network.

[0020] According to yet another embodiment, when the out-of-band network includes multiple L2 devices, the multiple L2 devices are synchronized between the L2 devices.

[0021] In yet another embodiment, the distributed routing system further includes a dedicated timing device associated with the cluster elements used to forward communication traffic, and the cluster elements used to forward communication traffic are directly connected to an out-of-band management network.

[0022] In yet another embodiment, the native management port of the cluster element used to forward communication traffic does not support the characteristics required to affect synchronization.

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments of the present disclosure and, together with the description, serve to explain the principles of these embodiments disclosed herein. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 illustrates a distributed routing system according to one embodiment of the present disclosure, where cluster elements included within the distributed routing system are synchronized to an internal plane by implementing a communication standard. [Figure 2] FIG. 1 illustrates a distributed routing system according to another embodiment of the present disclosure, where cluster elements included within the distributed routing system are synchronized to an internal plane by implementing a communication standard. DETAILED DESCRIPTION OF THE INVENTION

[0025] Some of the specific details and values ​​in the following detailed description illustrate particular examples of the present disclosure. However, this description is illustrative and is not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that the claimed methods and apparatus can be implemented by other techniques known in the art. Furthermore, the embodiments described herein include different steps, not all of which are required in all embodiments of the present invention. The scope of the present invention is summarized by reference to the appended claims.

[0026] The present invention aims to provide a solution in which the cluster includes multiple components that are synchronized to the level necessary to meet the requirements of a cellular network.

[0027] One of the fundamental principles of this disclosure is the creation of two synchronization planes. One plane is internal to the cluster and is used to synchronize all forwarding components that are part of the cluster. In this way, it becomes possible to apply accurate and consistent timestamps to all synchronization packets within the cluster forwarding components. Accurate timestamps make it possible to account for time errors introduced by the fact that a cluster configuration is implemented.

[0028] Once all cluster components are synchronized to the internal plane, the cluster management entity is ready to provide synchronization signals received from external customers to cluster members, while also achieving accurate timestamps of forwarded components, enabling synchronization of cluster members at a 5G-level (i.e., on the order of nanoseconds) level.

[0029] Preferably, all cluster components are synchronized internally by implementing standard synchronization techniques such as standard IEEE 1588 and Synchronous Ethernet (Sync-E). The cluster's control plane master in this example becomes the clock source for the cluster members.

[0030] There are various ways to implement the above configuration. For example, since the distributed cluster has an out-of-band management network, this example proposes using that out-of-band network as the intra-cluster synchronization network. All elements belonging to the out-of-band network must support synchronous Ethernet and must understand IEEE 1588. If the out-of-band network consists of several L2 devices, the L2 devices must also support synchronization, as shown in Figure 1.

[0031] According to a second embodiment, a method is provided whereby a dedicated timing device is associated (added) to a forwarding component and the forwarding component is directly connected to an out-of-band network. Such an example of implementation is useful when the native management port of the forwarding component does not support the properties necessary to influence timing operations. This example is illustrated in Figure 2.

[0032] The exact methods for implementing synchronization processes such as Synchronous Ethernet and IEEE 1588 are well known to those skilled in the art.

[0033] We now consider example steps that may be performed while establishing a cluster.

[0034] An external synchronization network is connected to the cluster either through the use of a customer-facing port or through the use of a dedicated synchronization port on the cluster's control plane master. The letters "M" and "S" shown in both Figures 1 and 2 refer to "master" and "slave" ports, respectively, and are responsible for implementing IEEE 1588 and / or Synchronous Ethernet operations.

[0035] When a cluster is started, the system's main oscillator (residing on the control plane master) goes into a "free-running" state, and the devices contained within the white boxes are configured to be fully unsynchronized and ignore synchronization-related packets at this stage. When a packet reaches the main oscillator, the main oscillator begins to align its phase and time to the clock source, and initiates synchronization of the devices contained within the white boxes. As the accuracy of the overall synchronization process increases, the devices contained within the white boxes begin time-stamping the packets. The implementation of time-stamping allows the main system oscillator to achieve higher synchronization accuracy and better synchronization of the white boxes. The synchronization process continues until the internal synchronization plane allows the overall synchronization plane to reach the required (predefined) level of accuracy.

[0036] The master NCM illustrated in Figures 1 and 2 is a network cluster management element, an out-of-band (OOB) L2 network component configured to manage a distributed cluster.

[0037] The present invention has been described using detailed descriptions of embodiments that are provided by way of example only and are not intended to limit the scope of the invention. The described embodiments include different configurations, and not all configurations are required in all embodiments of the invention. Some embodiments of the invention utilize only some of the configurations or possible combinations of configurations. Variations of the described embodiments of the invention, as well as embodiments of the invention that include different combinations of the configurations shown in the described embodiments, will be apparent to those skilled in the art. The scope of the invention is limited only by the following claims.

Claims

1. 1. A distributed routing system for use in a communications network, the distributed routing system including at least one cluster having a plurality of cluster elements, the distributed routing system characterized in that the cluster elements used to forward communications traffic are synchronized to a single clock among the cluster elements, thereby applying accurate and consistent timestamps to all synchronized packets within the cluster forwarding elements, and the cluster elements are then synchronized to external communications elements.

2. 2. The distributed routing system of claim 1, wherein all cluster elements used to forward communication traffic are configured to implement the IEEE 1588 standard and / or Synchronous Ethernet, Sync-E.

3. 3. The distributed routing system of claim 2, wherein all cluster elements used to forward communication traffic are configured to be synchronized by using their out-of-band network as an intra-cluster synchronization network.

4. 4. The distributed routing system of claim 3, wherein when the out-of-band network includes a plurality of layer 2 devices, the plurality of layer 2 devices are synchronized among the layer 2 devices.

Images