Highly Available Cluster Leader Election in Distributed Routing Systems

Abstract

A distributed routing system is provided for use in a communications network, the distributed routing system including at least one cluster, the at least one cluster including a first plurality of cluster elements, a second plurality of cluster elements selected from the first plurality of cluster elements, each cluster element in the second plurality of cluster elements configured to operate as a cluster leader candidate, and one of the second plurality of cluster elements selected to temporarily function as the cluster leader.

Description

[Technical Field]

[0001] This disclosure relates generally to the field of distributed computing, and more particularly to the operation of distributed routers.

[0002] BPCE: Backplane Cluster Element CE: Cluster Element CL: Cluster Leader CLC: Cluster Leader Candidate CM: Cluster Manager EM: Element Manager FCE: Forwarding Cluster Element LE: Leader Election Container: A container is an isolated execution environment that operates as a full-featured configuration with its own users, file system, processes, libraries, code, network stack, etc., ensuring that you get identical behavior from the software contained within such a container regardless of the system on which it runs. Data Plane: A logical layer containing all functions and processes related to the transfer of data packets / frames from one interface to another through the system. This definition includes, but is not limited to, routing through physical ports as well as routing through intermediate backplane ports that allow data packets to reach physical ports that are not immediately reachable. Control Plane: A logical layer that contains all applications related to all functions and processes that determine the routes used to manage the data plane, etc. This definition includes, but is not limited to, configuration engines, routing stacks, routing protocols, spanning trees, IdPs (identity providers), and user-facing services. [Background technology]

[0003] A distributed system is a system in which components are located on different networked computers, and communicate and coordinate their actions by sending messages to each other. The components interact with each other to achieve a common goal. Three important characteristics of distributed systems are the concurrency of components, the lack of a global clock, and the independent failure of components.

[0004] Computer programs that run in a distributed system are called distributed programs (and distributed programming is the process of writing such programs). Many different types of implementations exist for message passing mechanisms, including pure HTTP, RPC-like connectors, and message queues.

[0005] A computer cluster is a set of loosely or tightly connected computers that work together so that in many respects it is viewed as a single system. Unlike a grid computer, a computer cluster has a set of nodes that each perform the same tasks, controlled and scheduled by software.

[0006] The components of a cluster are typically connected to each other through a high-speed local area network, with each node (using a computer as a server) running its own instance of an operating system. In most situations, all nodes use the same hardware and the same operating system, but in some setups (e.g., using Open Source Cluster Application Resources (OSCAR)), each computer may use a different operating system or different hardware.

[0007] Clusters are typically deployed to improve performance and availability over a single computer, and are typically significantly more cost-effective than a single computer of similar speed or availability.

[0008] However, cluster computing technology poses several challenges, two of which stand out: the first is application complexity, and the second is the synchronicity of cluster elements.

[0009] Application complexity arises from the distributed nature of cluster computing: for example, the architecture applied must be able to solve the problem of how to use network elements when ongoing tasks are divided among them, while at the same time ensuring that from the customer's application perspective it appears to be communicating with a single logical unit.

[0010] On the other hand, component coherence relates to the internal integrity of the system: each data unit shared between components must be synchronized to ensure consistency of behavior across the cluster.

[0011] Such implementations of cluster computing applications use several methodologies to address these challenges. First, in systems where workload management is configured to enable effective load balancing among various cluster elements, application complexity is reduced by allowing cluster elements to remain as independent as possible. Second, a specific centralized entity can be used to handle element synchronization logic and manage all cluster elements. Such a cluster management entity allows for differentiation of information across the cluster and allows the information to be stored as part of the elements' local data and then utilized during cluster operation.

[0012] At any given time, only one cluster manager is active among the cluster elements contained within a cluster, since multiple active cluster managers would undoubtedly cause inconsistencies in outgoing cluster operations. To prevent this drawback, the present invention is directed to a solution in which a cluster leader is elected using a process that selects a single cluster element from all cluster elements contained within the cluster, which then acts as the leader element for that cluster. This leader element then hosts the active cluster manager and performs all other leader element functions necessary to operationally associate the cluster.

[0013] These problems are exacerbated when the cluster is a router. Different cluster elements have different responsibilities, but when working together, the cluster elements provide complete router functionality. Application components in such a cluster can be loosely categorized as belonging to the data plane or control plane, each residing in the appropriate cluster element. Thus, a router consists of multiple cluster elements, each forming part of the data plane or control plane. Figure 1 illustrates the data plane cluster elements in this type of cluster and how they are connected to neighboring routers. In such a cluster, the control plane must run on a single cluster element, which acts as the single source of truth, so it is natural for this cluster element to host the leading cluster manager as part of its control plane services.

[0014] To keep the cluster operational in the event of a control plane cluster element failure, some redundancy of cluster elements is required to be considered suitable candidates to replace the cluster leader. These cluster leader candidates (CLCs) must be highly synchronized to allow seamless failover without degrading routing functionality. It is also necessary that no control plane services remain running on the failed cluster leader. However, this failover should not cause communication issues between cluster leader candidates, and the leader election process should be resilient to minimize control plane migration instances. Finally, the leader election process must handle both cluster leader loss and cluster leader failure scenarios that are common in cluster applications.

[0015] Common deterministic consensus algorithms such as Paxos or Raft have limitations in distributed router applications. Taking a replicated state machine approach in such environments requires synchronization of the control plane stack between cluster leader candidates. The potential amount of data that needs to be synchronized in router applications can cause latency issues and saturate the interconnections of cluster leader candidates. Furthermore, the problem worsens as the number of cluster leader candidates increases, resulting in a quadratic increase in communication overhead between cluster leader candidates. Another limitation here arises from the fact that, unlike common Paxos-like implementations, only a small subset of all cluster elements participate in such approaches as cluster leader candidates, increasing the risk of instability if these select few cluster elements fail. Furthermore, if most cluster elements are not cluster leader candidates, the election algorithm can take advantage of this fact to increase reliability. Finally, from a purely operational perspective, a Paxos-like approach also requires special adaptation of all control plane applications to this specific design, a task that can present unique challenges for primarily software-defined solutions.

[0016] Given the limitations of prior art approaches, a new solution is needed to ensure proper selection of a highly available cluster leader in a distributed router system. Summary of the Invention [Problem to be solved by the invention]

[0017] The present disclosure can be summarized by reference to the appended claims.

[0018] It is an object of the present disclosure to provide a novel method and software for use in a communications network that enables the appropriate selection of a highly available cluster leader in a distributed router system.

[0019] Another object of the present disclosure is to provide novel methods and software for use in a distributed router system, the distributed router system including a cluster manager (CM) running on a cluster leader candidate (CLC) and element managers (EM) running on all cluster elements (CE), where the reliability of the element managers enables reliable, highly available leader election (LE), as well as seamless, lightweight migration of control plane services in the distributed router without impacting outbound operations.

[0020] Other objects of the present disclosure will become apparent from the following description. [Means for solving the problem]

[0021] According to a first embodiment of the present invention, there is provided a distributed routing system for use in a communications network, the distributed routing system comprising at least one cluster, the at least one cluster comprising a first plurality of cluster elements, a second plurality of cluster elements selected from the first plurality of cluster elements (i.e., the second plurality of cluster elements are a subset of the first plurality of cluster elements), each cluster element in the second plurality of cluster elements configured to operate as a cluster leader candidate (CLC), and one of the second plurality of cluster elements is selected to temporarily function as a cluster leader (CL).

[0022] As used herein and in the claims, the term "cluster" is used to refer to a set of loosely or tightly connected computing entities that operate together so that in many respects the cluster appears as a single system. Computer clusters each have a set of nodes that perform the same tasks, controlled and scheduled by software.

[0023] As used herein and in the claims, the term "cluster leader election" or variations thereof refers to a process performed in a distributed system that designates a single entity (or process) belonging to a cluster of similar entities as the organizer of several tasks distributed among multiple elements (nodes). Before a task is initiated, none of the network nodes know which node will act as the "leader" (or coordinator) of the task, or can communicate with the current coordinator. After the leader election algorithm is executed, each node in the network recognizes one and only one node as the task leader.

[0024] As used herein and in the claims, the term "cluster leader candidate" refers to a physical node that runs the cluster manager software during the leader election (LE) negotiation process and, if selected as the node to act as the cluster leader, also runs other control plane components.

[0025] As used herein and in the claims, the term "cluster manager" refers to a process that runs on a cluster leader candidate, i.e., on a designated node in a cluster that participates in the leader election (LE) process.

[0026] As used herein and in the claims, the term "cluster leader" refers to an elected cluster leader candidate node, which includes not only the "acting" cluster manager, but also the rest of the control plane stack.

[0027] Even if there is a currently elected cluster leader, the cluster manager process should still run on other cluster leader candidates to continually monitor for a "working" cluster manager (cluster leader). In this way, these other cluster leader candidates can react if the currently elected cluster leader fails.

[0028] According to another embodiment, the distributed routing system further includes management software configured to manage routing operations within the at least one cluster, the management software being divided into multiple fragments stored in a cluster manager and in managers of different cluster elements.

[0029] In yet another embodiment, each fragment belonging to said plurality of fragments is contained within a respective communication container.

[0030] According to yet another embodiment, the distributed routing system further includes at least one processor (e.g., associated with cluster leader candidates (CLCs)) configured to obtain information from reports provided by one or more cluster element managers (preferably all online cluster element managers), the information regarding which cluster element managers are associated with the second plurality of cluster elements that are eligible to be selected as the cluster leader candidates based on information obtained from each of the one or more cluster element managers.

[0031] In other embodiments, the reports are generated upon occurrence of a change in the distributed routing system and / or on demand.

[0032] In yet another embodiment, the reports are of one of two distinct types: reports on the logical state of a connection reaching a cluster manager, and reports on acknowledgments made by the cluster manager of messages sent along the connection to the cluster manager.

[0033] In another embodiment, if the report is determined to be inconclusive, the at least one processor further comprises initiating forwarding of messages within a time period selected by the cluster leader, the forwarding of the messages being performed by managers of one or more intermediate cluster elements, each of the messages being associated with a request to receive cluster visibility information from a cluster master receiving the message.

[0034] According to another embodiment, the at least one processor is further configured to divide the period during which the cluster leader is selected into a plurality of query periods, the lengths of the plurality of query periods being set according to the interconnection performance of the cluster leader candidates.

[0035] In yet another embodiment, the at least one processor is further configured to accept the cluster state at the end of a query period or start another query period if insufficient information was received during the previous query period.

[0036] According to another aspect of the present disclosure, there is provided a method for selecting a cluster element to function as a cluster leader in a distributed routing system operating in a communications network, the distributed routing system including at least one cluster, the at least one cluster including a first plurality of cluster elements, a second plurality of cluster elements being selected from the first plurality of cluster elements, the method comprising: providing information obtained from reports generated by managers of one or more cluster elements, the information regarding which managers of cluster elements belonging to the cluster are eligible to be selected as cluster leader candidates (i.e., eligible to belong to a second plurality of cluster elements) based on information obtained from each of the managers of the one or more cluster elements; initiating messages during which a cluster leader election is performed, each of these messages being associated with a request to receive cluster visibility information from the managers of all cluster elements receiving each of said messages; selecting a cluster leader from among the cluster leader candidates based on responses received to the message regarding the eligibility of each of the cluster leader candidates to serve as the cluster leader.

[0037] According to another embodiment of this aspect of the present disclosure, the report is generated upon occurrence of a change in the distributed routing system and / or on demand.

[0038] In yet another embodiment, the reports are of one of two distinct types: reports pertaining to the logical state of a connection reaching a cluster manager, and reports pertaining to acknowledgments by the cluster leader of messages sent along the connection to the cluster leader.

[0039] According to yet another embodiment, if the report is inconclusive, the method further comprises the cluster leader initiating messages within a selected period of time, the forwarding of the messages being performed by managers of one or more intermediate cluster elements, each of the messages being associated with a request to receive cluster visibility information from the cluster master receiving the message.

[0040] In another embodiment, the method further includes dividing the period during which the cluster leader is selected into a plurality of query periods, the lengths of the plurality of query periods being set according to the interconnection capabilities of the cluster leader candidates.

[0041] In yet another embodiment, the method further comprises deciding whether to accept the cluster state at the end of the query period or to start another query period if the information received during the previous query period is insufficient.

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments of the present disclosure and, together with the description, serve to explain the principles of these embodiments disclosed herein. [Brief explanation of the drawings]

[0043] [Figure 1] FIG. 1 illustrates a traffic transmission interconnection of distributed routers. [Figure 2] FIG. 1 illustrates the internal control traffic transmission interconnections of a distributed router. [Figure 3] FIG. 1 illustrates a system architecture including various element masters and cluster managers and their connectivity, according to an embodiment of the present invention. [Figure 4] FIG. 1 illustrates a method for carrying out an embodiment according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0044] Some of the specific details and values in the following detailed description illustrate particular examples of the present disclosure. However, this description is illustrative and is not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that the claimed methods and apparatus can be implemented by other techniques known in the art. Furthermore, the embodiments described herein include different steps, not all of which are required in all embodiments of the present invention. The scope of the present invention is summarized by reference to the appended claims.

[0045] The present invention aims to provide a novel partial synchronization approach that ensures the selection of a highly available cluster leader, thereby facilitating the operations that need to be performed by a cluster of communicating elements.

[0046] One of the underlying principles of this disclosure is the use of timeouts available in control plane management protocols. Typically, the time limits are loose enough to allow for complete teardown and startup of control plane services between cluster leader candidates (CLCs) without adversely affecting the state of their peers. Low-latency protocols negotiated over the physical interfaces are preferably handled by the forwarding cluster elements (FCEs) until the cluster manager (CM) is operational with the elected cluster leader (CL). As a result, customer-facing behavior remains unaffected by the transition of control plane services to the new cluster leader.

[0047] By relying on non-participating cluster elements, the proposed solution of this disclosure is intended to be resilient to Byzantine failures. If a cluster element erroneously reports a particular cluster leader candidate as unavailable, other cluster leader candidates can rely on the reports of other cluster elements to avoid the erroneous report. On the other hand, if a cluster element erroneously reports an unavailable cluster leader candidate, the cluster leader candidate can learn the true status of the reported cluster leader candidate by attempting to communicate through that cluster leader candidate.

[0048] Through the use of small-sized, infrequent messages, the proposed solution aims to impose low interconnection overhead. The aforementioned generous timeouts avoid the need for large data storage synchronization, thereby facilitating a low-overhead inter-cluster leader candidate communication protocol.

[0049] Finally, by splitting the leader election (LE) logic between the cluster manager and element-local application managers, microservices can be added and removed without affecting the leader election logic. Splitting the software logic between the element manager and cluster manager, each contained in the appropriate container, ensures the avoidance of Byzantine failures while simultaneously ensuring the isolation of the cluster leader service and the leader election logic.

[0050] FIG. 2 shows an example of a mesh connection of cluster elements that provides such interconnect reliability.

[0051] Having redundancy in switch elements maintains interconnect availability during individual switch failures, and these switch elements can be queried to generate visibility reports that rely on Link Layer Discovery Protocol (LLDP) adjacencies.

[0052] While the cluster manager runs specifically on the cluster leader candidates to negotiate the identity of the elected cluster leader (CL), it uses the cluster element managers running on all cluster elements to do so. After that, the leader election process is configuration agnostic in this respect, so you are free to change the configuration of the application layer, such as the control plane stack.

[0053] Figure 3 is used to further clarify the nature of the relationship between the element manager and the cluster manager, showing the location and relationship of these entities and highlighting the containerized environment to allow them to run alongside other applications on the cluster elements.

[0054] FIG. 4 illustrates a method for selecting cluster leader candidates to function as cluster leaders in a distributed routing system operating in a communication network according to an embodiment of the present disclosure.

[0055] First, information obtained from reports generated by one or more cluster element managers is provided (step 410). This information relates to which cluster leader candidates, i.e., which cluster managers are connected to the reporting element manager or which cluster managers are "visible" to each element manager, whether a particular cluster leader candidate-element manager is eligible to become the cluster leader, and the identity of the cluster leader, if known. The provisioning of these reports by the element manager (EM) to the cluster manager (CM) is preferably acted upon the occurrence of specific changes in the cluster elements or on demand. Thus, the cluster manager's visibility of the cluster elements is always up-to-date.

[0056] According to an embodiment of the present disclosure, requests to provide visibility reports can be classified into two different types: logical requests and immediate requests. The former requests depend on the logical state of the connection to the cluster manager, while the latter requests are requests sent along a specific connection, and the reports generated based on these requests are whether the target cluster manager acknowledged receipt of each message.

[0057] The next step involves initiating messages (step 420) during which cluster leader election is performed. Each of these messages is associated with a request to receive cluster visibility information from the cluster leader candidates that received the respective message. If the received reports are inconclusive, tally requests between cluster leader candidates can be used during leader election. The term "tally" as used here refers to a continuous count of something, in this case the number of element managers that a particular cluster manager has been able to confirm as currently operational, and then by proxy, make the cluster leader candidate the cluster leader (CL).

[0058] A message is sent to each discovered cluster manager through the intermediate element manager, requesting cluster visibility including a tally of the element manager's votes for the cluster leader's ID, unless the cluster manager's response has previously been requested by other cluster elements.

[0059] The Leader Election (LE) period is preferably divided into multiple query periods, the length of which is set by the interconnect capabilities of the cluster leader candidates, and finally the cluster status is either confirmed or rejected, subject to control plane routing protocol tolerances acting as logical or immediate visibility query timeouts. In the latter case, a tally is issued or another query period is initiated if the received information is insufficient.

[0060] Finally, a cluster leader is selected from among these cluster leader candidates based on the responses received to these messages regarding the eligibility of each of the cluster leader candidates to serve as the cluster leader (step 430). The tally is completed when the cluster element receives tally responses from most of the cluster managers it approached, after which the new leader's ID is announced along with all reachable element managers. Should a single cluster leader candidate not receive more tally responses than all other cluster leader candidates, a tally end is sent to the requested cluster manager, relieving the cluster manager from new tally requests, and a random timeout is imposed before attempting to start a new tally. Optionally, a predetermined criterion can be selected to resolve ties between two (or more) cluster leader candidates. For example, in the event of such a tie, the cluster leader candidate is selected that is the node with the lower ID number among the two (or more) cluster leader candidates that received the same number of tally responses.

[0061] The term "ID number" as used herein refers to the requirement that nodes included in a cluster have some kind of ID, used in the context of this disclosure, to correlate visibility reports with the reporters creating the reports. However, as noted above, it can also be used as a tiebreaker in the event of a tie in the number of votes received by two or more cluster leader candidates to become the cluster leader. There are typically no requirements for identifiers other than that they be comparable and unique.

[0062] The present invention has been described using detailed descriptions of embodiments that are provided by way of example only and are not intended to limit the scope of the invention. The described embodiments include different configurations, and not all configurations are required in all embodiments of the invention. Some embodiments of the invention utilize only some of the configurations or possible combinations of configurations. Variations of the described embodiments of the invention, as well as embodiments of the invention that include different combinations of the configurations shown in the described embodiments, will be apparent to those skilled in the art. The scope of the invention is limited only by the following claims.

Claims

1. 1. A distributed routing system for use in a communications network, the distributed routing system comprising: at least one cluster and at least one processor; the at least one cluster includes a first plurality of cluster elements, a second plurality of cluster elements are selected from the first plurality of cluster elements, each cluster element in the second plurality of cluster elements is configured to operate as a cluster leader candidate, and one of the second plurality of cluster elements is selected to temporarily function as a cluster leader; the at least one processor is configured to obtain information from reports provided by one or more cluster element managers, the information regarding which cluster element managers are associated with the second plurality of cluster elements that are eligible to be selected as the cluster leader candidate based on information obtained from each of the one or more cluster element managers; and If the report is inconclusive, the at least one processor further comprises: the cluster leader initiates message forwarding within a selected period of time; the message forwarding is performed by managers of one or more intermediate cluster elements; and each of the messages is associated with a request to receive cluster visibility information from the cluster manager receiving each of the messages.

2. 2. The distributed routing system of claim 1, further comprising management software configured to manage routing operations within the at least one cluster, the management software being divided into multiple fragments stored in a cluster manager and in managers of different cluster elements.

3. 3. The distributed routing system of claim 2, wherein each fragment belonging to the plurality of fragments is contained within a respective communication container.

4. 2. The distributed routing system of claim 1, wherein the reports are generated upon occurrence of a change in the distributed routing system and / or on request.

5. 2. The distributed routing system of claim 1, wherein the reports are of one of two distinct types: reports on the logical state of a connection reaching a cluster manager, and reports on acknowledgments made by a cluster leader of messages sent along the connection to the cluster leader.

6. 2. The distributed routing system of claim 1, wherein the at least one processor is further configured to divide the period during which the cluster leader is selected into a plurality of query periods, the lengths of the plurality of query periods being set by interconnect performance of the cluster leader candidates.

7. 7. The distributed routing system of claim 6, wherein the at least one processor is further configured to accept the cluster state at the end of a query period or to start another query period if insufficient information was received during a previous query period.

8. 1. A method for selecting a cluster element to function as a cluster leader in a distributed routing system operating in a communications network, the distributed routing system including at least one cluster and at least one processor, the at least one cluster including a first plurality of cluster elements, a second plurality of cluster elements being selected from the first plurality of cluster elements, the method comprising: providing information obtained from reports generated by one or more cluster element managers to the at least one processor, the information relating to which cluster element managers belonging to the cluster are eligible to be selected as cluster leader candidates based on information obtained from each of the one or more cluster element managers; initiating messages by said at least one processor during the period during which a cluster leader election is performed, each of said messages being associated with a request to receive cluster visibility information from the managers of all cluster elements receiving each of said messages; If the report is inconclusive, the method further includes the cluster leader initiating message forwarding within a selected time period, the message forwarding being performed by one or more intermediate cluster element managers, each of the messages being associated with a request to receive cluster visibility information from a cluster manager receiving the message; selecting a cluster leader from among the cluster leader candidates based on responses received to the message regarding the eligibility of each of the cluster leader candidates to serve as the cluster leader; A method comprising:

9. 9. The method of claim 8, wherein the reports are generated upon occurrence of a change in the distributed routing system and / or on demand.

10. 9. The method of claim 8, wherein the reports are of one of two distinct types: reports on the logical state of a connection reaching a cluster manager, and reports on acknowledgments made by a cluster leader of messages sent along the connection to the cluster leader.

11. 9. The method of claim 8, further comprising dividing the period during which the cluster leader is selected into multiple query periods, the lengths of the multiple query periods being set by interconnect performance of the cluster leader candidates.

12. 9. The method of claim 8, further comprising deciding whether to accept the cluster state at the end of a query period or to start another query period if insufficient information was received during the previous query period.

Images