A distributed cluster master selection method and server

Abstract

The present application relates to the field of computer technology, and in particular to a distributed cluster master selection method and server. The method comprises: a first master control node monitors the link between the first master control node and each service node; when an abnormality occurs in the link between the first master control node and at least P service nodes among all service nodes, the first master control node triggers a master selection process, and the master selection process is used to reselect a control node from multiple control nodes as a second master control node, and the second master control node is used to replace the first master control node and control the service node to execute the service, wherein P is an integer greater than or equal to 1. The method can improve the stability of the service node in executing the service.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a distributed cluster leader selection method and server. Background Art

[0002] With the development of computer technology and communication technology, many business systems adopt distributed cluster architecture to achieve better load balancing, etc., so as to better provide relevant services to users. In the distributed cluster architecture, different functions are distributed on different nodes in the cluster, and these nodes jointly implement the business. Generally, a distributed cluster includes a control subcluster composed of multiple control nodes and a business subcluster composed of multiple business nodes. Among them, the business nodes in the business subcluster are used to execute business. The control nodes in the control subcluster are used to control the business nodes, for example, to which business nodes the business request is assigned so that these business nodes can process the business request.

[0003] In the control subcluster, one control node serves as the master node, and the other control nodes serve as slave nodes. The master node is used to control the service node. When the status of the master node is abnormal, it is necessary to reselect the master node from the control subcluster. Currently, the status of the master node is monitored by the slave node. Therefore, it may happen that the monitoring result of the master node by the slave node shows that the status of the master node is normal, but the link between the master node and the service node is abnormal. In this case, the master node cannot effectively control the service node, and the control subcluster cannot re-elect the master node, which makes it difficult for the service subcluster to provide services. Summary of the invention

[0004] The embodiments of the present application provide a distributed cluster leader election method and server, which can improve the stability of service node execution.

[0005] In a first aspect, a distributed cluster master election method is provided, wherein the distributed cluster comprises multiple control nodes and multiple service nodes, wherein one of the multiple control nodes serves as a first master control node, and is used to control the service node to execute services; the method comprises: the first master control node monitors the link between the first master control node and each of the service nodes; when an abnormality occurs in the link between the first master control node and at least P service nodes among all the service nodes, the first master control node triggers a master election process, and the master election process is used to reselect a control node from the multiple control nodes as a second master control node, and the second master control node is used to replace the first master control node to control the service nodes to execute services, wherein P is an integer greater than or equal to 1.

[0006] The method determines whether to trigger the master election process by monitoring the link between the master control node and the business node, so that when the link between the master control node and the business node is abnormal, the master control node can be reselected or elected, thereby ensuring that the business node can be effectively controlled by the master control node and improving the stability of business execution.

[0007] In a possible implementation, the first master control node senses the state of the link between the first master control node and the service node through the heartbeat message between the first master control node and the service node; the first master control node monitors the link between the first master control node and each of the service nodes, including: the first master control node counts the number of lost heartbeat messages between the first master control node and the service node within a preset time period; when the number of lost heartbeat messages between the first master control node and the service node is not less than a first threshold, it is confirmed that an abnormality has occurred in the link between the first master control node and the service node.

[0008] In a possible implementation, when the multiple control nodes elect a master control node by voting, the method further includes: the first master control node does not vote for the first master control node; or, the distributed cluster further includes a controller, and the method further includes: delaying the start time of the first master control node attempting to create a temporary node under the controller path; or, the distributed cluster further includes a sentinel, and the method further includes: the sentinel selects a second master control node from other control nodes, and the other control nodes are nodes among the multiple control nodes other than the first master control node.

[0009] This implementation can reduce the possibility that the original master control node becomes the master control node again, thereby increasing the possibility that a good link exists between the new master control node and the service node.

[0010] In a possible implementation manner, the first master control node triggering a master election process includes: the first master control node instructing each of the multiple control nodes to enter the master election process.

[0011] In one possible implementation, the multiple control nodes include at least one first control node and at least two second control nodes, the first master control node belongs to the second control node, and the first master control node triggers the master election process including: the first master control node notifies the first control node; the first control node instructs each of the at least two second control nodes to enter the master election process, or the first control node designates one of the at least two second control nodes as the second master control node.

[0012] In a possible implementation, the method further includes: each of the multiple control nodes monitors the state of the first master control node, and enters the master election process when it is detected that the state of the first master control node is abnormal.

[0013] This implementation is compatible with the existing master election process triggering mechanism, thereby ensuring that the master election process can be triggered in a variety of situations.

[0014] In a second aspect, a first server is provided, wherein the first server is communicatively connected to multiple second servers, and the first server includes a processor and a receiving port coupled to the processor; wherein the receiving port is used to receive a heartbeat message from each second server; the processor is used to monitor the link between the first server and the second server based on the heartbeat message from the second server; and, when an abnormality occurs in the link between the first server and at least P of all the second servers, trigger a master election process, wherein the master election process is used to reselect a server from multiple control servers as a master control server, and the master control server is used to replace the first server to control the second server to perform services, wherein P is an integer greater than or equal to 1.

[0015] In a possible implementation, the processor is also used to: count the number of lost heartbeat messages between the first server and the second server within a preset time period; when the number of lost heartbeat messages between the first server and the second server is not less than a first threshold, confirm that an abnormality has occurred in the link between the first server and the second server.

[0016] In a possible implementation manner, the processor is further configured to: instruct each of the multiple control servers to enter the master selection process.

[0017] In a possible implementation, the first server is also connected to a management server for communication, and the processor is further used to notify the management server so that the management server instructs each of the control servers to perform a master selection process or designate one of the multiple control servers as a master control server.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method provided in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the structure of a distributed cluster provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the structure of a distributed cluster provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of a distributed cluster provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a distributed cluster provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of a distributed cluster leader election method provided in an embodiment of the present application;

[0025] Figure 6 A flowchart of a distributed cluster leader election method provided in an embodiment of the present application;

[0026] Fig. 7A A schematic diagram of a heartbeat message statistical result provided in an embodiment of the present application;

[0027] Figure 7B A schematic diagram of a heartbeat message statistical result provided in an embodiment of the present application;

[0028] Figure 7C A schematic diagram of a heartbeat message statistical result provided in an embodiment of the present application;

[0029] Fig.7D A schematic diagram of a heartbeat message statistical result provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a lost message statistics result provided in an embodiment of the present application;

[0031] Fig. 9A A schematic diagram of the structure of a distributed cluster switching master control node provided in an embodiment of the present application;

[0032] Fig. 9B A schematic diagram of the structure of a distributed cluster switching master control node provided in an embodiment of the present application;

[0033] Fig. 9C A schematic diagram of the structure of a distributed cluster switching master control node provided in an embodiment of the present application;

[0034] Fig.10 A schematic block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0036] In the description of the embodiments of the present application, "one embodiment" or "some embodiments" etc. means that one or more embodiments of the embodiments of the present application include a specific feature, structure or characteristic described in conjunction with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the different parts of the embodiments of the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0037] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0038] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0039] A distributed cluster consists of multiple nodes. These nodes may include multiple control nodes and multiple service nodes. Among them, multiple control nodes may include master nodes and slave nodes. The master node may also be called the cluster master or leader, which is used to control the service nodes to execute services, such as allocating service requests and monitoring whether the service nodes can work normally. The slave node may also be called a cluster backup, backup node, or follower. The slave node serves as a backup for the master node and is used to replace the abnormal master node and become the new master node when the master node is abnormal. The service node may also be called an observer or a cluster member, which is controlled by the master node and provides specific service services.

[0040] Taking a distributed storage cluster that provides data storage services as an example, the master control node can record a mapping table between business nodes and data. The mapping table includes multiple mappings, where the mapping indicates that the business node stores data mapped to the business node. When the master control node receives an access request, it can query the mapping table for the business node corresponding to the data requested by the access request, and then forward the access request to the business node.

[0041] The master node is crucial to the business implementation of the distributed cluster. When an abnormality occurs in the master node, it needs to be replaced as soon as possible to ensure that the cluster business can be implemented normally.

[0042] Through the master election process, a control node can be selected from multiple control nodes in the distributed cluster as the master node. Among them, common master election processes include the master election process under the zookeeper voting mechanism, the master election process under the kafka controller election mechanism, and the master election process under the redis sentinel designation mechanism. Among them, zookeeper is a distributed coordination service that can be used for service discovery, distributed locks, distributed master node election, configuration management, etc.

[0043] There are two scenarios that trigger the leader election process under the Zookeeper voting mechanism: the distributed cluster startup node and the master node exception during the operation of the distributed cluster. The specific process of the leader election process in these two scenarios is basically the same. The following takes the scenario of the master node exception during the operation of the distributed cluster as an example to explain the leader election process.

[0044] Each control node in the distributed cluster detects the status of the master node. Among them, the backup node monitors the status of the master node by monitoring the heartbeat message sent by the master node to the backup node. When the heartbeat message times out, the backup node can confirm that an abnormality has occurred in the master node. Among them, the heartbeat message timeout refers to the loss of N consecutive heartbeat messages. Among them, N is an integer greater than 1, and the value of N can be freely set by the developer. For the corresponding backup node, the heartbeat message timeout of the master node can indicate that the heartbeat of the master node is disconnected, that is, the backup node cannot detect the heartbeat of the master node.

[0045] When the backup node confirms that the master node is abnormal, the backup node can enter the election state. The election state can also be called the looking state, which is a state of electing the master node. The master node can monitor its own state by monitoring its own process, so as to find out whether the state of the master node is abnormal. When the master node finds that it is abnormal, it also enters the election state.

[0046] After entering the election state, each control node will vote for itself, called the master control node, and then broadcast the vote to other control nodes. Each control node receives the ballots sent by other control nodes and performs a legitimacy check. The legitimacy check includes checking the election round in which the ballot is located and checking the status of the control node that sent the ballot.

[0047] The control node compares the votes cast for itself and the votes cast for other control nodes. The specific comparison rules are as follows: compare the transaction identifiers (ZXID) of the control nodes, and the larger one wins; then, if the transaction identifiers of the votes are the same, compare the transaction identifiers (MYID) of the control nodes themselves, and the larger one wins. Among them, the control node of the loser casts the vote for the winner; then, the loser and the winner each broadcast their updated votes to other control nodes. If the number of votes of a node exceeds half of the total number of control nodes in the distributed cluster, the control node is elected as the new master node. If not, the next round of comparison and voting is carried out. Among them, when electing the master node, the transaction identifier of the control node is compared first, which can ensure that the data of the new master node is the latest. In addition, the number of control nodes in the distributed cluster using the zookeeper voting mechanism is an odd number to avoid the situation where the number of votes does not exceed half.

[0048] The new master control node updates its identity to the leader, and the identities of other control nodes are updated to or continue to be followers.

[0049] In the Kafka controller election mechanism, the control nodes in the distributed cluster can be divided into high-level control nodes and low-level control nodes, where there is at least one high-level control node and multiple low-level control nodes. The low-level control node can also be called a cache broker. The master election process under the Kafka controller election mechanism is that the high-level control node is used to select the master control node from the low-level control nodes. The specific process is as follows.

[0050] A high-level control node can be a controller. Multiple low-level control nodes use a competition mechanism to create temporary nodes under the controller path. The low-level control node that successfully creates a temporary node is called a master control node. The controller can monitor the status of the temporary nodes under its path. When the status of a temporary node is abnormal, the controller deletes the temporary node and notifies all low-level control nodes to use a competition mechanism to re-create temporary nodes under the path, thereby selecting a new master control node.

[0051] The control nodes in the distributed cluster using the redis sentinel designation mechanism can be divided into sentinels and ordinary control nodes. Among them, sentinels can also be understood as high-level control nodes, and ordinary control nodes can also be understood as low-level control nodes. Among them, there is at least one sentinel and multiple ordinary control nodes. The master selection process under the sentinel designation mechanism is: the sentinel designates the master control node from multiple ordinary control nodes and monitors the master control node and backup nodes. When the status of the master control node is abnormal (the sentinel receives the heartbeat message sent by the master control node for a timeout), the sentinel can again designate a new master control node from multiple ordinary control nodes.

[0052] In the above mechanism, the master election process is triggered only when the backup node or the advanced control node finds that the state of the master node is abnormal. In particular, in the zookeeper voting mechanism and the sentinel designation mechanism, the backup node or sentinel needs to monitor the heartbeat disconnection of the master node (that is, the heartbeat message timeout) to confirm that the state of the master node is abnormal, and then trigger the master election process. A common situation in reality is that the backup node or the advanced control node does not find the master node abnormal, and due to the link or network instability (such as congestion, packet loss) between the business node and the master node, the master node and the business node cannot communicate normally, which in turn affects the execution of the business.

[0053] The embodiment of the present application provides a distributed cluster master election method, which can monitor the link between the master control node and the business node, and trigger the master election process when an abnormality occurs in the link between the master control node and the business node, thereby selecting a new master control node to control the business node to execute the business, thereby ensuring that the business can be executed normally.

[0054] Next, the distributed cluster leader election method provided in the embodiment of the present application is specifically introduced.

[0055] First, the system architecture to which the distributed cluster leader election method provided in the embodiment of the present application can be applied is introduced.

[0056] Figure 1A schematic diagram of the structure of a distributed cluster 100 provided in an embodiment of the present application is shown. The distributed cluster 100 includes: a plurality of control nodes such as a control node 111, a control node 112, and a control node 113, and a plurality of service nodes such as a service node 121, a service node 122, a service node 123, a service node 124, a service node 125, and a service node 126. Among them, the plurality of control nodes can elect a master control node through the zookeeper voting mechanism. For example, the election control node 111 is used as the master control node. The master control node can control each of the plurality of service nodes to perform services. After the master control node is elected, the master control node can establish a communication link with each of the plurality of service nodes to exchange information with the service nodes through the communication link, thereby controlling the service nodes to perform services. Among them, the master control node also establishes a communication link with other control nodes, so that other control nodes can monitor the status of the master control node through the communication link.

[0057] Figure 2 The structural diagram of another distributed cluster 200 provided in an embodiment of the present application is shown. The distributed cluster 200 includes: a plurality of first-level control nodes such as a first-level control node 211, a first-level control node 212, and a first-level control node 213, and a plurality of service nodes controlled by the first-level control node such as a service node 221, a service node 222, and a service node 223. Among them, the plurality of first-level control nodes can elect a first-level master control node through the zookeeper voting mechanism. For example, the first-level control node 211 is elected as the master control node. Among them, the first-level master control node can control the service nodes 221, the service nodes 222, the service nodes 223, and the like, which are controlled by the first-level control node, to perform services. After the first-level master control node is elected, the first-level master control node can establish a communication link with each service node in the plurality of service nodes controlled by the first-level control node, so as to exchange information with the service node through the communication link, thereby controlling the service node to perform services. Among them, the first-level master control node also establishes a communication link with other first-level control nodes, so that other first-level control nodes can monitor the status of the first-level master control node through the communication link.

[0058] The distributed cluster 200 may further include: a plurality of secondary control nodes such as secondary control node 231 , secondary control node 232 , and secondary control node 233 , and a plurality of service nodes such as service node 241 , service node 242 , and service node 243 controlled by the secondary control nodes.

[0059] In some embodiments, the authority of the primary control node is greater than the authority of the secondary control node. That is, relative to the secondary control node, the primary control node is a high-level control node. In other words, relative to the primary control node, the secondary control node is a low-level control node. The authority of the primary control node is greater than the authority of the secondary control node. Specifically, the primary master control node elected from multiple primary control nodes can manage or control the secondary control node, for example, directly designate the master control node from multiple secondary control nodes.

[0060] In some embodiments, multiple secondary control nodes can elect a secondary master control node through the zookeeper voting mechanism. For example, the secondary control node 311 is elected as the master control node. Among them, the secondary master control node can control multiple service nodes controlled by the secondary control node, such as service node 241, service node 242, and service node 243, to perform services.

[0061] After the secondary master control node is elected, the secondary master control node can establish a communication link with each of the multiple service nodes controlled by the secondary control node to exchange information with the service node through the communication link, thereby controlling the service node to execute services. The secondary master control node also establishes a communication link with other secondary control nodes, so that other secondary control nodes can monitor the status of the secondary master control node through the communication link.

[0062] Figure 3 A schematic diagram of the structure of another distributed cluster 300 provided in an embodiment of the present application is shown. The distributed cluster 300 includes: multiple high-level control nodes such as high-level control node 311, high-level control node 312, and high-level control node 313, multiple low-level control nodes such as low-level control node 321, low-level control node 322, and low-level control node 323, and service nodes such as service node 331, service node 332, service node 333, service node 334, service node 335, and service node 336.

[0063] Among them, the multiple high-level control nodes can elect high-level master control nodes through the zookeeper voting mechanism, for example, electing high-level control node 311 as the high-level master control node. Among them, the high-level master control node is used as the controller in the kafka controller election mechanism, and multiple low-level control nodes adopt a competition mechanism to create temporary nodes under the controller path. The low-level control node that successfully creates the temporary node is used as the low-level master control node. The low-level master control node is used to control multiple business nodes in the cluster 300 to execute business.

[0064] The high-level master control node can establish a communication link with other high-level control nodes so that other high-level control nodes can monitor the status of the high-level master control node. In addition, the high-level master control node can establish a communication link with each low-level control node so that the low-level control node can create a temporary node under the high-level master control node path.

[0065] The low-level master control node and each service node can establish a communication link so that the low-level master control node can control the service node to perform services. In addition, the low-level master control node and other low-level control nodes can establish a communication link so that other low-level control nodes can monitor the status of the low-level master control node.

[0066] Figure 4 A schematic diagram of the structure of another distributed cluster 400 provided in an embodiment of the present application is shown. The distributed cluster 400 includes: a plurality of control nodes such as a control node 411, a control node 421, a control node 422, and a control node 423, and service nodes such as a service node 431, a service node 432, a service node 433, a service node 434, a service node 435, and a service node 436.

[0067] Among them, the distributed cluster 400 can adopt the redis sentinel designation mechanism to select the master control node. Among them, the control node 411 can be used as a sentinel, and the other control nodes are used as ordinary control nodes. The sentinel can designate a master control node from multiple ordinary control nodes. The master control node is used to control multiple service nodes in the cluster 400 to execute services.

[0068] The sentinels and control nodes 423 and 422 are connected to the master control node 421 to monitor the status of the master control node 421. In addition, each service node is connected to the master control node 421 so that the master control node 421 can control the service node to perform services.

[0069] The above examples introduce system architectures that can be used to implement the distributed cluster leader election method provided in the embodiments of the present application, which are not exhaustive. The distributed cluster leader election method provided in the embodiments of the present application can also be implemented in distributed clusters using other architectures, which are not listed here one by one.

[0070] In the embodiment of the present application, when no special distinction is made between control nodes, service nodes, high-level control nodes, low-level control nodes, and sentinels, they may be referred to simply as nodes.

[0071] In some embodiments, the node can be implemented by hardware. For example, the execution node can be a computing device, such as a server. For another example, the node can be one or more computing cores on a chip, a processor, or a multi-core processor. In some embodiments, the node can be implemented by software. For example, the node can be an application (application, APP). For another example, the node can be one or more processes (process). For another example, the node can be one or more threads (thread). The specific implementation form of the node is no longer specifically limited in the embodiments of the present application.

[0072] The above example introduces the system architecture of the distributed cluster and the implementation of the node. Figure 1-Figure 4 Taking the application in the distributed cluster shown as an example, the execution process of the distributed cluster leader election method provided by the embodiment of the present application is introduced.

[0073] See also Figure 5 , the method comprises the following steps.

[0074] Step 501: A first master control node monitors a link between the first master control node and each of the service nodes.

[0075] The first master node refers to the current master node of the distributed cluster. Figure 1 As an example, the first master control node refers to the control node 111 used as the master control node. Figure 2 Taking the architecture shown in the figure as an example, the first master control node refers to the primary control node 211 or the secondary control node 231. Figure 3 Taking the architecture shown in the figure as an example, the first master control node refers to the low-level control node 321. Figure 4 Taking the illustrated architecture as an example, the first master control node refers to the control node 421 .

[0076] The link between the master control node and the service node refers to a network channel for information exchange between the master control node and the service node. For example, the master control node can forward a service request to the service node through the link.

[0077] In step 501, the first master control node or the current master control node may monitor whether the link between it and each service node is abnormal.

[0078] In some embodiments, the first master control node can sense the state of the link between the first master control node and each service node through a heartbeat message between the first master control node and the service node. Figure 1Take the master control node 111 and the service node 121 in the illustrated architecture as an example. The service node 121 can send a heartbeat message to the master control node 111 so that the master control node 111 can sense the existence of the service node 121. Whether the heartbeat message sent by the service node 121 to the master control node 111 can reach the master control node 111 normally is limited by the state of the link A1 between the master control node 111 and the service node 121. Therefore, the master control node 111 can learn or judge the state of the link A1 between the master control node 111 and the service node 121 by receiving the heartbeat message sent by the service node 121. Among them, the state of the link A can be divided into a normal state and an abnormal state. The details are as follows.

[0079] The service node 121 may send a heartbeat message to the master control node 111 according to a preset sending cycle. That is, a heartbeat message is sent to the master control node 111 every time period t1. The time period t1 is the period length of the sending cycle. The length of the time period t1 may be preset, and the master control node 111 may obtain the time period t1.

[0080] The master control node 111 can count the number of lost messages between the master control node 111 and the service node 121 in the time period T1. The duration of the time period T1 is M times that of the time period t1. That is, in the time period T1, the service node 121 sends M heartbeat messages to the master control node 111. M is an integer greater than 1. Lost messages refer to messages sent by the sender but not received by the receiver. Lost messages between the master control node 111 and the service node 121 refer to messages sent by the service node 121 to the master control node 111 but not received by the master control node 111. Specifically, in the time period T1, the master control node 111 received a total of (MN) heartbeat messages from the service node 121. As mentioned above, in a time period T1, the service node 121 sent M heartbeat messages to the master control node. Then the number of lost messages = M-(MN) = N. N is an integer less than M and greater than or equal to 0.

[0081] The N lost messages may be N heartbeat messages that are lost continuously, or the sending time of the N lost heartbeat messages is adjacent to each other. That is to say, the master control node 111 does not receive the heartbeat message sent by the service node 121 in N consecutive time periods t1. The continuously lost heartbeat messages can be called continuously lost messages. In an example, Fig. 7A As shown, the duration of time period t1 can be set to 1 second (s). At the 4th, 5th and 6th seconds, no heartbeat message sent by service node 121 is received, and there are 3 consecutive heartbeat messages lost between master node 111 and service node 121. In another example, Figure 7BAs shown, the duration of time period t1 can be set to 1 second. At the 10th second, the 11th second and the 12th second, no heartbeat message sent by the service node 121 is received, and there are 3 consecutive heartbeat messages lost between the master control node 111 and the service node 121.

[0082] The N lost messages may be N heartbeat messages lost at intervals, that is, the sending time of the N lost messages is not consecutive. Heartbeat messages lost at intervals can be called interval lost messages. In an example, Figure 7C As shown, the time period t1 is set to 1 second. At the 1st second, the 3rd second, the 4th second, the 6th second and the 8th second, no heartbeat message sent by the service node 121 is received. There are 5 heartbeat messages lost between the master node 111 and the service node 121. In another example, Fig.7D As shown, with the time period t1 set to 1 second, no heartbeat message sent by the service node 121 is received at the 6th second, the 8th second, the 9th second, the 11th second and the 13th second, and there are 5 heartbeat messages lost at intervals between the master control node 111 and the service node 121.

[0083] In some embodiments, Figure 7A-7D As shown, a sliding window statistical mechanism can be used, with time period T1 as the time window and the length of time period t as the step length (that is, the start time of a time window differs from the start time of the previous time window by a time period t, that is, a heartbeat message sending interval), and the number of lost messages in each time window is counted in turn.

[0084] In some embodiments, the duration of time period T1 is less than the heartbeat timeout duration between the master control node and the monitoring node on the control plane. The monitoring node on the control plane of the master control node refers to the backup node or sentinel corresponding to the master control node. Figure 1 Taking the architecture shown in FIG. 1 as an example, the monitoring nodes on the control plane of the master control node 111 refer to the control node 112 and the control node 113. Figure 4 Taking the architecture shown as an example, the monitoring node on the control plane of the master control node 4211 refers to the control node 411. In this way, when the heartbeat between the master control node and the monitoring node on the control plane has not timed out, the link abnormality between the master control node and the service node can be monitored, thereby triggering the master election process.

[0085] Back to Figure 5The distributed cluster master election method provided in the embodiment of the present application also includes: step 502, when an abnormality occurs in the link between the first master control node and at least P of the service nodes among all the service nodes, the first master control node triggers a master election process, the master election process is used to reselect a control node from the multiple control nodes as the second master control node, the second master control node is used to replace the first master control node, and control the service nodes to execute services. Among them, all service nodes refer to all service nodes controlled by the first master control node in the distributed cluster.

[0086] P is an integer greater than or equal to 1, and P is less than or equal to the number of service nodes controlled by the first master control node. Exemplarily, P is a preset value and can be freely configured, for example, P is 3, 5, or 7, etc. Exemplarily, the size of P is positively correlated with the number M1 of service nodes controlled by the first master control node. Wherein, the ratio of P to M1 is less than or equal to 1. In one example, the ratio of P to M1 is equal to 1 / 3. In another example, the ratio of P to M1 is equal to 1 / 2. And so on, which are not listed here one by one.

[0087] In step 502, the first master control node can obtain or determine the state of the link between the first master control node and the service according to the monitoring result of step 501, and when the state of the link between the first master control node and P service nodes is abnormal, the first master control node triggers the master election process.

[0088] Next, the specific process of triggering the master election process is described by taking the master control node 111 as an example. For the convenience of description, a service node whose link with the master control node is in an abnormal state may be referred to as an abnormal node.

[0089] See also Figure 6 In step 601 , the master control node 111 may determine abnormal nodes and count the number of abnormal nodes.

[0090] Taking determining whether the service node 121 is an abnormal node as an example, whether the service node is an abnormal node can be determined by method 1, which is: determining whether the number of lost messages between the control node 111 and the service node 121 is not less than the threshold value Y1 within the time period T1. The threshold value Y1 is a preset value. When the length of the time period is M times the time period t1, the threshold value Y1 is an integer less than or equal to M and greater than or equal to 1. When the number of lost messages within the time period T1 is not less than the threshold value Y1, it can be confirmed that the state of the link A1 is an abnormal state. When the number of lost messages within the time period T1 is less than the threshold value Y1, it can be confirmed that the state of the link A1 is a normal state, that is, no abnormality occurs.

[0091] In some embodiments, whether the service node is an abnormal node can also be determined by mode 2. As described above, the lost message can be a continuous lost message and an interval lost message. Accordingly, the threshold Y1 can be divided into a threshold Y11 and a threshold Y12, wherein the threshold Y11 is less than the threshold Y12. Among them, when the number of continuous lost messages is large, it reflects that the link may be interrupted. Therefore, the state of the link can be determined according to the number of continuous lost messages (mode 21). The number of interval lost messages reflects the packet loss rate of the link. Therefore, the state of the link can be determined according to the number of interval lost messages (mode 22). As long as either mode 21 or mode 22 determines that the service node is an abnormal node, the service node is an abnormal node. The accuracy of the service node status judgment can be ensured by jointly determining whether the service node is abnormal by two methods.

[0092] In other embodiments, it is also possible to determine whether a service node is an abnormal node only through method 1 or method 2, or to determine whether a service node is an abnormal node only through method 21 or method 22. This solution can reduce the computational burden to a certain extent.

[0093] When the number of lost packets in the time period T1 obtained by counting is the number of consecutive lost packets, it is determined whether the number of consecutive lost packets is not less than the threshold value Y11. When the number of consecutive lost packets is not less than the threshold value Y11, it can be confirmed that the state of link A1 is abnormal. In an example, if Fig. 7A As shown, in the time period T1 starting at the 0th second and ending at the 10th second, there are three consecutive heartbeat messages lost between the master node 111 and the service node 121. When the threshold Y11 is 3, that is, the number of consecutively lost messages is not less than the threshold Y11, it can be confirmed that an abnormality has occurred in link A1, that is, the state of link A1 is abnormal. In another example, Figure 7B As shown, in the time period T1 starting at the 2nd second and ending at the 12th second, there are three consecutively lost heartbeat messages between the master node 111 and the service node 121. When the threshold Y11 is 3, that is, the number of consecutively lost messages is not less than the threshold Y11, it can be confirmed that an abnormality has occurred in link A1.

[0094] When the number of lost messages in the time period T1 obtained by counting is the number of interval lost messages, it is determined whether the number of interval lost messages is not less than the threshold value Y12. When the number of interval lost messages is not less than the threshold value Y12, it can be confirmed that the state of link A1 is abnormal. In one example, Figure 7CAs shown, in the time period T1 starting at the 0th second and ending at the 10th second, there are 5 heartbeat messages lost between the master node 111 and the service node 121. When the threshold Y12 is 5, that is, the number of messages lost at the interval is not less than the threshold Y12, it can be confirmed that an abnormality has occurred in the link A1. In another example, Fig.7D As shown, in the time period T1 starting at the 3rd second and ending at the 13th second, there are 5 heartbeat messages lost between the master node 111 and the service node 121. When the threshold Y12 is 5, that is, the number of messages lost at intervals is not less than the threshold Y12, it can be confirmed that an abnormality has occurred in link A1.

[0095] In some embodiments, as described above, a sliding window statistics mechanism is used, with time period T1 as the time window, and the duration of time period t as the step length (a difference of one heartbeat message sending interval), and the number of lost messages in each time window is counted in sequence. In this way, whenever Y1 lost messages are counted, and the time interval between the first lost message and the last lost message in the Y1 lost messages is not greater than the duration of time period T1, it can be confirmed that the number of lost messages in a time period T1 is not less than the threshold value Y1, so that it can be confirmed that an abnormality has occurred in the link.

[0096] If an abnormality occurs in the link between the master control node 111 and the service node 121, the service node 121 can be determined to be an abnormal node. If no abnormality occurs in the link between the master control node 111 and the service node 121, the service node 121 can be determined not to be an abnormal node.

[0097] Thus, referring to the way in which the master control node 111 determines whether the service node 111 is an abnormal node, the master control node 111 determines whether other service nodes (such as the service node 122) are abnormal nodes, which will not be described in detail here.

[0098] Then, the master control node 111 may count the number of abnormal nodes.

[0099] Continue reading Figure 6 , the master control node 111 can determine whether the number of abnormal nodes is greater than or equal to P. If the number of abnormal nodes is greater than or equal to P, the master control node 111 triggers the master election process. The value of P is specifically described above and will not be repeated here. In addition, in the embodiment of the present application, "greater than or equal to" can be understood as "not less than".

[0100] In some embodiments, Figure 1 or Figure 2In the architecture shown, in the scenario where the distributed cluster adopts the zookeeper voting mechanism, when the number of abnormal nodes is greater than or equal to P, the first master control node can instruct the backup node to enter the master election process by killing the main process, thereby triggering the master election process. Among them, the function of the main process includes sending heartbeat messages to the backup node. After the main process is killed, the first master control node no longer sends heartbeat messages to the backup node. In this way, when the heartbeat message times out, the backup node can enter the election state, and then can enter the master election process. Among them, when the first master control node is control node 111, the backup nodes are control node 112 and control node 113.

[0101] In some embodiments, Figure 1 or Figure 2 In the architecture shown, in the scenario where the distributed cluster adopts the zookeeper voting mechanism, when the number of abnormal nodes is greater than or equal to P, the first master control node can notify each backup node to enter the election state, thereby entering the master election process.

[0102] In some embodiments, Figure 1 or Figure 2 In the architecture shown, in the scenario where the distributed cluster adopts the zookeeper voting mechanism, when the number of abnormal nodes is greater than or equal to P, the first master control node can instruct the backup node to enter the master election process by stopping sending heartbeat messages, thereby triggering the master election process. Among them, stopping sending heartbeat messages means no longer sending heartbeat messages to the backup node. In this way, when the backup node times out in the heartbeat message, it can enter the election state and then enter the master election process.

[0103] Among them, Figure 1 or Figure 2 In the illustrated architecture, when the first master control node is the primary control node 211, the backup nodes are the primary control node 212 and the primary control node 213. When the first master control node is the secondary control node 231, the backup nodes are the secondary control node 232 and the secondary control node 233. In addition, the first master control node can actively enter the election state to enter the master election process. That is, the first master control node can instruct itself to enter the master election process.

[0104] In some embodiments, Figure 2 In the architecture shown, when the number of abnormal nodes is greater than or equal to P, and the first master control node is the secondary master control node 231, the first master control node (i.e., the secondary master control node 231) can send a request message to the primary master control node 211, and the request information is used to request the primary master control node to re-designate the secondary master control node, thereby triggering the master election process.

[0105] In some embodiments, Figure 3In the architecture shown, that is, in the scenario where the distributed cluster adopts the Kafka controller election mechanism, when the number of abnormal nodes is greater than or equal to P abnormalities, the first master control node (i.e., the low-level master control node 321) can send a request message to the high-level master control node (i.e., the high-level control node 311), and the request information is used to request the high-level master control node to delete the temporary node under the controller path, thereby triggering or instructing each backup node (low-level control node 322, low-level control node 323, and low-level control node 321) to enter the master election process.

[0106] In some embodiments, Figure 4 In the architecture shown, that is, in the scenario where the distributed cluster adopts the redis sentinel designation mechanism, when the number of abnormal nodes is greater than or equal to P, the first master control node (i.e., the master control node 421) can send a request message to the sentinel (i.e., the advanced control node 411), and the request message is used to request the sentinel to reselect the master control node, thereby triggering the master election process.

[0107] The master selection process is used to reselect a control node from multiple control nodes as the second master control node, and the second master control node is used to replace the first master control node to control the service node to execute the service.

[0108] In some embodiments, after the master election process is triggered, a new master node can be selected according to the master election process specified in the existing zookeeper voting mechanism, kafka controller election mechanism or redis sentinel designation mechanism. For details, please refer to the above introduction and will not be repeated here.

[0109] In some embodiments, the possibility of the first master control node becoming the second master control node is less than the possibility of other control nodes becoming the second master control node, and the other control nodes are nodes other than the first master control node among the multiple control nodes. In other words, the possibility of the original master control node becoming the new master control node is less than the possibility of other control nodes becoming the new master control node. The details are as follows.

[0110] exist Figure 1 or Figure 2 In the architecture shown, see Figure 6, the node entering the election state can determine whether the node is the original master control node (for example, master control node 111) before voting for the master election. If it is not the original master control node, execute step 6032a and vote for yourself according to the existing zookeeper voting mechanism. If it is the original master control node, execute step 6032b not to vote for yourself, thereby reducing the possibility of the original master control node becoming the new master control node. Then, compare, and the loser will vote for the winner. After that, execute step 6033 to determine whether a new master control node is elected, that is, whether the number of votes for any control node exceeds half of the total number of control nodes. If not, perform the next round of comparison and voting. Until a new master control node is selected. After the new master control node is elected, the new master control node can execute step 6034 to replace the original master control node and continue to control the business node to execute the business.

[0111] In one example, see Fig. 9A Through the above solution, the control node 112 can be selected as the new master control node. The control node 112 can replace the control node 111 and continue to control the service node to execute the service.

[0112] exist Figure 3 In the architecture shown, when each low-level control node creates a temporary node under the controller path through a competition mechanism, the original master control node attempts to create a temporary node under the controller path after extending the time period T2 compared to other low-level control nodes, thereby reducing the possibility of the original master control node becoming the new master control node.

[0113] In one example, see Fig. 9B , through the above solution, the low-level control node 322 can be selected as a new master control node. The low-level control node 322 can replace the low-level control node 321 and continue to control the service node to execute the service.

[0114] exist Figure 4 In the architecture shown, when the sentinel designates a new master node, it does not select the original master node, but designates the master node from the backup nodes.

[0115] In one example, see Fig. 9C , through the above solution, the control node 422 can be selected as the new master control node. The control node 422 can replace the control node 421 and continue to control the service node to execute the service.

[0116] As described above, the distributed cluster leader election method provided in the embodiment of the present application does not conflict with the existing leader election process trigger mechanism (i.e., the leader election process trigger mechanism under the zookeeper voting mechanism, the leader election process trigger mechanism under the kafka controller election mechanism, and the leader election process trigger mechanism under the redis sentinel designation mechanism). Therefore, the distributed cluster leader election method provided in the embodiment of the present application can be compatible with the existing leader election process trigger mechanism.

[0117] Specifically, in Figure 1 In the illustrated architecture, the control nodes 112 and 113 can still monitor the status of the master control node 111, and when the status of the master control node 111 is found to be abnormal and each control node is not in the master election process, they can enter the election state and start re-election of the master control node. For details, please refer to the above introduction.

[0118] exist Figure 3 In the architecture shown, the controller can still monitor the status of the temporary nodes under its path. When the status of the temporary node is abnormal and the control nodes are not in the selection process, the controller deletes the temporary node and notifies all lower-level control nodes to adopt a competition mechanism to re-create the temporary node under the path, thereby selecting a new master control node.

[0119] exist Figure 4 In the architecture shown, the sentinel can still monitor the master node and the backup node. When the status of the master node is abnormal and the common control nodes are not in the master election process, the sentinel can again designate a new master node from multiple common control nodes.

[0120] The distributed cluster master election method provided in the embodiment of the present application can trigger the master election process when an abnormality occurs in the link between the master control node and the service node. The method avoids the problem that the backup node or the advanced control node does not find the abnormality of the master control node, and the link or network between the service node and the master control node is unstable (such as congestion, packet loss), resulting in the inability to communicate normally between the master control node and the service node, thereby affecting the execution of the service. In addition, the method can reduce the original master control node from becoming the master control node again, thereby ensuring that there is a good link between the new master control node and the service node.

[0121] The embodiment of the present application also provides a server 1000. The server 1000 can be connected to multiple second servers for communication. Fig.10As shown, the server 100 includes a processor 1010, and a receiving port 1020 coupled to the processor 1010. The receiving port 1020 is used to receive the heartbeat message of each second server; the processor 1010 is used to monitor the link between the first server and the second server based on the heartbeat message of the second server; and, when an abnormality occurs in the link between the first server and at least P second servers among all the second servers, trigger a master selection process, wherein the master selection process is used to reselect a server from a plurality of control servers as a master control server, and the master control server is used to replace the first server and control the second server node to perform services, wherein P is an integer greater than or equal to 1. The functions of the receiving port 1020 and the processor 1010 can be implemented with reference to the above description of the operations performed by the first master control node, and will not be described one by one here. The second server may be the service node described above. The control server may be the control node described above. In one example, the plurality of control servers may be Figure 1 In one example, multiple control servers can be Figure 2 In one example, the multiple control servers may be Figure 3 In one example, multiple control servers can be Figure 4 Other control nodes except control node 111 in the illustrated architecture.

[0122] The present application also provides a computing device, which includes a processor, and the processor can execute the following Figure 5 The method shown enables the computing device to have the technical effect of the above-mentioned protected technical solution.

[0123] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer executes the above Figure 5 The method shown.

[0124] In an embodiment of the present application, a computer program product is also provided, wherein the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the above Figure 5 The method shown.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed cluster leader selection method, characterized in that: The distributed cluster includes multiple control nodes and multiple service nodes, wherein one of the multiple control nodes serves as a first master control node, a link is provided between the first master control node and the service node to control the service node to execute services, and a link is provided between the first master control node and other control nodes to enable other control nodes to monitor the status of the first master control node; the method includes: The first master control node monitors a link between the first master control node and each of the service nodes; When an abnormality occurs in a link between the first master control node and at least P service nodes among all the service nodes, the first master control node triggers a master selection process, wherein the master selection process is used to reselect a control node from the multiple control nodes as a second master control node, and the second master control node is used to replace the first master control node to control the service nodes to execute services, where P is an integer greater than or equal to 1; In the case where the multiple control nodes elect a master control node by voting, the method further includes: the first master control node does not vote for the first master control node; or, The distributed cluster further includes a controller, and the method further includes: delaying the start time of the first master control node attempting to create a temporary node under the controller path; or, The distributed cluster further includes a sentinel, and the method further includes: the sentinel selects a second master control node from other control nodes, and the other control nodes are nodes among the multiple control nodes except the first master control node.

2. The method according to claim 1, characterized in that The first master control node senses the state of the link between the first master control node and the service node through a heartbeat message between the first master control node and the service node; The first master control node monitoring the link between the first master control node and each of the service nodes includes: The first master control node counts the number of lost heartbeat messages between the first master control node and the service node within a time period of a preset duration; When the number of lost heartbeat messages between the first master control node and the service node is not less than a first threshold, it is confirmed that an abnormality occurs in the link between the first master control node and the service node.

3. The method according to any one of claims 1 to 2, characterized in that: The process of triggering the master election by the first master control node includes: The first master control node instructs each of the multiple control nodes to enter the master election process.

4. The method according to any one of claims 1 to 2, characterized in that The multiple control nodes include at least one first control node and at least two second control nodes, the first master control node belongs to the second control node, and the first master control node triggers a master election process including: The first master control node notifies the first control node; The first control node instructs each of the at least two second control nodes to enter the master election process, or the first control node designates one of the at least two second control nodes as the second master control node.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Each of the multiple control nodes monitors the state of the first master control node, and enters the master election process when it is detected that the state of the first master control node is abnormal.

6. A first server, characterized in that: The first server is in communication connection with a plurality of second servers, and the first server comprises a processor and a receiving port coupled to the processor; wherein, The receiving port is used to receive the heartbeat message of each second server; The processor is used to monitor the link between the first server and the second server based on the heartbeat message of the second server; and is used to trigger a master selection process when an abnormality occurs in the link between the first server and at least P second servers among all the second servers, wherein the master selection process is used to reselect a server from a plurality of control servers as a master control server, and the master control server is used to replace the first server and control the second server to execute the service, wherein P is an integer greater than or equal to 1; In the case where the multiple control servers elect a master control server by voting, the processor is further configured to: not vote for the first server; or, The processor is further configured to delay a start time at which the first server attempts to create a temporary node under a controller path.

7. The first server according to claim 6, characterized in that: The processor is further configured to: Counting the number of lost heartbeat messages between the first server and the second server within a preset time period; When the number of lost heartbeat messages between the first server and the second server is not less than a first threshold, it is confirmed that an abnormality occurs in the link between the first server and the second server.

8. The first server according to claim 6 or 7, characterized in that: The processor is further configured to instruct each of the plurality of control servers to enter the master selection process.

9. The first server according to claim 6 or 7, characterized in that: The first server is also connected to the management server for communication, and the processor is further used to notify the management server so that the management server instructs each of the control servers to perform a master selection process or designates one of the multiple control servers as a master control server.

Images