High-real-time cluster management method and system

Abstract

The invention discloses a high-real-time cluster management method and system, and the method comprises the steps: respectively building long connection between a master node, a slave node and a standby node in a cluster and an arbitration node after the master node, the slave node and the standby node are started; the master node, the slave node and the standby node respectively send application messages to the arbitration node through the corresponding long connection, and the application messages comprise the initial role and the initial state of each node; the arbitration node makes a decision based on an arbitration protocol according to the application message, sends response messages to the master node, the slave node and the standby node, and determines current roles, current states and current rounds corresponding to the master node, the slave node and the standby node; in the normal working process of the cluster, the master node, the slave node and the standby node report the operation state to the arbitration node regularly, and the arbitration node sends global state messages to all the nodes in the cluster regularly. The decision is made by combining the arbitration node with the cluster global state, the double-point fault prevention in the master-slave-slave working mode is facilitated, and the data synchronization efficiency between the master and slave is high.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a high real-time cluster management method and system. Background Art

[0002] The current mainstream distributed system algorithms are all proposed based on the distributed consistency theory. Classic algorithms include Paxos and Raft. A typical scenario is that in a distributed database system, if the initial state of each node is consistent and each node executes the same sequence of operations, they can finally get a consistent state. To ensure that each node executes the same sequence of commands, it is necessary to execute a "consistency algorithm" on each instruction to ensure that the instructions seen by each node are consistent. The consistency algorithm can be applied in many scenarios and is an important issue in distributed computing. Since it was proposed, the Paxos protocol has been the standard protocol for distributed consistency algorithms, but it is not easy to understand and is very complicated for engineers to implement. As a result, there has been no complete implementation plan for nearly 30 years since the algorithm was proposed. Many implementations in the industry are Paxos-like. Raft algorithm was proposed by Stanford University in 2014. The name of the paper is "In Search of an Understandable Consensus Algorithm". The name Raft comes from the acronym of "Reliable, Replicated, Redundant, And Fault-Tolerant". Raft belongs to Paxos-like algorithm. With its easy-to-understand and easy-to-implement features, Raft has promoted the widespread application of distributed consistency algorithms. In the Raft algorithm, all nodes in the cluster are divided into two roles: Leader and Follower. The implementation mainly includes the Leader election process and the data synchronization and confirmation process. Leader election is when the current leader crashes or the cluster is initialized, and a new leader is elected. The data synchronization and confirmation process is that after the leader election is completed, the leader sends the data to be synchronized to each follower, and the follower sends a confirmation response to the leader after receiving it. When most nodes in the cluster send a confirmation response to the leader, the leader performs the commit operation and sends a commit message to the follower. After receiving the commit message from the leader, the follower commits the data. In the Raft algorithm, the leader ensures the consistency of distributed data through data synchronization and confirmation.

[0003] However, in scenarios with high real-time requirements, such as real-time trading systems, although both Paxos and Raft algorithms include election and data synchronization and confirmation processes, the trading system itself has very high real-time requirements and is particularly sensitive to delays. Once a node goes down and needs to be re-elected, it will inevitably cause serious delay jitter in the trading system, causing a huge impact on market transactions. Therefore, Paxos and Raft algorithms cannot be applied to scenarios with high real-time requirements, and new reliable mechanisms need to be proposed for high-real-time cluster management.

[0004] Based on the above technical problems, the applicant proposed the technical solution of this application. Summary of the invention

[0005] In order to achieve the above object, the present invention discloses a high real-time cluster management method, comprising the following steps:

[0006] After startup, the master node, slave node, and standby node in the cluster establish a persistent connection with the arbitration node respectively;

[0007] Through the corresponding long connection, the master node, the slave node and the standby node respectively send an application message to the arbitration node, wherein the application message includes the initial role and initial state of each node;

[0008] The arbitration node makes a decision based on the arbitration protocol according to the application message, sends a response message to the master node, the slave node and the standby node, and determines the current role, current state and current round corresponding to the master node, the slave node and the standby node;

[0009] During normal operation of the cluster, the master node, slave node and standby node respectively report their operating status to the arbitration node regularly, and the arbitration node regularly sends global status messages to all nodes in the cluster.

[0010] Preferably, the high real-time cluster management method further comprises the following steps:

[0011] The master node, slave node, and standby node in the cluster receive the pre-data from the pre-data source respectively;

[0012] The master node generates a sequence message for the pre-data, writes the sequence message into an index stream, and sends the index stream to the slave node and the standby node;

[0013] The master node, the slave node and the standby node queue the preceding data according to the order in the index stream to form a queue stream with slave-standby consistency;

[0014] The master node, the slave node and the standby node perform service processing according to the queuing flow respectively to generate a result flow of the service processing, and the slave node outputs the result flow.

[0015] Preferably, the high real-time cluster management method further comprises the following steps:

[0016] After the master node, slave node, and backup node are all working normally, when a cluster failure occurs, determine the location of the failed node;

[0017] If the failed node is the master node, the slave node will be switched to a single node, and then the single node will be switched to the master node, and the standby node will be switched to the slave node to form a new master node and slave node, so that the cluster enters the master-slave dual-point working mode;

[0018] If the failed node is a slave node, switch the master node to a single node, then switch the single node to the master node, and switch the standby node to the slave node to form a new master node and slave node, so that the cluster enters the master-slave dual-point working mode;

[0019] If the failed node is a standby node, the cluster will not be changed;

[0020] If the failed node is an arbitration node, restart the arbitration node.

[0021] Preferably, the high real-time cluster management method further comprises the following steps:

[0022] When a node failure occurs after the cluster enters the master-slave dual-point working mode, determine the location of the failed node;

[0023] If the failed node is the primary node, the slave node will be switched to a single node, so that the cluster enters a single-machine single-point working mode;

[0024] If the failed node is a slave node, switch the master node to a single node, so that the cluster enters a single-machine single-point working mode.

[0025] Preferably, the high real-time cluster management method further comprises the following steps:

[0026] After receiving the global status information, the master node, slave node and backup node make decisions according to the arbitration protocol and send a state switching application message to the arbitration node. After receiving the state switching application message, the arbitration node makes a judgment and decision, and sends a state switching response message or a suicide instruction according to the node corresponding to the state switching application message. Preferably, the application message includes at least the node identification number, role, status, round, index stream submission number, and index stream confirmation number, the response message includes at least the node identification number, target role, target status and current round, the global status message includes at least the current round in the group, cluster node information, and the cluster node information includes the node identification number, role, status, index stream submission number, and index stream confirmation number.

[0027] Preferably, the role switching instruction message includes at least a node identification number, a switching role, a switching state and a current round, and the suicide instruction message includes at least a node identification number, a switching role and a switching state.

[0028] Preferably, during the cluster operation, the arbitration node records the node status information, wherein the node status information includes the last reporting time of the node and the last queuing time of the node;

[0029] If the last reporting time of a node exceeds the reporting threshold, the corresponding node will be disconnected;

[0030] If the last queuing time of the slave node exceeds the queuing threshold, the arbitration node will make a decision after determining that the slave node has timed out.

[0031] The present invention also discloses a high real-time cluster management system, comprising a master node, a slave node, a standby node and an arbitration node;

[0032] After startup, the master node, slave node, and standby node in the cluster establish a persistent connection with the arbitration node respectively;

[0033] Through the corresponding long connection, the master node, the slave node and the standby node respectively send an application message to the arbitration node, wherein the application message includes the initial role and initial state of each node;

[0034] The arbitration node makes a decision based on the arbitration protocol according to the application message, sends a response message to the master node, the slave node and the standby node, and determines the current role, current state and current round corresponding to the master node, the slave node and the standby node;

[0035] During normal operation of the cluster, the master node, slave node and standby node respectively report their operating status to the arbitration node regularly, and the arbitration node regularly sends global status messages to all nodes in the cluster.

[0036] Preferably, the master node, slave node and standby node in the cluster receive the pre-data from the pre-data source respectively;

[0037] The master node generates a sequence message for the pre-data, writes the sequence message into an index stream, and sends the index stream to the slave node and the standby node respectively;

[0038] The master node, the slave node and the standby node queue the preceding data according to the order in the index stream to form a queue stream with slave-standby consistency;

[0039] The master node, the slave node and the standby node perform service processing according to the queuing flow respectively to generate a result flow of the service processing, and the slave node outputs the result flow.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The high real-time cluster management method and system provided by the present invention respectively establish long connections between the arbitration node and the master and slave nodes, so that the master and slave nodes actively report to the arbitration node and apply for roles and status. The arbitration node makes a decision after obtaining the global status message, and determines the role conversion of the master and slave nodes based on the global status, thereby achieving flexible adaptation of the working mode of each node in a high real-time scenario and avoiding jitter delay and interference when a node failure occurs.

[0042] 2. The high real-time cluster management method and system provided by the present invention perform data sequencing only by the master node, and send the sequencing message to the slave node and the standby node through the master node. The slave node and the standby node queue up according to the sequence specified by the master node, and there is no need to send data confirmation messages to the master node, thereby ensuring that the number of network hops for the cluster in data synchronization confirmation is 1 hop, so that the cluster supports double-point failure protection and has high data synchronization efficiency.

[0043] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a data flow diagram of the high real-time cluster management method of the present invention.

[0045] Figure 2 It is a schematic diagram of master node sequencing of the high real-time cluster management method of the present invention.

[0046] Figure 3 It is a schematic diagram of the role switching path of the high real-time cluster management method of the present invention.

[0047] Figure 4It is a schematic diagram of the master-slave startup process of the high real-time cluster management method of the present invention.

[0048] Figure 5 It is a schematic diagram of the master-slave application work of the high real-time cluster management method of the present invention.

[0049] Figure 6 It is a schematic diagram of a dual-active scenario failure in the high real-time cluster management method of the present invention.

[0050] Figure 7 It is a schematic diagram of a double-slave scenario failure in the high real-time cluster management method of the present invention.

[0051] Figure 8 It is a schematic diagram of a host free scenario failure in the high real-time cluster management method of the present invention.

[0052] Fig. 9 This is another schematic diagram of a host-free scenario failure in the high-real-time cluster management method of the present invention.

[0053] Fig.10 It is a schematic diagram of the node architecture of the high real-time cluster management system of the present invention. DETAILED DESCRIPTION

[0054] In order to make the technical means, creative features, objectives and effects of the invention easier to understand, the invention is further described below with reference to specific diagrams. However, the invention is not limited to the following implementation cases.

[0055] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0056] High-real-time systems are particularly sensitive to delays. The Paxos and Raft algorithms used in existing clusters both include election and data synchronization and confirmation processes. When a node goes down and is re-elected, it will inevitably cause serious delay jitter in the high-real-time system, which will have a huge impact on external users of the high-real-time system and the entire market. The data synchronization and confirmation process involves three types of messages, namely data synchronization messages, confirmation messages of data synchronization messages, and data submission messages. In specific applications, data submission information is often added to the data synchronization message to save network hops. At this time, the number of internal network hops is approximately 2 hops. However, when the cluster is not busy, the above three types of messages need to be sent. For the follower node, it actually needs to go through 3 hops of network transmission to achieve data consistency. Taking a cluster of 3 nodes as an example, the data synchronization and confirmation process in the Paxos and Raft algorithms involves approximately 2 hops of network hops, and only supports single-point failures, not double-point failures. When a single point failure (leader failure) occurs, a new leader needs to be elected again before the cluster can work normally again. The election process will cause delays, which makes it difficult to adapt to scenarios with high real-time requirements.

[0057] The first embodiment of the present application discloses a high real-time cluster management method, such as Figure 1 As shown, after startup, the master node, slave node and standby node in the cluster respectively establish long connections with the arbitration node; through the corresponding long connections, the master node, slave node and standby node respectively send application messages to the arbitration node, and the application messages include the initial role and initial status of each node; the arbitration node makes a decision based on the arbitration protocol according to the application message, sends a response message to the master node, slave node and standby node, and determines the current role, current status and current round corresponding to the master node, slave node and standby node; during the normal operation of the cluster, the master node, slave node and standby node respectively regularly report the operating status to the arbitration node, and the arbitration node regularly sends a global status message to all nodes in the cluster.

[0058] The cluster works in master-slave and arbitration mode, with master-slave three nodes deployed to prevent double-point failure. The master-slave request roles and report status to the arbitration. The arbitration obtains the cluster status through the node report, and performs cluster status checks and judgment decisions based on the arbitration protocol.

[0059] The master node, slave node and standby node in the cluster receive the pre-data from the pre-data source respectively; the master node generates a sequenced message for the pre-data, writes the sequenced message into an index stream, and sends the index stream to the slave node and standby node respectively; the master node, slave node and standby node queue the pre-data according to the order in the index stream to form a queued stream with slave-standby consistency; the master node, slave node and standby node perform business processing according to the queued stream respectively to generate a result stream of the business processing, and the slave node outputs the result stream.

[0060] In this embodiment, the pre-data source is the data input source to be synchronized. The pre-data source can be numbered, and the data between different pre-data sources are uniquely identified by a timestamp. The pre-data source publishes the pre-data to the master node, slave node, and standby node via UDP broadcast. After the master, slave, and standby receive the pre-data, only the master node sequences the input data of all pre-data sources, writes the sequenced message into the index stream, and sends the index stream to the slave node and the standby node via UDP broadcast. The master and slave will generate a consistent queue stream for all pre-data according to the index stream generated by the master node. The master and slave perform business processing according to the queue stream and generate a business processing result stream. Only the slave node publishes the business processing result stream to all pre-data via UDP broadcast. The cluster supports master-slave dual-node deployment, which can prevent single point failures. The master and slave subscribe to the pre-stream, the master node sequences and generates the index stream, the master and slave obtain the consistent queue stream according to the index stream for business processing, and the slave node is responsible for publishing the result stream to the outside. The cluster also supports stand-alone single-point deployment. The stand-alone machine subscribes to the front-end stream, generates the index stream in sequence, generates the queue stream according to the index stream, performs business processing and publishes the result stream to the outside. Among them, the front-end stream is the data input source of the master and slave nodes, that is, the front-end input data to be synchronized. The index stream is the sequencing stream for each front-end stream, which is used to queue the front-end stream. The index stream submission sequence number represents the index stream sequence number received by the node, and the index stream confirmation sequence number represents the index stream sequence number that the node has completed queuing. The node has completed queuing means that the sequencing message about the front-end stream in the index stream has been processed and the front-end stream message has been written into the queue stream. The index stream submission sequence number and the index stream confirmation sequence number are fields in the application message sent by the node to the arbitration. The queue stream is the consistency queue stream generated by the master and slave according to the received front-end stream and the order in the index stream. The result stream is the business result stream generated by the master and slave according to the queue stream after business processing.

[0061] like Figure 2As shown in , three front data sources are used as an example. The master, slave, and standby nodes will subscribe to the front stream published by the front data source, and the front stream will be used as the data input source. The specific process of the master node performing the sequencing operation according to the front stream is that the sequencing message [f1:3] in the index stream represents that three new messages are added to the front (front) 1, [f2:2], [f3:1], and so on. The host publishes the index stream to the slave and standby nodes, and the master, slave, and standby nodes will generate consistent queue streams based on the sequencing messages in the index stream. The host executes the sequencing operation, and the slave and standby nodes obey the queue order specified by the host without confirming with the host.

[0062] The above cluster management architecture formed by the master, slave, standby and arbitration nodes can achieve the purpose of preventing double-point failures at most. When a node failure occurs, the entire cluster can quickly resume work without generating large jitter delays, nor causing major interference to external users of high-real-time systems and the entire market. In the cluster, only the master node performs data sequencing and sends sequencing messages to the slave nodes and standby nodes. After receiving the sequencing messages, the slave nodes and standby nodes queue up according to the sequence specified by the master node, and there is no need to send data confirmation messages to the master node, ensuring that the number of network hops in the cluster is always 1 hop when confirming data synchronization, reducing the possibility of network delay. In addition, the master node, slave node and standby node regularly report the current node operation status to the arbitration node, and the arbitration node periodically checks whether the cluster is working normally. It can make decisions based on the global status of the cluster operation to ensure the stability of the cluster operation.

[0063] The cluster management architecture disclosed in this embodiment can be called "master-slave-standby". Master-slave means that when the master, slave and standby three nodes are deployed, the host will sequence the input data, and the slave and standby will queue the input data according to the sequence specified by the host. Slave-slave means that only the slave will publish the business processing result message to the outside. Standby-standby means that the standby is equivalent to the hot standby of the slave. In some scenarios, arbitration takes the standby status into decision-making reference, but arbitration does not directly use the standby status for judgment. It should be noted that master, slave and standby refer to the master node, slave node and standby node respectively, arbitration refers to the arbitration node, and the arbitration node makes decisions based on the arbitration protocol. The arbitration protocol can obtain the global status of the cluster based on the reports of the master, slave and standby. When a failure occurs, the arbitration node sends instructions to the master, slave and standby according to the global status to realize automatic switching of nodes in the cluster to prevent double-point failures.

[0064] Specifically, the master node, slave node, and backup node are all set with multiple attributes, including node identification number, role, state, and round. The node identification number indicates the node number. The roles include master M (Master), slave S (Slave), backup B (Backup), unique D (Dictator), and null N (NULL). The master M indicates that it is responsible for data sequencing and publishing sequencing messages. The slave S indicates that it is responsible for publishing business processing result messages. The backup B indicates that it follows the slave, which is equivalent to the hot standby of the slave. The unique D indicates data sequencing and publishing business processing result messages. The null N indicates offline, and the role is required to commit suicide. The states include normal N (Normal), working W (Working), and offline O (Offline). Normal N indicates the initial state, working W indicates the working state, and offline O indicates being ordered to commit suicide. The round is used to indicate the situation of each role switching and state switching in the current cluster. It is a key variable in the arbitration protocol. When there is a split-brain scenario with dual masters and dual slaves, the round helps the arbitration to make judgments and decisions. In addition, the attributes also include queuing rounds. When the master or standalone machine (queuer) in the cluster enters the working state, the queuing rounds are updated with the current rounds of the master / standalone. When a network failure causes a host to be isolated and the new master / standalone machine has a network outage, the queuing rounds help the arbitration to make judgments and decisions.

[0065] After the master, slave, and backup are started, they will first establish a connection with the arbitration and apply for roles and status. After the arbitration agrees, it sends the corresponding role and initial status to each node. If the role or status changes in the cluster, it will cause the accumulation of rounds, for example, increasing by 1 each time. The rules followed by role switching include that the master and slave can switch to the standby, only the standby can switch to the slave, only the standby can switch to the master, and when the arbitration orders the node to commit suicide, the role can be switched to empty.

[0066] In one example, the master node, slave node and standby node make decisions according to the arbitration protocol after receiving the global status information, and send a state switching application message to the arbitration node. After receiving the state switching application message, the arbitration node makes a judgment and decision, and sends a state switching response message or a suicide command according to the node corresponding to the state switching application message. After the master node, slave node and standby node establish a long connection with the arbitration node, they first send an application message to the arbitration node, and the application message is used to apply for the initial role and initial state, and the initial state is generally N. After receiving the response message sent by the arbitration node to the application message, a state switching application message will be sent to the arbitration node, for example, the initial state will be switched from N to W, indicating that the node enters the working state. In general, the master, slave and standby send a role application message to the arbitration when they are just connected to the arbitration. Only when the standby machine is in the working state can the standby machine send a role application message to the arbitration to apply for the slave role.

[0067] The following is a detailed introduction to decisions made based on arbitration agreements.

[0068] After the master-slave standby system works normally, first determine where the faulty node is. When a single point of failure occurs, the specific processing method of the arbitration protocol is: if the faulty node is the master node, switch the slave node to a single node, then switch the single node to the master node, and switch the standby node to the slave node to form a new master node and slave node, so that the cluster enters the master-slave dual-point working mode; if the faulty node is a slave node, switch the master node to a single node, then switch the single node to the master node, and switch the standby node to a slave node to form a new master node and slave node, so that the cluster enters the master-slave dual-point working mode; if the faulty node is a standby node, the cluster is not changed; if the faulty node is an arbitration node, restart the arbitration node. The cluster does not rely on the arbitration node. If the arbitration node goes down, the cluster can work normally without being affected. Restarting is also supported after the arbitration node fails. The master-slave standby node re-establishes a TCP connection with the arbitration node, and the cluster resumes the anti-dual-point mode.

[0069] When a node failure occurs after the cluster enters the master-slave dual-point working mode, if the failed node is the master node, the slave node will be switched to a single node, so that the cluster enters the single-machine single-point working mode; if the failed node is a slave node, the master node will be switched to a single node, so that the cluster enters the single-machine single-point working mode. For details, see Figure 3 The role switching path diagram shown in the figure. The node state depends on the role. When the node is just connected to the arbitration, it always applies for the specified role and initial state (normal N). The arbitration decision agrees to its application and sends a response message. When the arbitration orders the node to commit suicide, the role is switched to empty and the state is switched to offline state O.

[0070] During the operation of the master and slave, the roles, states and rounds may change at any time, such as Figure 4 As shown in the figure, each dotted box in the figure represents an application response process between the node and the arbitration. The round number increases by 1 when the role and status change is agreed. The change in the round number also reflects the order of node application. It can be seen from the figure that the initial status of the role is applied by the node, and after the arbitration decision is agreed, the node will report the status to the arbitration regularly. The working status of the role is also applied by the node, and the arbitration decision is agreed. The arbitration obtains global information through the node's report, and sends global status notifications to all nodes in the cluster regularly. After obtaining the status, the node decides whether to apply for the role / status.

[0071] like Figure 5 As shown in the figure, in order to ensure that the cluster enters the working state independently of the startup order of the master, slave, and standby, the timing of the master, slave, and standby applying for the working state is determined by the arbitration protocol. After the slave obtains the global state, it finds that there is a master in the cluster and will apply for the slave working state (S, W), which will take effect after the arbitration decision is agreed. The master finds that the slave in the cluster is working (S, W), and will apply for the master working state (M, W), which will take effect after the arbitration decision is agreed. The standby finds that there is a master in the cluster and will apply for the standby working state (B, W).

[0072] For example, the rules for entering the working state include: the slave finds that the master is connected to the arbitration and applies for a change from (S, N) to (S, W); after the master finds (S, W), the master applies for a change from (M, N) to (M, W); the standby finds that the master is connected to the arbitration, and the state of the master is (M, N) or (M, W), then the standby applies for a change from (B, N) to (B, W); any single node of the master, slave, and standby connected to the arbitration will not automatically apply to enter the working state; only the slave (master) establishes a connection with the arbitration, and the slave (master) will not enter the working state; regardless of the startup order of the master and slave, they enter the working state in the order of (S, W), (M, W). The process of a node applying for a role or status is as follows: the node applies for a role status; the arbitration receives the application and makes a decision; the node regularly reports local information; the arbitration obtains global information and regularly notifies the cluster of the current global status; the node receives the global status and decides whether to apply for a role / status. When the role or status of the cluster changes, the round increases by 1.

[0073] After the master node, slave node and standby node are started with the specified role, they will actively establish a TCP long connection with the arbitration node. Through the TCP long connection, the master, slave and standby will apply for roles to the arbitration. After receiving the application message, the arbitration will make a logical judgment. After agreeing to the application, it will send a request response message to the applicant node through the TCP long connection. Subsequently, each node will report the current node status information to the arbitration regularly (200ms). The node reports the status regularly. If the arbitration passes the check, the node status information will be saved. Through the node report status, the arbitration can grasp the global status of the cluster. The arbitration regularly (200ms) checks the status of each node in the cluster, makes judgments and decisions, and sends global status notification messages to all nodes in the cluster through the TCP connection between the master, slave and standby. When checking each node, if the reporting time of the node exceeds 2s, it means that there may be a problem with the node, and the arbitration will disconnect the TCP connection with the node. When the master, slave and standby are working in the cluster, the last queue time of the slave is checked. When the last queue time of the slave exceeds 2s, the arbitration makes a decision based on the current global status.

[0074] It should be noted that the order in which the master, slave, and arbitration establish TCP connections is related to the startup order of the master and slave. The order in which the master, slave, and arbitration establish TCP connections has no effect on the normal operation of the entire cluster. The cluster does not depend on the startup order of the master and slave. When the arbitration sends the global status to the master and slave through the TCP connection, the master and slave will send it. The arbitration response message or suicide command will only be sent to the corresponding node.

[0075] The application message sent by the master-slave standby to the arbitration includes at least the node identification number, role, status, round, index stream submission number, and index stream confirmation number. The response message sent by the arbitration to the master-slave standby includes at least the node identification number, target role, target status, and current round. The global status message sent by the arbitration to all nodes in the cluster includes at least the current round in the group and cluster node information. The cluster node information includes the node identification number, role, status, index stream submission number, and index stream confirmation number. The role switching instruction message includes at least the node identification number, switching role, switching status, and current round. The suicide instruction message includes at least the node identification number, switching role, and switching status.

[0076] In order to ensure that the arbitration node makes preparation decisions, during the normal operation of the cluster, the arbitration node will record the node status information and cluster status information. The node status information includes the node identification number, role, status, index stream submission number, index stream confirmation number, last report time and last queue time. The last report time represents the time when the node's last report is recorded. The last queue time represents when the index stream confirmation number increases, indicating that the node is queuing. This moment can be recorded as the node's last queue time. Cluster status information includes rounds and queue rounds.

[0077] When making decisions, the arbitration node checks and processes the application message of the node. The specific inspection rules include role validity check, master / single mutual exclusion check, duplicate role check, and queue round check. After passing all the above checks, the arbitration will record the applicant's role and status. For example, the role validity check means that if the applicant applies for a role other than single, master, slave, or backup, the applicant is ordered to commit suicide; the master / single mutual exclusion check means that if there is a master (single) role and the applicant is single (master), the applicant is ordered to commit suicide; the duplicate role check means that if the applicant's role already exists, the applicant is ordered to commit suicide; the queue round detection means that if the applicant is master / single, the application status is the initial status (normal N), and the applicant's round is less than the cluster's queue round, the applicant is ordered to commit suicide.

[0078] In some examples, the arbitration node also checks and updates the node report, including role validity check, updating the node's last report time, updating the node's index stream submission sequence number, if the index stream confirmation sequence number in the report is greater than the record value in the arbitration, then the node's index stream confirmation sequence number and the node's last queuing time are updated, if the node report comes from the master / slave, and the master / slave is in working state, when its turn is greater than the cluster queuing round, the cluster queuing round is updated.

[0079] In some examples, the arbitration node also performs cluster checks regularly, checks all nodes in the cluster, and disconnects the corresponding connection if the report time of a node has timed out. When the cluster is in a master-slave standby working state, check the last queue time of the slave node. When the slave node queue times out, if the index stream submission sequence number and index stream confirmation sequence number of the slave and the standby are the same, it indicates that the slave and the standby have the same reception confirmation status for the index stream. It is highly likely that the cluster is abnormal due to a network failure of the host. The host is ordered to commit suicide and the slave is switched to standalone. If the standby exists and the index stream reception confirmation status is different from that of the slave, the slave is ordered to commit suicide and the host is switched to standalone. When the cluster is in a master-slave working state, check the last queue time of the slave. If the slave queue times out, order the slave to commit suicide and the host to switch to standalone. When only the master exists in the cluster and the slave has failed and disconnected, check the last queue time of the slave. If the slave queue times out, order the master to switch to standalone. When the master in the cluster has failed and disconnected, and the slave exists, check the last queue time of the slave. If the slave queue times out, order the slave to switch to standalone. When the standalone machine in the cluster is in working state and the slave machine is in initial state, command the standalone machine to switch to the master state.

[0080] Table 1 below summarizes the arbitration decision-making process and subsequent cluster status changes by taking the master-slave-standby three-point startup method as an example.

[0081]

[0082]

[0083] Table 1. Summary of quorum decision and cluster status changes

[0084] In some examples, the master-slave backup system works. When the arbitration orders a node to commit suicide or switch, the node executes the arbitration instruction and updates the role status. After the switch, the cluster role becomes the standby. At this time, the node initiates an application according to its own role, as follows:

[0085] □ The working status of the independent machine application is (D,N)->(D,W). After the arbitration is approved, the independent machine starts working;

[0086] □The standby machine receives the arbitration notification message and finds that the standby machine is working alone (D, W) and both the master and slave do not exist. The standby machine applies to switch to the slave: (B, W) -> (S, N);

[0087] □ The arbitration agrees to the standby machine application. After checking, it is found that the standalone machine is working (D, W) and the slave machine is in the initial state (S, N). The standalone machine is ordered to switch to the master state: (D, W) -> (M, N).

[0088] In this state, the cluster returns to the initial master-slave state, and the arbitration notifies each node of the global state. For example, after (M,N), the slave applies for (S,N)->(S,W), and after (S,W), the master applies for (M,N)->(M,W).

[0089] If the cluster experiences a single point failure switch, the master-slave working mode will be restored. If a node continues to fail in the master-slave mode, the last two rows in Table 1 will be shown:

[0090] The slave fails, detects the master is working, the slave queue times out, and commands the master to switch to (M, W)-> (D, N). After receiving the command, the node updates the role status and applies (D, N)-> (D, W);

[0091] The master fails, the slave is detected to be working, the slave queue times out, and the slave is ordered to switch to independent (S, W) -> (D, N). After receiving the command, the node updates the role status and applies (D, N) -> (D, W).

[0092] After experiencing a double-point failure, the master-slave cluster can be switched to a single-machine single-point working state. In addition to the master-slave failure, if the arbitration fails, it will not affect the normal operation of the entire cluster. At this time, the arbitration can be restarted. The master-slave will re-establish a connection with the arbitration, apply for the role status, and the cluster will quickly resume working status.

[0093] It should be noted that the timing is limited in this implementation. After the master and slave nodes successfully apply for the role, they report the local status every 200ms. The arbitration is scheduled for 200ms for inspection and decision-making, and sends global status notifications to the master and slave. The TCP connection between the node and the arbitration has a heartbeat of 200ms. If no message is received within the time limit, the connection will be disconnected, and the timeout period is 1s. When the arbitration is checking, if the node reports a timeout, the timeout period is 2s and the connection will be disconnected. If the node queue times out, the timeout period is 2s, and a decision will be made based on the global status.

[0094] The following uses multiple network failure scenarios as examples to illustrate.

[0095] like Figure 6In the dual-master scenario shown, the initial master-slave standby works normally, with a round of 6; the failure is the host network failure, causing the host to be isolated, the host and the arbitration are disconnected, and the slave standby cannot receive the index stream published by the host. After the arbitration decision, the cluster switches to form a new master-slave and starts working. After a round of switching, the new master-slave round is 12; the network of the isolated host is partially restored, and the index stream can be published, because the round in the index stream message is 6. The node receiving the index stream in the cluster only receives the message with the same round as this node, and the message published by the isolated host will be discarded. In this scenario, the isolated host only performs queuing business processing to generate the result stream, but does not publish the result stream to the outside, so it will not affect the external front-end receiving the result stream. When the network for publishing the index stream returns to normal, because the round is different from that of the new master-slave, the index stream sequencing message it publishes will be discarded, and will not affect the new cluster.

[0096] like Figure 7 In the dual-slave scenario shown, the initial master-slave standby works normally, with a round of 6; the fault condition is that the network between the slave and the arbitration is disconnected. After the arbitration decision, the cluster switches to form a new master-slave and starts working. After a round of switching, the new master-slave round is 12; the round in the index flow message published by the new master is 12. The node 2 slave receives the index flow message and determines that the round in the message is inconsistent with the current node. The index flow message will be discarded, resulting in the node 2 slave being unable to queue and subsequently unable to process business, and the result flow no longer grows. In this scenario, the network between the slave and the arbitration fails, the cluster switches, the round changes, and the node 2 slave lacks the queued message of the index flow and cannot queue for business processing. The result flow no longer grows, and it will not affect the external front-end.

[0097] like Figure 8 In the scenario shown in the figure, the master node is free and the new independent working network is disconnected. The initial master-slave standby working round is 6, and the host network failure is free. After the arbitration decision, the slave is switched to independent and the independent machine works. The current cluster queue round is 8; the new independent network is disconnected, the independent host is connected to the arbitration, and the independent host applies for (M, N), and its round is 6; the arbitration determines that the round of the independent host is less than the current cluster queue round, and orders the independent host to commit suicide; after the independent machine network is restored and reconnected to the arbitration, because its round is 8, the arbitration determines that the independent machine application is passed, and the cluster resumes the master-slave work.

[0098] like Fig. 9In the scenario shown in the figure where the master node is free and the new master working network is disconnected, the initial master-slave standby working round is 6, and the host network failure is free. The arbitration decision judges that after the cluster switch, the new master and slave start working, and the current cluster queue round is 12; the new master network is disconnected, the free host connects to the arbitration, and the free host applies for (M, N), and its round is 6; the arbitration judges that the round of the free host is less than the queue round of the current cluster, and orders the free host to commit suicide; the new master network is restored, and after reconnecting to the arbitration, because its round is 12, the arbitration judges that the new master application is approved, and the cluster resumes work.

[0099] exist Figure 8 and Fig. 9 In the two scenarios, if the host is freed and replaced by a standalone, the new master (new standalone) works, and the network between the new master (new standalone) and the arbitration is disconnected. The free standalone connects to the arbitration and applies for the role. Because the round of applying for the standalone is less than the queuing round of the cluster, the arbitration orders the free standalone to commit suicide.

[0100] This type of failure scenario reflects the importance of queuing rounds. Rounds cannot be used for judgment here. For example, the master and slave are connected to arbitration first, and the master and slave work with round 4 and queuing round 4. The host network is disconnected, and the standby machine is connected to arbitration, and the cluster round becomes 5. The host network is restored and connected to arbitration. The host applies for (M, N) round 4, and arbitration makes a decision: If the cluster round judgment (4<5) is used, the host may be mistakenly killed, while the queuing round can make an accurate judgment. In order to avoid interference to the cluster caused by network disconnection, the concept of queuing rounds is introduced.

[0101] The second embodiment of the present invention discloses a high real-time cluster management system, such as Fig.10 As shown, it includes a master node, a slave node, a standby node and an arbitration node. The master node receives the pre-data from multiple pre-data sources by subscribing to the external pre-data sources, and the slave node outputs the result stream after the business processing through the external publishing terminal.

[0102] After startup, the master node, slave node, and standby node in the cluster establish a persistent connection with the arbitration node respectively;

[0103] Through the corresponding long connection, the master node, the slave node and the standby node respectively send an application message to the arbitration node, wherein the application message includes the initial role and initial state of each node;

[0104] The arbitration node makes a decision based on the arbitration protocol according to the application message, sends a response message to the master node, the slave node and the standby node, and determines the current role, current state and current round corresponding to the master node, the slave node and the standby node;

[0105] During normal operation of the cluster, the master node, slave node and standby node respectively report their operating status to the arbitration node regularly, and the arbitration node regularly sends global status messages to all nodes in the cluster.

[0106] The master node, slave node, and standby node in the cluster receive the pre-data from the pre-data source respectively;

[0107] The master node generates a sequence message for the pre-data, writes the sequence message into an index stream, and sends the index stream to the slave node and the standby node respectively;

[0108] The master node, the slave node and the standby node queue the preceding data according to the order in the index stream to form a queue stream with slave-standby consistency;

[0109] The master node, the slave node and the standby node perform service processing according to the queuing flow respectively to generate a result flow of the service processing, and the slave node outputs the result flow.

[0110] Since the first embodiment corresponds to this embodiment, this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and the technical effects that can be achieved in the first embodiment can also be achieved in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0111] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A high real-time cluster management method, characterized in that: The following steps are involved: After startup, the master node, slave node, and standby node in the cluster establish a persistent connection with the arbitration node respectively; Through the corresponding long connection, the master node, the slave node and the standby node respectively send an application message to the arbitration node, wherein the application message includes the initial role and initial state of each node; The arbitration node makes a decision based on the arbitration protocol according to the application message, sends a response message to the master node, the slave node and the standby node, and determines the current role, current state and current round corresponding to the master node, the slave node and the standby node; During normal operation of the cluster, the master node, slave node and standby node respectively report their operating status to the arbitration node regularly, and the arbitration node regularly sends global status messages to all nodes in the cluster.

2. The high real-time cluster management method according to claim 1, characterized in that: The following steps are also included: The master node, slave node, and standby node in the cluster receive the pre-data from the pre-data source respectively; The master node generates a sequence message for the pre-data, writes the sequence message into an index stream, and sends the index stream to the slave node and the standby node respectively; The master node, the slave node and the standby node queue the pre-data according to the order in the index stream to form a queue stream with master-slave-standby consistency; The master node, the slave node and the standby node perform service processing according to the queuing flow respectively to generate a result flow of the service processing, and the slave node outputs the result flow.

3. The high real-time cluster management method according to claim 1, characterized in that: The following steps are also included: After the master node, slave node, and backup node are all working normally, determine the location of the faulty node when a cluster failure occurs; If the failed node is the master node, the slave node will be switched to a single node, and then the single node will be switched to the master node, and the standby node will be switched to the slave node to form a new master node and slave node, so that the cluster enters the master-slave dual-point working mode; If the failed node is a slave node, switch the master node to a single node, then switch the single node to the master node, and switch the standby node to the slave node to form a new master node and slave node, so that the cluster enters the master-slave dual-point working mode; If the failed node is a standby node, the cluster will not be changed; If the failed node is an arbitration node, restart the arbitration node.

4. The high real-time cluster management method according to claim 3, characterized in that: The following steps are also included: When a node failure occurs after the cluster enters the master-slave dual-point working mode, determine the location of the failed node; If the failed node is the primary node, the slave node will be switched to a single node, so that the cluster enters a single-machine single-point working mode; If the failed node is a slave node, switch the master node to a single node, so that the cluster enters a single-machine single-point working mode.

5. The high real-time cluster management method according to claim 1, characterized in that: The following steps are also included: After receiving the global status information, the master node, slave node and standby node make a decision and send a status switching application message to the arbitration node; The arbitration node makes a decision after receiving the state switching application message, and sends a state switching response message or a suicide instruction according to the node corresponding to the state switching application message.

6. The high real-time cluster management method according to claim 1, characterized in that: The application message includes at least the node identification number, role, status, round, index stream submission number, and index stream confirmation number; the response message includes at least the node identification number, target role, target status, and current round; the global status message includes at least the current round in the group and cluster node information; the cluster node information includes the node identification number, role, status, index stream submission number, and index stream confirmation number.

7. The high real-time cluster management method according to claim 5, characterized in that: The state switching response message includes at least a node identification number, a switching role, a switching state and a current round, and the suicide instruction includes at least a node identification number, a switching role and a switching state.

8. The high real-time cluster management method according to claim 1, characterized in that: During the normal operation of the cluster, the arbitration node records the node status information, which includes the node's last report time and the node's last queue time; If the last reporting time of a node exceeds the reporting threshold, the corresponding node will be disconnected; If the last queuing time of the slave node exceeds the queuing threshold, the arbitration node will make a decision after determining that the slave node has timed out.

9. A high real-time cluster management system, characterized in that: Includes master node, slave node, backup node and arbitration node; After startup, the master node, slave node, and standby node in the cluster establish a persistent connection with the arbitration node respectively; Through the corresponding long connection, the master node, the slave node and the standby node respectively send an application message to the arbitration node, wherein the application message includes the initial role and initial state of each node; The arbitration node makes a decision based on the arbitration protocol according to the application message, sends a response message to the master node, the slave node and the standby node, and determines the current role, current state and current round corresponding to the master node, the slave node and the standby node; During normal operation of the cluster, the master node, slave node and standby node respectively report their operating status to the arbitration node regularly, and the arbitration node regularly sends global status messages to all nodes in the cluster.

10. The high real-time cluster management system according to claim 9, characterized in that: The master node, slave node, and standby node in the cluster receive the pre-data from the pre-data source respectively; The master node generates a sequence message for the pre-data, writes the sequence message into an index stream, and sends the index stream to the slave node and the standby node respectively; The master node, the slave node and the standby node queue the preceding data according to the order in the index stream to form a queue stream with slave-standby consistency; The master node, the slave node and the standby node perform service processing according to the queuing flow respectively to generate a result flow of the service processing, and the slave node outputs the result flow.

Images