A multi-node dual-path serial high-availability cluster system and management method

Abstract

The application relates to a multi-node dual-path serial high-availability cluster system and a management method, and belongs to the cluster design field. The application flexibly realizes the increase and deletion of the master and standby nodes of the serial multi-node cluster by establishing a communication route configuration on the nodes, and meets the demand of different flow service on different node numbers. The application realizes the rapid propagation of the fault information in the cluster through a longitudinal and transverse double notification mechanism of the fault information, and ensures the master and standby switching performance of the whole serial cluster when the node is faulty. The application adopts a time-sharing dual-redundancy heartbeat mechanism, avoids the brain split problem of the cluster system, considers the heartbeat monitoring performance and the occupation of the system resources, and guarantees the stable and reliable operation of the cluster system. Compared with the prior art, the technical method provided by the application provides a high-availability cluster system and a management method suitable for serial flow service, provides three different node intercommunication modes for selection, and flexibly configures the node number.

Description

A multi-node dual-path serial high-availability cluster system and its management method Technical Field

[0001] This invention belongs to the field of cluster design, specifically relating to a multi-node dual-path serial high-availability cluster system and management method. Background Technology

[0002] Typical high-availability cluster systems are mostly composed of two or more interconnected nodes. The interconnection topology between nodes is generally a direct connection between two nodes (as shown in Figure 1) or a star network interconnection between multiple nodes (as shown in Figure 2). High-availability cluster management is also based on this topology. For clusters composed of two nodes, the nodes are directly interconnected, and high-availability management generally adopts a primary-backup or primary-primary mode. For multi-node clusters, the nodes are interconnected through switches, and the communication between nodes is peer-to-peer. High-availability management generally adopts a multi-primary-one-backup, one-primary-multiple-backup, or multi-primary-multiple-backup mode.

[0003] Existing high-availability cluster systems and their management are all based on topologies of direct connection between two nodes or peer-to-peer interconnection between multiple nodes, which are not suitable for systems with multi-node serial pipelined services. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] The technical problem to be solved by this invention is how to provide a multi-node dual-path serial high-availability cluster system and management method, so as to solve the problem that the existing high-availability cluster systems and their management are carried out in the topology of dual-node direct connection or multi-node peer interconnection, which cannot be applied to systems with multi-node serial pipelined services.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical issues, this invention proposes a multi-node dual-path serial high-availability cluster system. This system comprises two serial nodes suitable for pipelined services, with each path containing at least three serial nodes.

[0008] Two serial nodes at the same location are used for primary and backup dual-node redundancy to achieve high availability;

[0009] The communication link between each serial node can be any of the following: network, serial port, or SRIO. However, the communication method between two nodes in the same location must be consistent, and the communication method cannot be changed during system operation. Two serial nodes in the same location form a master-slave node, which is directly connected through the network and serial port to achieve dual redundancy heartbeat.

[0010] This invention also provides a method for managing a multi-node dual-path serial high-availability cluster, the method comprising the following steps:

[0011] S101. Set the default state of the nodes; by default, each serial node of the first path is the master node, and each serial node of the second path is the backup node.

[0012] S102. Determine the communication method between nodes; Based on the actual hardware connection, determine the communication method between each node in the form of a configuration file, and establish the communication routing link between nodes.

[0013] S103. Start status monitoring on each node; including the internal operating status of the node and the communication status of the nodes connected to the node on the same serial link;

[0014] S104. Start time-sharing dual-redundant heartbeat monitoring on each node; that is, network heartbeat monitoring and serial port heartbeat monitoring; the two are not performed at the same time. Network heartbeat monitoring is performed first, and serial port heartbeat monitoring is enabled only when the network heartbeat is abnormal.

[0015] S105. Determine the role status of the nodes; Each node determines its initial role status based on its link number, status monitoring results, and heartbeat monitoring results, specifically: primary status, backup status, or fault status.

[0016] S106. Node role status coordination: Each node notifies its own determined role status to adjacent nodes on the serial link. Upon receiving the role status information from adjacent nodes, the node adjusts its own role status. Finally, the primary node link, backup node link, or faulty link of the system is determined. By default, if there are no abnormal nodes or links, all nodes on the first path are primary nodes, that is, the first path is the primary link, and the second path is the backup link, as shown in Figure 4. Figure 5 illustrates other possible primary and backup node role statuses.

[0017] S107. Node role state changes and actions: Based on the changed role state, the node executes different services and configurations, which serves as the basis for the high availability master-slave mode switching of services.

[0018] This invention also provides a method for managing a multi-node dual-path serial high-availability cluster, the method comprising the following steps:

[0019] S201. The system determines the primary node link and the backup node link;

[0020] S202. Handling node failures; When a node in the link fails, the fault handling procedure is as follows:

[0021] A) Nodes at the same location on the serial link detect the other's failure through heartbeat monitoring between the primary and backup nodes, and proceed to step C;

[0022] B) If any node on the serial link detects an anomaly in the link communication with its neighboring node, proceed to step K;

[0023] C) The node determines its own role status. If it is in the primary state, it means that the backup node has crashed. No action is taken and the process ends. If it is in the backup state, it means that the primary node has crashed. Proceed to step D.

[0024] D) Set your own role status as the main status node and proceed to step E;

[0025] E) Complete the node role status settings and proceed to steps F and G;

[0026] F) Notify adjacent nodes in the same link of the role status change information, and proceed to step H;

[0027] G) Notify another node at the same location in another link of the role status change information, and proceed to step H;

[0028] H) Node receives role status change information and proceeds to step I;

[0029] I) The node that receives the role status change information makes a decision to change the role status in advance, and determines whether the role to be changed is the same as the current role status based on the decision result. If they are the same, the process ends; if they are different, proceed to step J.

[0030] J) Change your own character status to the desired character status, and then proceed to step E;

[0031] K) If a node detects a link communication anomaly, determine whether it is a complete link anomaly; if it is a complete link anomaly, proceed to step L; otherwise, proceed to step M.

[0032] L) The node sets its own role status to a fault state and proceeds to step E;

[0033] M) If a node detects an incomplete link anomaly, it determines whether it is the end node. If it is the end node, proceed to step N; if it is not the end node, the node does not perform a role state change and the fault process ends.

[0034] N) The end node determines whether the abnormal link is a communication link with the master status node of the previous node. If so, proceed to step L; otherwise, no role state change is performed and the fault handling process ends.

[0035] S203, Handling non-downtime link failures at nodes;

[0036] Proceed directly to step S202 above;

[0037] S204. Handling non-failure faults of the node;

[0038] Proceed directly to step L above.

[0039] (III) Beneficial Effects

[0040] This invention proposes a multi-node, dual-path serial high-availability cluster system and its management method. By establishing communication routing configurations on nodes, this invention flexibly implements the addition and deletion of primary and backup nodes in the serial multi-node cluster, meeting the different node requirements of various traffic flows. Through a vertical and horizontal dual notification mechanism for fault information, this invention achieves rapid propagation of fault information within the cluster, ensuring the primary / backup switchover performance of the entire serial cluster in the event of node failure. This invention employs a time-division dual-redundancy heartbeat mechanism, which avoids the split-brain problem in the cluster system while balancing heartbeat monitoring performance and system resource utilization, ensuring the stable and reliable operation of the cluster system.

[0041] Compared with existing technologies, the technical method proposed in this invention provides a highly available cluster system and management method suitable for serial pipelined services, offering three different inter-node communication methods to choose from, and allowing for flexible configuration of the number of nodes. Attached Figure Description

[0042] Figure 1 shows the interconnection topology of a dual-node high-availability cluster;

[0043] Figure 2 shows the interconnection topology of a multi-node high-availability cluster.

[0044] Figure 3 shows the node connection topology of the present invention;

[0045] Figure 4 shows the system's default primary / standby status;

[0046] Figure 5 shows the possible states of the system;

[0047] Figure 6 shows the main management process. Detailed Implementation

[0048] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0049] The purpose of this invention is to provide a multi-node dual-path serial high-availability cluster system and management method, which realizes primary and backup high-availability management suitable for multi-node serial pipeline business systems; supports three communication forms between serial nodes, including network, serial port, and SRIO; supports primary and backup linkage switching of dual-link serial nodes, that is, realizes primary and backup high-availability management; and supports flexible addition and reduction of the number of primary and backup nodes.

[0050] Figure 3 shows the topology of the multi-node dual-path serial high-availability cluster system in the technical solution of this invention.

[0051] As shown in Figure 3, the multi-node dual-path serial high-availability cluster system proposed in this invention includes the following:

[0052] (1) The system contains two serial nodes suitable for continuous business, and each serial node has more than or equal to 3 nodes.

[0053] (2) Two serial nodes at the same location are used as primary and backup nodes for redundancy to achieve high availability;

[0054] (3) The communication link between each serial node can be any of the following: network, serial port, or SRIO. However, the communication method between two nodes at the same location must be consistent, and the communication method cannot be changed during system operation. Two serial nodes at the same location (i.e., primary and backup nodes) are directly connected via network and serial port to achieve dual redundancy heartbeat. When the communication method between nodes Y and y at the end of the two links and the previous nodes X and x is network, these two pairs of serial nodes are allowed to achieve primary and backup node cross communication, as shown in the gray area of ​​Figure 3.

[0055] The multi-node dual-path serial high-availability cluster system management method proposed in this invention includes the following:

[0056] S101. Set the default node status. By default, each serial node in the first path is the master node, and each serial node in the second path is the backup node.

[0057] S102. Determine the communication method between nodes. Based on the actual hardware connection, determine the communication method between each node in the form of a configuration file, and establish the communication routing links between nodes.

[0058] S103. Start status monitoring on each node. This includes the internal operating status of the node and the communication status of nodes connected to that node on the same serial link;

[0059] S104. Start time-sharing dual-redundancy heartbeat monitoring on each node. That is, network heartbeat monitoring and serial port heartbeat monitoring. The two are not performed simultaneously. Network heartbeat monitoring is performed first, and serial port heartbeat monitoring is activated only when the network heartbeat is abnormal.

[0060] S105. Determine the role status of the nodes. Each node determines its initial role status based on its link number (link 1 or link 2), status monitoring (S103 above), and heartbeat monitoring (S104 above), specifically: primary status, backup status, or fault status.

[0061] S106. Inter-node role status coordination. Each node notifies its determined role status to adjacent nodes on the serial link. Upon receiving the role status information from its neighbors, each node adjusts its own role status. This ultimately determines the primary node link, backup node link, or faulty link of the system. By default, if there are no node or link anomalies, all nodes on the first path are primary nodes, i.e., the first path is the primary link, and the second path is the backup link, as shown in Figure 4. Figure 5 illustrates other possible primary and backup node role statuses.

[0062] S107. Node Role State Changes and Actions. Based on the changed role state, nodes execute different services and configurations, serving as the basis for high-availability primary / standby mode switching. For example, when a node becomes the primary state, the floating IP needs to be set to the current node; when a node changes to the standby state or a fault state, the floating IP needs to be removed from the current node.

[0063] Figure 6 illustrates the main workflow for managing a multi-node dual-path serial high-availability cluster system according to the technical solution of this invention. As shown in Figure 6, the management method for a multi-node dual-path serial high-availability cluster system includes the following steps:

[0064] S201. The system determines the primary node link and the backup node link.

[0065] S202. Handling Node Failures. When a node in the link experiences a failure, the fault handling procedure is as follows:

[0066] A) Nodes at the same location on the serial link detect each other's downtime fault through heartbeat monitoring between the primary and backup nodes (vertical diagram in Figure 3), and proceed to step C.

[0067] B) If any node on the serial link detects an anomaly in the communication with the adjacent node (shown horizontally in Figure 3), proceed to step K.

[0068] C) The node determines its own role status. If it is in the primary state, it means that the backup node has crashed. No action is taken and the process ends. If it is in the backup state, it means that the primary node has crashed. Proceed to step D.

[0069] D) Set your own role status as the main state node and proceed to step E.

[0070] E) Complete the node role status setting, then proceed to steps F and G.

[0071] F) Notify the adjacent nodes in the same link (shown horizontally in Figure 3) of the role status change information, and proceed to step H.

[0072] G) Notify another node at the same location in another link (vertical diagram in Figure 3) of the role status change information, and proceed to step H;

[0073] When node H receives the role status change information, it proceeds to step I.

[0074] I) The node that receives the role status change information makes a decision to change the role status in advance, and determines whether the role to be changed is the same as the current role status based on the decision result. If they are the same, the process ends; if they are different, proceed to step J.

[0075] J) Change your own role status to the pre-changed role status, and then proceed to step E.

[0076] K) Nodes that detect abnormal link communication determine whether it is a complete link failure. If it is a complete link failure, proceed to step L; otherwise, proceed to step M. A complete link failure refers to two situations: when the abnormal link is a single link, it is a complete link failure (as shown in Figure 3, node A detects abnormal communication with node B); when the link allows cross-communication, a complete link failure occurs when both links are abnormal (as shown in Figure 3, node X detects abnormal communication between nodes XY and between nodes Xy).

[0077] Node L) sets its own role status to fault state and proceeds to step E.

[0078] M) If a node detects an incomplete link anomaly, it determines whether it is the end node. If it is the end node, it proceeds to step N; if it is not the end node, the node does not perform a role state change and the fault handling process ends.

[0079] N) The end node determines whether the abnormal link is a communication link with the primary state node of the preceding node. If so, proceed to step L; otherwise, no role state change is performed, and the fault handling process ends. (As shown in Figure 3, assuming X is the primary and x is the backup, node Y determines whether the abnormal link is between Y and X or between Y and x. If it is between Y and X, proceed to L; if it is between Y and x, no action is taken because the current state does not affect the service flow.)

[0080] S203, Handling non-downtime link failures at nodes.

[0081] Proceed directly to step S202 above.

[0082] S204, Handling other faults other than node downtime.

[0083] Proceed directly to step L above.

[0084] This invention relates to a multi-node, dual-path serial high-availability cluster system and management method designed for serial pipelined services. Key aspects include the following three points:

[0085] By configuring communication routes on the nodes, the addition and deletion of primary and backup nodes in a serial multi-node cluster can be flexibly implemented to meet the different number of nodes required by different pipeline services.

[0086] By employing a dual notification mechanism for fault information, both vertically and horizontally (steps F and G in the scheme's process), fault information is rapidly propagated throughout the cluster, ensuring the performance of the entire serial cluster's master-slave failover when a node fails.

[0087] By adopting a time-sharing dual-redundancy heartbeat mechanism, the split-brain problem of the cluster system is avoided, and the heartbeat monitoring performance and system resource consumption are balanced, thus ensuring the stable and reliable operation of the cluster system.

[0088] Compared with existing technologies, the technical method proposed in this invention provides a highly available cluster system and management method suitable for serial pipelined services, offering three different inter-node communication methods to choose from, and allowing for flexible configuration of the number of nodes.

[0089] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-node dual-path serial high-availability cluster system, characterized in that, The system comprises two serial nodes suitable for continuous operation, with each serial node having at least three nodes. By default, the serial nodes in the first serial link are master nodes, and the serial nodes in the second serial link are backup nodes. Two corresponding nodes at the same position on the two serial links are redundantly configured as master and backup nodes for high availability. The communication link between each serial node can be any of the following: network, serial port, or SRIO. All corresponding nodes at the same position on the two serial links are directly connected via network and serial port to achieve dual-redundant heartbeat, and the connection method between nodes cannot be changed during system operation. Specifically, when the communication method of the serial nodes in the first serial link is network communication, and the communication method of the serial nodes in the second serial link is also... For network communication, the terminal node of the first path is allowed to communicate with the previous neighbor node of the terminal node of the second path, and the previous neighbor node of the terminal node of the first path is allowed to communicate with the terminal node of the second path, thus realizing cross-communication between the terminal nodes of the first and second paths and their previous neighbor nodes. Network heartbeat monitoring and serial port heartbeat monitoring are started on each node. Network heartbeat monitoring is performed first, and serial port heartbeat monitoring is enabled only when the network heartbeat is abnormal. When a node fails, the whole system switches in a coordinated manner. The state change of each node is consistent. That is, when any node in each serial link switches to the master state or the backup state, all other nodes on the corresponding serial link must also switch to the master state or the backup state.

2. A multi-node dual-path serial high-availability cluster management method based on the cluster system of claim 1, characterized in that, The method includes the following steps: S101, setting the default state of the nodes; S102, determining the communication form between nodes; based on the actual hardware connection, determining the communication form between each node in the form of a configuration file, and establishing the communication routing link between the nodes; S103, starting status monitoring on each node; including the internal operating status of the node and the communication status of connected nodes on the same serial link; S104, starting time-division dual-redundancy heartbeat monitoring on each node; Specifically, network heartbeat monitoring and serial port heartbeat monitoring; The two are not performed simultaneously. Network heartbeat monitoring is performed first, and serial port heartbeat monitoring is activated only when the network heartbeat is abnormal. S105: Determine the role status of the nodes. Each node determines its initial role status based on its link number, status monitoring results, and heartbeat monitoring results, specifically: primary, backup, or fault status. S106: Coordinate the role status between nodes. Each node notifies its determined role status to the adjacent nodes on the serial link. Upon receiving the role status information from the adjacent nodes, the node adjusts its own role status. Finally, the primary node link, backup node link, or fault link of the system is determined. By default, if there are no node or link abnormalities, all nodes on the first path are primary nodes, i.e., the first path is the primary link, and the second path is the backup link. S107: Changes and actions of node role status: Nodes execute different services and configurations based on their changed role status, which serves as the basis for switching between primary and backup modes for high availability services.

3. A multi-node dual-path serial high-availability cluster management method based on the cluster system of claim 1, characterized in that, The method includes the following steps: S201, the system determines the primary node link and the backup node link; S202, handle node failure; when a node in the link fails, the fault handling process is as follows: A) Nodes at the same position on the serial link detect the failure of the other party through the heartbeat between the primary and backup nodes, and proceed to step C; B) Any node on the serial link detects an abnormality in the link communication with the adjacent node, and proceeds to step K; C) The node determines its own role status. If it is in the primary state, it means the backup node has crashed, no action is taken, and the process ends. If it is in the backup state, it means the primary node has crashed, and proceeds to step D. D) The node sets its own role status to the primary state node and proceeds to step E. E) The node completes the role status setting and proceeds to steps F and G. F) The node notifies adjacent nodes in the same link of the role status change information and proceeds to step H. G) The node notifies another node in the same position in another link of the role status change information and proceeds to step H. H) The node receives the role status change information and proceeds to step I. I) The node that receives the role status change information makes a decision on the pre-change role status and determines whether the pre-change role is the same as the current role status based on the decision result. If they are the same, the process ends. If they are different, proceed to step J; J) Change its own role status to the pre-changed role status, and then proceed to step E; K) Nodes that detect abnormal link communication determine whether it is a complete link failure; if it is a complete link failure, proceed to step L; otherwise, proceed to step M; L) Nodes set their own role status to a fault state and proceed to step E. M) Nodes that detect incomplete link anomalies determine whether they are end nodes. If they are end nodes, proceed to step N. If it is not an end node, no role state change processing is performed, and the fault process ends; N) The end node determines whether the abnormal link is a communication link with the master state node of the previous node. If so, proceed to step L. Otherwise, no role state change is performed, and the fault handling process ends; S203, handle non-failed link faults of the node; proceed directly to step B); S204, handle other non-failed faults of the node; proceed directly to step L); In step K, when the abnormal link is a single link, the communication abnormality of the link is a complete link abnormality; when the link is a cross-communication link, the abnormality of both links is a complete link abnormality.

Images