Keepalived-based split-brain problem solving method, medium and equipment

Abstract

The invention provides a Keepalived-based split-brain problem solving method, a medium and equipment, and belongs to the technical field of cloud computing. The Keepalived enters an initial state, the priority of each node is judged, the nodes are set as Mater master nodes according to the priorities, and the master nodes send VRRP heartbeat; the Keepalived-expander sends a USR2 signal to the Keepalived host process at a fixed time, so that the Keepalived host process updates a state configuration file / tmp / Keepalived. Stats, and the Keepalived-expander sends a USR2 signal to the Keepalived host process at a fixed time; when / tmp / Keepalived. Stats read-write ramming is carried out, the standby node cannot receive the VRRP heartbeat sent by the main node, an arbitration mechanism is started for detection, and the main node and the standby node are switched according to the ping connection condition between the main node and the standby node and the priority comparison condition between the main node and other main nodes. The system reliability and fault tolerance are improved, automatic fault transfer is achieved, the service interruption time is shortened, and management is simplified.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing, and in particular relates to a solution, medium and device for solving the split-brain problem based on Keepalived. Background Art

[0002] Keepalived is an open-source high-availability solution, primarily used to achieve high availability and load balancing for services. However, due to design flaws in Keepalived itself, when disk IO is jammed on the virtual or physical machine where it resides, VRRP (Virtual Router Redundancy Protocol) heartbeats fail to be sent, resulting in split-brain and multiple-master and multiple-VIP (Virtual IP) servers. This leads to the following high-availability issues:

[0003] (1) Service conflict: When two high-availability servers cannot detect each other's heartbeat, they may each think they are the master node and start services at the same time, resulting in a service conflict.

[0004] (2) Data inconsistency or loss: If a split-brain occurs on a stateful service (such as a database), it may cause data inconsistency or loss because two nodes may write to the same data at the same time.

[0005] (3) IP address conflict: Two servers may occupy the same VIP at the same time, similar to the IP address conflict within the local area network, causing one or two nodes to fail to work properly, affecting user access to the server.

[0006] (4) Service interruption: Due to competition between nodes, service interruption may occur, and users may not be able to access or use the services originally managed by Keepalived.

[0007] (5) Resource competition: When a split-brain occurs, the two nodes will compete for shared resources, which may result in the resources being divided and the service being unable to start or run normally.

[0008] (6) Data corruption: If two nodes read and write to shared storage at the same time, data corruption may occur, especially for critical services such as databases.

[0009] There is currently no effective solution to the above Keepalived issues. Summary of the Invention

[0010] The present application aims at the deficiencies in the prior art, and provides a Keepalived-based brain split problem solving method, medium and equipment. The present application is applicable to related applications of Keepalived, such as master-slave switching (failover), load balancing, health check, network layer fault detection, application layer fault detection and various scenes, and can effectively solve the Keepalived VRRP heartbeat sending failure caused by disk IO jamming in the Keepalived running process, and the brain split problem of multiple masters (multiple master nodes) and multiple VIPs.

[0011] To achieve the above object, the present application adopts the following technical solutions:

[0012] In a first aspect, the present application provides a Keepalived-based brain split problem solving method, comprising the following steps:

[0013] Keepalived enters an initial state, judges the priority of each node, sets the node as a Mater master node according to the priority, and the master node sends a VRRP heartbeat;

[0014] Keepalived-exporter sends a USR2 signal to Keepalived master process at a regular time to make it update the state configuration file / tmp / Keepalived.stats of Keepalived, which is used to provide Keepalived state information;

[0015] When reading and writing / tmp / Keepalived.stats, the backup node cannot receive the VRRP heartbeat sent by the master node, starts an arbitration mechanism for detection, and switches the master and backup nodes according to the ping-through condition between the master and backup nodes and the priority comparison condition between the master node and other master nodes.

[0016] Optionally, the node is set as the Mater master node according to the priority, and specifically, if the priority is 255, the node becomes the Mater master node directly.

[0017] Optionally, the VRRP heartbeat is sent once every 2 seconds, and a total of 3 times.

[0018] Optionally, the arbitration mechanism is started for detection, and specifically:

[0019] The master node sends a ping message to the backup node at the same time;

[0020] If the backup node can be pinged normally, the backup node is not promoted to the master node, still maintains the backup node state, and does not obtain the VIP.

[0021] If the backup node cannot be pinged normally, the priority of the current master node is compared, if the priority is greater than that of other nodes, the state of the current master node is maintained, if the priority is less than that of other nodes, the backup node is switched, and if the priority is the same, the master and backup nodes are switched by comparing the interface host IP.

[0022] Optionally, the switching of the master and backup nodes by comparing the interface host IP comprises:

[0023] If the priority is the same, the interface host IP is compared, if the IP is the maximum or consistent with the VRRP backup group virtual IP, the state of the current master node is maintained, and if the interface host IP is not the maximum, the backup node is switched.

[0024] Optionally, the ping message is 1 time per second, and a total of 3 times.

[0025] In a second aspect, the application provides a computer readable storage medium, which stores a computer program, and the computer program enables a computer to execute the Keepalived-based split-brain problem solving method according to the first aspect.

[0026] In a third aspect, the application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the Keepalived-based split-brain problem solving method according to the first aspect is implemented.

[0027] The application has the following beneficial effects:

[0028] (1) Improving system reliability and fault tolerance: after adopting the scheme of the application, Keepalived can ensure that only one node provides services in a multi-node high-availability cluster, avoiding conflicts and data inconsistency problems caused by multiple nodes providing services at the same time. In this way, Keepalived can improve the reliability and fault tolerance of the entire system, ensuring the continuity and stability of services.

[0029] (2) Automatic failover: when the Master node fails, Keepalived can automatically switch traffic to the backup node, realizing seamless switching of services and minimizing service interruption time, thereby improving service availability.

[0030] (3) Reducing service interruption time: after adopting the scheme of the application, the split-brain problem caused by disk ramming can be effectively solved, thereby significantly reducing service interruption time.

[0031] (Four) Simplify management: after adopting the scheme of the application, the administrator does not need to manually intervene in the failover process, all fault detection and transfer are automatically completed by Keepalived, which greatly simplifies the management complexity. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Is the generation mechanism of the split-brain problem based on Keepalived.

[0033] Figure 2 Is the strace Keepalived master process open stats file stuck problem.

[0034] Figure 3 Is the USR1USR2 signal sent by Keepalived-exporter to Keepalived master process.

[0035] Figure 4 Is the Keepalived.stats file record Keepalived state information.

[0036] Figure 5 Is the open file flag in Keepalived code, vrrp_print.c file and vrrp_print_stats function.

[0037] Figure 6 Is the split-brain problem solving method based on Keepalived. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application.

[0039] Keepalived generally runs in a multi-node environment, has a master-standby mode and a master-multiple standby mode, the master node has a VIP to provide services externally, the standby node does not provide VIP services externally, and the master node sends VRRP heartbeat packets to the standby node. In the case that the standby node fails to receive the master node heartbeat for three times, the standby node is upgraded to the master node, and itself has a VIP. At this time, the Keepalived cluster has multiple VIPs to provide services externally, thereby causing split-brain.

[0040] In VRRP, the Master node is the node currently handling network traffic, while the Backup node is in standby state, ready to take over in the event of Master node failure. Each node in Keepalived has a priority, and when the Master node is unavailable, the Backup node with the highest priority will become the new Master. Nodes in Keepalived can be in different states, such as INIT (initialization), MASTER (master), BACKUP (backup), and FAULT (fault).

[0041] / tmp / keepalived.stats is the state configuration file of Keepalived, which records the statistical file of state information. This file usually contains some statistical data of its runtime, such as the state of VRRP instance, priority, virtual IP address, etc. The file is mainly exported to Prometheus Zabbix monitoring system through Keepalived-exporter to display the monitoring state.

[0042] As shown in Figure 1 , under normal circumstances, the master sends heartbeats to the backup server every 2 seconds. When the master writes a file, it needs to wait for a short period of time, during which it cannot send heartbeats to the backup. If the backup does not receive a heartbeat from the master within 6 seconds, it will upgrade to the master, broadcast free VRRP, VIP brain split, and the switch will judge IP conflict. Figure 2 For distributed storage clusters, when IO jam occurs due to node power failure, strace Keepalived master process open stats file appears to be stuck. From this, it can be seen that it has been stuck for about 13s. Figure 3 For Keepalived-exporter to send USR1 and USR2 signals to Keepalived master process (through String grabbing). Figure 4 For Keepalived.stats file, which mainly records Keepalived state information. Figure 5 For the open file flags in Keepalived code, the vrrp_print.c file, and the vrrp_print_stats function.

[0043] In an embodiment, the present application proposes a Keepalived-based brain split problem solving method. By introducing a three-party arbitration mechanism, it can ensure that in the case of IO jam in the underlying storage system, i.e. when the / tmp / keepalived.stats file is stuck in write-only open, the problem of blocking the master process heartbeat mechanism can be effectively solved. The logic flow chart is as Figure 6 . Specifically, the steps include:

[0044] (1) keepalived enters the initial initial state, first judge the priority of master and backup nodes, if the priority is 255, the timer ends and switches to Master, and sends VRRP state message.

[0045] (2) Keepalived-exporter timing (every min, configurable) sends USR2 signal to Keepalived master process to update / tmp / Keepalived.stats file, and provides Keepalived state information to Prometheus Zabbix monitoring system.

[0046] (3) After Keepalived master process receives USR2 signal, the storage system fails or produces IO jam, tamp master problem, open / tmp / Keepalived.stats file fails, such as distributed storage cluster node suddenly power off, or storage system disk, network accident failure, etc. cause VRRP heartbeat sending failure.

[0047] (4) The backup node cannot receive the VRRP heartbeat sent by the master node (once every 2 seconds, a total of 3 times, configurable).

[0048] The following is the arbitration mechanism added when / tmp / Keepalived.stats is read and written to tamp:

[0049] (5) The master node sends ping message to the backup node at the same time, once per second (configurable), a total of 3 times (configurable).

[0050] (6) If the backup node can be pinged normally, the backup node does not become master, still maintains the backup node state, and does not obtain VIP.

[0051] (7) If the backup node cannot be pinged normally, compare the node priority, and if the priority is greater than other nodes, keep the Master node state.

[0052] (8) If the priority is less than other nodes, become Backup node directly.

[0053] (9) If the priority is the same, compare the interface host IP, and if the IP is the largest or consistent with the VRRP backup group virtual IP, keep the Master node state.

[0054] (10) If the interface host IP address is smaller, become Backup node directly.

[0055] Through the above network connectivity detection arbitration, the problem of brain split caused by the fact that the Keepalived master process cannot send VRRP heartbeat due to the fact that / tmp / Keepalived.stats cannot be written because of the storage card segment is eliminated.

[0056] In fact, the core requirement of Keepalived detection is to detect the network connectivity of different nodes in the cluster and the network state of the nodes, and then generate various states such as master-slave node switching and brain split. The storage card segment should not be a factor affecting the Keepalived brain split problem.

[0057] The application is suitable for multiple scenarios using Keepalived, is widely applied, and the typical application scenarios are as follows:

[0058] (1) High availability (HA) scenario: Keepalived handles load balancer failover by implementing Virtual Router Redundancy Protocol (VRRP). In the event of a primary server failure, Keepalived can automatically switch traffic to a backup server, ensuring uninterrupted service.

[0059] (2) Load balancing (LB): Keepalived can be used in conjunction with LVS to provide four-layer load balancing functionality. Traffic can be distributed to multiple backend servers, improving throughput and ensuring high availability of services.

[0060] (3) Health check: Keepalived ensures that only healthy servers provide services by performing health checks on backend servers. This can be done via TCP, HTTP, or custom scripts. Once a server is found to be problematic, Keepalived automatically removes it from the load balancing pool.

[0061] (4) Virtual IP (VIP) management: Keepalived can manage virtual IPs on multiple servers, making external access independent of specific physical servers. When a server goes down, the VIP can automatically float to another server, ensuring service continuity.

[0062] (5) Distributed storage cluster: In distributed storage systems such as Ceph, GlusterFS, etc., Keepalived can ensure the high availability of storage nodes by monitoring the status of storage nodes and automatically switching storage services in the event of node failure, avoiding data loss.

[0063] In another embodiment, the application proposes a computer readable storage medium storing a computer program, the computer program causing a computer to execute the Keepalived-based brain split problem solving method of the preceding embodiment.

[0064] In another embodiment, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the Keepalived-based split-brain problem solution method of the foregoing embodiments when executing the computer program.

[0065] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium which can contain or store programs for use by or in connection with an instruction execution system, apparatus or device. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer storage medium can include one or more wires, portable computer disks, hard drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disk read-only memory (CDROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0066] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0067] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application shall be considered within the protection scope of the present application.

Claims

1. A solution to the split-brain problem based on Keepalived, characterized in that: The steps include: Keepalived enters the initial state, determines the priority of each node, sets the node as the master node according to the priority, and the master node sends VRRP heartbeat; Keepalived-exporter periodically sends USR2 signals to the Keepalived main process to update the Keepalived status configuration file / tmp / Keepalived.stats, which is used to provide Keepalived status information; When the read and write of / tmp / Keepalived.stats is stuck, the standby node cannot receive the VRRP heartbeat sent by the master node, and the arbitration mechanism is started for detection. The master and standby nodes are switched based on the ping success between the master and standby nodes and the priority comparison between the master node and other master nodes.

2. The Keepalived-based split-brain solution according to claim 1, characterized in that: The node is set as the master node according to the priority, specifically: if the priority is 255, it directly becomes the master node.

3. The method for solving the split-brain problem based on Keepalived according to claim 1, characterized in that: The VRRP heartbeat occurs once every 2 seconds, 3 times in total.

4. The method for solving the split-brain problem based on Keepalived according to claim 1, wherein: The arbitration mechanism is started to perform detection, specifically: The master node sends a ping message to the standby node at the same time; If the standby node can be pinged normally, the standby node will not be promoted to the primary node and will remain in the standby node status without obtaining VIP. If the standby node cannot be pinged normally, the priority of the current active node is compared. If the priority is higher than that of other nodes, the current active node is maintained. If the priority is lower than that of other nodes, it will switch to the backup node; If the priorities are the same, the master and backup nodes are switched by comparing the interface host IP addresses.

5. The method for solving the split-brain problem based on Keepalived according to claim 4, characterized in that: The switching of the primary and backup nodes is performed by comparing the interface host IP addresses, specifically: If the priorities are the same, the interface host IP is compared. If the IP is the largest or the same as the VRRP backup group virtual IP, the current master node status is maintained; if the interface host IP is not the largest, the node is switched to the backup node.

6. The method for solving the split-brain problem based on Keepalived according to claim 4, characterized in that: The ping message is sent once per second, three times in total.

7. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the Keepalived-based split-brain problem solution according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the split-brain solution based on Keepalived is implemented as described in any one of claims 1 to 6.

Images