Server fault detection processing method and device

Abstract

The invention is suitable for the technical field of Internet, and provides a server fault detection processing method and device. In the invention, a management node is periodically connected to a master node in a node cluster through an SSH (Secure Shell), checks heartbeat information of the master node or executes an SQL (Structured Query Language) query to confirm a health state of the master node, and when the master node fails, the management node is responsible for managing a scheduling switching process, determining a target slave node and lifting the target slave node as a new master node; and redirecting all other slave nodes to the new master node, and maintaining the configuration information of the slave nodes. The problem that data synchronization is not timely or data is inconsistent due to the fact that the health state of the main node is abnormal is solved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a server fault detection and processing method and device. Background Art

[0002] With the rapid development of information technology, server clusters have become an indispensable infrastructure in modern data centers and cloud computing environments. Server clusters combine multiple servers into a logical unit to jointly process tasks, which not only improves the processing power of the system, but also enhances the reliability and scalability of the system. In server clusters, data consistency and high availability are crucial, especially when processing critical business data and real-time transactions.

[0003] In a typical server cluster architecture, a node cluster consists of multiple data nodes, which are usually divided into a master node and several slave nodes. The master node is responsible for handling the main read and write operations, while the slave node provides data redundancy and backup by replicating the master node's data to ensure high data availability. The master node and the slave node maintain communication through a heartbeat link to synchronize data status and ensure data consistency.

[0004] However, in actual applications, server clusters face a variety of potential failure risks, such as hardware failure, software errors, network interruptions, etc., which may lead to abnormal node health status, untimely data synchronization, or data inconsistency. These problems will not only affect the normal operation of the system, but may also cause data loss or business interruption, causing significant losses to the enterprise.

[0005] In order to meet these challenges, the industry has developed a variety of server fault detection and processing technologies. However, existing technologies often have problems such as untimely detection, incomplete processing or low automation, which makes it difficult to meet complex and changing business needs. Especially in large-scale, high-concurrency server cluster environments, how to quickly and accurately detect faults, automatically switch master nodes and restore data consistency has become an urgent problem to be solved. Summary of the invention

[0006] In view of this, the present application provides a server fault detection and processing method and device to achieve rapid completion of fault switching when a primary server fails, and to ensure data consistency during the fault switching process.

[0007] In a first aspect, the present application provides a server fault detection and processing method, which is applied to a management node MHA Manager of a node cluster, wherein the node cluster further includes a plurality of data nodes, wherein the data nodes are divided into a master node and a plurality of slave nodes, and the master node and the slave nodes synchronize data via a heartbeat link; the method comprises:

[0008] Monitor each node in the node cluster, and issue an alarm message when an abnormality is determined, wherein the abnormality includes abnormal health status of the master node, untimely data synchronization between the master node and the slave node, and inconsistent data between the master node and the slave node;

[0009] Detect the health status of the master node, and when it is determined that the master node fails, determine a target slave node from the slave nodes according to a preset rule, determine the target slave node as the master node, and synchronously modify the configuration information of the slave node;

[0010] Re-establish the heartbeat link between the master node and the slave node to ensure data synchronization and consistency.

[0011] Optionally, determining that the master node fails includes:

[0012] The management node periodically connects to the master node through SSH, and when the master node fails to respond, it is determined that the master node is faulty; or,

[0013] The replication delay time between the master node and the slave node is determined, and when the replication delay time exceeds a threshold, it is determined that the master node is faulty.

[0014] Optionally, determining the target slave node from the slave nodes according to a preset rule includes:

[0015] The CPU occupancy rate and memory occupancy rate of each slave node are determined, and the slave node with the lowest CPU occupancy rate or memory occupancy rate is determined as the target slave node.

[0016] Optionally, the method further includes:

[0017] The slave nodes are synchronously managed according to the user's operation instructions, and the configuration information of the slave nodes is maintained, wherein the operation instructions include deletion instructions, addition instructions or modification instructions.

[0018] Optionally, determining the target slave node as a master node includes:

[0019] Stop writing to the master node, and record the name, location, and system status information of the binary log file of the master node;

[0020] Use an SQL command to stop the replication process of the target slave node, modify the configuration file of the slave node, and determine the target slave node as the master node for operation.

[0021] A second aspect of the present application provides a server fault detection and processing device, which is applied to a management node MHA Manager of a node cluster, wherein the node cluster further includes a plurality of data nodes, wherein the data nodes are divided into a master node and a plurality of slave nodes, and the master node and the slave nodes synchronize data via a heartbeat link; the device includes:

[0022] A node monitoring unit is used to monitor each node in the node cluster and issue an alarm message when an abnormality is determined, wherein the abnormality includes abnormal health status of the master node, untimely data synchronization between the master node and the slave node, and inconsistent data between the master node and the slave node;

[0023] A master node adjustment unit, configured to detect the health status of the master node, and when determining that the master node fails, determine a target slave node from the slave nodes according to a preset rule, determine the target slave node as the master node, and synchronously modify the configuration information of the slave node;

[0024] The heartbeat link reconstruction unit is used to re-establish the heartbeat link between the master node and the slave node to ensure data synchronization and consistency.

[0025] Optionally, the determining of the master node failure in the master node adjustment unit includes:

[0026] The management node periodically connects to the master node through SSH, and when the master node fails to respond, it is determined that the master node is faulty; or,

[0027] The replication delay time between the master node and the slave node is determined, and when the replication delay time exceeds a threshold, it is determined that the master node is faulty.

[0028] Optionally, the determining of the target slave node from the slave nodes according to a preset rule in the master node adjustment unit includes:

[0029] The CPU occupancy rate and memory occupancy rate of each slave node are determined, and the slave node with the lowest CPU occupancy rate or memory occupancy rate is determined as the target slave node.

[0030] Optionally, the device further comprises:

[0031] The configuration information maintenance unit is used to synchronously manage the slave nodes according to the user's operation instructions and maintain the configuration information of the slave nodes, wherein the operation instructions include deletion instructions, addition instructions or modification instructions.

[0032] Optionally, determining the target slave node as the master node in the master node adjustment unit includes:

[0033] Stop writing to the master node, and record the name, location, and system status information of the binary log file of the master node;

[0034] Use an SQL command to stop the replication process of the target slave node, modify the configuration file of the slave node, and determine the target slave node as the master node for operation.

[0035] In the embodiment provided by the present application, the management node periodically connects to the master node in the node cluster via SSH, checks its heartbeat information or executes SQL queries to confirm its health status. When the master node fails, the management node is responsible for managing the scheduling switching process, determining the target slave node and promoting it to the new master node, and then redirecting all other slave nodes to the new master node, and maintaining the configuration information of the slave nodes. This solves the problem of delayed data synchronization or data inconsistency due to abnormal health status of the master node.

[0036] This technical solution has some significant advantages over the traditional master-slave replication structure:

[0037] Automatic failover: MHA can automatically monitor the status of the MySQL master host and automatically failover when a failure is detected. This greatly reduces the need for manual intervention by the DBA (database administrator) and improves the availability and reliability of the system.

[0038] Data consistency: MHA can automatically repair consistency issues between slaves during failover, ensuring that all slaves are consistent with the new master. This avoids potential problems caused by data inconsistency.

[0039] Faster recovery time: MHA can complete the database failover operation within 0 to 30 seconds, and during the failover process, MHA can ensure data consistency to the greatest extent. This fast recovery time is very important for applications that require high availability.

[0040] Reduce the risk of data loss: During the MHA automatic failover process, MHA will try to save the binary log from the downed master server to ensure that data is not lost to the greatest extent possible.

[0041] Flexibility and scalability: MHA supports a variety of configuration options and tools that can be customized and optimized according to specific needs. In addition, as the business grows, MHA can be easily expanded to support more slave servers and more complex deployment scenarios.

[0042] In contrast, although the traditional MySQL master-slave replication structure also has certain advantages in data redundancy and performance improvement, it is usually not as flexible and efficient as MHA in terms of failover, data consistency guarantee and recovery time. Therefore, in scenarios that require high availability, fast recovery and data consistency, MHA is usually a better choice.

[0043] Combined with instant communication software, automatic alarms are configured to notify relevant technical and business personnel in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of the method provided in the embodiment of the present application;

[0045] Figure 2 A structural diagram of a device provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] The present application provides a server fault detection and processing method and device to solve the problem of untimely data synchronization or data inconsistency caused by abnormal health status of the master node.

[0051] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0052] like Figure 1 As shown, it is a flow chart of a server fault detection and processing method provided by the present application, which is applied to the management node MHA Manager of the node cluster, and the node cluster also includes a plurality of data nodes, and the data nodes are divided into a master node and a plurality of slave nodes. The master node and the slave nodes synchronize data through a heartbeat link, and the process includes:

[0053] Step S101, monitor each node in the node cluster, and issue an alarm message when an abnormality is determined. The abnormality includes abnormal health status of the master node, untimely data synchronization between the master node and the slave node, and inconsistent data between the master node and the slave node.

[0054] In this embodiment, the MHA Manager can periodically connect to the master node via SSH to check its heartbeat information or execute SQL queries to confirm its health status, and promptly alarm when an abnormality occurs;

[0055] At the same time, MHA Manager can also monitor the replication delay time between the slave node and the master node. When the replication delay time exceeds the preset threshold, it determines that the master node has a fault and issues an alarm in time.

[0056] You can also use the pt-table-checksum tool to regularly check the master node and slave node. When data inconsistency occurs, determine the abnormality and issue an alarm in time.

[0057] Step S102, detecting the health status of the master node, and when determining that the master node fails, determining a target slave node from the slave nodes according to a preset rule, determining the target slave node as the master node, and synchronously modifying the configuration information of the slave node.

[0058] In this embodiment, there are many methods to determine whether the master node is faulty, such as:

[0059] 1. Heartbeat detection: The master node and the slave node maintain communication through the heartbeat mechanism, and regularly send heartbeat signals to confirm each other's existence and status. The management node MHA Manager can monitor these heartbeat signals. If the heartbeat signal of the master node is not received within a certain period of time, or the frequency and content of the heartbeat signal are abnormal, it can be preliminarily judged that the master node may have a fault.

[0060] 2. Performance monitoring: The management node MHA Manager also continuously monitors the performance indicators of the master node, such as CPU usage, memory occupancy, disk I / O, network bandwidth, etc. If any one or more of these indicators exceeds the preset threshold and lasts for a period of time (such as a few minutes to a few hours), this may indicate that the master node is experiencing a performance bottleneck or resource exhaustion, which may cause a failure.

[0061] 3. Error log analysis: The error log of the master node is an important source for diagnosing problems. The management node MHAManager can regularly scan and analyze these log files to find error messages, warnings, or exceptions related to failures. These logs may contain detailed information about software crashes, hardware failures, configuration errors, and other issues, which helps to quickly locate the cause of the problem.

[0062] 4. Service status check: The management node MHA Manager will also check the status of key services (such as database services, Web services, etc.) running on the master node. By calling the status query interface provided by the service or using system commands (such as systemctl status, service status, etc.), you can obtain information such as the service's running status, start time, process ID, etc. If the key service is not running or the status is abnormal, it can be determined that the master node has failed.

[0063] After the failure is determined, the target slave node is determined from the slave nodes according to a preset rule (such as randomly, or in numerical order). After the target slave node is determined, a series of preparations need to be performed on it. This may include: pausing the data replication process on the target slave node to avoid data conflicts during the switch. Raising the priority or role of the target slave node to indicate that it will become the new master node. Checking the configuration file of the target slave node to ensure that its configuration is suitable for running as a master node, including network settings, storage configuration, log level, etc. Then perform the master-slave switch: performing necessary cleanup work on the original master node, such as shutting down the database service, stopping write operations, etc., to ensure that there are no new data changes. Update the replication configuration of other slave nodes to point them to the new master node (that is, the previous target slave node). This is usually done through the CHANGEMASTER TO command or other similar mechanisms. Start the database service on the new master node and ensure that it can accept and process read and write requests normally. Verify that data synchronization between the new master node and other slave nodes has started and is proceeding smoothly.

[0064] In another embodiment, determining the target slave node from the slave nodes according to a preset rule comprises:

[0065] The CPU occupancy rate and memory occupancy rate of each slave node are determined, and the slave node with the lowest CPU occupancy rate or memory occupancy rate is determined as the target slave node.

[0066] In this embodiment, the performance of the slave node can also be determined based on the disk I / O of the slave node, data synchronization delay, stability of the network connection, etc., and then the slave node with the highest performance is determined as the target slave node.

[0067] In another embodiment, determining the target slave node as a master node comprises:

[0068] Stop writing to the master node, and record the name, location, and system status information of the binary log file of the master node;

[0069] Use an SQL command to stop the replication process of the target slave node, modify the configuration file of the slave node, and determine the target slave node as the master node for operation.

[0070] Step S103, re-establishing the heartbeat link between the master node and the slave node to ensure data synchronization and consistency.

[0071] In this embodiment, first, it is necessary to confirm that the new master node has successfully taken over the work of the original master node, including processing read and write requests, managing data changes, etc. Then, the configuration information of all slave nodes needs to be updated to point to the new master node. This usually involves modifying the replication configuration of the slave node, including the IP address, port number, replication user and password of the master node, etc. On the slave node, the replication threads (IO threads and SQL threads) need to be started or restarted so that they can connect to the new master node and start receiving and applying data changes. Finally, through monitoring tools or logs, verify whether the new master node is sending heartbeat signals to the slave node and whether the slave node can successfully receive and respond to these signals. The frequency and response time of the heartbeat signal should meet the preset expected values. In order to verify the effectiveness and reliability of re-establishing the heartbeat link, it is recommended to perform fault recovery drills regularly. This can help discover and solve potential problems and optimize the failover and recovery process.

[0072] It is important to note that during the process of re-establishing the heartbeat link, special attention should be paid to data consistency and service continuity. During the switchover process, it should be ensured that no data is lost or corrupted and that applications can continue to access database services. In addition, it should be ensured that all slave nodes can successfully connect to the new master node and synchronize data correctly.

[0073] In another embodiment, the above method further includes:

[0074] The slave nodes are synchronously managed according to the user's operation instructions, and the configuration information of the slave nodes is maintained, wherein the operation instructions include deletion instructions, addition instructions or modification instructions.

[0075] In this embodiment, since the slave nodes in the node cluster may increase or decrease, this embodiment can realize the synchronous management of each slave node by modifying the configuration information after the slave nodes are increased or decreased. And when processing these operation instructions, the control node can provide sufficient logging and error handling mechanisms so that problems can be quickly located and solved when they occur. In addition, in order to ensure data consistency and service continuity, these operations should be performed during low-load periods and minimize the impact on existing services.

[0076] So far, completed Figure 1 The process shown.

[0077] In the embodiment of the present application, the management node periodically connects to the master node in the node cluster via SSH, checks its heartbeat information or executes SQL queries to confirm its health status. When the master node fails, the management node is responsible for managing the scheduling switching process, determining the target slave node and promoting it to the new master node, and then redirecting all other slave nodes to the new master node, and maintaining the configuration information of the slave nodes. This solves the problem of delayed data synchronization or data inconsistency due to abnormal health status of the master node.

[0078] The present application also provides a server fault detection and processing device, which is applied to the management node MHAManager of the node cluster. The node cluster also includes several data nodes, and the data nodes are divided into a master node and several slave nodes. The master node and the slave nodes synchronize data through a heartbeat link.

[0079] like Figure 2 As shown, the device comprises:

[0080] The node monitoring unit 201 is used to monitor each node in the node cluster and issue an alarm message when an abnormality is determined. The abnormality includes abnormal health status of the master node, untimely data synchronization between the master node and the slave node, and inconsistent data between the master node and the slave node;

[0081] A master node adjustment unit 202 is used to detect the health status of the master node. When it is determined that the master node fails, a target slave node is determined from the slave nodes according to a preset rule, the target slave node is determined as the master node, and the configuration information of the slave node is synchronously modified;

[0082] The heartbeat link reconstruction unit 203 is used to re-establish the heartbeat link between the master node and the slave node to ensure data synchronization and consistency.

[0083] In another embodiment, determining the master node failure in the master node adjustment unit includes:

[0084] The management node periodically connects to the master node through SSH, and when the master node fails to respond, it is determined that the master node is faulty; or,

[0085] The replication delay time between the master node and the slave node is determined, and when the replication delay time exceeds a threshold, it is determined that the master node is faulty.

[0086] In another embodiment, determining the target slave node from the slave nodes according to a preset rule in the master node adjustment unit includes:

[0087] The CPU occupancy rate and memory occupancy rate of each slave node are determined, and the slave node with the lowest CPU occupancy rate or memory occupancy rate is determined as the target slave node.

[0088] In another embodiment, the apparatus further comprises:

[0089] The configuration information maintenance unit 204 is used to perform synchronization management on the slave nodes according to the user's operation instructions and maintain the configuration information of the slave nodes, wherein the operation instructions include deletion instructions, addition instructions or modification instructions.

[0090] In another embodiment, determining the target slave node as the master node in the master node adjustment unit includes:

[0091] Stop writing to the master node, and record the name, location, and system status information of the binary log file of the master node;

[0092] Use an SQL command to stop the replication process of the target slave node, modify the configuration file of the slave node, and determine the target slave node as the master node for operation.

[0093] The above embodiment of the present invention provides a server fault detection and processing method, and based on this method provides a server fault detection and processing device. Through the above method and device, the problem of untimely data synchronization or data inconsistency caused by abnormal health status of the master node is solved.

[0094] This embodiment also discloses a computer device, such as Figure 3 As shown, the computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement any of the above-mentioned server fault detection and processing methods.

[0095] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A server fault detection and processing method, characterized in that: A management node MHAManager applied to a node cluster, wherein the node cluster further includes a plurality of data nodes, wherein the data nodes are divided into a master node and a plurality of slave nodes, and data is synchronized between the master node and the slave nodes via a heartbeat link; the method includes: Monitor each node in the node cluster, and issue an alarm message when an abnormality is determined, wherein the abnormality includes abnormal health status of the master node, untimely data synchronization between the master node and the slave node, and inconsistent data between the master node and the slave node; Continuously monitor the performance indicators of the master node. When the performance indicators exceed a preset threshold, determine that the master node is faulty, determine the target slave node from the slave nodes according to preset rules, suspend the data replication process on the target slave node, increase the priority of the target slave node, and check the configuration file of the target slave node to ensure that its configuration is suitable for running as a master node. Then, perform master-slave switching to determine the target slave node as the master node, and synchronously modify the configuration information of the slave node, wherein the performance indicators include CPU usage, memory occupancy, disk I / O, and network bandwidth; Re-establish the heartbeat link between the master node and the slave node to ensure data synchronization and consistency.

2. The method according to claim 1, characterized in that Determining the master node failure includes: The management node periodically connects to the master node through SSH, and when the master node fails to respond, it is determined that the master node is faulty; or, The replication delay time between the master node and the slave node is determined, and when the replication delay time exceeds a threshold, it is determined that the master node is faulty.

3. The method according to claim 1, characterized in that: Determining the target slave node from the slave nodes according to a preset rule comprises: The CPU occupancy rate and memory occupancy rate of each slave node are determined, and the slave node with the lowest CPU occupancy rate or memory occupancy rate is determined as the target slave node.

4. The method according to claim 1, characterized in that: The method further comprises: The slave nodes are synchronously managed according to the user's operation instructions, and the configuration information of the slave nodes is maintained, wherein the operation instructions include deletion instructions, addition instructions or modification instructions.

5. The method according to claim 1, characterized in that Determining the target slave node as a master node comprises: Stop writing to the master node, and record the name, location, and system status information of the binary log file of the master node; Use an SQL command to stop the replication process of the target slave node, modify the configuration file of the slave node, and determine the target slave node as the master node for operation.

6. A server fault detection and processing device, characterized in that: A management node MHAManager applied to a node cluster, wherein the node cluster further includes a plurality of data nodes, wherein the data nodes are divided into a master node and a plurality of slave nodes, and data is synchronized between the master node and the slave nodes via a heartbeat link; the device includes: A node monitoring unit is used to monitor each node in the node cluster and issue an alarm message when an abnormality is determined, wherein the abnormality includes abnormal health status of the master node, untimely data synchronization between the master node and the slave node, and inconsistent data between the master node and the slave node; A master node adjustment unit is used to continuously monitor the performance index of the master node. When the performance index exceeds a preset threshold, the master node is determined to be faulty, a target slave node is determined from the slave nodes according to a preset rule, the data replication process on the target slave node is suspended, the priority of the target slave node is increased, and the configuration file of the target slave node is checked to ensure that its configuration is suitable for running as a master node. Then, a master-slave switch is performed to determine the target slave node as the master node, and the configuration information of the slave node is modified synchronously, wherein the performance index includes CPU usage, memory occupancy, disk I / O, and network bandwidth; The heartbeat link reconstruction unit is used to re-establish the heartbeat link between the master node and the slave node to ensure data synchronization and consistency.

7. The device according to claim 6, characterized in that Determining the master node failure in the master node adjustment unit includes: The management node periodically connects to the master node through SSH, and when the master node fails to respond, it is determined that the master node is faulty; or, The replication delay time between the master node and the slave node is determined, and when the replication delay time exceeds a threshold, it is determined that the master node is faulty.

8. The device according to claim 6, characterized in that Determining the target slave node from the slave nodes according to a preset rule in the master node adjustment unit includes: The CPU occupancy rate and memory occupancy rate of each slave node are determined, and the slave node with the lowest CPU occupancy rate or memory occupancy rate is determined as the target slave node.

9. The device according to claim 6, characterized in that The device also includes: The configuration information maintenance unit is used to synchronously manage the slave nodes according to the user's operation instructions and maintain the configuration information of the slave nodes, wherein the operation instructions include deletion instructions, addition instructions or modification instructions.

10. The device according to claim 6, characterized in that Determining the target slave node to be the master node in the master node adjustment unit includes: Stop writing to the master node, and record the name, location, and system status information of the binary log file of the master node; Use an SQL command to stop the replication process of the target slave node, modify the configuration file of the slave node, and determine the target slave node as the master node for operation.

Images