High-availability service split-brain state processing method, system and device and storage medium

Abstract

The invention relates to the technical field of cloud native storage, and particularly provides a high-availability service split-brain state processing method, system and device and a storage medium, and the method comprises the steps: monitoring the states of a main node and a standby node through a third-party node, and appointing an arbitration main node name based on a monitoring result; the main node and the standby node mutually monitor the states of opposite ends; when the main node or the standby node confirms that the state of the opposite end is abnormal, the main node or the standby node obtains the name of the arbitration main node from a third-party node, and whether the main node or the standby node provides main node service is confirmed by comparing the consistency of the name of the arbitration main node and the name of the main node or the standby node; a split-brain detection rule is configured in the switching auxiliary scripts of the main node and the standby node, so that the main node and the standby node execute split-brain detection before switching the main state and the standby state, and the split-brain detection rule comprises obtaining the opposite-end state. According to the method, the split-brain state of the high-availability service is effectively solved, the stability of the high-availability service is improved, and data loss is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud native, and particularly relates to a method, system, device and storage medium for processing the split-brain state of highly available services. Background Art

[0002] Keepalived is a powerful open-source software mainly used to achieve the high availability and load balancing of servers. In a highly available NFS environment, services are provided through the VIP of Keepalived. When the NFS service of the primary node or the standby node fails, Keepalived will drift the VIP to the node where the service is available to continue providing services.

[0003] However, if the network connection between the primary and standby nodes fails, the primary and standby nodes of Keepalived will split into two VIPs because they cannot connect to each other. In a highly available NFS environment, since the disks used by NFS on the primary and standby nodes require DRBD for data synchronization, if two VIPs appear simultaneously, the NFS on the primary and standby nodes in the cluster will read and write to the data disk at the same time, resulting in inconsistent data and causing problems such as Drbd being unable to synchronize data, which will lead to data loss and other issues.

[0004] The current methods for solving the split-brain problem of Keepalived mainly involve designating an arbitration node and deploying monitoring scripts on the primary node and the standby node to regularly send node information to the arbitration node, such as IP addresses, instance status, health check information, priorities, etc. The arbitration node generates an arbitration result based on the received information and publishes the arbitration result to the current primary node and standby node. The primary node and the standby node adjust their service status according to the arbitration result.

[0005] However, this method has some problems. On the one hand, due to the need to regularly send a large amount of node information, it will increase the network transmission burden of the primary node and the standby node, and at the same time, it will occupy a certain amount of computing resources during the information preparation and sending process, which will affect the normal business processing of servers with high performance requirements. On the other hand, the arbitration process of the arbitration node is not flexible enough. For example, when the arbitration node arbitrates at a fixed time interval, it may frequently make arbitration decisions due to slight fluctuations in the node status, which will cause the primary and standby nodes to frequently switch states, thus affecting the stability and continuity of the service. Summary of the Invention

[0006] In view of the above deficiencies of the prior art, the present invention provides a method, system, device and storage medium for processing the split-brain state of highly available services to solve the above technical problems.

[0007] In the first aspect, the present invention provides a method for processing the split-brain state of highly available services, including: Monitor the status of the primary node and the standby node through a third - party node, and specify the name of the arbitration primary node based on the monitoring results; The primary node and the standby node monitor the status of each other's peer; When the primary node or the standby node confirms that the status of the peer is abnormal, obtain the name of the arbitration primary node from the third - party node, and confirm whether it provides the primary node service by comparing the consistency between the name of the arbitration primary node and its own name; Configure the split - brain detection rules in the switching auxiliary scripts of the primary node and the standby node respectively, so that the primary node and the standby node perform split - brain detection before switching the primary - standby status, and the split - brain detection rules include obtaining the status of the peer.

[0008] In an alternative embodiment, the method further includes: Deploy the nginx service on the third - party node, and configure the communication interface for the primary node and the communication interface for the standby node based on the nginx service; After the primary node and the standby node perform the primary - standby switch, report the status switch event to the third - party node through the corresponding communication interface; The third - party node parses the current primary node information and the current standby node information from the status switch event, and updates the current primary node information and the current standby node information to the node information list.

[0009] In an alternative embodiment, monitoring the status of the primary node and the standby node through a third - party node and specifying the name of the arbitration primary node based on the monitoring results includes: Set a timing task on the third - party node, and the timing task includes periodically sending an echo request data packet to the monitoring target based on the node information list, and the monitoring target includes the primary node and the standby node; Wait to receive the response data packet of the monitoring target to the echo request data packet. If the response data packet is not received within the specified time limit, it is determined that the monitoring target is abnormal. If the response data packet is received within the specified time limit, it is determined that the monitoring target is normal; Create a structure, and store the name of the primary node in the node information list as the default value in the structure; If the primary node is normal and the standby node is abnormal, do not update the structure; If the primary node is normal and the standby node is normal, do not update the structure; If the primary node is abnormal and the standby node is normal, update the node name of the standby node to the structure; If the primary node is abnormal and the standby node is abnormal, update the structure to none.

[0010] In an alternative embodiment, the method further includes: Set up an election system to select the node with the most idle resources from the nodes other than the primary node and the standby node as the third-party node; Start a monitoring thread on the third-party node, and the monitoring thread collects the running parameters and running logs of the third-party node; Compare the running parameters with preset standard thresholds, and generate abnormal record data for the running parameters that do not match the standard thresholds. The abnormal record data includes the corresponding running parameter items, parameter values, and collection times; Extract error data from the running logs using keyword extraction technology; Integrate the abnormal record data and the error data into a data set, and perform deduplication processing on the data set; Based on the correlation degree and time correlation degree between the parameter items in the data set, use a pre-trained random forest model to divide the data in the data set into multiple data groups; Package multiple data groups into multiple query statements; Input multiple query statements into a pre-constructed fault diagnosis knowledge graph in sequence to obtain the diagnosis result of the third-party node; If it is confirmed that the diagnosis result reaches the preset fault level, re-execute the election to obtain a new third-party node; The current third-party node synchronizes the local node information list and structure to the new third-party node; After the current third-party node receives the receipt information from the new third-party node, delete the node information list and structure stored locally; Publish the information of the new third-party node to the primary node and the slave nodes.

[0011] In an optional implementation manner, the primary node and the standby node monitor the status of each other, including: The primary node and the standby node send heartbeat signals to each other at a agreed first frequency; If the primary node or the standby node confirms that it has not received the heartbeat signal from the other party within the agreed time, increase the first frequency to the second frequency; The second frequency is higher than the first frequency.

[0012] In an optional implementation manner, confirm whether to provide the primary node service by comparing the consistency of the arbitration primary node name with its own name, including: Confirm that the arbitration primary node name is the same as its own name, and continue to provide the primary node service; Confirm that the arbitration primary node name is different from its own name, control the high-concurrency service to enter the fault state, and delete the virtual address.

[0013] In an optional implementation manner, the brain split detection rule includes: Before switching the primary and standby states of a high-concurrency service, obtain the peer state by sending an echo request to the peer; If it is confirmed that the peer state is abnormal, obtain the arbitration node name from a third-party node; If it is confirmed that the arbitration node name is inconsistent with its own name, do not configure its own distributed replicated block device as the primary state, so that its distributed replicated block device does not have the write permission for data.

[0014] In a second aspect, the present invention provides a high-availability service split-brain state processing system, including: An arbitration module, configured to monitor the states of the primary node and the standby node through a third-party node, and specify the arbitration primary node name based on the monitoring results; A monitoring module, configured to monitor the states of the peer by the primary node and the standby node; A processing module, configured to, when the primary node or the standby node confirms that the state of the peer is abnormal, obtain the arbitration primary node name from a third-party node, and confirm whether it provides the primary node service by comparing the consistency between the arbitration primary node name and its own name; A pre-check module, configured to configure split-brain detection rules in the switching auxiliary scripts of the primary node and the standby node respectively, so that the primary node and the standby node perform split-brain detection before switching the primary and standby states, and the split-brain detection rules include obtaining the peer state.

[0015] In a third aspect, a device is provided, including: A memory, configured to store a high-availability service split-brain state processing program; A processor, configured to implement the steps of the high-availability service split-brain state processing method provided in the first aspect when executing the high-availability service split-brain state processing program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, on which a high-availability service split-brain state processing program is stored, and when the high-availability service split-brain state processing program is executed by a processor, the steps of the high-availability service split-brain state processing method provided in the first aspect are implemented.

[0017] The beneficial effects of the present invention are as follows. The method, system, device, and storage medium for processing the split-brain state of highly available services provided by the present invention design a third-party arbitration logic. This arbitration logic only needs to obtain the ping status of the primary node and the standby node, reducing the amount of data processing and data interaction. Moreover, by enabling the primary node and the standby node to monitor each other's status and initiate arbitration actively based on the status monitoring results, it effectively avoids the adverse effects on the service caused by unreasonable arbitration issued by the arbitration node. In addition, by configuring the split-brain detection rule in the switching auxiliary script, problems such as the failure of the primary-standby switch and data loss due to split-brain during the primary-standby switch are avoided. In short, the present invention effectively solves the split-brain state of highly available services, improves the stability of highly available services, and avoids data loss.

[0018] In addition, the design principle of the present invention is reliable, the structure is simple, and it has a very wide application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention.

[0021] Figure 2 It is a schematic logical architecture diagram of the method according to an embodiment of the present invention.

[0022] Figure 3 It is a schematic block diagram of the system according to an embodiment of the present invention.

[0023] Figure 4 It is a schematic structural diagram of a device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs. The terms used in the specification of this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention.

[0026] The following explains the key terms that appear in this invention.

[0027] High-availability NFS: (Network File System) A file system mechanism that can store and retrieve data from multiple disks and directories through a shared network. The network file system enables local users to access remote data and files in the same way as local access. High-availability NFS refers to a set of high-availability NFS services that utilize the VIP capabilities of Keepalived and the data synchronization capabilities of Drbd to form a primary and standby switching and automatic fault repair function.

[0028] VIP, virtual address.

[0029] Keepalived runs on the active LVS router and one or more optional standby LVS routers. The active (primary) router uses the Virtual Router Redundancy Protocol (VRRP) to notify the backup routers of its active status. This protocol requires the primary router to send announcements regularly. If the active router stops sending announcements, a new primary router will be selected.

[0030] VRRP (Virtual Router Redundancy Protocol) is a network protocol mainly used to achieve high availability of network devices. VRRP allows multiple physical routers to form a virtual router group. These physical routers jointly bear network traffic and appear as a logical virtual router externally. In this way, when a router in the group fails, other routers can automatically take over its work to ensure the continuous operation of the network and avoid single points of failure.

[0031] DRBD (Distributed Replicated Block Device) is a network-based block device mirroring technology mainly used to achieve data redundancy and high availability between servers.

[0032] Nginx is a lightweight, high-performance Web server, reverse proxy server, and email (IMAP / POP3) proxy server.

[0033] The high-availability service split-brain status processing method provided by the embodiments of this invention is executed by a computer device. Correspondingly, the high-availability service split-brain status processing system runs in the computer device.

[0034] Figure 1 is a schematic flowchart of the method according to an embodiment of the present invention. Among them, Figure 1 The execution subject can be a high-availability service split-brain state processing system. According to different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0035] As Figure 1 shown, the method includes: S1, monitor the status of the primary node and the standby node through a third-party node, and specify the arbitration primary node name based on the monitoring results.

[0036] Select an independent and stable server as the third-party node, ensure its reliable network connection, and have sufficient computing resources and storage resources to handle the monitoring tasks. This node should be reachable on the network with the primary node and the standby node, and is not affected by the failures of the primary and standby nodes.

[0037] Install appropriate monitoring tools on the third-party node. For example, write a custom monitoring script using a scripting language (such as Python), or use open-source monitoring software (such as Prometheus combined with relevant Exporters).

[0038] If a custom script is used, network probing tools (such as using the ping command to detect node connectivity, and detecting whether a service is running through TCP connections on specific ports, etc.) are needed to obtain the status information of the primary node and the standby node.

[0039] For Prometheus, corresponding Exporters need to be deployed on the primary node and the standby node to expose the status metrics of the nodes (such as CPU usage, memory usage, service running status, etc.) for Prometheus to collect.

[0040] Determine the key metrics to be monitored, including but not limited to the network connectivity of the nodes, the usage of system resources (CPU, memory, disk space, etc.), and the running status of specific services (such as Keepalived, Nginx, etc.).

[0041] For example, check the network connectivity of the nodes by periodically executing the ping command. If the nodes cannot be pinged continuously for multiple times, it is considered that the network status is abnormal; check whether a specific service is running by checking the process list. If the service process does not exist, it is determined that the service status is abnormal.

[0042] Write arbitration logic based on the monitored metrics. If the primary node can be pinged, the primary node remains unchanged. If the primary node cannot be pinged, the primary node is modified.

[0043] Store the name of the arbitration master node in a shared storage location (such as a database, a distributed file system, etc.), or create a configuration file on a third-party node to record the name of the arbitration master node so that the master node and the standby node can obtain this information.

[0044] S2. The master node and the standby node mutually monitor the status of the peer.

[0045] Configure monitoring scripts on the master node and the standby node respectively, and use network communication tools (such as ping, TCP connection test, etc.) and process detection tools (such as using the pidof command to check whether a process exists) to monitor the status of the peer node.

[0046] For example, by periodically pinging the IP address of the peer node to check network connectivity; attempting to establish a TCP connection to a specific service port of the peer to determine whether the service is running properly; using a script to query the process ID of the key service of the peer to confirm whether the service is running.

[0047] Set an appropriate monitoring frequency according to the actual situation. For example, perform network connectivity detection every 1 - 5 seconds and service status detection every 10 - 30 seconds. Too high a frequency may increase network and system resource consumption, and too low a frequency may result in untimely detection of status changes.

[0048] Record the status information of the peer node on the master node and the standby node respectively. These information can be recorded in a log file to facilitate subsequent troubleshooting and fault analysis. The log content should include the monitoring time, the status of the peer node (normal / abnormal), and possible reasons for abnormalities (such as network timeout, service process not found, etc.).

[0049] S3. When the master node or the standby node confirms that the status of the peer is abnormal, obtain the name of the arbitration master node from a third-party node, and confirm whether it provides the master node service by comparing the consistency of the name of the arbitration master node with its own name.

[0050] After each monitoring, the monitoring scripts on the master node and the standby node judge whether the status of the peer node is abnormal according to the set thresholds and rules. For example, if the peer node times out continuously for 3 times when pinging, or the key service process of the peer is not found continuously for 2 times, it is determined that the status of the peer is abnormal.

[0051] After confirming that the status of the peer is abnormal, the master node or the standby node obtains the name of the arbitration master node from a third-party node through a network request (such as an HTTP request, a database query, etc.).

[0052] If the arbitration master node name is stored in the database, the node obtains the name through the database connection string and the corresponding query statement; if it is stored in the configuration file of the third-party node, the node can obtain the content of the file through an HTTP request and parse out the arbitration master node name.

[0053] Compare the obtained arbitration master node name with its own node name. If the names are the same, it is considered that the node itself should provide the master node service; if they are different, it indicates that the node itself should not provide the master node service and should maintain the standby node status or make corresponding adjustments.

[0054] If it is confirmed that the node itself should provide the master node service, the node needs to ensure that relevant services (such as Keepalived taking over the virtual IP, starting key business services, etc.) are running properly to provide the complete master node function externally.

[0055] If it is confirmed that the node itself should not provide the master node service, the node should stop operations that may interfere with the master node service (such as releasing the virtual IP, pausing some high-priority business processes, etc.) and maintain the standby node status.

[0056] S4. Configure the split-brain detection rules in the switching auxiliary scripts of the master node and the standby node respectively, so that the master node and the standby node perform split-brain detection before switching the master-standby status. The split-brain detection rules include obtaining the status of the peer end.

[0057] Write switching auxiliary scripts on the master node and the standby node respectively. This script will be called before the master-standby status is switched. The script can be written in languages such as Shell and Python and has the functions of obtaining the status of the peer end and executing the split-brain detection logic.

[0058] To configure the split-brain detection rules in the script, first obtain the status of the peer end. The method and code logic for monitoring the status of the peer end in S2 can be reused to obtain the latest status information of the peer end by calling the monitoring script or directly executing the monitoring command.

[0059] In addition to obtaining the status of the peer end, other detection logics need to be configured, such as checking whether the communication between the node itself and the third-party node is normal (ensuring that the obtained arbitration master node name is the latest and reliable), and comparing the consistency between the current status of the node itself and the arbitration master node name.

[0060] If abnormal situations are found during the split-brain detection process (such as both nodes thinking that they are the master node, or the communication with the third-party node is interrupted and the arbitration information cannot be obtained, etc.), the script should operate according to the preset processing logic.

[0061] For example, pause the primary and standby state switching operation, record detailed exception information in a log file, and send an alarm message to the system administrator (such as by email, SMS, etc.) to promptly handle potential split-brain problems.

[0062] If the split-brain detection passes, the script continues to execute the relevant operations for primary and standby state switching, such as adjusting node priorities, starting or stopping specific services, taking over or releasing virtual IPs, etc.

[0063] Please refer to Figure 2 , introduce a third-party node. When split-brain occurs in Keepalived, provide a master election service through the third-party node, and only keep the Keepalived service on the elected master node, and turn off the Keepalived on the standby node, so as to ensure that only one VIP provides services at the same time, and thus ensure the data consistency of the NFS service and solve the Keepalived split-brain problem under NFS high availability.

[0064] In an embodiment of the present invention, before executing S1, it is necessary to deploy the communication between the third-party node and the primary and standby nodes. The following will give a possible embodiment to non-restrictively elaborate on its specific implementation scheme.

[0065] 1. Deploy the nginx service on the third-party node, and configure the communication interfaces for the primary node and the standby node based on the nginx service.

[0066] On the third-party node, first ensure that the operating system meets the installation requirements of Nginx. For example, for the Linux system, a package manager (such as apt-get or yum) can be used for installation.

[0067] Open the Nginx configuration file (usually located at / etc / nginx / nginx.conf), and make necessary modifications to meet the requirements of the current architecture. A new server block can be created or an existing server block can be modified. Listen to the Nginx service on port 80 and record requests in a specified log file.

[0068] Add a location directive in the Nginx server block to create a dedicated API endpoint for communication with the primary node. This endpoint will receive status update information from the primary node. To ensure communication security, SSL / TLS encryption can be enabled, configure Nginx to listen on port 443 and use an SSL certificate. Similar to the interface configuration of the primary node, add another location directive for the standby node.

[0069] 2. After the primary node and the standby node perform the primary and standby switch, report the status switch event to the third-party node through the corresponding communication interface.

[0070] Write a status switching script on the master node, which will be triggered when the master-backup switch occurs. For example, call this script in the notify_master or notify_backup script of Keepalived. This script uses the curl tool to send a POST request containing the information of the master node (node type, IP address, timestamp, and service status) in JSON format to the / api / main-node-status interface of the third-party node. Add the above status switching script to the Keepalived configuration file. Similar to the master node, the standby node also has a corresponding script, which is triggered when its status switches, and add this script to the standby node configuration of Keepalived.

[0071] 3. The third-party node parses the current master node information and the current standby node information from the status switching event, and updates the current master node information and the current standby node information to the node information list.

[0072] On the third-party node, a backend service is required to handle requests from the master node and the standby node. Specifically, use Flask to create a Web service that receives POST requests from the master node and the standby node, parses the JSON data, and stores it in the node_info_list. If the node information already exists (judged according to the node type), update the corresponding information; if not, add it to the list.

[0073] In the update_node_info_list function, a logging function can be added to record the update of node information for subsequent analysis and monitoring. At the same time, monitoring tools such as Prometheus can be used to monitor the performance of this service and the request processing situation. By adding Summary and Counter metrics in the code, the request processing time and the total number of requests are counted, and monitoring can be carried out by starting the Prometheus service.

[0074] The existing third-party node and the master-standby node usually adopt the common TCP communication method in the cluster. In this application, by deploying the Nginx service, an independent arbitration communication channel is opened for the third-party node and the master-standby node. This arbitration communication channel has communication flexibility and can quickly establish a communication connection with the new master-standby node. Specifically, its reverse proxy function supports the third-party node to communicate with any master-standby node without exposing the real IP.

[0075] In an embodiment of the present invention, based on step S1, a possible embodiment will be given below to non-restrictively elaborate on its specific implementation scheme.

[0076] S101. Set a scheduled task on a third-party node. The scheduled task includes periodically sending an echo request data packet to a monitoring target based on the node information list. The monitoring target includes a primary node and a standby node.

[0077] In the Linux system, cron can be used to set a scheduled task. Open the cron table editor and add the following entry. Taking the monitoring task being executed every 5 minutes as an example: * / 5 * * * * / usr / bin / python3 / path / to / monitoring_script.py; Every 5 minutes, the system will call / usr / bin / python3 to execute the / path / to / monitoring_script.py script.

[0078] Write the primary node address and standby node address in the node information list into the target address variable of the / path / to / monitoring_script.py script.

[0079] S102. Wait to receive the response data packet of the monitoring target to the echo request data packet. If the response data packet is not received within the specified time limit, it is determined that the monitoring target is abnormal. If the response data packet is received within the specified time limit, it is determined that the monitoring target is normal.

[0080] In the send_echo_requests function of the / path / to / monitoring_script.py script, use the -W parameter of the ping command to set the timeout time to ensure that the node can be determined abnormal in time when no response is received within the specified time.

[0081] def send_echo_requests(): node_info_list = get_node_info_list()timeout = 2 # timeout is 2 secondsfor node in node_info_list: node_name = node['name'] node_ip = node['ip']result = subprocess.run(['ping', '-c', '3', '-W',str(timeout), node_ip], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) if result.returncode == 0: print(f"The reply data packet of the {node_name} node is received normally.") else: print(f"The {node_name} node did not send a reply data packet within {timeout} seconds, an exception may occur.")

[0082] S103. Create a structure, and store the master node name in the node information list as a default value in the structure.

[0083] Using Python classes as structures: class NodeStatus: def __init__(self, node_info_list): self.current_master = None self.current_backup = None self.update_status(node_info_list) def update_status(self, node_info_list): for node in node_info_list: if node['node_type'] == 'MASTER': self.current_master = node['node_name'] elif node['node_type'] == 'BACKUP': self.current_backup = node['node_name'] def main(): node_info_list = get_node_info_list() node_status = NodeStatus(node_info_list) print(f"Initial structure: Master node: {node_status.current_master}, Backup node: {node_status.current_backup}") if __name__ == "__main__": main(); A NodeStatus class is created as a structure, and when initialized, it will find the master node name from the node information list as the default value of current_master.

[0084] S104. Update the content stored in the structure.

[0085] If the master node is normal and the backup node is abnormal, do not update the said structure: class NodeStatus: def __init__(self, node_info_list): self.current_master = None self.current_backup = None self.update_status(node_info_list)def update_status(self, node_info_list): master_status = check_node_status('MASTER', node_info_list) backup_status = check_node_status('BACKUP', node_info_list) if master_status == 'Normal' and backup_status == 'Abnormal': print("The master node is normal and the backup node is abnormal. The structure is not updated.") else: # Other update logic pass def check_node_status(node_type, node_info_list): # Assume that the check is based on the node type and node information listfor node in node_info_list: if node['node_type'] == node_type: node_name= node['node_name']node_ip = node['ip'] result = subprocess.run(['ping', '-c', '3', '-W', '2', node_ip], stdout=subprocess.PIPE, stderr=subprocess.PIPE,text=True) if result.returncode == 0: return 'normal' else: return 'exception' defmain(): node_info_list = get_node_info_list() node_status = NodeStatus(node_info_list) if __name__ == "__name__": main().

[0086] If the primary node is normal and the backup node is normal, the structure is not updated: class NodeStatus: def __init__(self, node_info_list): self.current_master = None self.current_backup = None self.update_status(node_info_list)def update_status(self, node_info_list): master_status = check_node_status('MASTER', node_info_list) backup_status = check_node_status('BACKUP', node_info_list) if master_status == 'Normal' and backup_status == 'Abnormal': print("The master node is normal and the backup node is abnormal. The structure is not updated.") elif master_status == 'Normal' and backup_status == 'Normal': print("The master node is normal and the backup node is normal. The structure is not updated.") elif master_status == 'Abnormal' and backup_status == 'Normal': self.update_master_to_backup(node_info_list) elif master_status == 'Abnormal' and backup_status == 'Exception':self.update_to_none() def update_master_to_backup(self, node_info_list): fornode in node_info_list: if node['node_type'] == 'BACKUP': self.current_master= node['node_name']print(f"Update the master node to the backup node: {self.current_master}") defupdate_to_none(self): self.current_master = None self.current_backup = Noneprint("Update the structure to none.") def check_node_status(node_type, node_info_list): for node in node_info_list: if node['node_type'] == node_type: node_name = node['node_name']node_ip = node['ip'] result = subprocess.run(['ping','-c', '3', '-W', '2', node_ip], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) if result.returncode == 0: return 'Normal' else:return 'Abnormal' def main(): node_info_list = get_node_info_list() node_status= NodeStatus(node_info_list) if __name__ == "__main__": main()。

[0087] If the primary node is abnormal and the standby node is normal, update the node name of the standby node to the structure: In the update_master_to_backup function, after updating the standby node to the primary node, notify the relevant services or update the log: def update_master_to_backup(self, node_info_list): for node in node_info_list: if node['node_type'] == 'BACKUP': self.current_master = node['node_name']print(f"Updating the primary node to the standby node: {self.current_master}") # Notify the relevant services, for example, call a notification function notify_service(f"The primary node has been updated to {self.current_master}") def notify_service(message): # Here, you can use email, SMS, or other notification methods to notify the administrator or relevant systems print(f"Sending notification: {message}")。

[0088] If the primary node is abnormal and the standby node is abnormal, update the structure to none: def update_to_none(self): self.current_master = None self.current_backup = None print("Update the structure to none.") # Trigger an alert, such as sending an email or SMS send_alert("Both the primary node and the standby node are abnormal, and the service may be interrupted!") def send_alert(message): # Assume using smtplib to send email alerts import smtplib from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart sender_email = "sender@example.com" receiver_email = "receiver@example.com" password = "password" message = MIMEMultipart("alternative") message["Subject"] = "Node Abnormality Alert" message["From"] = sender_email message["To"] = receiver_email text = f"""\ Alert: {message} """ part1 = MIMEText(text, "plain") message.attach(part1) with smtplib.SMTP("smtp.example.com", 587) as server: server.starttls() server.login(sender_email, password) server.sendmail(sender_email, receiver_email, message.as_string()).

[0089] In the update_to_none function, an alert trigger is added. This alert can trigger a split-brain detection initiated by the primary node and the standby node. Split-brain detection means that the primary and standby sides send heartbeat signals to each other, and if a heartbeat signal is not received in time, it is determined that there is a split-brain situation.

[0090] The input variables of this arbitration method are only the ping status of the primary and standby nodes, which greatly reduces the data processing volume. Moreover, the arbitration result is saved to a structure instead of being published to the primary and standby nodes, which is a passive arbitration and effectively avoids the impact of unreasonable arbitration on the service continuity of the primary and standby nodes.

[0091] Based on the above embodiments, in order to further improve the reliability of the third-party node, as an implementable method, in one embodiment, a management method for the third-party node is provided, and the specific method is as follows: (1)Set an election system to select the node with the most idle resources from the nodes other than the primary node and the standby node as the third-party node.

[0092] It is necessary to determine the metrics for evaluating the idle resources of the node, and these metrics may include but are not limited to CPU idle rate, memory free amount, disk free space, remaining network bandwidth, etc. For example, use system monitoring tools (such as the top, free, df, etc. commands) to obtain the current resource usage of the node.

[0093] The following script (taking the Linux system as an example) can be used to obtain this information: # Obtain the CPU idle rate cpu_idle=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}') # Obtain the memory free amount (in MB) mem_free=$(free -m | awk 'NR==2{print $4}') # Obtain the disk free space (in GB) disk_free=$(df -BG / | awk 'NR==2{print$4}' | sed's / G / / ') # Obtain the remaining network bandwidth (assuming tools such as `iftop` or `nload` are used for evaluation) # This is only for illustration, and the actual operation may be more complex and different network cards and traffic types need to be considered net_bandwidth=$(iftop -t -s 5 | awk ' / Total send rate: / {print $4}').

[0094] Traverse all available nodes and exclude the primary node and the standby node. For each candidate node, collect the above resource metrics and calculate the comprehensive resource score according to a certain weight.

[0095] For example, set the CPU weight to 0.4, the memory weight to 0.3, the disk weight to 0.2, and the network weight to 0.1. The formula for calculating the comprehensive score can be: score = (cpu_idle * 0.4) + (mem_free * 0.3) + (disk_free * 0.2) +(net_bandwidth * 0.1).

[0096] Compare the scores of all candidate nodes and select the node with the highest score as the third-party node. Assume that node information is stored in a list: import subprocess def get_node_resources(node): try: # Get CPU idle rate cpu_output = subprocess.check_output("top -bn1 | grep 'Cpu(s)' | awk '{print $8}'", shell=True).decode().strip() cpu_idle = float(cpu_output) # Get free memory (in MB) mem_output = subprocess.check_output("free -m |awk 'NR==2{print $4}'", shell=True).decode().strip() mem_free = float(mem_output) # Get free disk space (in GB) disk_output = subprocess.check_output("df -BG / | awk 'NR==2{print $4}' | sed 's / G / / '", shell=True).decode().strip() disk_free = float(disk_output) # Assume network bandwidth is not considered temporarily and set it to 0 net_bandwidth = 0 return (cpu_idle * 0.4) + (mem_free * 0.3) + (disk_free * 0.2) + (net_bandwidth * 0.1) except Exception as e: print(f"Error getting resources for node {node}: {e}") return 0 def select_third_party_node(all_nodes, master_node, backup_node): candidates = [node for node in all_nodes if node not in [master_node, backup_node]] scores = [(node, get_node_resources(node)) for node in candidates]scores.sort(key=lambda x: x[1], reverse=True) if scores: return scores[0][0] else: return None # Example call all_nodes = ['node1', 'node2', 'node3', 'node4'] master_node = 'node1' backup_node = 'node2' third_party_node = select_third_party_node(all_nodes, master_node, backup_node) print(f"Selected third-party node: {third_party_node}").

[0097] (2)Start a monitoring thread on the third-party node, and the monitoring thread collects the running parameters and running logs of the third-party node.

[0098] On the third-party node, the threading mechanism of the programming language (such as the threading module in Python) can be used to create a dedicated monitoring thread. In addition to using system commands, dedicated system performance monitoring tools such as the psutil library (Python) can also be used, which provides a more convenient interface to obtain system performance parameters.

[0099] For log collection, it can be read according to the format and storage location of the log file, and log management tools such as logrotate can also be used to manage the rolling and expiration of log files.

[0100] (3)Compare the running parameters with the preset standard thresholds, and generate abnormal record data for the running parameters that do not match the standard thresholds. The abnormal record data includes the corresponding running parameter items, parameter values, and collection times.

[0101] Set standard thresholds for each running parameter. For example, it is abnormal when the CPU usage rate exceeds 80%, it is abnormal when the memory usage rate exceeds 90%, and it is abnormal when the disk usage rate exceeds 85%. Store the abnormal records in a database, file, or message queue for subsequent processing.

[0102] (4)Extract error data from the running logs using keyword extraction technology.

[0103] Use natural language processing (NLP) technology, such as using the nltk or spaCy library in Python to extract keywords.

[0104] Use regular expressions to extract specific error messages and keywords. For example, extract log entries containing ERROR or WARNING and extract the key information therein.

[0105] (5) Integrate the exception record data and error data into a data set and perform deduplication processing on the data set.

[0106] Store the exception record data and error data in a data structure, such as a list or a dictionary. Use the set data structure in Python to deduplicate the elements in the data set. However, since the data may be a dictionary, it needs to be converted into a hashable form, such as converting the dictionary into a tuple.

[0107] (6) Based on the correlation degree and time correlation degree between parameter items in the data set, use a pre-trained random forest model to divide the data in the data set into multiple data groups.

[0108] For the data in the data set, convert it into a feature vector suitable for the random forest model. Encode the parameter items and time information into numerical features. For example, convert the time into a timestamp and encode the parameter items according to their types.

[0109] For different parameter items in the data set, use one-hot encoding to represent categorical parameters. This can prevent the model from wrongly assuming a size relationship between different encoded values. Statistically count the number of exceptions or the frequency of error keywords occurring within a specific time window (such as the past 1 hour) and use it as an additional time correlation feature.

[0110] Use correlation analysis (such as the Pearson correlation coefficient) to analyze the linear correlation degree between different parameters. For non-linear correlations, more complex correlation metrics, such as mutual information, can be used. Add the correlation matrix or relevant features as new features to the original feature set, or as new feature dimensions, to enhance the model's understanding of the correlation between parameter items.

[0111] Use a pre-trained random forest model to divide the data set into different groups according to the features. Different groups may represent different types of faults or abnormal patterns.

[0112] Specifically, the structure of the random forest model includes: 100 decision trees. A decision tree includes internal nodes and leaf nodes. Internal node: contains a feature test condition, and according to this condition, the data is divided into different sub-nodes. For example, for a feature X1, the internal node may check X1 < 5, and divide the data that meets the condition into the left sub-node and the data that does not meet the condition into the right sub-node. Leaf node: represents the final decision result. For a classification problem, the leaf node contains a class label; for a regression problem, the leaf node contains a predicted value.

[0113] When constructing each decision tree, a part of the features (max_features) are randomly selected from all the features as candidate features, which helps to increase the diversity of the model and reduce overfitting. For the training of each tree, bootstrap sampling is used to randomly draw samples from the training dataset. Approximately 63.2% of the samples will be selected, and the remaining unselected samples can be used for subsequent performance evaluation (OOB error rate estimation).

[0114] The random forest model is trained using a dataset of historical fault data. The trained random forest model can distinguish different types of faults or abnormal patterns. The model is evaluated using a validation set, and the model parameters, such as the number of trees, depth, splitting criterion, etc., are adjusted according to the evaluation results to improve the grouping performance.

[0115] The specific training process using the scikit-learn library includes: from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score import numpy as np import pandas as pd # Assume our data set is a DataFrame containing features and labels # Example data, which needs to be replaced with real data in actual use data = pd.DataFrame({ 'feature1': np.random.rand(1000), 'feature2': np.random.rand(1000), 'feature3': np.random.rand(1000), 'label': np.random.choice([0, 1, 2], 1000) # Assume there are 3 classes for the label: 0, 1, 2}) # Separate features and labels X = data.drop('label', axis = 1) y = data['label'] # Split the training set and test set X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42) # Initialize the random forest classifier # n_estimators: Number of trees # max_depth: Maximum depth of the tree to prevent overfitting # random_state: Random seed to ensure reproducibility of results clf = RandomForestClassifier(n_estimators = 100, max_depth = 10, random_state = 42) # Train the model clf.fit(X_train, y_train) # Predict the test set y_pred = clf.predict(X_test) # Evaluate the model accuracy = accuracy_score(y_test, y_pred) print(f"Model accuracy: {accuracy}") # Adjust model parameters (using grid search) from sklearn.from model_selection import GridSearchCV # Define the parameter grid param_grid = { 'n_estimators': [50, 100, 200],'max_depth': [None, 10, 20, 30],'min_samples_split': [2, 5, 10],'min_samples_leaf': [1, 2, 4]} # Use grid search to find the best parameters grid_search = GridSearchCV(estimator=clf, param_grid=param_grid, cv=5, scoring='accuracy') grid_search.fit(X_train, y_train) # Output the best parameters print(f"Best parameters: {grid_search.best_params_}") # Retrain the model with the best parameters best_clf = grid_search.best_estimator_ best_clf.fit(X_train, y_train) # Re-evaluate the model y_pred_best = best_clf.predict(X_test) accuracy_best = accuracy_score(y_test, y_pred_best) print(f"Accuracy of the model after adjusting parameters: {accuracy_best}").

[0116] Among them, data is a DataFrame containing features and labels, X is the feature matrix, and y is the label vector.

[0117] The train_test_split function divides the data into training set and test set for training and evaluating the model.

[0118] The RandomForestClassifier class is used to initialize the random forest classifier and set parameters such as n_estimators and max_depth. clf.fit(X_train, y_train) performs model training.

[0119] accuracy_score is used to calculate the accuracy of the model on the test set.

[0120] GridSearchCV is used to perform grid search, traverse the parameter combinations in param_grid, and find the best parameters through cross-validation (cv = 5).

[0121] The process of grouping data using a pre-trained random forest model includes: # Assume we already have a trained random forest model best_clf # Feature engineering to convert the data set into a feature matrix acceptable to the model # Here, assume the data set is a list containing multiple dictionaries, each dictionary contains different features data_set = [ {'feature1': 0.5, 'feature2': 0.3, 'feature3':0.8}, {'feature1': 0.2, 'feature2': 0.7, 'feature3': 0.1}, {'feature1': 0.9,'feature2': 0.4, 'feature3': 0.6} ] # Convert the data set to a feature matrix X_new =pd.DataFrame(data_set) # Use the trained random forest model to group the data groups = best_clf.predict(X_new) # Output the grouping results print(f"Grouping results: {groups}") # Add grouping information to each data item for i, group in enumerate(groups): data_set[i]['group'] = group print(f"The grouped data set: {data_set}").

[0122] Convert the data set to DataFrame format and use best_clf.predict(X_new) for prediction to obtain the grouping results.

[0123] (7) Package multiple said data groups into multiple query statements.

[0124] According to the grouping results, package each group of data into a query statement, which can be used to query the fault diagnosis knowledge graph or database.

[0125] According to the structure and storage method of the knowledge graph, use a dynamic query statement generator to allow more flexible combinations of query conditions. For example, if the knowledge graph supports multiple query conditions and filters, filter conditions can be dynamically added according to the characteristics of the data groups.

[0126] (8) Input multiple said query statements into a pre-constructed fault diagnosis knowledge graph in sequence to obtain the diagnosis results of the third-party nodes.

[0127] The fault diagnosis knowledge graph is a graph database (Neo4j) and is queried using the corresponding query language.

[0128] The construction method of the fault diagnosis knowledge graph includes: 1) Determine the scope and domain of the knowledge graph First, clarify that the knowledge graph to be constructed is for a single node of the cluster and needs to cover fault knowledge of server hardware (CPU, memory, hard disk, etc.), operating system, application programs, etc.

[0129] 2) Knowledge acquisition Expert knowledge: Consult domain experts to obtain their experience and knowledge in fault diagnosis, including common fault types, fault causes, fault manifestations, and solutions, etc.

[0130] Document materials: Collect equipment manuals, operation manuals, troubleshooting guides, technical white papers, etc., and extract the fault diagnosis information from them.

[0131] Historical data: Analyze historical fault records, including fault logs, maintenance records, fault work orders, etc., and extract the patterns and relevant information of fault occurrence from them.

[0132] Online resources: Collect relevant fault information from online resources such as technical forums, knowledge bases, and open source communities.

[0133] 3) Data modeling Define entity content: Equipment: Such as servers, routers, switches, storage devices, etc.

[0134] Fault types: Such as hardware faults, software faults, network faults, etc.

[0135] Fault symptoms: Such as server crashes, network latency, service unavailability, etc.

[0136] Solutions: Such as restarting the equipment, updating the software, replacing the hardware, etc.

[0137] Other entities: Other entities can be added as needed, such as system metrics (CPU usage rate, memory usage rate, etc.), services (Web service, database service, etc.), etc.

[0138] Define relationship types: Equipment - Fault type: Represents the possible fault types of the equipment. For example, server - hardware fault.

[0139] Fault type - Fault symptom: Represents the symptoms caused by the fault. For example, hardware fault - server crash.

[0140] Fault Type - Solution: Represents the method to solve the fault. For example, Hardware Fault - Replace Hardware.

[0141] Device - System Metrics: Represents the performance metrics of the device. For example, Server - CPU Usage.

[0142] 4) Knowledge Representation Use RDF (Resource Description Framework) or property graph model to represent knowledge.

[0143] For RDF, use the form of triples (subject, predicate, object) to represent knowledge. For example, (Server 1, has a fault, Hardware Fault), (Hardware Fault, causes symptom, Server crashes), etc.

[0144] For the property graph, use the attributes of nodes and edges to represent information. For example, the attributes of node Server 1 can include IP address, hardware configuration, etc., and the edge has a fault can have attributes such as occurrence time, etc.

[0145] 5) Data Storage and Management Select Neo4j graph database to store the knowledge graph.

[0146] 6) Knowledge Extraction and Transformation For knowledge in text form (such as documents, logs, etc.), use natural language processing (NLP) techniques to extract information. For example, use named entity recognition (NER) techniques to extract entities such as devices, faults, symptoms, etc., and use relation extraction techniques to extract the relationships between entities.

[0147] 7) Knowledge Graph Population Import the knowledge obtained from various sources into the graph database through programs or tools. For structured data (such as CSV, Excel, etc.), you can use ETL (Extract, Transform, Load) tools to convert the data into the format required by the knowledge graph and import it.

[0148] For the application of the fault diagnosis knowledge graph, for graph databases such as Neo4j, you can use the Cypher query language for querying. For example, query the fault type of server crashes: MATCH (s: Fault Symptom {name: "Server crashes"})<-[: causes symptom]-(f: Fault Type) RETURN f.name.

[0149] Use a rule engine or inference algorithm to perform reasoning based on the existing knowledge graph. For example, based on the known device faults and relationships, infer possible fault symptoms and solutions.

[0150] Building a fault diagnosis knowledge graph requires determining the scope, obtaining knowledge from multiple sources, performing data modeling, representation, storage, and import, storing using a graph database, and using query and inference mechanisms to support fault diagnosis.

[0151] (9) If it is confirmed that the diagnosis result reaches the preset fault level, re - execute the election to obtain a new third - party node.

[0152] Define the thresholds for different fault levels, which can be based on the assessment of the severity and impact scope of the fault. Specifically, divide the fault levels into three levels: low, medium, and high, corresponding to different fault indicators or scores respectively.

[0153] Use a comprehensive fault score (failure_score) to measure the severity of the fault, which can be calculated from multiple factors in the diagnosis result, such as the severity of the fault type, the number of affected system components, the duration, etc.

[0154] Set the fault score thresholds corresponding to different fault levels. When the calculated fault score exceeds the threshold, trigger the re - election of the third - party node. For example, set the threshold for the high - fault level to 80, the medium - fault level to 50, and the low - fault level to 20.

[0155] Re - execute the election process in step (1), and select the node with the most idle resources from the nodes other than the primary node and the standby node as the new third - party node.

[0156] (10) The current third - party node synchronizes the local node information list and structure to the new third - party node.

[0157] Use a network protocol (such as HTTP, RPC, message queue, etc.) to transfer data from the current third - party node to the new third - party node. Before transferring the data, ensure that the data structure can be serialized for transmission over the network. Convert the data to a string using JSON or other serialization formats.

[0158] (11) After the current third - party node receives the receipt information from the new third - party node, delete the node information list and structure stored locally.

[0159] The current third - party node needs to listen for the receipt information from the new third - party node to confirm that the data has been successfully received and processed. Set up a simple HTTP service on the current third - party node to receive the receipt.

[0160] In the delete_local_data() function, perform corresponding deletion operations according to the data storage method.

[0161] (12) Publish the information of the new third - party node to the master node and the slave node.

[0162] The third - party node uses the interface of the nginx service to send its own information to the master node and the standby node. Ensure that the sent information contains the necessary information of the new third - party node, such as the node name, IP address, etc.

[0163] This method generates a third - party node through an election method, avoiding the failure of the arbitration function caused by the failure of a fixed third - party node. And to avoid the waste of cluster resources caused by frequently changing the third - party node, a fault self - detection function for the third - party node is further added. This function combines a random forest model and a knowledge graph. Compared with the existing fault detection methods that completely rely on the knowledge graph or neural network models, the fault self - detection function provided in this application first uses the random forest model to group the data collected by the monitoring thread, and then encapsulates each data group into a query statement respectively, reducing the interference between different fault types, improving the diagnostic accuracy of the fault diagnosis knowledge graph, and reducing the single - time data processing volume of the fault diagnosis knowledge graph, thus improving the fault diagnosis efficiency.

[0164] In an embodiment of the present invention, based on step S2, a possible embodiment will be given below to non - restrictively elaborate on its specific implementation scheme.

[0165] S201. The master node and the standby node send heartbeat signals to each other at a predefined first frequency.

[0166] The heartbeat signal is a signal regularly sent between the master node and the standby node to confirm whether the other node is in a normal operating state. It can be a simple data packet containing the basic information of the sending node, such as the node identifier, timestamp, etc., to indicate that the sending node is still alive and working properly.

[0167] The sending and receiving of the heartbeat signal can use various network protocols, such as TCP, UDP, or a custom protocol. In this system, the master node and the standby node send heartbeat signals through a pre - set communication link.

[0168] The setting of the first frequency usually needs to consider factors such as system performance, network latency, and resource consumption. If the system does not have a very high real - time requirement for fault detection and hopes to minimize the occupation of network and system resources, a relatively low frequency can be selected. For example, the first frequency can be set to send a heartbeat signal every 10 seconds.

[0169] The setting of this frequency should be optimized according to the actual situation during the system design stage to balance the relationship between resource consumption and the timeliness of fault detection.

[0170] Send heartbeat signals using the TCP protocol: import socket import time def send_heartbeat_tcp(host, port, node_id): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) while True: try:s.connect((host, port)) timestamp = time.time() heartbeat_message = f"HEARTBEAT|{node_id}|{timestamp}" s.sendall(heartbeat_message.encode())s.close() time.sleep(10) # First frequency, send once per secondexcept Exception as e:print(f"Exception occurred when sending heartbeat signal: {e}") time.sleep(1) # Example callif __name__ == "__main__": # Fake device node IP and port backup_node_host = '192.168.1.2' backup_node_port = 12345 main_node_id = 'MainNode1' send_heartbeat_tcp(backup_node_host, backup_node_port, main_node_id).

[0171] S202. If the master node or the backup node confirms that it has not received the heartbeat signal from the other end within the agreed time, it increases the first frequency to a second frequency; the second frequency is higher than the first frequency.

[0172] The agreed time is the time window to determine whether the heartbeat signal of the other end is received normally. It should be set reasonably according to the network delay and the first frequency. For example, if the first frequency is to send a heartbeat signal every 10 seconds, the agreed time can be set to 30 seconds, which allows for a certain network delay and packet loss.

[0173] This time is set to ensure that under normal network conditions, the heartbeat signal has enough time to arrive, but not too long to delay fault detection.

[0174] Heartbeat signal receiving mechanism: import socket import time def receive_heartbeat_tcp(port, timeout): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) last_heartbeat_time = time.time() while True: try: conn, addr = s.accept() data = conn.recv(1024).decode() if data.startswith("HEARTBEAT"): last_heartbeat_time = time.time() print("Received heartbeat signal:", data) conn.close() except Exception as e: print(f"Exception occurred while receiving heartbeat signal: {e}") # Check if timeout if time.time() - last_heartbeat_time > timeout: print("Heartbeat signal timeout, increase frequency") increase_heartbeat_frequency() def increase_heartbeat_frequency(): # Logic to increase frequency pass # Example call if __name__ == "__main__": # Listening port listen_port = 12345 timeout = 30 # Agreed time is 30 seconds receive_heartbeat_tcp(listen_port, timeout)

[0175] When it is confirmed that the heartbeat signal from the peer is not received within the agreed time, increase the first frequency to the second frequency. The second frequency can be set according to specific requirements. For example, increase the frequency from once every 10 seconds to once every 2 seconds to more closely monitor the peer status.

[0176] The following is an example of implementing the frequency increase in the increase_heartbeat_frequency function: import threading import time def send_heartbeat_tcp(host, port, node_id, frequency): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) defsend(): while True: try: s.connect((host, port)) timestamp = time.time()heartbeat_message = f"HEARTBEAT|{node_id}|{timestamp}" s.sendall(heartbeat_message.encode()) s.close() time.sleep(frequency) except Exception as e:print(f"Exception occurred when sending heartbeat signal: {e}") time.sleep(1) def increase_heartbeat_frequency(): global heartbeat_thread # Stop the original heartbeat sending thread heartbeat_thread.join() # Increase the frequency, for example from once every 10 seconds to once every 2 seconds new_frequency = 2heartbeat_thread = threading.Thread(target=send_heartbeat_tcp, args=('192.168.1.2', 12345, 'MainNode1', new_frequency)) heartbeat_thread.start()if __name__ == "__main__": # Fake device node IP and port backup_node_host = '192.168.1.2' backup_node_port = 12345 main_node_id = 'MainNode1' # The initial frequency is once every 10 seconds initial_frequency = 10 heartbeat_thread = threading.Thread(target=send_heartbeat_tcp, args=(backup_node_host, backup_node_port, main_node_id,initial_frequency)) heartbeat_thread.start().

[0177] In an embodiment of the present invention, based on step S3, a possible embodiment will be given below to non - restrictively elaborate on its specific implementation.

[0178] Confirm that the name of the arbitration master node is the same as its own name, and continue to provide the master node service; confirm that the name of the arbitration master node is inconsistent with its own name, control the high - concurrency service to enter the fault state, and delete the virtual address.

[0179] Specifically, the master - standby node timing task is responsible for detecting the status of Keepalived and drbd of this node. If it is found that Keepalived is the master and the peer cannot be connected, it is considered that a brain split has indeed occurred at this time. Then, the arbitration node is accessed, and the relevant processing logic is as follows: If the third - party node cannot be accessed, close the keepalived - nfs service; When the third - party node returns the nodename of the peer, close the keepalived - nfs service; When the third - party node returns its own nodename, do nothing; When the third - party node returns none, close the keepalived service.

[0180] In an embodiment of the present invention, based on step S4, a possible embodiment will be given below to non - restrictively elaborate on its specific implementation.

[0181] The brain - split detection rule includes: before the high - concurrency service switches the master - standby state, obtain the status of the peer by sending an echo request to the peer; confirm that the status of the peer is abnormal, then obtain the name of the arbitration node from the third - party node; confirm that the name of the arbitration node is inconsistent with its own name, then do not configure its own distributed replicated block device as the main state, so that its own distributed replicated block device does not have the write permission for data.

[0182] Specifically, improve the notify_master script of Keepalived and add the split-brain judgment logic when Keepalived switches to the master. If the peer is found to be unreachable at this time, access the arbitration node. If the arbitration node does not return the current node, do not set Drbd to the master state. In this way, even if there are two VIPs, the standby NFS cannot provide services, ensuring data consistency.

[0183] The notify_master script is a customizable script provided by Keepalived and is triggered when the status of the master node changes (becomes the master node). Adding the split-brain judgment logic in the notify_master script of Keepalived is to perform additional checks during the master node switch to avoid problems caused by split-brain.

[0184] When Keepalived performs the master-slave switch and the current node is about to become the master node, it will first check whether the peer (the original master node or the standby node) is reachable. This is because in the normal master-slave switch process, the original master node should have stopped serving. If the peer is still reachable, it may mean that there is a network partition or other abnormalities, posing a risk of split-brain.

[0185] When the peer is found to be unreachable, the current node will access the arbitration node and request the arbitration node to make a ruling on the current situation.

[0186] If the result returned by the arbitration node is not the current node, it means that the current node should not become the master node, even if Keepalived thinks that the current node should switch to the master node.

[0187] In this case, do not set Drbd (Distributed Replicated Block Device, a tool for implementing data replication) to the master, that is, do not mount the data and start the service, to avoid the situation where two nodes simultaneously have the VIP and provide services.

[0188] Through the above mechanism, even if there is a situation where both nodes think they are the master node (there are two VIPs), since the standby NFS (Network File System) service is not started, the standby node cannot provide services, thus avoiding the situation where two nodes simultaneously read and write to the storage device and ensuring data consistency.

[0189] In some embodiments, the high-availability service split-brain state processing system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the high-availability service split-brain state processing system may be stored in the memory of a computer device and executed by at least one processor to perform (see Figure 1 description) the functions of high-availability service split-brain state processing.

[0190] In this embodiment, according to the functions it performs, the high-availability service split-brain state processing system may be divided into multiple functional modules, as Figure 3 shown. The functional modules of the system may include: an arbitration module, a monitoring module, a processing module, and a pre-check module. The module referred to in the present invention means a series of computer program segments that can be executed by at least one processor and can complete fixed functions, and are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0191] The arbitration module is used to monitor the states of the primary node and the standby node through a third-party node, and specify the arbitration primary node name based on the monitoring results; The monitoring module is used for the primary node and the standby node to mutually monitor the states of each other's counterparts; The processing module is used when the primary node or the standby node confirms that the state of the counterpart is abnormal, to obtain the arbitration primary node name from the third-party node, and confirm whether it provides the primary node service by comparing the consistency between the arbitration primary node name and its own name; The pre-check module is used to configure split-brain detection rules in the switching auxiliary scripts of the primary node and the standby node respectively, so that the primary node and the standby node perform split-brain detection before switching the primary and standby states, and the split-brain detection rules include obtaining the state of the counterpart.

[0192] Figure 4 The high-availability service split-brain state processing method provided by the embodiments of the present application can be applied to a device. Those skilled in the art can understand that the device structure involved in the embodiments of the present invention does not constitute a limitation on the device. The device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In the embodiments of the present invention, the device includes but is not limited to laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described herein and / or claimed.

[0193] Among them, the device 400 may include: a processor 410, a memory 420, and a communication unit 430. These components communicate via one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation to the present invention. It can be a bus structure, a star structure, and may also include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0194] Among them, the memory 420 can be used to store the execution instructions of the processor 410. The memory 420 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. When the execution instructions in the memory 420 are executed by the processor 410, the device 400 can execute some or all of the steps in the above method embodiments.

[0195] The processor 410 is the control center of the storage device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing the software programs and / or modules stored in the memory 420, and by calling the data stored in the memory, it executes various functions of the electronic device and / or processes data. The processor may be composed of an integrated circuit (IC). For example, it may be composed of a single packaged IC, or may be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 410 may only include a central processing unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core or may include multiple operation cores.

[0196] The communication unit 430 is used to establish a communication channel, so that the storage device can communicate with other devices. It receives user data sent by other devices or sends user data to other devices.

[0197] The present invention also provides a computer storage medium. Among them, the computer storage medium can store a program, and when the program is executed, it may include some or all of the steps in the embodiments provided by the present invention. The storage medium may be a magnetic disk, an optical disc, a read-only memory (ROM), or a random access memory (RAM), etc.

[0198] Those skilled in the art can clearly understand that the technologies in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc, etc., various media that can store program codes, including several instructions for causing a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0199] For the same or similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the descriptions in the method embodiments.

[0200] In several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the systems or modules can be in an electrical, mechanical or other forms.

[0201] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place, or they can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, the various functional modules in the various embodiments of the present invention can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0203] Although the present invention has been described in detail by reference to the accompanying drawings and in conjunction with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should all be within the scope of the present invention / Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention.

Claims

1. A method for processing a split-brain state of a high-availability service, characterized in that: include: Monitor the status of the master node and the standby node through a third-party node, and designate the arbitration master node name based on the monitoring results; The master node and the backup node monitor each other's status; When the master node or the standby node confirms that the status of the other end is abnormal, it obtains the arbitration master node name from the third-party node, and confirms whether it provides the master node service by comparing the consistency between the arbitration master node name and its own name; The split-brain detection rule is configured in the switching auxiliary script of the master node and the standby node respectively, so that the master node and the standby node perform split-brain detection before switching the master-standby state, and the split-brain detection rule includes obtaining the peer state.

2. The method according to claim 1, characterized in that The method further comprises: Deploy the nginx service on the third-party node, and configure the communication interface to the master node and the communication interface to the standby node based on the nginx service; After the master node and the backup node perform the master-slave switch, they report the state switching event to the third-party node through the corresponding communication interface; The third-party node parses the current master node information and the current standby node information from the state switching event, and updates the current master node information and the current standby node information to the node information list.

3. The method according to claim 2, characterized in that Monitor the status of the master node and standby node through a third-party node, and specify the arbitration master node name based on the monitoring results, including: Setting a timed task at a third-party node, the timed task comprising periodically sending an echo request data packet to a monitoring target based on the node information list, the monitoring target comprising a primary node and a backup node; Waiting to receive a response data packet from the monitoring target to the echo request data packet, if the response data packet is not received within a specified period, determining that the monitoring target is abnormal, if the response data packet is received within the specified period, determining that the monitoring target is normal; Create a structure, and store the master node name in the node information list as a default value in the structure; If the master node is normal and the backup node is abnormal, the structure is not updated; If the primary node is normal and the backup node is normal, the structure is not updated; If the master node is abnormal and the standby node is normal, the node name of the standby node is updated to the structure; If the master node is abnormal and the backup node is abnormal, the structure is updated to none.

4. The method according to claim 2, characterized in that: The method further comprises: Set up an election system to select the node with the most idle resources from nodes other than the primary node and the backup node as the third-party node; Start a monitoring thread on the third-party node, where the monitoring thread collects operating parameters and operating logs of the third-party node; Comparing the operating parameters with a preset standard threshold, generating abnormal record data for the operating parameters that do not meet the standard threshold, the abnormal record data including the corresponding operating parameter item, parameter value and acquisition time; Extracting error data from the operation log using keyword extraction technology; Integrate the abnormal record data and the error data into a data set, and perform deduplication processing on the data set; Based on the correlation between parameter items in the data set and the time correlation, the data in the data set is divided into multiple data groups using a pre-trained random forest model; Encapsulating the plurality of data groups into a plurality of query statements; Inputting the plurality of query statements into the pre-built fault diagnosis knowledge graph in sequence to obtain the diagnosis result of the third-party node; If it is confirmed that the diagnosis result reaches a preset fault level, the election is re-executed to obtain a new third-party node; The current third-party node synchronizes the local node information list and structure to the new third-party node; After receiving the receipt information from the new third-party node, the current third-party node deletes the node information list and structure stored in itself; Publish the information of the new third-party node to the master node and the slave node.

5. The method according to claim 1, characterized in that The master node and the backup node monitor each other's status, including: The master node and the backup node send heartbeat signals to each other at the agreed first frequency; If the master node or the standby node confirms that it has not received the heartbeat signal from the other end within the agreed time, the first frequency is increased to the second frequency; The second frequency is higher than the first frequency.

6. The method according to claim 1, characterized in that Confirming whether it provides master node services by comparing the name of the arbitration master node with its own name, including: Confirm that the arbitration master node name is consistent with its own name and continue to provide master node services; Confirm that the arbitration master node name is inconsistent with its own name, control the high-concurrency service to enter a fault state, and delete the virtual address.

7. The method according to claim 1, characterized in that The split-brain detection rules include: Before the high-concurrency service switches to the active / standby state, it obtains the peer status by sending an echo request to the peer. If the peer status is confirmed to be abnormal, obtain the arbitration node name from the third-party node; If it is confirmed that the arbitration node name is inconsistent with its own name, its own distributed replication block device is not configured as a master state, so that its own distributed replication block device does not have write permission to the data.

8. A high-availability service split-brain state processing system, characterized in that: include: The arbitration module is used to monitor the status of the master node and the standby node through a third-party node, and designate the name of the arbitration master node based on the monitoring results; Monitoring module, used for the master node and the backup node to monitor the status of each other; A processing module, used for the master node or the standby node to obtain the arbitration master node name from a third-party node when confirming that the state of the other end is abnormal, and to confirm whether it provides the master node service by comparing the consistency between the arbitration master node name and its own name; The pre-check module is used to configure the brain split detection rules in the switching auxiliary scripts of the master node and the backup node respectively, so that the master node and the backup node perform brain split detection before switching the master-slave state, and the brain split detection rules include obtaining the peer state.

9. A device, characterized in that: include: A memory for storing a high-availability service split-brain state handler; A processor, configured to implement the steps of the high availability service split brain state processing method as described in any one of claims 1 to 7 when executing the high availability service split brain state processing program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a high-availability service split-brain state processing program, and when the high-availability service split-brain state processing program is executed by the processor, the steps of the high-availability service split-brain state processing method according to any one of claims 1 to 7 are implemented.

Images