High-availability cluster system and failover and service self-healing method

Abstract

This invention discloses a highly reliable, highly self-healing, and highly available cluster system and a fault switching and service self-healing method. By introducing an independent arbitration node, an odd-numbered sentinel cluster, a middleware status synchronization proxy, and a dynamic priority adjustment mechanism, it achieves second-level fault switching and seamless business recovery. The highly available cluster system includes: an arbitration server and at least two business servers (the number of business servers is even), with communication connections between the business servers and the arbitration server; the arbitration server is equipped with monitoring components to monitor the operational status of the business servers, including a Redis sentinel instance; the business servers are equipped with business applications, Redis sentinel instances, a virtual IP management module, and a middleware status synchronization proxy module, and business data is synchronized between the business servers.

Description

Technical Field

[0001] This invention relates to the field of computer software high availability and disaster recovery technology, specifically to a high availability cluster system and a fault switching and service self-healing method. Background Technology

[0002] Current mainstream high-availability solutions mostly adopt a dual-machine hot standby architecture, relying on shared storage or master-slave replication to achieve failover. For example, solutions such as RoseHA and Pacemaker+Corosync trigger virtual IP migration through heartbeat detection, with switchover times typically ranging from 30 seconds to 1 minute. However, such solutions have the following technical drawbacks: The risk of split-brain remains unresolved: When a dual-node system partitions the network, it cannot determine which node should continue to provide services, which may lead to dual-master writes, causing data conflicts or loss.

[0003] Middleware state cannot be migrated: MQTT broker services such as EMQX and Mosquitto maintain client sessions and subscription relationships. This state is usually stored in local memory. After a master-slave switch, the new master cannot inherit the original session, resulting in a large number of devices disconnecting and reconnecting, and business interruption.

[0004] Insufficient database synchronization reliability: Traditional MySQL master-slave replication is based on binlog position. If the slave node is offline for a long time, the slave cannot catch up after the master purges the log, and manual intervention is required to rebuild it. Even with a dual-master architecture, auto-incrementing ID conflicts can easily lead to replication interruption.

[0005] Limited alarm mechanisms: Most solutions only provide log recording and lack proactive, multi-channel (email, SMS, IM) real-time alarm capabilities, which affects the speed of fault response.

[0006] Therefore, there is an urgent need for a new type of high-availability cluster solution that can simultaneously address the four major pain points of arbitration decision-making, strong data consistency, middleware state recovery, and intelligent alarms. Summary of the Invention

[0007] The purpose of this invention is to overcome the defects in the existing technology and provide a highly reliable, self-healing, and highly available cluster system and a fault switching and service self-healing method. By introducing an independent arbitration node, an odd number of sentinel clusters, a middleware state synchronization proxy, and a priority dynamic adjustment mechanism, it can achieve second-level fault switching and seamless service recovery.

[0008] To achieve the above objectives, a first aspect of the present invention provides a highly available cluster system, comprising: An arbitration server and at least two business servers, wherein the number of business servers is even, and communication connections exist between each business server and the arbitration server. The arbitration server is equipped with a monitoring component that monitors the operating status of the business server, including a Redis Sentinel instance; The business server is equipped with business applications, a Redis Sentinel instance, a virtual IP management module, and a middleware status synchronization proxy module, and business data is synchronized between the business servers. The arbitration server and each Redis Sentinel instance on the business server constitute an odd-numbered Redis Sentinel cluster. The Redis Sentinel cluster performs a master node election among the business servers to determine the business server as the master node. The virtual IP management module is used to manage the virtual IP address that provides a unified service entry point to the outside world, and dynamically bind the virtual IP address to the business server with the highest priority based on the priority of the business server. The priority of the business server is determined based on whether the business server is determined as the master node and the business port status of the business server. The middleware state synchronization proxy module is used to obtain the session information of online clients accessing through the virtual IP address from the middleware database after the server is bound to the virtual IP address, so as to restore the session with the online client on the server. The session information of the online client is synchronized to the middleware database by the business application on the business server bound to the virtual IP address before the business server bound to the virtual IP address is switched.

[0009] Furthermore, the business applications include web services, Java application services, MQTT message brokers, and database instances.

[0010] Furthermore, when synchronizing business data between various business servers, the database instance is used to implement master-slave replication between the business servers based on a global transaction identifier to complete the business data synchronization.

[0011] Furthermore, when the middleware state synchronization agent module obtains the session information of the online client accessing through the virtual IP address from the middleware database in order to restore the session with the online client on this server, it specifically calls the HTTP management API or command-line tool provided by the MQTT message broker to query the client identifier, authentication credentials, and subscription topic list of the current online client in the middleware database as session information, and injects the session information into the session storage of the MQTT message broker on this server to restore the session with the online client on this server.

[0012] Furthermore, the virtual IP management module includes a Keepalived service; when dynamically binding the virtual IP address to the highest priority business server, the virtual IP management module specifically uses the Keepalived service to bind the highest priority business server to the virtual IP address based on the VRRP protocol.

[0013] Furthermore, the monitoring component also includes a Keepalived auxiliary monitoring module and a service health check module configured in the Keepalived service on each business server, used to periodically probe the TCP connectivity of the ports of the business application and obtain port status results.

[0014] Furthermore, the virtual IP management module is also used to obtain the port status results of this server and the current master node business server; if the port status results of this server show that each business port is connectable and this server is the master node, then the priority of this server is set to the first priority; if the port status results of this server show that each business port is connectable and this server is not the master node, then the priority of this server is set to the second priority; if the port status results of this server show that there are unconnectable business ports, then the priority of this server is set to the third priority; wherein, the first priority is higher than the second priority, and the second priority is higher than the third priority.

[0015] Furthermore, the detection cycle of the service health check module is 1 second, and the virtual IP management module is also used to determine that the server service is abnormal when the service health check module fails to detect for a preset number of consecutive times, and to set the priority of the server to the third priority.

[0016] Furthermore, the monitoring component also includes a multi-channel alarm listening program, which is used to send alarm messages via at least two of the following communication methods: email, SMS and instant messaging tools, when the business server crashes or the business server bound to the virtual IP address is switched.

[0017] A second aspect of the present invention provides a fault switching and service self-healing method, applied to the high-availability cluster system described in the first aspect, comprising: When a business server acting as the master node fails, causing the Redis Sentinel instance to go offline, the Redis Sentinel cluster performs a master node election among the business servers to determine the business server that will become the new master node. The virtual IP management module of each business server dynamically adjusts the local priority based on whether the server is identified as the master node and the connectivity status of the local critical business service port, and binds the virtual IP address to the target business server with the highest priority. The middleware state synchronization proxy module of the target business server obtains the session information of the online client accessing through the virtual IP address from the middleware database, and restores the session with the online client on this server. The session information of the online client is synchronized to the middleware database by the business application on the business server bound to the virtual IP address before the business server bound to the virtual IP address is switched.

[0018] The advantages and beneficial effects of this invention are as follows: By introducing an independent arbitration server, which together with the Redis Sentinel instances on an even number of business servers forms an odd-numbered Redis Sentinel cluster, the cluster can avoid split-brain through a majority decision-making mechanism when the network is partitioned. At the same time, a virtual IP management module is set up to dynamically adjust the priority and bind the virtual IP address based on the master node role and port status of the business server, realizing differentiated and rapid response to node-level and service-level failures. Furthermore, by setting up a middleware state synchronization proxy module, after the virtual IP address is bound to a new server, the session information of online clients is obtained from the middleware database and restored, solving the problem of middleware state migration in traditional solutions. This enables seamless business switching, effectively solving the risk of split-brain in the cluster system, and achieving seamless self-healing of external business disconnection. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the architecture of a high-availability cluster system according to the present invention; Figure 2 Based on Figure 1 A schematic diagram of the specific deployment architecture shown; Figure 3 This is a flowchart of the fault switching and service self-healing method of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0021] according to Figure 1 As shown, the high-availability cluster system of the present invention includes: An arbitration server and at least two business servers, wherein the number of business servers is even, and communication connections exist between each business server and the arbitration server. The arbitration server is equipped with a monitoring component that monitors the operating status of the business server, including a Redis Sentinel instance; The business server is equipped with business applications, a Redis Sentinel instance, a virtual IP management module, and a middleware status synchronization proxy module, and business data is synchronized between the business servers. The arbitration server and each Redis Sentinel instance on the business server constitute an odd-numbered Redis Sentinel cluster. The Redis Sentinel cluster performs a master node election among the business servers to determine the business server as the master node. The virtual IP management module is used to manage the virtual IP address that provides a unified service entry point to the outside world, and dynamically bind the virtual IP address to the business server with the highest priority based on the priority of the business server. The priority of the business server is determined based on whether the business server is determined as the master node and the business port status of the business server. The middleware state synchronization proxy module is used to obtain the session information of online clients accessing through the virtual IP address from the middleware database after the server is bound to the virtual IP address, so as to restore the session with the online client on the server. The session information of the online client is synchronized to the middleware database by the business application on the business server bound to the virtual IP address before the business server bound to the virtual IP address is switched.

[0022] In this embodiment, two business servers (Business Server A and Business Server B) and one arbitration server are used as examples. Business Server A and Business Server B are respectively deployed with business applications, Redis Sentinel instances, virtual IP management modules, and middleware status synchronization proxy modules, and business data is synchronized between the two business servers. The arbitration server is deployed with monitoring components to monitor the running status of the business servers, including a Redis Sentinel instance. The Redis Sentinel instance on the arbitration server and the Redis Sentinel instances on the two business servers together form a 3-node odd-numbered Redis Sentinel cluster.

[0023] This Redis Sentinel cluster is used to perform master node election among business servers to determine the business server to serve as the master node. Specifically, three Sentinel instances monitor the health status of the Redis instances on each business server through mutual communication. When the Redis instance on the business server that should serve as the master node fails, the Sentinel cluster elects a new master node through a voting mechanism. The Arbitration Server, acting as an arbitration node that only participates in the voting election, forms an odd-numbered arbitration group with each business server node to satisfy the majority decision principle. The Arbitration Server is configured as a low-resource-consuming node and does not run business loads. Based on the availability status of the Redis instances on the business servers it monitors, the Arbitration Server ensures that the newly elected master node is available, thereby ensuring high availability at the data layer.

[0024] The virtual IP management module deployed on each business server manages the virtual IP addresses that provide a unified service entry point to the outside world. In this embodiment, the virtual IP management module is specifically the Keepalived service, which dynamically binds virtual IP addresses between business server A and business server B based on the VRRP protocol. The virtual IP management module dynamically binds the virtual IP address to the business server with the highest priority based on the priority of the business server. The priority of the business server is dynamically determined based on two factors: first, whether the business server has been identified as the master node by the Redis Sentinel cluster; and second, the connectivity status of the business port of the business server (e.g., whether the Web service port 80, MQTT service port 1883, Java application port 8080, etc., are reachable).

[0025] The middleware state synchronization proxy module deployed on each business server is used to retrieve session information of online clients accessing through that virtual IP address from the middleware database after the server is bound to a virtual IP address, in order to restore the session with the online clients on the local server. The online client session information is synchronized to the middleware database by the business application on the business server that was previously bound to the virtual IP address before the business server is switched. Specifically, when a device accesses through a virtual IP address, the MQTT message broker in the business application triggers a connection event. The business system listens for this event and writes the session information (including client identifier, authentication credentials, subscription topic list, etc.) to the middleware database. The middleware database can be deployed as a local instance on each business server and synchronized via master-slave replication, or as an independent remote database. Both deployment methods ensure that the latest online client session information is accessible whenever any business server becomes the master node.

[0026] In the following two typical fault scenarios, this system can achieve fault switching and self-healing: Scenario 1: The business server acting as the master node completely crashes (hardware failure). When business server A (the current master node) crashes due to a hardware failure, the Redis Sentinel instance deployed on it also goes offline. The Sentinel instances on the arbitration server and business server B detect A's loss of connection and elect business server B as the new master node through a voting mechanism. Simultaneously, since business server B's business port is in normal status and it has been elected as the new master node, it receives the highest priority and automatically takes over the virtual IP address. The middleware state synchronization proxy module on business server B is triggered immediately after the virtual IP is bound, retrieves the session information of currently online clients from the middleware database, and injects the session information into the local in-memory session storage by calling the MQTT message broker's HTTP management API or command-line tools, thus restoring the session with online clients. In this scenario, the data layer master node and the access layer traffic entry point synchronously switch to the same server, the system quickly restores service, and the client is almost unaware of the changes.

[0027] Scenario 2: Network failure causes master node service unavailability, but the Redis instance remains online. Assume that business server A experiences a network failure, making its external service ports (web service, MQTT service, etc.) unreachable, but its Redis instance is still running and communicating normally with the arbitration server and business server B. At this time, the Redis Sentinel cluster detects that A's Redis instance is still running and therefore still designates A as the master node. However, the Keepalived service on business server A detects through its health check module that the local business port is unreachable and automatically lowers its local priority (e.g., from first priority 100 to third priority 80). The Keepalived service on business server B detects that its local priority (second priority 90) is higher than A's current priority and therefore binds the virtual IP address to itself. After business server B acquires the virtual IP, its middleware state synchronization proxy module is triggered, retrieves session information from the middleware database, injects it into local MQTT memory, and begins providing services externally. Once the network of business server A is restored and the business port becomes accessible again, its Keepalived service will restore its priority to the highest priority of 100, and rebind the virtual IP address back to A according to the preemption strategy (such as enabling preemption mode). Simultaneously, the middleware state synchronization proxy module on A triggers a recovery session, completing the service rollback. During this process, since the Redis master node remains on A, the data layer does not switch, avoiding data inconsistency. The access layer achieves rapid service recovery and rollback through virtual IP migration, ensuring business continuity while avoiding the risk of split-brain.

[0028] This invention introduces an independent arbitration server, which, together with Redis Sentinel instances on an even number of business servers, forms an odd-numbered Redis Sentinel cluster. This enables the cluster to avoid split-brain scenarios during network partitioning through a majority decision-making mechanism. Simultaneously, a virtual IP management module dynamically adjusts priorities and binds virtual IP addresses based on the master node role and port status of the business servers, achieving differentiated and rapid responses to node-level and service-level failures. Furthermore, by setting up a middleware state synchronization proxy module, after the virtual IP address is bound to a new server, the session information of online clients is retrieved from the middleware database and restored. This solves the problem of middleware state migration in traditional solutions, enabling seamless business switching and effectively mitigating the risk of split-brain scenarios in cluster systems. It also achieves seamless self-healing from external business disconnections.

[0029] To ensure the consistency and recoverability of business data, a preferred embodiment of this invention includes a business application comprising a Web service, a Java application service, an MQTT message broker, and a database instance. When synchronizing business data between the various business servers, master-slave replication is specifically implemented between the business servers using the database instance based on a Global Transaction Identifier (GTID) to complete the business data synchronization.

[0030] The web service provides the HTTP access point, the Java application service handles business logic, the MQTT message broker is responsible for message communication with IoT devices, and the database instance persistently stores business data. These components together constitute the infrastructure of an enterprise-level business system. In actual deployment, these components can run on the business server in a containerized or traditional manner.

[0031] In this embodiment, the database instances of each business server form a distributed database subsystem (such as MySQL). For example, by enabling gtid_mode=ON and enforce_gtid_consistency=ON in the MySQL configuration file, the MySQL master and slave nodes enable GTID (Global Transaction Identifier) ​​mode, establishing a master-slave replication channel. Each transaction is assigned a globally unique Global Transaction Identifier (GTID). When the original master node fails and comes back online, its MySQL instance automatically joins the replication topology as a slave, automatically identifying and synchronizing missing transactions through GTID. This eliminates the need for manual comparison of binlog positions, achieving plug-and-play fault recovery.

[0032] To quickly restore the session with the online client, a preferred embodiment of the present invention is that, when the middleware state synchronization proxy module obtains the session information of the online client accessing through the virtual IP address from the middleware database to restore the session with the online client on the local server, it specifically calls the HTTP management API or command-line tool provided by the MQTT message broker to query the client identifier, authentication credentials, and subscription topic list of the current online client in the middleware database as session information, and injects the session information into the session storage of the MQTT message broker on the local server to restore the session with the online client on the local server.

[0033] Taking EMQX as an example, this MQTT message broker provides a standard HTTP management API (such as / api / v4 / clients). The middleware state synchronization broker module can use this API to inject client identifiers, subscription relationships, and other information obtained from the middleware database into EMQX's in-memory session storage. This middleware state synchronization broker module is deployed independently as a daemon process, triggered only after a virtual IP address is bound to the local machine. It is read-only and has a single responsibility, enabling it to quickly restore sessions with online clients on the business server that takes over the virtual IP, while ensuring the system's simplicity and reliability.

[0034] To ensure mature and reliable management of virtual IPs, a preferred embodiment of the present invention includes a Keepalived service in which the virtual IP management module dynamically binds the virtual IP address to the highest-priority business server using the Keepalived service based on the VRRP protocol.

[0035] Keepalived, a mature high-availability solution, enables virtual IP migration between multiple business servers via the VRRP protocol. In this system, Keepalived not only handles heartbeat detection but also dynamically adjusts priorities through custom scripts linked to the health check module and Redis sentinel election results.

[0036] To achieve fine-grained monitoring of business service ports, a preferred embodiment of the present invention includes a Keepalived auxiliary monitoring module and a service health check module configured in the Keepalived service on each business server, used to periodically probe the TCP connectivity of the ports of the business application and obtain port status results.

[0037] The Keepalived auxiliary monitoring module is deployed on the arbitration server to listen for cluster status events (such as virtual IP switching and node failures) and trigger alarms. The service health check module periodically probes the TCP connectivity of critical service ports (Nginx, port 80, MQTT, port 1883, and Java application, port 8080) using custom scripts (such as the health check script, implemented via vrrp_script) and reports the results back to the Keepalived main process.

[0038] To achieve reasonable virtual IP migration through a dynamically adjustable priority system, a preferred embodiment of the present invention is that the virtual IP management module is further configured to obtain the port status results of the local server and the current master node business server; if the port status results of the local server show that each business port is connectable and the local server is the master node, then the priority of the local server is set to the first priority; if the port status results of the local server show that each business port is connectable and the local server is not the master node, then the priority of the local server is set to the second priority; if the port status results of the local server show that there are unconnectable business ports, then the priority of the local server is set to the third priority; wherein, the first priority is higher than the second priority, and the second priority is higher than the third priority.

[0039] In a specific example, the first priority is set to 100, the second priority to 90, and the third priority to 80. This ensures that the primary node and the server that is providing normal service have the highest priority in holding the virtual IP, the secondary node and the server that is providing normal service have the next highest priority, and any server with abnormal service has the lowest priority. Even if it is the primary node, it will trigger the virtual IP to migrate to other nodes. To improve the anti-interference capability and accuracy of port fault detection, a preferred embodiment of the present invention is that the detection period of the service health check module is 1 second, and the virtual IP management module is further used to determine that the server service is abnormal when the service health check module fails to detect for a preset number of consecutive times, and to set the priority of the server to the third priority.

[0040] In this embodiment, the preset number of attempts can be set to 3, that is, if the detection fails for 3 consecutive seconds, the service is judged to be abnormal. This ensures a fast response (second-level) and avoids misjudgment caused by network jitter.

[0041] To improve operational and maintenance response efficiency, a preferred embodiment of the present invention is that the monitoring component further includes a multi-channel alarm listening program, which is used to send alarm messages through at least two of the following communication methods: email, SMS and instant messaging tools, when the business server crashes or the business server bound to the virtual IP address is switched.

[0042] The multi-channel alarm monitoring program is deployed on the arbitration server. It obtains cluster events by listening to Redis Pub / Sub or local sockets. Once a failover is detected, it immediately calls the SMTP email service, WeChat / DingTalk Webhook or third-party SMS API to send alarms, ensuring that operations and maintenance personnel are informed as soon as possible.

[0043] Figure 2The detailed deployment architecture further refines the deployment location and interaction details of each component, including how specific services or components such as Nginx, MinIO, and MongoDB are integrated into the high-availability cluster system architecture of this embodiment. External requests enter the system via the Keepalived virtual IP and virtual IP management module, and are forwarded to the backend application by the Nginx reverse proxy; the backend application accesses MySQL, MongoDB, and Redis databases; MinIO is used to store file data; each database and middleware is configured in master-slave replication mode to ensure data consistency and high availability; the listener is a multi-channel alarm listener, with the listener on the arbitration server monitoring the service availability status of the business servers and virtual IP address switching events through the listener nodes on each business server, etc.

[0044] like Figure 3 As shown, the fault switching and service self-healing method of the present invention, applied to the high-availability cluster system described in the above embodiment, includes: S101. When the business server acting as the master node fails and causes the Redis Sentinel instance to go offline, the Redis Sentinel cluster performs a master node election among the business servers to determine the business server as the new master node. S102. The virtual IP management module of each business server dynamically adjusts the local priority based on whether the server is identified as the master node and the connectivity status of the local key business service port, and binds the virtual IP address to the target business server with the highest priority. S103. The middleware state synchronization proxy module of the target business server obtains the session information of the online client accessing through the virtual IP address from the middleware database, and restores the session with the online client on this server. The session information of the online client is synchronized to the middleware database by the business application on the business server bound to the virtual IP address before the business server bound to the virtual IP address is switched.

[0045] This method, through the steps described above, enables the system to automatically switch and self-heal under two fault scenarios: In hardware failure scenarios (complete failure of the master node), the election of a new master node and virtual IP migration occur simultaneously in the Sentinel cluster. The new master node simultaneously holds data control and traffic entry, and quickly restores services through state synchronization. There is no need to switch back after the switch, and the system runs stably on the new master node.

[0046] In service failure scenarios (master node network or service anomalies but Redis survives), the Sentinel cluster retains the original node as the master to ensure data consistency, while the virtual IP is migrated to a healthy slave node via priority comparison, where the slave temporarily takes over traffic. Once the original node recovers, the virtual IP is switched back, and the state synchronization proxy resumes the session, achieving complete self-healing of the service. This two-layer decision-making mechanism effectively avoids split-brain while ensuring business continuity and data consistency.

[0047] In one implementation, the specific steps include: Initialize the cluster: Deploy three servers and assign them fixed internal network IPs; apply for virtual IPs; install business suites on the business servers and deploy sentinels and alarm programs on the arbitration server.

[0048] Configure Keepalived: Set the primary node priority to 100 and the standby node priority to 90; configure a health check script so that if any critical port becomes unreachable, the local machine priority is reduced to 80, triggering virtual IP migration.

[0049] To enable GTID replication: Enable gtid_mode=ON and enforce_gtid_consistency=ON in the MySQL configuration file to establish a master-slave replication channel.

[0050] Fault detection: When the master node loses connection due to hardware failure or network interruption, the backup node Keepalived takes over the virtual IP after three consecutive heartbeat timeouts.

[0051] Service Rebuild: The standby node automatically starts all business processes (Nginx, Java App, MQTT).

[0052] State self-healing: After the MQTT service starts up, the state synchronization agent is triggered to execute the SQL query SELECT client_id, username, password, topics FROM device_sessions WHERE status='online', and the result is injected into the local session storage through the MQTT management API.

[0053] Alarm notification: When the arbitration server detects a virtual IP switching event via Redis Pub / Sub or a local socket, it immediately calls the multi-channel alarm module to send a notification.

[0054] Node recovery: After the original master node is repaired and brought back online, its MySQL instance is automatically added to the GTID replication topology as a slave, and missing transactions are incrementally synchronized.

[0055] Figure 3 The fault switching and service self-healing method shown has similar technical effects to the system embodiments described above, and will not be repeated here.

[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0057] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A highly available cluster system, characterized in that, include: An arbitration server and at least two business servers, wherein the number of business servers is even, and communication connections exist between each business server and the arbitration server. The arbitration server is equipped with a monitoring component that monitors the operating status of the business server, including a Redis Sentinel instance; The business server is equipped with business applications, a Redis Sentinel instance, a virtual IP management module, and a middleware status synchronization proxy module, and business data is synchronized between the business servers. The arbitration server and each Redis Sentinel instance on the business server constitute an odd-numbered Redis Sentinel cluster. The Redis Sentinel cluster performs a master node election among the business servers to determine the business server as the master node. The virtual IP management module is used to manage the virtual IP address that provides a unified service entry point to the outside world, and dynamically bind the virtual IP address to the business server with the highest priority based on the priority of the business server. The priority of the business server is determined based on whether the business server is determined as the master node and the business port status of the business server. The middleware state synchronization proxy module is used to obtain the session information of online clients accessing through the virtual IP address from the middleware database after the server is bound to the virtual IP address, so as to restore the session with the online client on the server. The session information of the online client is synchronized to the middleware database by the business application on the business server bound to the virtual IP address before the business server bound to the virtual IP address is switched.

2. The high-availability cluster system according to claim 1, characterized in that, The business applications include web services, Java application services, MQTT message brokers, and database instances.

3. The high-availability cluster system according to claim 2, characterized in that, When synchronizing business data between various business servers, the database instance is used to implement master-slave replication between the business servers based on a global transaction identifier to complete the business data synchronization.

4. The high-availability cluster system according to claim 2, characterized in that, When the middleware state synchronization agent module retrieves session information of online clients accessing through the virtual IP address from the middleware database to restore the session with the online client on the local server, it specifically calls the HTTP management API or command-line tool provided by the MQTT message broker to query the client identifier, authentication credentials, and subscription topic list of the current online client in the middleware database as session information, and injects the session information into the session storage of the MQTT message broker on the local server to restore the session with the online client on the local server.

5. The high-availability cluster system according to claim 1, characterized in that, The virtual IP management module includes the Keepalived service; when dynamically binding the virtual IP address to the highest priority business server, the virtual IP management module specifically uses the Keepalived service to bind the highest priority business server to the virtual IP address based on the VRRP protocol.

6. The high-availability cluster system according to claim 5, characterized in that, The monitoring component also includes a Keepalived auxiliary monitoring module and a service health check module configured in the Keepalived service on each business server, which is used to periodically probe the TCP connectivity of the ports of the business application and obtain port status results.

7. The high-availability cluster system according to claim 6, characterized in that, The virtual IP management module is also used to obtain the port status results of this server and the current master node business server; if the port status results of this server show that each business port is connectable and this server is the master node, then the priority of this server is set to the first priority; if the port status results of this server show that each business port is connectable and this server is not the master node, then the priority of this server is set to the second priority; if the port status results of this server show that there are unconnectable business ports, then the priority of this server is set to the third priority; wherein, the first priority is higher than the second priority, and the second priority is higher than the third priority.

8. The high-availability cluster system according to claim 7, characterized in that, The detection cycle of the service health check module is 1 second. The virtual IP management module is also used to determine that the server service is abnormal when the service health check module fails to detect for a preset number of consecutive times, and to set the priority of the server to the third priority.

9. The high-availability cluster system according to claim 1, characterized in that, The monitoring component also includes a multi-channel alarm listening program, which is used to send alarm messages via at least two of the following communication methods: email, SMS and instant messaging tools, when the business server crashes or the business server bound to the virtual IP address is switched.

10. A fault switching and service self-healing method, applied to the high-availability cluster system as described in any one of claims 1 to 9, characterized in that, include: When a business server acting as the master node fails, causing the Redis Sentinel instance to go offline, the Redis Sentinel cluster performs a master node election among the business servers to determine the business server that will become the new master node. The virtual IP management module of each business server dynamically adjusts the local priority based on whether the server is identified as the master node and the connectivity status of the local critical business service port, and binds the virtual IP address to the target business server with the highest priority. The middleware state synchronization proxy module of the target business server obtains the session information of the online client accessing through the virtual IP address from the middleware database, and restores the session with the online client on this server. The session information of the online client is synchronized to the middleware database by the business application on the business server bound to the virtual IP address before the business server bound to the virtual IP address is switched.

Images