A high availability service method and system based on dual-machine hot standby

Abstract

The present invention discloses a high-availability service method based on dual-machine hot standby, which includes a master server and a slave server as service nodes, and the applications on the two servers are constructed as cross-hardware multi-process applications based on the Microsoft Orleans framework, and the application communicates with the master server and the slave server through the VRRP protocol; one of the two servers runs in the host mode, and when triggered by the VRRP heartbeat failure, the host mode is switched from the current server to another server. The present invention solves the problem of insufficient continuous protection data in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of server backup, and in particular to a high-availability service method and system based on dual-machine hot standby. Background Art

[0002] As the use of information technology continues to expand, data has covered the entire human life and has become an important asset for enterprises and institutions. Due to the fragility of data itself, research on data protection has always been an important topic in the industry. In industries with particularly high data value, higher requirements are placed on data protection, which has led to the need for continuous data protection. However, there are still some problems with the existing technology of server continuous data protection:

[0003] (1) The existing technology of continuous data protection is limited to single-machine protection. Most of the current continuous protection is for single-machine protection. However, in actual production applications, a dual-machine environment is very common: two business machines are divided into a master and a slave, and a data disk is mounted on the master and slave machines using a dual-machine system. When the host fails, the master and slave machines will automatically switch, so that business processing is not interrupted and the data falls on the same data disk. The current single-machine protection is difficult to meet the needs of dual-machine business. Because in an environment with a dual-machine system, the real data disk is generally not allowed to install third-party continuous data protection products, and can only be installed on the master and slave business machines. If a third-party continuous data protection product is installed on the master and slave business machines, the master and slave machines are objectively protected as two disks in single-machine protection mode. When the master and slave machines are switched, the data will inevitably be lost.

[0004] (2) Dual-machine protection is limited to non-continuous protection. The data backup of dual-machine protection is generally based on periodic data replication. The backup granularity of periodic data replication is much larger than real-time level protection and block-level protection. Data between two backup cycles may be lost. Block-level protection can make the snapshot disk and the real disk at the corresponding moment of the snapshot reach block-level consistency, making the data more complete. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a high-availability service method and system based on dual-machine hot standby, which is used to solve the problem of insufficient continuous protection of data in the prior art.

[0006] In order to achieve the above object, the technical solution of the present invention is:

[0007] A high-availability service method based on dual-machine hot standby includes a master server and a slave server as service nodes. Applications on the two servers are built as cross-hardware multi-process applications based on the Microsoft Orleans framework. The application communicates with the master server and the slave server through the VRRP protocol. One of the two servers runs in host mode, and when triggered by a VRRP heartbeat failure, the host mode is switched from the current server to another server.

[0008] Furthermore, the two servers are both installed with the same PostgreSQL database, Redis cache, MinIO file server, gateway program, portainer container management tool, pgwatch database tool, and application program;

[0009] The portainer container management tool provides a docker management interface for the gateway program;

[0010] The application is run as a business program in the docker container according to the service request, and the gateway program coordinates the operation of the business program and provides reverse proxy services, so that the same application has only one running instance on the two service nodes;

[0011] The pgwatch database tool sets the PostgreSQL database to master-slave mode and starts the backup stream; the Redis cache runs in master-slave mode; and the MinIO file server runs in a cluster mode in mirror mode.

[0012] Furthermore, for the PostgreSQL database, the host mode switching in the master and slave servers is implemented as follows:

[0013] 1) Use the Keepalived tool to perform the same routing configuration on the master and slave servers:

[0014] In non-preemptive mode, both the master and slave servers are configured with the VRRP protocol's backup router, network interface, virtual router ID, priority, VRRP heartbeat packet check cycle, notification script, authentication, PostgreSQL database virtual IP, and heartbeat trigger failure conditions;

[0015] The notification script records the current status of the PostgreSQL database and the switching log of the current virtual IP. The application only uses the PostgreSQL database virtual IP for access, and the virtual IP only points to the PostgreSQL database of the server running in host mode.

[0016] 2) The pgwatch database tool processing process includes:

[0017] Step a: Check the connection status of the PostgreSQL database. If the connection is normal, proceed to step b; if the connection times out, proceed to step e.

[0018] Step b: Check the operation mode of the PostgreSQL database. If it is the host mode, proceed to step c; if it is the standby mode, proceed to step f;

[0019] Step c: Check the switch log of the current virtual IP through the notification script. If the switch log record value indicates that the current node state is the master, proceed to step d; otherwise, it is determined that the current PostgreSQL database is running in the master mode, but the master-slave relationship is broken, and the database is restored to the follower state;

[0020] Step d: Check the backup stream. If the backup stream status is normal, it is determined that the current PostgreSQL database is running normally in the host mode; if the backup stream status is interrupted, start troubleshooting and restore the backup stream;

[0021] Step e: Check whether the number of timeouts has reached the maximum value. If so, modify the record value in the notification script to indicate that the PostgreSQL database is not running normally, and then start troubleshooting and recovery;

[0022] Step f, check the switch log of the current virtual IP through the notification script. If the switch log record value indicates that the current node state is a standby machine, proceed to step g; if the switch log record value indicates that the current node state is a host, switch the PostgreSQL database of the current node to the host mode; if the switch log record value indicates that the PostgreSQL database is not running normally, start troubleshooting and recovery;

[0023] Step g: Check the backup stream status. If the backup stream is interrupted, start troubleshooting and restoring the backup stream. If the backup stream is normal, the PostgreSQL database of the current node runs in standby mode.

[0024] The present invention also provides a high-availability service system based on dual-machine hot standby, which is characterized by comprising:

[0025] A processor; a memory for storing instructions executable by the processor;

[0026] The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in the present invention.

[0027] Beneficial effects of the present invention:

[0028] 1. A comprehensive dual-machine hot standby solution for database (PostgreSQL), cache / message queue (Redis), file server (MinIO), gateway program and application program, suitable for servers of multiple business types.

[0029] 2. Under the coordination of the gateway program, all applications are in a single-instance running state, and there will be no running state conflicts. If it is detected that an application has no running instance, the startup command of the application will be automatically initiated. If the primary and backup machines start the same application at the same time, the built-in mutually exclusive startup mechanism can ensure that only one instance can be successfully started. This single-node application implementation method is suitable for small and medium-sized program application scenarios, and has lower requirements for hardware resources.

[0030] 3. Support scenarios of multiple continuous automatic switching. Except for the situation where human intervention is required to restore the status after the database switches from primary to standby, other scenarios can achieve unattended automatic recovery. During the operation, the keepalived service on the failed node will not be stopped. As long as the primary and standby machines do not fail at the same time, long-term stable operation can be maintained.

[0031] The solution of the present invention constructs a flexible and lightweight dual-machine hot standby implementation solution under the condition of only two servers. This solution can automatically switch various necessary services including databases, caches, file servers and applications without human intervention when one of the hosts fails, and can be used as an infrastructure to support common application deployment scenarios. Compared with the common high-availability solutions on the market, this solution only requires two-state servers to achieve the dual-machine hot standby function, and the threshold for hardware configuration is low. This solution is built based on open source and free tools, with low cost, high customizability and strong flexibility, and each component can be freely matched and functionally expanded according to one's specific needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a processing logic flow chart of the pgwatch database tool for the PostgreSQL database in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments disclosed in this application to clearly and completely describe the technical solutions in the embodiments disclosed in this application. Obviously, the described embodiments are only part of the embodiments disclosed in this application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the disclosure of this application and its application or use. Based on the embodiments disclosed in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection disclosed in this application.

[0034] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0035] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0036] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0037] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0038] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0040] The present invention is based on the VRRP protocol and the Microsoft 0rleans framework.

[0041] VRRP protocol: VRRP stands for Virtual Router Redundancy Protocol, which is a protocol for achieving high availability of routers. N routers that provide the same functions are grouped into a router group. There is a master and multiple backups in this group. The master has a VIP (Virtual IP Address) that provides services to the outside world. The master will send multicasts. When the backup cannot receive the VRRP packet, it is considered that the master is down. At this time, it is necessary to elect a backup as the master according to the priority of VRRP. In this way, the high availability of the router can be guaranteed. Keepalived achieves high availability through the VRRP protocol.

[0042] Microsoft Orleans: Orleans is a cross-platform framework for building reliable and scalable distributed applications. Distributed applications are defined as applications that span multiple processes, typically using peer-to-peer communication to transcend hardware boundaries. Orleans scales from a single on-premises server to hundreds or thousands of distributed, highly available applications in the cloud. Orleans extends familiar concepts and C# idioms to multi-server environments. Orleans is elastically scalable by design. When a host joins a cluster, it can accept new activations. When a host leaves the cluster due to scale-down or machine failure, previous activations on that host are reactivated on the remaining hosts as needed.

[0043] The present invention provides a high-availability service method based on dual-machine hot standby, which includes a master server and a slave server as service nodes. The applications on the two servers are built as cross-hardware multi-process applications based on the Microsoft Oracle framework. The application communicates with the master server and the slave server through the VRRP protocol. One of the two servers runs in the host mode, and when triggered by a VRRP heartbeat failure, the host mode is switched from the current server to another server.

[0044] The present invention only requires two servers, and implements a high-availability service that automatically switches to another server to continue working when one of the servers cannot continue working due to downtime or network failure, thereby ensuring the uninterrupted operation of important applications.

[0045] The two servers are both installed with the same PostgreSQL database, Redis cache, MinIO file server, gateway program, portainer container management tool, pgwatch database tool, and application program;

[0046] The portainer container management tool provides a docker management interface for the gateway program;

[0047] The application is run as a business program in the docker container according to the service request, and the gateway program coordinates the operation of the business program and provides reverse proxy services, so that the same application has only one running instance on the two service nodes;

[0048] The pgwatch database tool sets the PostgreSQL database to master-slave mode and starts the backup stream; the Redis cache runs in master-slave mode; and the MinIO file server runs in a cluster mode in mirror mode.

[0049] The following is a detailed description of the solution using a typical application scenario as an example.

[0050] Example 1: PostgreSQL database application scenario:

[0051] ①keepalived configuration

[0052]

[0053] The main logic of the above notification script (pgkpstate.sh) is to record the current status of the database and the switch log of the current virtual IP:

[0054]

[0055] The virtual IP state switching of keepalived is triggered by heartbeat failure, which is usually caused by network failure or server downtime. In addition, the database needs to perform regular health status checks and execute corresponding subsequent processing logic under certain conditions.

[0056] The application always uses the virtual IP to access the database service, and the virtual IP always points to the server node where the database runs in host mode. After a failure occurs, the database service can remain available as long as there is a node with the database running in host mode and the virtual IP correctly points to the node.

[0057] ②pgwatch supporting processing logic

[0058] The supporting processing logic of pgwatch is the periodic health check logic. Its core purpose is to upgrade the database on the server pointed to by the virtual IP from the original standby operation mode to the main operation mode after the keepalived switch occurs, so as to continue to support the normal operation of the business program (the database is read-only in the standby operation mode). The processing logic of a single cycle is as follows: Figure 1 As shown:

[0059] Step a: Check the connection status of the PostgreSQL database. If the connection is normal, proceed to step b; if the connection times out, proceed to step e.

[0060] Step b: Check the operation mode of the PostgreSQL database. If it is the host mode, proceed to step c; if it is the standby mode, proceed to step f;

[0061] Step c: Check the switch log of the current virtual IP through the notification script. If the switch log record value indicates that the current node state is the master, proceed to step d; otherwise, it is determined that the current PostgreSQL database is running in the master mode, but the master-slave relationship is broken, and the database is restored to the follower state;

[0062] Step d: Check the backup stream. If the backup stream status is normal, it is determined that the current PostgreSQL database is running normally in the host mode; if the backup stream status is interrupted, start troubleshooting and restore the backup stream;

[0063] Step e: Check whether the number of timeouts has reached the maximum value. If so, modify the record value in the notification script to indicate that the PostgreSQL database is not running normally, and then start troubleshooting and recovery;

[0064] Step f, check the switch log of the current virtual IP through the notification script. If the switch log record value indicates that the current node state is a standby machine, proceed to step g; if the switch log record value indicates that the current node state is a host, switch the PostgreSQL database of the current node to the host mode; if the switch log record value indicates that the PostgreSQL database is not running normally, start troubleshooting and recovery;

[0065] Step g: Check the backup stream status. If the backup stream is interrupted, start troubleshooting and restoring the backup stream. If the backup stream is normal, the PostgreSQL database of the current node runs in standby mode.

[0066] ③Description of manual processing after database switching

[0067] The initial state is that the primary and standby databases are running normally in the master mode and standby mode respectively, and a backup stream is established. The standby follows the master in real time through the backup stream. Under normal conditions, the data of the primary and standby databases are complete.

[0068] After the first switch occurs, the original standby machine becomes the master machine, and the data is complete. At this time, the steps for manual intervention to restore the master-standby state are to back up the database of the current master machine and restore it to the original master machine, and then run the original master machine database in standby mode following the current master machine.

[0069] If the state is not restored during the first switch, and at least one switch occurs afterwards, the databases of both the primary and backup machines are no longer complete. It is necessary to merge the data to the current host according to the business logic, and then restore the data to another host based on the current host. The other host can then run in backup mode following the current host.

[0070] Example 2: Application scenario of Redis cache and MinIO file server:

[0071] ①keepalived configuration

[0072]

[0073]

[0074] down_timer_adverts 100#The number of VRRP timeout check intervals. The timeout period is down_timer_adverts*advert_int. If the heartbeat packet verification fails but the heartbeat is restored in time within the timeout period, it is considered as not timed out and the notify is not triggered. This configuration avoids unnecessary switching caused by short jitter.

[0075]

[0076] Heartbeat failure can directly lead to virtual IP status switching. The reasons for heartbeat failure are generally network failure or server downtime. The failure of the master node minio or redis-server process will cause the current node priority to decrease, and then cause the current node to be downgraded from the master node to the slave node, and the original slave node to be promoted to the master node, which indirectly leads to virtual IP status switching.

[0077] The application always uses the virtual IP to access the cache and file server, and the virtual IP always points to the server node where the database runs in host mode. After a failover occurs, since Mini0 runs in a cluster in mirror mode, no subsequent processing is required. Redis runs in master-slave mode, and supporting operations need to be performed to make the redis service running on the node pointed to by the virtual IP the master node, and the redis service value of the other node the slave node and follow the master node (the redis of the slave node is read-only). After completing the supporting processing, as long as there is still a node whose cache and file server are running normally and the virtual IP correctly points to the node, the cache and file server can remain available.

[0078] ②PGWAt CH supporting processing logic

[0079] The redis master-slave relationship check and recovery logic needs to be executed regularly. The following is the core code for a single execution of this logic (for the sake of brevity, the exception handling logic is removed and only the core processing logic is retained):

[0080]

[0081]

[0082]

[0083] In short, it checks whether the current state of redis of the two nodes is consistent with the target state. If it is inconsistent, execute the command to convert it to the target state.

[0084] Unlike databases, the switching of cache and file servers can be completed automatically without manual processing, so multiple switches can be performed unattended without manual data restoration and status recovery.

[0085] Example 3: Gateway program application scenario: ① keepalived configuration

[0086]

[0087] The main logic of the above notification script (svcchange.sh) is to record the switch log of the current virtual IP:

[0088]

[0089] Heartbeat failure can directly lead to virtual IP status switching. The reasons for heartbeat failure are generally network failure or server downtime. The failure of the master node gateway process will cause the current node priority to decrease, and then cause the current node to be downgraded from the master node to the slave node, and the original slave node to be promoted to the master node, which indirectly leads to virtual IP status switching.

[0090] ②Gateway program core logic description

[0091] The core logic of the gateway program is to coordinate the operation of business programs and provide reverse proxy services. The corresponding core code segments are extracted below for illustration.

[0092] A. Coordinate the operation of business procedures

[0093] Use 0RLEANS to maintain the business program running information. The two gateway nodes share the same information and persist the information to the redis cache. The relevant logic is located in the program startup link of the gateway program. The code is as follows:

[0094]

[0095]

[0096]

[0097] When a business program accesses the gateway service, the following logic must be executed during startup to ensure that only one instance of the same business program on two nodes can be started, and that it maintains a heartbeat with the server as a client and continuously reports its own status. If the heartbeat fails, the current business program is stopped:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] The definition of the custom load balancing service is as follows, and its strategy is to forward requests to the currently active business program node:

[0107]

[0108]

[0109] Example of a reverse proxy configuration file:

[0110]

[0111] In the above example configuration, if the gateway program virtual IP is 10.1.2.200, the gateway program port is 5010, and the business program physical address is http: / / 10.1.3.165∶5700 / and

[0112] http: / / 10.1.2.135:5700 / , then in hot standby mode, you should

[0113] http: / / 10.1.2.200∶5010 / vm to access the business program.

[0114] The present invention also provides a high-availability service system based on dual-machine hot standby, which is characterized by comprising:

[0115] A processor; a memory for storing instructions executable by the processor;

[0116] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above method of the present invention.

[0117] The technical solutions provided by the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the embodiments of the present invention. The description of the above embodiments is only applicable to help understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, according to the embodiments of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A high-availability service method based on dual-machine hot standby, characterized in that: It includes a master server and a slave server as service nodes, wherein the applications on the two servers are built as cross-hardware multi-process applications based on the Microsoft Orleans framework, and the application communicates with the master server and the slave server through the VRRP protocol; One of the two servers operates in a host mode, and when triggered by a VRRP heartbeat failure, switches the host mode from the current server to the other server; The two servers are both installed with the same PostgreSQL database, Redis cache, MinIO file server, gateway program, portainer container management tool, pgwatch database tool, and application program; The portainer container management tool provides a docker management interface for the gateway program; The application is run as a business program in the docker container according to the service request, and the gateway program coordinates the operation of the business program and provides reverse proxy services, so that the same application has only one running instance on the two service nodes; The pgwatch database tool sets the PostgreSQL database to master-slave mode and starts the backup stream; the Redis cache runs in master-slave mode; the MinIO file server runs in a cluster mode in mirror mode; For the PostgreSQL database, the host mode switching in the master and slave servers is implemented as follows: 1) Use the Keepalived tool to perform the same routing configuration on the master and slave servers: In non-preemptive mode, both the master and slave servers are configured with the VRRP protocol's backup router, network interface, virtual router ID, priority, VRRP heartbeat packet check cycle, notification script, authentication, PostgreSQL database virtual IP, and heartbeat trigger failure conditions; The notification script records the current status of the PostgreSQL database and the switching log of the current virtual IP. The application only uses the PostgreSQL database virtual IP for access, and the virtual IP only points to the PostgreSQL database of the server running in host mode. 2) The pgwatch database tool processing process includes: Step a: Check the connection status of the PostgreSQL database. If the connection is normal, proceed to step b; if the connection times out, proceed to step e. Step b: Check the operation mode of the PostgreSQL database. If it is the host mode, proceed to step c; if it is the standby mode, proceed to step f; Step c, check the switching log of the current virtual IP through the notification script. If the switching log record value indicates that the current node state is the master, proceed to step d; otherwise, it is determined that the current PostgreSQL database is running in the master mode, but the master-slave relationship is broken, and the database is restored to the follower state; Step d: Check the backup stream. If the backup stream status is normal, it is determined that the current PostgreSQL database is running normally in the host mode; if the backup stream status is interrupted, start troubleshooting and restore the backup stream; Step e: Check whether the number of timeouts has reached the maximum value. If so, modify the record value in the notification script to indicate that the PostgreSQL database is not running normally, and then start troubleshooting and recovery; Step f, check the switch log of the current virtual IP through the notification script. If the switch log record value indicates that the current node state is a standby machine, proceed to step g; if the switch log record value indicates that the current node state is a host, switch the PostgreSQL database of the current node to the host mode; if the switch log record value indicates that the PostgreSQL database is not running normally, start troubleshooting and recovery; Step g: Check the backup stream status. If the backup stream is interrupted, start troubleshooting and restoring the backup stream. If the backup stream is normal, the PostgreSQL database of the current node runs in standby mode.

2. A high availability service method based on dual-machine hot standby according to claim 1, characterized in that: Specifies the notification delay for network interface status changes. The value of this notification delay is less than the countdown broadcast interval and check interval values ​​in all virtual network instance configurations.

3. A high availability service method based on dual-machine hot standby according to claim 2, characterized in that: For the Redis cache and MinIO file server, the host mode switching in the master and slave servers is implemented as follows: 1) Start the process monitoring of Redis cache and MinIO file server; 2) In non-preemptive mode, both the master and slave servers are configured with the BACKUP router, network interface, virtual router ID, priority, VRRP heartbeat packet check cycle, notification script, authentication, PostgreSQL database virtual IP, and heartbeat trigger failure conditions in the VRRP protocol; 3) When the Redis cache or MinIO file server process in host mode crashes, the current node is downgraded from host mode to standby mode, and the other node is upgraded from standby mode to host mode, and the virtual IP status is switched; 4) Regularly perform Redis cache master-slave relationship check and recovery: Step a1: Obtain the target state and actual state of the master-slave relationship of the Redis cache of the current node. If both the target state and the actual state are the master node, but there is no slave node, make the Redis cache of another node the slave node of the Redis cache of the current node; Otherwise, go to step b1; Step b1, if the target state is the master node, and the actual state is not the master node, then the Redis cache of the current node is made the master node, and the Redis cache of another node is made the slave node of the Redis cache of the current node; Otherwise, go to step c1; Step c1: If the target state is a slave node, and the actual state of the other node is a master node, but there is no connected slave node, then make the Redis cache of the current node a slave node of the other node; Otherwise, go to step d1; Step d1, if the target state is a slave node and the actual state of the other node is not a master node, make the Redis cache of the other node a master node and make the Redis cache of the current node a slave node of the other node; otherwise, go to step d1.

4. A high-availability service method based on dual-machine hot standby according to claim 3, characterized in that: For the gateway program, the host mode switching in the master and slave servers is implemented as follows: 1) Start the process monitoring of the gateway; 2) In non-preemptive mode, both the master and slave servers are configured with the BACKUP router, network interface, virtual router ID, priority, VRRP heartbeat packet check cycle, notification script, authentication, PostgreSQL database virtual IP, and heartbeat trigger failure conditions in the VRRP protocol; 3) The two gateway nodes share the same business program running information, which is persisted in the redis cache. The gateway program coordinates the business program including: a2. When a business program accesses the gateway service, check the number of instances started by the same business program to ensure that only one instance is running. The gateway program maintains a heartbeat with the server as a client and continuously reports its own status. If the heartbeat fails, the current business program is stopped. b2. The gateway program of the master node periodically checks the running status of each business program. If it is not running, it sends a start command to start the business program; c2. When the gateway program provides reverse proxy service, it performs load balancing scheduling: forwarding the request to the currently active business program node.

5. A high-availability service system based on dual-machine hot standby, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-4.

Images