Dual-availability zone disaster recovery method, device, electronic device, and storage medium

Abstract

The present invention provides a dual-availability zone disaster recovery method, apparatus, electronic device, and storage medium, relating to the field of database technology. The method includes: upon determining that the dual-availability zone connection is normal, a leader node elected from all nodes in the dual-availability zone receives a request and synchronizes logs generated in response to the request to the dual-availability zone. Compared to existing technologies, the present invention provides disaster recovery capabilities with a recovery point objective of zero.

Description

Technical Field

[0001] The present application relates to the field of database technology, and more specifically, to a dual-availability zone disaster recovery method, device, electronic device, and storage medium. Background Art

[0002] Natively distributed relational databases (as opposed to database sharding middleware solutions like MySQL or Postgres SQL) generally use distributed consensus algorithms like Raft and Paxos to implement multi-replica disaster recovery and rely on these consensus algorithms to achieve high availability with automatic failover capabilities. For example, Raft is a voting system that follows a majority agreement. In a Raft group, a member becomes a leader when it receives a majority vote. In other words, when a Raft group maintains a majority, it can elect a leader to provide external services.

[0003] In a dual-zone Raft deployment, the total number of members in the primary and backup zones must be an odd number, and the number of members in the primary zone must exceed the number of members in the backup zone to meet the "majority" requirement. However, in existing technologies, if the primary zone fails, the backup zone cannot meet data consistency requirements. Summary of the Invention

[0004] Embodiments of the present invention provide a disaster recovery method, device, electronic device, and storage medium for a dual-availability-zone distributed system that overcomes the above-mentioned problems or at least partially solves the above-mentioned problems.

[0005] In a first aspect, a dual-availability zone disaster recovery method is provided, the method comprising:

[0006] If it is determined that the dual-availability zone connection is normal, the leader node elected based on all nodes in the dual-availability zone receives the request and synchronizes the logs generated based on the request to the dual-availability zone.

[0007] In one possible implementation, the disaster recovery method further includes:

[0008] If a fault is detected in the backup zone, the system checks whether the backup zone has recovered within a preset time period and stops accepting new requests within the preset time period.

[0009] If the backup availability zone still fails within the preset period, the leader node elected based on all nodes in the primary availability zone will receive the new request and synchronize the logs generated based on the new request to the primary availability zone.

[0010] In one possible implementation, the disaster recovery method further includes:

[0011] If the standby availability zone returns to normal within the preset time, the leader node elected by all nodes in both availability zones will receive the new request and synchronize the logs generated based on the new request to both availability zones.

[0012] In one possible implementation, the disaster recovery method further includes:

[0013] If the backup zone is determined to be back to normal after the preset period of time, the primary zone will synchronize the logs that have not been synchronized to the backup zone to the backup zone.

[0014] In one possible implementation, logs that have not been synchronized to the standby availability zone are synchronized to the standby availability zone, and then the following steps are included:

[0015] If log synchronization between the primary and backup zones is complete, the leader node elected from all nodes in both zones receives the request and synchronizes the logs generated based on the request to both zones.

[0016] In one possible implementation, the disaster recovery method further includes:

[0017] If the primary availability zone in the dual availability zones is faulty, a switch is performed between the primary and backup availability zones.

[0018] The leader node elected by all nodes in the new primary availability zone generated after the switch receives the request and synchronizes the logs generated based on the request to the new primary availability zone generated after the switch.

[0019] In one possible implementation, switching between the primary availability zone and the backup availability zone may further include:

[0020] If it is determined that the new standby availability zone generated after the switch has returned to normal, the new primary availability zone generated after the switch will synchronize the logs that have not been synchronized to the new standby availability zone to the new standby availability zone; and

[0021] If it is determined that the logs of the new primary availability zone and the new backup availability zone generated after the switch are synchronized, the leader node elected by all nodes in the dual availability zones after the switch receives the request and synchronizes the logs generated according to the request to the dual availability zones after the switch.

[0022] In one possible implementation, synchronizing logs generated based on requests to two availability zones includes:

[0023] Generate logs based on received requests;

[0024] Send logs to each follower node in both availability zones, instructing the follower nodes to replicate the logs and then provide response information.

[0025] If it is determined that the number of received response messages meets the preset conditions and there is response information fed back by at least one follower node in a different availability zone, the request corresponding to the log is executed.

[0026] In one possible implementation, the disaster recovery method further includes:

[0027] Set up at most one submission group for each node in dual availability zones, and nodes in the same submission group must be in the same availability zone.

[0028] The response information also includes the information of the submission group to which the following node belongs;

[0029] The response information sent by at least one follower node in a different availability zone is determined to include:

[0030] If it is determined that the submission group information included in the received response information is different from the submission group information of the leader node, it is determined that there is response information sent by at least one follower node in a different availability zone.

[0031] In a second aspect, a disaster recovery device for a dual-availability zone distributed system is provided, comprising:

[0032] The dual-availability zone synchronization module is used to, if it is determined that the dual-availability zone connection is normal, have the leader node elected based on all nodes in the dual-availability zone receive the request and synchronize the logs generated based on the request to the dual-availability zone.

[0033] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method provided in the first aspect are implemented.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method provided in the first aspect when executed by a processor.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device performs the steps of implementing the method provided in the first aspect.

[0036] The dual-availability zone disaster recovery method, device, electronic device and storage medium provided by the embodiments of the present invention determine that the dual-availability zone connection is normal, and the leader node elected based on all nodes in the dual-availability zone receives the request and synchronizes the log generated based on the request to the dual-availability zone. This is different from the existing technology in which the nodes in the primary availability zone have met most conditions and the logs are not synchronized to the backup availability zone. Therefore, when the primary availability zone fails, the embodiments of the present application can provide a disaster recovery capability with a recovery point objective (RPO) of 0 compared to the existing technology because the backup availability zone and the primary availability zone are data synchronized. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0038] Figure 1 A schematic diagram of an exemplary system architecture provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a process flow for a disaster recovery method for a dual-availability zone distributed system provided in an embodiment of the present application;

[0040] Figure 3 This is an illustration of switching between different working modes in a disaster recovery method provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a process for synchronizing logs to dual availability zones provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a log transmission path provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of another log transmission path provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the structure of a disaster recovery device for a dual-availability zone distributed system provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present invention.

[0047] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any one of the units and all combinations of one or more associated listed items.

[0048] The solution of the present application is applicable to any distributed system, for example, the distributed system may be a distributed storage system, a distributed computing system, etc.

[0049] For example, a distributed system might need to periodically complete settlement reconciliation. Each time a settlement reconciliation task occurs, the distributed system must identify a primary server from among multiple servers capable of performing settlement reconciliation and complete the settlement reconciliation task through that primary server. Another example is when a distributed system needs to deliver messages to users, a primary server must be identified to execute the message delivery task.

[0050] Among the many distributed consensus algorithms, the Raft algorithm is used as an example. The Raft algorithm itself does not limit the number of members (i.e., nodes) in a Raft group. This number of members can be any positive integer. When the number of members is n, the reliability of a Raft group is as follows:

[0051] If n is an odd number, the Raft group can tolerate (n-1) / 2 members failing simultaneously;

[0052] If n is an even number, the Raft group can tolerate (n / 2)-1 members failing simultaneously;

[0053] In general, the number of members in a Raft group is set to an odd number for the following reasons:

[0054] Avoid wasting storage space: Three members can tolerate the failure of one member (i.e., n=3). After adding one member to become four members (i.e., n=4), only one member failure can be tolerated, and the disaster recovery capability remains unchanged.

[0055] Raft groups have stronger reliability: when the number of members is even, if a network isolation occurs and the number of members on both sides of the isolation is n / 2, since neither side has a majority of members, it is impossible to elect a member as a leader to provide services. This network isolation will directly lead to the unavailability of the entire service.

[0056] When the number of members is odd, if a network isolation occurs, one of the two sides of the network isolation will always have the majority of members, so the side with the majority of members can elect a member as the leader to continue providing services.

[0057] Based on the reliability characteristics of the Raft group described above, it can be seen that the native Raft protocol does not support even-numbered availability zones (AZs). 3 availability zones may be the most suitable high availability and disaster recovery solution for deploying Raft groups.

[0058] In reality, however, most deployment environments rarely accommodate three availability zones within a specific region, such as within a single city. For example, in the highly digitized banking industry, due to the point-to-point replication characteristics of traditional standalone systems, most banks have only built infrastructure with two availability zones within a single city or three availability zones across two locations.

[0059] Deploy native Raft in dual availability zones within the same city. Three members are allocated to the primary and backup zones in a 2:1 ratio, meaning two members are configured in the primary zone and one in the backup zone. Due to network latency differences, asynchronously replicated members are present in the backup zone. If the primary zone fails and cannot be recovered, the single remaining member in the backup zone cannot guarantee CAP (Consistency, Availability, and Partition Tolerance).

[0060] The disaster recovery method, device, electronic device, and computer-readable storage medium for a dual-availability-zone distributed system provided in this application are intended to solve the above-mentioned technical problems in the prior art.

[0061] The embodiment of the present application provides a new disaster recovery method based on the Raft algorithm, which can, to a certain extent, overcome the problem that it is difficult to restore consistent data using the backup availability zone after a failure in the primary availability zone.

[0062] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0063] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0064] Figure 1 Schematic diagram showing an exemplary system architecture to which the embodiments of the present application can be applied. Figure 1 As shown, the system architecture 100 may include network devices (such as Figure 1 1 , a switch 101, a router 102, and a gateway device 103 are shown, and other network devices may also be used. A network 104 and a multi-node system 110 are shown. Multi-node system 110 includes a first server 111, a second server 112, and a third server 113. Each server in multi-node system 110 is a node in multi-node system 110. Network 104 is a medium for providing communication links between terminal devices and task processing nodes in multi-node system 110. Network 104 can include various connection types, such as wired communication links, wireless communication links, and the like.

[0065] It should be understood that Figure 1 The number of terminal devices, networks, and nodes in the multi-node system is merely illustrative. Depending on implementation requirements, any number of terminal devices and networks can be used, and any number of task processing nodes can be included in the multi-node system. Each task processing node can even be composed of a cluster. For example, the first server 111 can be a server cluster consisting of multiple servers.

[0066] See Figure 2 , which exemplarily shows a flow chart of a disaster recovery method for a dual-availability zone distributed system according to an embodiment of the present application, as shown in the figure, including:

[0067] S101. Determine whether the dual-zone connection is normal.

[0068] Since the embodiment of the present application is implemented based on the Raft algorithm, it should be understood that, in the initial state, the primary availability zone includes a majority of nodes (for example, 2n nodes, where n is a positive integer), and the backup availability zone includes a minority of nodes (for example, 2n-1 nodes, where n is a positive integer).

[0069] In the Raft protocol, a node is in one of the following three states at any time:

[0070] leader node

[0071] follower node

[0072] candidate node candidate

[0073] In the initial state, all nodes start in the follower state. If any follower node does not receive a heartbeat message from the leader within a period of time, all followers switch to the candidate state and initiate an election to select a leader by voting. If a candidate receives a majority vote (including its own vote), it switches to the leader state. If a node finds that other nodes are updated more promptly than its own state, for example, the data logs on one or more other nodes are updated later than the data log on its own node, the node will actively withdraw from the leader competition and switch from candidate to follower after the vote or when the election-voting process is interrupted.

[0074] The system has at most one leader. If any follower discovers that there is no leader for a period of time (for example, if the follower fails to receive heartbeat messages from the leader), all followers become candidates and elect a new leader through an election-vote process. If the election-vote process discovers that the system already has a leader, the election-vote process is interrupted, and all candidates are reset to followers. The leader continuously sends heartbeat messages to followers to indicate its survival status. If the leader fails, the follower becomes a candidate and initiates an election-vote process to re-elect a leader.

[0075] Therefore, the embodiment of the present application determines whether the dual-availability zone connection is normal by determining whether the nodes belonging to the two availability zones can normally send heartbeat packets and receive heartbeat responses.

[0076] Taking the leader node located in the primary availability zone as an example, if any node in the primary availability zone can receive a heartbeat response returned by any node in the backup availability zone, then it can be determined that the dual availability zone connection is normal. If any node in the primary availability zone cannot receive a heartbeat response returned by all nodes in the backup availability zone, then it is determined that the connection between the primary and backup availability zones is abnormal.

[0077] S102: If it is determined that the dual availability zones are connected normally, the leader node selected from all nodes in the dual availability zones receives the request and synchronizes the logs generated according to the request to the dual availability zones.

[0078] It should be understood that when the dual availability zones are connected normally, the leader node is determined by election by all nodes in the dual availability zones. The election process may include the following steps:

[0079] Each node casts its own vote and sends RequestVote RPCs to other nodes in parallel;

[0080] Wait for responses from other nodes;

[0081] During this process, depending on the messages from other nodes, two possible outcomes may occur:

[0082] Result 1: If a node receives a majority vote (including its own vote), it wins the election and becomes the leader.

[0083] Result 2: A node is informed that someone else has been elected, and then switches to follower automatically.

[0084] If a node does not receive a majority vote within a period of time, it remains in the candidate state and re-issues the election.

[0085] For the first result, after winning the election, the new leader will immediately send a message to all other nodes to inform them that the leader has been elected, so as to prevent the other nodes from triggering a new election.

[0086] For the second outcome, consider a system consisting of three nodes, A, B, and C. Nodes A and B initiate an election simultaneously, but the election message from node A reaches node C first, and node C immediately casts a vote for node A. Later, when the message from node B reaches node C, node C no longer votes for node B because node C has already voted for node A. Since neither node A nor B will vote for the other, node A wins the vote. After winning, node A sends heartbeat messages to nodes B and C. Node B discovers that node A's term is no less than its own term, and knowing that a leader already exists in the system, node B transitions to a follower.

[0087] As can be seen from the above embodiments, the leader node can be a node in the primary availability zone or a node in the backup availability zone. In the embodiment of the present application, the leader node receives the request, generates a log according to the request, and synchronizes the log to the dual availability zones. This is different from the prior art in which the node in the primary availability zone has met most conditions and the log is not synchronized to the backup availability zone. Therefore, when the primary availability zone fails, the embodiment of the present application can provide a disaster recovery capability with a recovery point objective (RPO) of 0 compared to the prior art because the backup availability zone is data synchronized with the primary availability zone.

[0088] Based on the above embodiments, as an optional embodiment, after step S101, the following steps are further included:

[0089] S102′: If it is determined that the backup zone in the dual availability zones has failed, determine whether the backup zone has returned to normal within a preset time period, and stop accepting new requests within the preset time period.

[0090] It should be understood that if a failure occurs in the backup zone in the dual availability zone, resulting in a communication failure between the primary availability zone and the backup availability zone, then the embodiment of the present application adopts a blocking window method to stop receiving information requests. The blocking window is used to block the submission of new requests within a preset time period when a dual availability zone communication failure is sent, to ensure that the dual availability zones continue to maintain data synchronization within the preset time period.

[0091] For example, if the connection between the two availability zones is normal at 10:00 AM on a certain day, the logs in the primary and backup availability zones are synchronized. That is, the logs stored in the primary availability zone are also stored in the backup availability zone. If the communication between the two availability zones fails at 10:01 AM, and the preset duration is 2 hours, then the two availability zones will no longer accept new requests within two hours, keeping the logs in both availability zones consistent.

[0092] S103. If the backup availability zone still fails within the preset time, the leader node selected from all nodes in the primary availability zone receives the new request and synchronizes the log generated according to the new request to the primary availability zone.

[0093] If the backup zone still fails after the preset duration, for example, due to communication interruption between the backup and primary zones, the nodes in the primary and backup zones will each conduct elections according to the Raft algorithm. However, because the nodes in the backup zone do not meet the majority criteria, the backup zone will not be able to elect a leader node, while the primary zone can. After the preset duration, the leader node elected by all nodes in the primary zone will receive the request and synchronize the logs generated by the request to all nodes in the primary zone.

[0094] The following describes the disaster recovery method according to an embodiment of the present application with reference to a specific example.

[0095] In the dual availability zone of the present embodiment, the primary availability zone includes nodes 1 through 4, for a total of four nodes, and the backup availability zone includes nodes 4 through 7, for a total of three nodes. Initially, the dual availability zones are connected normally, so nodes 1 through 7 elect node 2 as the leader node. Node 2 receives the request and synchronizes the logs generated based on the request to nodes 2 through 7 in the dual availability zone.

[0096] If communication between the primary availability zone and the backup availability zone is interrupted during the operation of the system, node 2 will no longer receive new requests within the preset time period. If communication returns to normal within the preset time period, node 2 will continue to receive new requests after the restoration and synchronize them to all nodes. The embodiment of the present application does not re-elect the leader node during the preset time period, which lays the foundation for log synchronization immediately after communication is normal within the preset time period and saves the computing power consumption of each node.

[0097] If the backup availability zone still fails after the preset period of time, for example, communication between the primary and backup availability zones is interrupted, all nodes in the primary availability zone, that is, nodes 1 to 4, will re-elect to obtain a new leader node, assuming it is node 1. Node 1 will continue to receive new requests and generate logs, and synchronize the logs to nodes 2 to 4.

[0098] Based on the above embodiments, as an optional embodiment, it is determined whether the backup zone has returned to normal within a preset time period, and then the following steps are further included:

[0099] If the standby availability zone returns to normal within the preset time, the leader node elected by all nodes in both availability zones will receive the new request and synchronize the logs generated based on the new request to both availability zones.

[0100] Based on the above embodiments, as an optional embodiment, if it is determined that the backup zone has returned to normal after a preset period of time, the primary zone will synchronize logs that have not been synchronized to the backup zone to the backup zone.

[0101] It should be noted that since the primary and backup zones begin receiving new requests after a preset period of time, each node in the primary and backup zones will store some logs that have not been synchronized with the backup zone before communication is restored. In this embodiment of the application, if the backup zone is determined to be back to normal after a preset period of time, the primary zone will synchronize the logs that have not been synchronized to the backup zone to the backup zone, gradually restoring log synchronization between the two zones.

[0102] Specifically, if the time when communication between the primary and backup availability zones is interrupted is defined as T1, the time after the preset period is T2, and the time when communication between the primary and backup availability zones is restored is T3 (T3 is later than T2), then the primary availability zone receives requests in the time period from T2 to T3, and synchronizes the logs generated according to the requests to each node in the primary availability zone.

[0103] The embodiment of the present application may continue to receive new requests after time T3, or may wait for data synchronization between the primary and backup available zones before receiving new requests. The embodiment of the present application does not make any specific limitation.

[0104] As an optional embodiment, after time T3, the leading node continues to receive requests and synchronizes the logs generated according to the requests to each node in the dual availability zones, and on the other hand synchronizes the logs generated in the time period from T2 to T3 to each node in the backup availability zone. The speed of synchronizing the logs generated in the time period from T2 to T3 to each node in the backup availability zone can be higher than the speed of generating logs for the received requests after time T3, so that the logs stored in the nodes in the backup availability zone can gradually catch up with the logs stored in the nodes in the primary availability zone.

[0105] Based on the above embodiments, as an optional embodiment, if it is determined that the log synchronization of the primary availability zone and the backup availability zone is completed, the leader node elected based on all nodes in the dual availability zones receives the request and synchronizes the logs generated according to the request to the dual availability zones.

[0106] That is to say, after confirming the log synchronization of the primary and backup availability zones, the embodiment of the present application can re-elect the leader node by all nodes in all dual availability zones, and the re-elected leader node receives the request.

[0107] Based on the above embodiments, as an optional embodiment, the disaster recovery method of the embodiment of the present application further includes:

[0108] If the primary availability zone in the dual availability zones is faulty, a switch is performed between the primary and backup availability zones.

[0109] The leader node elected by all nodes in the new primary availability zone generated after the switch receives the request and synchronizes the logs generated based on the request to the new primary availability zone generated after the switch.

[0110] That is, if the primary availability zone fails, a failover is performed, with the failed primary availability zone becoming the backup availability zone and the original backup availability zone becoming the new primary availability zone. All nodes in the new primary availability zone elect a leader node, which receives requests and synchronizes the logs generated by the requests to all nodes in the new availability zone.

[0111] Based on the above embodiments, as an optional embodiment, switching between the primary availability zone and the backup availability zone is performed, and then further includes:

[0112] If it is determined that the new standby availability zone generated after the switch has returned to normal, the new primary availability zone generated after the switch will synchronize the logs that have not been synchronized to the new standby availability zone to the new standby availability zone; and

[0113] If it is determined that the logs of the new primary availability zone and the new backup availability zone generated after the switch are synchronized, the leader node elected by all nodes in the dual availability zones after the switch receives the request and synchronizes the logs generated according to the request to the dual availability zones after the switch.

[0114] See Figure 3 , which exemplarily shows a schematic diagram of switching between different working modes in the disaster recovery method of an embodiment of the present application, as shown in the figure, including:

[0115] First, after initializing the distributed system, it enters dual-availability zone synchronization mode. In this mode, the dual-availability zones are connected normally. The leader node elected by all nodes in the dual-availability zones receives requests and synchronizes the logs generated by the requests to both zones.

[0116] If a disaster recovery zone (i.e., a backup zone) fails, the system waits for the blocking window to time out. Before the blocking window times out, the system stops accepting new requests and continuously determines whether the backup zone has returned to normal. If the failure is recovered, the system re-enters the dual-zone synchronization mode. If the backup zone still has not recovered after the blocking window times out, the system enters the majority member mode. In the majority member mode of the embodiment of the present application, all nodes in the primary zone elect a leader node, which receives requests and synchronizes the logs generated based on the requests to the dual zones.

[0117] If the disaster recovery zone fails after entering majority member mode, the system will enter recovery synchronization mode. In this embodiment of the present application, in recovery synchronization mode, the primary zone will synchronize logs that have not been synchronized to the backup zone to the backup zone. After log synchronization is completed, the present embodiment of the present application will enter dual zone synchronization mode. It is understood that if a disaster recovery zone failure occurs during rewriting in recovery synchronization mode, the system will be updated from recovery synchronization mode to majority member mode.

[0118] If a primary availability zone fails when the embodiment of the present application is in dual availability zone synchronization mode, then the embodiment of the present application will be updated from the dual availability zone synchronization mode to the disaster recovery mode. In the disaster recovery mode, switching is performed between the primary availability zone and the backup availability zone. The leader node elected by all nodes in the new primary availability zone generated after the switch receives the request and synchronizes the log generated according to the request to the new primary availability zone generated after the switch.

[0119] When the backup availability zone fails, the automatic failover capability of the native distributed consensus algorithm is maintained; after the disaster and backup availability zone failures are recovered, no manual operation is required and it can automatically recover to the dual availability zone synchronous replication mode with extremely high disaster tolerance capabilities.

[0120] After the failure of the primary availability zone before the switch is recovered, the embodiment of the present application enters the recovery synchronization mode. It can be seen from the above embodiment that when the log synchronization of the primary and backup availability zones is completed, the dual availability zone synchronization mode is entered again.

[0121] Based on the above embodiments, as an optional embodiment, see Figure 4 , synchronize the logs generated based on the request to both availability zones, including:

[0122] S201. Generate a log according to the received request;

[0123] S202: Send the log to each follower node in the dual availability zones, and instruct the follower node to replicate the log and then feedback response information;

[0124] S203: If it is determined that the number of received response information meets the preset condition and there is response information sent by at least one follower node in a different availability zone, execute the request corresponding to the log.

[0125] In the embodiment of the present application, the request corresponding to the execution log needs to satisfy the preset condition that the number of response information received meets the response information returned by the majority of nodes. More importantly, there is response information fed back by at least one follower node in a different availability zone (a different availability zone from the leader node). Due to the limitation of this condition, the request corresponding to the log can only be executed after the log is synchronized to two availability zones, thereby ensuring data security.

[0126] Based on the above embodiments, as an optional embodiment, the embodiment of the present application can set up at most one submission group for each node in dual availability zones, and the nodes in the same submission group are located in the same availability zone.

[0127] See Figure 5 , which exemplarily illustrates a log transmission path of an embodiment of the present application. As shown in the figure, the primary availability zone AZ0 includes four nodes, namely node 0, node 1, node 2, and node 5, and the backup availability zone AZ1 includes three nodes, namely node 3, node 4, and node 6. In the embodiment of the present application, all nodes in AZ0 (node 0, node 1, node 2, and node 5) are set as nodes in the submission group Group0, and all nodes in AZ1 (node 3, node 4, and node 6) are set as nodes in the submission group Group1.

[0128] like Figure 5 As shown in the figure, after node 0 generates a log, it is sent to nodes 1, 2, and 5 in sequence to form a majority. However, once the entire availability zone AZ 0 is damaged, the latest changes will be lost.

[0129] The commit group in this embodiment organizes nodes into different groups according to certain rules. Furthermore, it stipulates that modifications must be committed by a majority of members and replicated to at least two groups before they can be committed. In this case, the aforementioned nodes 0, 1, 2, and 5 do not meet these requirements. While this combination can meet the majority of member commitments, it only includes one availability zone, AZ0.

[0130] See Figure 6 , which is a schematic diagram of another log transmission path of an embodiment of the present application. As shown in the figure, node 0 sends the log to 1, 2 and 6. Nodes 0, 1, 2, and 6 form a member combination, which contains 4 nodes, meeting the requirements of most nodes being implemented, and these 4 members are distributed in two different availability zones AZ0 and AZ1, so the request corresponding to the log can be executed.

[0131] The embodiment of the present application provides a disaster recovery device for a distributed system with dual availability zones, such as Figure 7 As shown, the device may include: a dual-availability zone synchronization module, specifically:

[0132] The dual-availability zone synchronization module 101 is configured to receive the request from a leader node selected from all nodes in the dual-availability zones if it is determined that the dual-availability zone connection is normal, and synchronize the logs generated according to the request to the dual-availability zones.

[0133] The embodiment of the present invention provides a disaster recovery device for a distributed system with dual availability zones, which specifically executes the above-mentioned method embodiment process. For details, please refer to the content of the above-mentioned disaster recovery method embodiment for a distributed system with dual availability zones, which will not be repeated here. The disaster recovery device for a distributed system with dual availability zones provided by the embodiment of the present invention determines that the connection between the dual availability zones is normal, and the leader node elected based on all nodes in the dual availability zones receives the request and synchronizes the log generated based on the request to the dual availability zones. This is different from the prior art in which the nodes in the primary availability zone have met most of the conditions, but the logs are not synchronized to the backup availability zone. Therefore, when the embodiment of the present application encounters a failure in the primary availability zone, since the backup availability zone and the primary availability zone are data synchronized, it can provide a disaster recovery capability with a recovery point target of 0 compared to the prior art.

[0134] Based on the above embodiments, as an optional embodiment, the disaster recovery device further includes:

[0135] a request pausing module, configured to, if it is determined that the backup zone in the dual availability zones has failed, determine whether the backup zone has returned to normal within a preset time period, and stop receiving new requests within the preset time period;

[0136] The main synchronization module is used to receive a new request from a leader node elected by all nodes in the main availability zone if the backup availability zone still fails after the preset time period, and synchronize the log generated according to the new request to the main availability zone.

[0137] Based on the above embodiments, as an optional embodiment, the disaster recovery device includes:

[0138] The full synchronization module is used to receive the new request from the leader node elected by all nodes in the dual availability zones if the standby availability zone returns to normal within the preset time period, and synchronize the log generated according to the new request to the dual availability zones.

[0139] Based on the above embodiments, as an optional embodiment, the disaster recovery device further includes:

[0140] The primary-backup synchronization module is configured to synchronize the logs that have not been synchronized to the backup zone to the backup zone if it is determined that the backup zone has returned to normal after the preset time period.

[0141] Based on the above embodiments, as an optional embodiment, the disaster recovery device further includes:

[0142] If it is determined that the log synchronization of the primary availability zone and the backup availability zone is completed, the leader node elected based on all nodes in the dual availability zones receives the request and synchronizes the logs generated based on the request to the dual availability zones.

[0143] Based on the above embodiments, as an optional embodiment, the disaster recovery device further includes:

[0144] A primary-backup switching module, configured to switch between the primary and backup availability zones if a failure in the primary availability zone of the dual availability zones is determined;

[0145] The new primary availability zone synchronization module is used to receive requests from the leader node elected by all nodes in the new primary availability zone generated after the switch, and synchronize the logs generated according to the requests to the new primary availability zone generated after the switch.

[0146] Based on the above embodiments, as an optional embodiment, the disaster recovery device further includes:

[0147] a resynchronization module configured to synchronize logs that have not been synchronized to the new standby available zone generated after the switch to the new primary available zone generated after the switch if it is determined that the new standby available zone has returned to normal; and

[0148] The complete synchronization module is used to receive the request from the leader node elected by all nodes in the dual availability zones after the switch if it is determined that the log synchronization of the new primary availability zone and the new backup availability zone generated after the switch is completed, and synchronize the logs generated according to the request to the dual availability zones after the switch.

[0149] Based on the above embodiments, as an optional embodiment, the dual-availability zone synchronization module includes:

[0150] A log generating unit, configured to generate the log according to the received request;

[0151] a log sending unit, configured to send the log to each follower node in the dual availability zones, and instruct the follower node to replicate the log and then feed back response information;

[0152] The execution unit is configured to execute the request corresponding to the log if it is determined that the number of the received response information meets a preset condition and there is response information fed back by at least one follower node in a different availability zone.

[0153] Based on the above embodiments, as an optional embodiment, the disaster recovery device further includes:

[0154] a submission group determination module, configured to set at most one submission group for each node in the dual availability zones, and the nodes in the same submission group are located in the same availability zone;

[0155] The execution unit is specifically used to: if it is determined that the information of the submission group included in the received response information is different from the information of the submission group of the leader node, then determine that there is the response information sent by at least one of the follower nodes in the different availability zone.

[0156] An embodiment of the present application provides an electronic device, which includes: a memory and a processor; at least one program, stored in the memory, for being executed by the processor. Compared with the prior art, it can achieve: by determining that the connection between the dual availability zones is normal, the leader node elected based on all nodes in the dual availability zones receives the request, and synchronizes the log generated according to the request to the dual availability zones. This is different from the prior art in which the nodes in the primary availability zone have met most conditions and the logs are not synchronized to the backup availability zone. Therefore, when encountering a failure in the primary availability zone, the embodiment of the present application can provide a disaster recovery capability with a recovery point objective (RPO) of 0 compared with the prior art because the backup availability zone and the primary availability zone are data synchronized.

[0157] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0158] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0159] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0160] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0161] The memory 4003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0162] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when run on a computer, enables the computer to execute the corresponding contents of the aforementioned method embodiment. Compared with the prior art, by determining that the connection between the dual availability zones is normal, the leader node elected based on all the nodes in the dual availability zones receives the request, and synchronizes the log generated based on the request to the dual availability zones. This is different from the prior art in which the nodes in the primary availability zone have met most of the conditions, but the logs are not synchronized to the backup availability zone. Therefore, when encountering a failure in the primary availability zone, the embodiment of the present application can provide a disaster recovery capability with a recovery point target of 0 compared to the prior art, because the backup availability zone is data synchronized with the primary availability zone.

[0163] An embodiment of the present application provides a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When the processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device executes the content shown in the aforementioned method embodiment. Compared with the prior art, by determining that the connection between the dual availability zones is normal, the leader node elected based on all the nodes in the dual availability zones receives the request, and synchronizes the log generated based on the request to the dual availability zones. This is different from the prior art in which the nodes in the primary availability zone have met most of the conditions, but the logs are not synchronized to the backup availability zone. Therefore, when encountering a failure in the primary availability zone, the embodiment of the present application can provide a disaster recovery capability with a recovery point target of 0 compared to the prior art because the backup availability zone is data synchronized with the primary availability zone.

[0164] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0165] The above are only some of the embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A dual-availability zone disaster recovery method, characterized in that: The dual availability zones include a primary availability zone and a backup availability zone. In an initial state, the primary availability zone includes a majority of nodes, and the backup availability zone includes a minority of nodes. The dual-availability zone disaster recovery method includes: If it is determined that the dual availability zones are connected normally, a leader node selected from all nodes in the dual availability zones receives the request and synchronizes logs generated according to the request to the dual availability zones; Synchronizing the logs generated according to the request to the dual availability zones includes: generating the log according to the received request; Sending the log to each follower node in the dual availability zones, instructing the follower node to replicate the log and then feed back response information; If it is determined that the response information returned by the majority of follower nodes in the dual availability zones is received and there is response information fed back by at least one follower node in a different availability zone, the request corresponding to the log is executed, where the different availability zone refers to an availability zone in the dual availability zones that is different from the leader node.

2. The disaster recovery method according to claim 1, wherein: Also includes: If it is determined that the backup zone in the dual availability zones fails, determine whether the backup zone has returned to normal within a preset time period, and stop accepting new requests within the preset time period; If the backup availability zone still fails after the preset period of time, the leader node elected based on all nodes in the primary availability zone receives the new request and synchronizes the log generated based on the new request to the primary availability zone.

3. The disaster recovery method according to claim 2, wherein: Also includes: If the standby availability zone returns to normal within the preset time period, the leader node elected based on all nodes in the dual availability zones receives the new request and synchronizes the log generated based on the new request to the dual availability zones.

4. The disaster recovery method according to claim 2, wherein: Also includes: If it is determined that the backup zone has returned to normal after the preset time period, the primary zone will synchronize the logs that have not been synchronized to the backup zone to the backup zone.

5. The disaster recovery method according to claim 4, characterized in that: Synchronizing the logs that have not been synchronized to the standby available zone to the standby available zone further includes: If it is determined that the log synchronization of the primary availability zone and the backup availability zone is completed, the leader node elected based on all nodes in the dual availability zones receives the request and synchronizes the logs generated based on the request to the dual availability zones.

6. The disaster recovery method according to claim 1, wherein: Also includes: If it is determined that the primary availability zone in the dual availability zones fails, switching between the primary availability zone and the backup availability zone is performed; The leader node selected by all nodes in the new primary availability zone generated after the switch receives the request and synchronizes the log generated according to the request to the new primary availability zone generated after the switch.

7. The disaster recovery method according to claim 6, characterized in that: The switching between the primary availability zone and the backup availability zone further includes: If it is determined that the new standby availability zone generated after the switch has returned to normal, the new primary availability zone generated after the switch will synchronize the logs that have not been synchronized to the new standby availability zone to the new standby availability zone; and If it is determined that the log synchronization of the new primary availability zone and the new backup availability zone generated after the switch is completed, the leader node elected by all nodes in the dual availability zones after the switch receives the request and synchronizes the logs generated according to the request to the dual availability zones after the switch.

8. The disaster recovery method according to claim 7, characterized in that: Also includes: Setting at most one submission group for each node in the dual availability zones, and the nodes in the same submission group are located in the same availability zone; The response information also includes information about the submission group to which the follower node belongs; The response information sent by at least one follower node in a different availability zone is determined to include: If it is determined that the submission group information included in the received response information is different from the submission group information of the leader node, it is determined that there is the response information sent by at least one of the follower nodes in the different availability zone.

9. A dual-availability zone disaster recovery device, characterized in that: The dual availability zones include a primary availability zone and a backup availability zone. In an initial state, the primary availability zone includes a majority of nodes, and the backup availability zone includes a minority of nodes. The dual-availability zone disaster recovery device includes: A dual-availability zone synchronization module is configured to, if it is determined that the dual-availability zone connection is normal, cause a leader node elected from all nodes in the dual-availability zone to receive the request and synchronize logs generated according to the request to the dual-availability zone; Synchronizing the logs generated according to the request to the dual availability zones includes: generating the log according to the received request; Sending the log to each follower node in the dual availability zones, instructing the follower node to replicate the log and then feed back response information; If it is determined that the response information returned by the majority of follower nodes in the dual availability zones is received and there is response information fed back by at least one follower node in a different availability zone, the request corresponding to the log is executed, where the different availability zone refers to an availability zone in the dual availability zones that is different from the leader node.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the dual-availability zone disaster recovery method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of the dual-availability zone disaster recovery method according to any one of claims 1 to 8.

Images