Database cluster exception handling method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for handling exceptions in a database cluster. The method comprises: if a current database node is selected as a master node after network communication is restored and other database nodes are selected as slave nodes after network communication is restored, the current database node is switched to a slave node, and the other database nodes are instructed to switch to proxy master nodes, data operation records of the other database nodes are obtained to synchronize data with the current database node, and data operations received during the period when the other database nodes serve as proxy master nodes are synchronized; after synchronization is completed, the current database node is switched to a master node and the other database nodes are switched to slave nodes. The other database nodes can be used as proxy master nodes to ensure that external services are available at any time, and the current database node is first used as a slave node to synchronize all differential data before being converted to a master node, so as to ensure the integrity of the data on the final master node.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to an exception handling method, device, and storage medium for a database cluster. Background Art

[0002] Cloud databases are typically deployed in database clusters consisting of multiple database nodes. When a database cluster experiences network or node failures, a split-brain problem can occur. This problem means that different database nodes are unable to communicate or synchronize with each other. The previously unified database cluster is split into two independent node groups. These two groups then compete for shared resources, leading to data inconsistencies and conflicts.

[0003] In current database clusters, it's difficult to simultaneously avoid split-brain issues, ensure data integrity, and guarantee service availability. For example, during data recovery after a split-brain occurs, the master node needs to synchronize inconsistent data with slave nodes during the split-brain period, rendering the database cluster unable to provide external services. Summary of the Invention

[0004] In order to solve the above technical problems, the present application at least provides a database cluster exception handling method, device and storage medium.

[0005] In a first aspect, the present application provides an exception handling method for a database cluster, wherein the database cluster includes at least two database nodes. The method includes: in response to a network communication anomaly between a current database node and other database nodes, recording the time when the current database node becomes a master node during the network communication anomaly to obtain a master-becoming time record, and recording data operations received by the current database node after becoming the master node during the network communication anomaly to obtain a data operation record; if network communication between the current database node and the other database nodes is restored, and the current database node is selected as the master node after the network communication is restored, and the other database nodes are selected as slave nodes after the network communication is restored, detecting whether the other database nodes have master-becoming time records during the network communication anomaly; if the other database nodes have master-becoming time records during the network communication anomaly, switching the current database node to a slave node, instructing the other database nodes to switch to proxy master nodes, and obtaining data operation records of the other database nodes; synchronizing data with the current database node using the data operation records of the other database nodes, and synchronizing data operations received by the other database nodes during the period when they became proxy master nodes to the current database node; and in response to completion of the synchronization, switching the current database node to a master node and instructing the other database nodes to switch to slave nodes.

[0006] In one embodiment, the method further includes: if other database nodes do not have a master time record during the network communication abnormality, directly switching the current database node to the master node and instructing other database nodes to switch to slave nodes.

[0007] In one embodiment, after detecting that the network communication between the current database node and other database nodes is restored, the method further includes: performing master node election for the current database node and other database nodes based on the time when the current database node and other database nodes last became master nodes, and / or the database node that served as the master node when the network communication was restored, and / or the Internet Protocol addresses of the current database node and other database nodes.

[0008] In one embodiment, a database node in a database cluster is communicatively connected to a container cloud, and each database node has a corresponding node state, the node state including a master node state corresponding to the database node being a master node and a slave node state corresponding to the database node being a slave node. After detecting a network communication anomaly between a current database node and other database nodes, the method further comprises: upon detecting that the current database node is selected as a master node in the slave node state, switching the current database node from a slave node to a master node; feeding back the node state of the current database node to the container cloud so that the container cloud stops the database cluster from processing data operations when it detects that two or more database nodes are master nodes based on the node states of each database node; if an event that stops the database cluster from processing data operations is detected, the current database node exits the master node state and determines a time for a next attempt to become a master. In response to the time for the next attempt to become a master being reached, the current database node is switched back to a master node and the node state of the current database node is sent to the container cloud to continue detecting whether an event that stops the database cluster from processing data operations is detected; if no event that stops the database cluster from processing data operations is detected, the current database node is maintained as the master node.

[0009] In one embodiment, data operation records of other database nodes are used to synchronize data with the current database node, including: based on the master time records of the current database node and other database nodes, the data operation records of the current database node and other database nodes are spliced in chronological order to obtain the full data operation records during the period of network communication abnormality; and the full data operation records are used to synchronize data with the current database node.

[0010] In one embodiment, determining the time for the next attempt to become the master includes: counting the number of times the current database node switches from the master node to the slave node during the network communication abnormality to obtain the number of master conflicts; and / or detecting the network communication quality corresponding to the current database node; calculating the backoff time interval based on the number of master conflicts and / or the network communication quality; and using the backoff time interval to calculate the time for the next attempt to become the master.

[0011] In one embodiment, the backoff time interval is calculated based on the number of master conflicts and / or the network communication quality, including: generating a target value selection range based on the number of master conflicts and / or the network communication quality; and performing random value selection within the target value selection range to obtain the backoff time interval.

[0012] In one embodiment, a target numerical selection range is generated based on the number of master conflicts and / or the quality of network communication, including: obtaining an initial numerical selection range, and calculating a numerical growth value based on the number of master conflicts and / or the quality of network communication; wherein the number of master conflicts is directly proportional to the numerical growth value, and the quality of network communication is inversely proportional to the numerical growth value; and using the numerical growth value to modify the initial numerical selection range to obtain a target numerical selection range.

[0013] In one embodiment, a random value is selected within a target value selection range to obtain a backoff time interval, including: selecting a random value within the target value selection range to obtain a value to be verified, and setting a maximum backoff time based on network communication quality; if the value to be verified is greater than the maximum backoff time, the maximum backoff time is used as the backoff time interval; if the value to be verified is not greater than the maximum backoff time, the value to be verified is used as the backoff time interval.

[0014] The second aspect of the present application provides an exception handling device for a database cluster, the device comprising: a data recording module for recording the time when the current database node becomes a master node during the network communication anomaly in response to an anomaly in network communication between the current database node and other database nodes, obtaining a master time record, and recording data operations received after the current database node becomes a master node during the network communication anomaly, obtaining a data operation record; a master detection module for detecting the other database nodes if the network communication between the current database node and other database nodes is restored, and the current database node is selected as the master node after the network communication is restored, and the other database nodes are selected as slave nodes after the network communication is restored. Whether there is a master time record during the network communication anomaly; a record acquisition module, which is used to switch the current database node to a slave node if other database nodes have a master time record during the network communication anomaly, and instruct other database nodes to switch to proxy master nodes, and obtain data operation records of other database nodes; a data synchronization module, which is used to synchronize data with the current database node using the data operation records of other database nodes, and synchronize data operations received during the period when other database nodes became proxy master nodes to the current database node; a master-slave switching module, which is used to switch the current database node to the master node in response to the completion of synchronization, and instruct other database nodes to switch to slave nodes.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-mentioned database cluster exception handling method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned exception handling method for the database cluster when the program instructions are executed by a processor.

[0017] The above scheme records the time when the current database node becomes the master node during the network communication anomaly to obtain a master time record, and records the data operations received by the current database node after becoming the master node during the network communication anomaly to obtain a data operation record. If the network communication between the current database node and other database nodes is restored, and the current database node is selected as the master node after the network communication is restored, and the other database nodes are selected as slave nodes after the network communication is restored, then detect whether the other database nodes have master time records during the network communication anomaly. If the other database nodes have master time records during the network communication anomaly, the current database node is switched to a slave node, and the other database nodes are instructed to switch to proxy master nodes. The data operation records of the other database nodes are obtained to synchronize data with the current database node, and the data operations received by the other database nodes during the period when they became proxy master nodes are synchronized to the current database node. In response to the synchronization being completed, the current database node is switched to the master node, and the other database nodes are instructed to switch to slave nodes. The other database nodes can be used as proxy master nodes to ensure that external services are available at any time. The current database node first synchronizes all differential data as a slave node before being converted to a master node, which can ensure the integrity of the data on the final master node.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0020] Figure 1 This is a flowchart of a method for handling exceptions in a database cluster, shown in an exemplary embodiment of the present application;

[0021] Figure 2 This is a flowchart of a master node election after network communication is restored, shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of state transition of a database node shown in an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of network flow cutting shown in an exemplary embodiment of the present application;

[0024] Figure 5 This is a flowchart of an exemplary embodiment of the present application showing a current database node attempting to become a master;

[0025] Figure 6is a schematic diagram of data storage shown in an exemplary embodiment of the present application;

[0026] Figure 7 This is a flowchart of data synchronization after network communication is restored, shown in an exemplary embodiment of the present application;

[0027] Figure 8 is a block diagram of an exception handling device for a database cluster shown in an exemplary embodiment of the present application;

[0028] Figure 9 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0029] Figure 10 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0031] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0032] The term "and / or" in this article is merely information describing the association of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0033] A cloud database is usually deployed in a database cluster consisting of multiple database nodes. The cloud database can be implemented based on a cloud storage system. In this case, the database nodes are deployed to the nodes of the cloud storage system. The cloud database can also be implemented based on a container cloud system. In this case, the database nodes are deployed to the POD of the container cloud system. This application does not limit the specific implementation method of the cloud database.

[0034] A database split-brain problem occurs when communication between database nodes is interrupted, causing the database cluster to split into multiple independent node groups, resulting in data inconsistency or service conflicts.

[0035] Under normal circumstances, database nodes that communicate with each other can perceive each other's status. At this time, one database node will be elected as the master node, and other database nodes will be elected as slave nodes. The master node provides services to the outside world.

[0036] However, when there is a communication interruption between database nodes, the interrupted database nodes cannot sense each other's status and are unable to know whether the master node has failed. At this time, in order to ensure that external services can be provided normally, the interrupted database nodes will elect a master node, and a new master node will appear. If the original master node is not faulty, this will result in two master nodes in the database cluster, causing the database cluster to split into two independent node groups. The databases on different node groups may operate independently, leading to data conflicts and inconsistencies. For example, if the database of one node group updates certain data, and another node group also updates the same data, and the two updates cannot be synchronized due to the communication failure, data inconsistencies and conflicts will occur.

[0037] In order to solve the above technical problems, the present application at least provides a database cluster exception handling method, device and storage medium.

[0038] The following describes the exception handling method for the database cluster provided in the embodiment of the present application.

[0039] See also Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application, illustrating a method for handling exceptions in a database cluster. This method can be applied to a database cluster and can be executed by either the current database node in the database cluster or the management terminal corresponding to the database cluster. This embodiment does not restrict the execution entity of this method.

[0040] like Figure 1 As shown, the database cluster exception handling method includes at least steps S110 to S150, which are described in detail as follows:

[0041] Step S110: In response to a network communication anomaly between the current database node and other database nodes, record the time when the current database node becomes the master node during the network communication anomaly to obtain a master time record, and record the data operations received after the current database node becomes the master node during the network communication anomaly to obtain a data operation record.

[0042] A database cluster contains at least two database nodes. When the network communication between the database nodes is normal, there is one database node in the database cluster as the master node, and the other database nodes are slave nodes. The master node provides external data operation services (such as adding, deleting, and modifying data), while the slave node is mainly used to synchronize and back up the data of the master node (or provide data query services). After detecting a failure of the master node, the slave node can be converted to the master node to continue providing external data operation services.

[0043] However, when network communication between database nodes in a database cluster fails, the original master node and slave nodes become disconnected. If the original master node's database is normal, the original master node can still provide external data operation services. However, the original slave node detects that it has lost connection with the master node and determines that the original master node cannot provide external data operation services. Therefore, a new master node is elected from the original slave nodes, resulting in multiple master nodes in a database cluster, which is a split-brain phenomenon.

[0044] The current database node refers to any database node in the database cluster, and other database nodes refer to database nodes in the database cluster except the current database node.

[0045] After detecting that the current database node has a network communication anomaly with other database nodes, since a split-brain phenomenon may occur in the database cluster due to the network communication anomaly, each database node records the time when it becomes the master node during the network communication anomaly, and obtains the master time record corresponding to each database node. In addition, each database node records the data operations received after becoming the master node during the network communication anomaly, and obtains the data operation records corresponding to each database node.

[0046] For example, the start and end times when database node 1 becomes the master node during a network communication anomaly are recorded to obtain a master time record. Furthermore, the data addition, deletion, and modification operations received by database node 1 after becoming the master node during the network communication anomaly are recorded to obtain a data operation record. This data can be recorded using binary logs or other file types, and this application does not limit this.

[0047] Step S120: If the network communication between the current database node and other database nodes is restored, and the current database node is selected as the master node after the network communication is restored, and other database nodes are selected as slave nodes after the network communication is restored, then detect whether there is a master time record for other database nodes during the network communication abnormality.

[0048] After network communication between database nodes is restored, the primary node needs to be determined.

[0049] For example, based on the time when the current database node and other database nodes last became master nodes, and / or the database node that serves as the master node when network communication is restored, and / or the Internet Protocol addresses of the current database node and other database nodes, the current database node and other database nodes are elected as master nodes.

[0050] For example, the current database node is node 1, and other database nodes are node 2. Figure 2 , Figure 2 FIG. 1 is a flowchart of a master node election after network communication is restored, as shown in FIG. 2 . Figure 2 Said, including:

[0051] Step S201: Node 1 detects whether node 2 is online. If not, steps S202 to S203 are executed; if so, steps S204 to S206 are executed.

[0052] If node 2 is offline, it means that network communication with node 2 is impossible.

[0053] Step S202: Count the cumulative offline time of node 2 and detect whether the cumulative offline time exceeds the preset maximum time threshold. If so, execute step S203; if not, continue to execute step S201;

[0054] Step S203: Node 1 becomes the master node.

[0055] Step S204: Determine whether node 2 is the master node. If so, execute step S205; if not, execute step S206.

[0056] Step S205: Node 1 becomes a slave node.

[0057] Step S206: Based on the time when node 1 and node 2 last became master nodes, and / or the database node that serves as the master node when network communication is restored, and / or the Internet Protocol addresses of node 1 and node 2, node 1 and node 2 are elected as master nodes.

[0058] Specifically, check whether node 1 and node 2 meet conditions 1 to 8. If any one of conditions 1 to 4 is met, node 1 becomes the master node; if any one of conditions 5 to 8 is met, node 2 becomes the master node.

[0059] Condition 1: The relational database management system (MySQL) on node 1 was the primary node last time, and the MySQL on node 2 was not the primary node last time.

[0060] Condition 2: The Database High Availability (DBHA) node on node 1 was previously the primary node, and the DBHA node on node 2 was not previously the primary node.

[0061] Condition 3: The DBHA of node 1 was the master node last time, and the DBHA of node 2 was also the master node last time, but the duration of the last time that the DBHA of node 1 became the master node is greater than the duration of the last time that the DBHA of node 2 became the master node.

[0062] Condition 4: The Internet Protocol (IP) address of node 1 is greater than the IP address of node 2.

[0063] Condition 5: The MySQL server on node 1 was not the master node last time, and the MySQL server on node 2 was the master node last time.

[0064] Condition 6: The last time the DBHA of node 1 was active, node 2 was active.

[0065] Condition 7: The DBHA of node 1 was the master node last time, and the DBHA of node 2 was also the master node last time, but the duration of the last time that the DBHA of node 1 became the master node was shorter than the duration of the last time that the DBHA of node 2 became the master node.

[0066] Condition 8: The Internet Protocol (IP) address of node 1 is smaller than the IP address of node 2.

[0067] Of course, in addition to the master node election method exemplified above, other methods can also be used to elect the master node, such as electing the master node based on the network stability and software operation stability of each database node. This application does not limit this.

[0068] If the current database node is selected as the master node after network communication is restored, and other database nodes are selected as slave nodes after network communication is restored, it is detected whether other database nodes have master time records during the network communication abnormality period.

[0069] During periods of network communication anomalies, multiple database nodes may become master nodes, resulting in inconsistent data stored in multiple database nodes after the network is restored. If the data of the master node is directly synchronized to the slave node after the master node is selected, the difference data recorded by the slave node during the period of network communication anomaly will be lost.

[0070] Therefore, in order to ensure data integrity, after the current database node is selected as the master node after network communication is restored, it is detected whether other database nodes have master time records during the network communication anomaly. If other database nodes have master time records during the network communication anomaly, it means that other database nodes have processed data operations as master nodes during the network communication anomaly, that is, there are different data in other database nodes.

[0071] Step S130: If other database nodes have a master time record during the network communication abnormality, the current database node is switched to a slave node, and other database nodes are instructed to switch to proxy master nodes to obtain data operation records of other database nodes.

[0072] If other database nodes have master time records during the network communication abnormality, in order to ensure data integrity, the difference data of other database nodes needs to be synchronized to the current database node.

[0073] Since the master node must suspend external data operation services to ensure data accuracy when performing data synchronization, the present application further takes service availability into consideration. After detecting that the differential data of other database nodes needs to be synchronized, the current database node is not directly converted into a master node. Instead, the current database node is first switched to a slave node, and other database nodes are instructed to switch to proxy master nodes, so that other database nodes can first provide external data operation services as proxy master nodes, so that the current database node can first synchronize the differential data of other database nodes as a slave node based on the data operation records of other database nodes.

[0074] In some embodiments, if other database nodes do not have a master time record during the network communication abnormality, it indicates that there is no difference data in other database nodes that needs to be synchronized, then the current database node is directly switched to the master node, and the other database nodes are instructed to switch to slave nodes, and subsequently the other database nodes act as slave nodes to fully synchronize the data in the master node (current database node).

[0075] Step S140: synchronizing data on the current database node using data operation records of other database nodes, and synchronizing data operations received during the period when other database nodes become proxy master nodes to the current database node.

[0076] First, based on the data operation records of other database nodes, the differential data formed by other database nodes during the network communication abnormality is synchronized, and then based on the data operations received during the period when other database nodes became proxy master nodes, the differential data formed during the period when other database nodes became proxy master nodes are synchronized.

[0077] Optionally, since different data operations may have certain logical associations in the time dimension, in order to ensure the accuracy of data synchronization, the data stored in the current database node can be rolled back to a specified time point, such as rolling back to the starting time point of the network communication abnormality, or rolling back to the starting time point of becoming the primary node for the first time. Then, the data operation records of other database nodes and the data operation records of the current database node after the specified time point are obtained, and the data after the rollback of the current database node are synchronously restored (such as additions, deletions, and modifications) in the time order of these data operation records.

[0078] Step S150: In response to synchronization completion, the current database node is switched to a master node, and other database nodes are instructed to switch to slave nodes.

[0079] After the synchronization of the current database node is completed, the data in the current database node is the most complete in the database cluster. At this time, the current database node is switched to the master node, and other database nodes are instructed to switch to slave nodes.

[0080] Of course, after other database nodes switch to slave nodes, all data of the master node (current database node) will be fully synchronized to ensure data integrity and consistency.

[0081] In this application, after the current database node is selected as the master node after the network communication is restored and it is detected that other database nodes have master time records during the network communication abnormality, the current database node is first switched to a slave node and the other database nodes are instructed to switch to proxy master nodes. Then, the current database node synchronizes the difference data of other database nodes during the network communication abnormality and the difference data during the period of becoming the proxy master node. After the synchronization is completed, the current database node is switched to the master node and the other database nodes are instructed to switch to slave nodes. Other database nodes can be used as proxy master nodes to ensure that external services are available at any time, and the current database node first synchronizes the difference data as a slave node and then is converted to a master node, which can ensure the integrity of the data on the final master node.

[0082] Next, some embodiments of the present application are described in detail.

[0083] In some embodiments, a database node in a database cluster is communicatively connected to a container cloud, and each database node has a corresponding node status, which includes at least a master node status corresponding to when the database node is a master node and a slave node status corresponding to when the database node is a slave node. After detecting a network communication anomaly between the current database node and other database nodes, the method further includes:

[0084] Step S310: If it is detected that the current database node is selected as the master node in the slave node state, the current database node is switched from the slave node to the master node.

[0085] Step S320: Send the node status of the current database node to the container cloud, so that the container cloud stops the database cluster from processing data operations when it detects that there are two or more database nodes as master nodes based on the node status of each database node.

[0086] Step S330: If an event that stops the database cluster from processing data operations is detected, the current database node exits the master node state and determines the time for the next attempt to become the master. In response to the time for the next attempt to become the master, the current database node is switched back to the master node, and the node status of the current database node is sent to the container cloud to continue detecting whether there is an event that stops the database cluster from processing data operations.

[0087] Step S340: If it is detected that there is no event that stops the database cluster from processing data operations, the current database node is kept as the master node.

[0088] For example, see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a state transition of a database node shown in an exemplary embodiment of the present application. Figure 3 As shown in the figure, the database service of the database node is started. After successful initialization, it enters the initialization (init) state.

[0089] In the "init" state, the database service self-checks are continuously performed. If the database service is normal and the data information is complete, the database service self-check passes and the state is changed to the ready state. Otherwise, the "init" state is maintained until the database service self-check passes.

[0090] In the "prepare" state, based on the master-slave election results, determine whether the current database node is elected as a master node or a slave node, and switch to the master node (master) state or the slave node (slave) state respectively.

[0091] In the "master" state (or slave state), if the database service is abnormal, such as being unable to connect to the container cloud or MySQL being abnormal, exit the "master" state and switch back to the "init" state to perform a database service self-test.

[0092] In the "slave" state, if the master node is detected to be abnormal (if the original master node is judged to be abnormal due to network communication abnormalities, the database service of the original master node may not be abnormal, that is, the original master node has not yet exited the "master" state) and it allows itself to become the master, it will switch to the "master" state; after switching to the "master" state, if it is detected that the container cloud has cut off the network flow (that is, the database cluster stops processing data operations), it will adaptively yield, the current database node will exit the "master" state, switch back to the "prepare" state (or switch back to the "slave" state), and try to become the master again after a certain period of time.

[0093] It should be noted that in the above steps, after the database node is converted from the "slave" state to the "master" state, it will exit the "master" state only after it senses that the container cloud has cut off the network flow within the preset time interval; and the database node that was originally in the "master" state does not need to be processed and remains in the "master" state. Among them, the database node that has been in the "master" state for more than the above preset time interval will be determined to be a database node that was originally in the "master" state.

[0094] This application performs network flow control through the container cloud. After sensing that there are multiple database nodes in the master node state, the database cluster can be prohibited from providing external services through network flow control. After sensing the network flow control, the database node can exit the master node state in time to avoid brain split in the database, thereby avoiding system chaos, data conflicts and inconsistencies caused by the two master nodes competing for shared resources.

[0095] For example, see Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a network flow cutoff shown in an exemplary embodiment of the present application. Figure 4 As shown in the figure, assuming that the database cluster includes nodes 1 and 2, if node 1 is in the "master" state, the ready state (ready) is fed back to the container cloud; if it is in any state other than the "master" state, the non-ready state (notready) is fed back to the container cloud.

[0096] The container cloud checks the status of each database node in the database cluster at a preset interval (such as 0.5 seconds). If a multi-ready state occurs, it is determined that there are two or more database nodes as master nodes, and the network traffic is immediately cut off to prohibit the database cluster from providing external services.

[0097] For example, a database cluster includes node 1 and node 2, where node 1 is a master node and node 2 is a slave node.

[0098] Scenario 1: Network communication between Node 1 and Node 2 is abnormal

[0099] 1) Node 2 detects a network communication anomaly with Node 1, switches to the "master" state, and reports its readiness status to the container cloud.

[0100] 2) The container cloud detects a dual-ready state and performs network traffic cutoff.

[0101] 3) Node 1 senses the network traffic cutoff operation on the container cloud and remains in the "master" state.

[0102] 4) Node 2, which becomes the master later, detects the network traffic cutoff operation on the container cloud, exits the "master" state, switches to the "prepare" state, and determines the time for the next attempt to become the master.

[0103] 5) When the next attempt to become the master arrives, repeat the above steps until the network communication between node 1 and node 2 is normal.

[0104] Scenario 2: Network communication between Node 1 and Node 2 is restored

[0105] 1) Node 1 remains in the master state.

[0106] 2) If node 2 is in the "prepare" state, it directly switches to the "slave" state and performs data adaptive recovery (i.e., synchronizes the data of node 1); if node 2 is in the "master" state, a master node election is performed. If node 2 is elected as a slave node, node 2 still switches to the "slave" state. If node 2 is elected as the master node, data recovery is performed according to the above steps S120 to S150.

[0107] The above embodiment can effectively avoid the split-brain problem in the database cluster.

[0108] In addition, when the container cloud detects that there are two or more database nodes in the database cluster as master nodes, it will perform network traffic cutting. If the nodes in the database cluster cannot communicate normally for a long time, the original slave node will frequently switch between the "master" state and other states (such as the "prepare" state), causing the container cloud to frequently perform network traffic cutting, affecting the use of database services.

[0109] Therefore, in order to solve the above problem, it is necessary to adaptively adjust the time when the original slave node attempts to become the master next time.

[0110] In some embodiments, determining the time of the next attempt to become the master in step S330 includes:

[0111] Step S331: Count the number of times the current database node switches from a master node to a slave node during the network communication abnormality to obtain the number of master conflicts; and / or detect the network communication quality corresponding to the current database node.

[0112] During the period of network communication anomaly, each time the current database node attempts to enter the "master" state and fails, the number of master conflicts is accumulated by 1.

[0113] Detect the network communication quality corresponding to the current database node, such as detecting parameters such as the network load and / or network response speed of the current database node to obtain the network communication quality.

[0114] For example, the network load can be assessed by counting the amount of data packets transmitted in the network corresponding to the current database node. The greater the amount of data packets transmitted, the greater the network load, and the smaller the amount of data packets transmitted, the smaller the network load. The network response speed can also be assessed by measuring the transmission delay of the heartbeat data packets corresponding to the current database node. The greater the transmission delay of the heartbeat data packets, the slower the network response speed, and the smaller the transmission delay of the heartbeat data packets, the faster the network response speed. By combining parameters such as the network load and network response speed of the current database node, the network communication quality can be obtained. The greater the network load and the slower the network response speed, the lower the network communication quality, while the smaller the network load and the faster the network response speed, the higher the network communication quality.

[0115] Of course, other methods may also be used to evaluate the network communication quality, and this application does not limit this.

[0116] Step S332: Calculate a backoff time interval based on the number of master conflicts and / or network communication quality.

[0117] Exemplarily, based on the number of master conflicts and / or network communication quality, a target value selection range is generated; and a random value is selected within the target value selection range to obtain a backoff time interval.

[0118] The target value selection range contains a plurality of consecutive selectable values, and a random value is selected within the target value selection range to obtain a backoff time interval.

[0119] For example, the minimum selectable value and the maximum selectable value can be determined based on the number of master conflicts and / or the network communication quality, and the interval from the minimum selectable value to the maximum selectable value is used as the target value selection range. Specifically, the greater the number of master conflicts and the lower the network communication quality, the larger the minimum selectable value and the maximum selectable value; the smaller the number of master conflicts and the higher the network communication quality, the smaller the minimum selectable value and the maximum selectable value.

[0120] For another example, an initial numerical selection range is obtained, and a numerical growth value is calculated based on the number of master conflicts and / or the network communication quality; wherein the number of master conflicts is directly proportional to the numerical growth value, and the network communication quality is inversely proportional to the numerical growth value; the numerical growth value is used to modify the initial numerical selection range to obtain a target numerical selection range.

[0121] The initial value selection range may be pre-set based on experience or flexibly calculated.

[0122] For example, the initial value selection range is set according to the network communication quality. For example, the lower the network communication quality, the larger the value in the initial value selection range; the higher the network communication quality, the smaller the value in the initial value selection range.

[0123] The numerical increment is calculated based on the number of master conflicts and / or the network communication quality. A greater number of master conflicts and / or lower network communication quality results in a greater numerical increment, while a smaller number of master conflicts and / or higher network communication quality results in a smaller numerical increment.

[0124] Use the increment value to modify the initial range to create the target range. For example, you can add the increment value to each value in the initial range to create the target range. You can also add the increment value to the maximum value in the initial range to create the new maximum value, and use the range from the original minimum to the new maximum value in the initial range as the target range.

[0125] Optionally, if the current database node fails in the first attempt to become the master, a value is selected within the initial value selection range to obtain a backoff time interval.

[0126] Optionally, a maximum backoff time is set based on the network communication quality, and a value randomly selected within the target value selection range is used as the value to be verified; if the value to be verified is greater than the maximum backoff time, the maximum backoff time is used as the backoff time interval; if the value to be verified is not greater than the maximum backoff time, the value to be verified is used as the backoff time interval.

[0127] Step S333: Calculate the time for the next attempt to become the master using the backoff time interval.

[0128] Take the number of conflicts as an example to illustrate, please refer to Figure 5 , Figure 5 FIG. 1 is a flowchart of an exemplary embodiment of the present application showing a current database node attempting to become a master node. Figure 5 Shown, including:

[0129] Step S501: Update to the main conflict number.

[0130] Step S502: Determine whether the number of primary conflicts is greater than a preset number threshold. If not, execute step S503; if greater, execute step S504.

[0131] The preset number threshold can be pre-set based on experience, such as the preset number threshold is 9.

[0132] Step S503: randomly selecting a value from the first value selection range to obtain a backoff time interval.

[0133] For example, the time for the next attempt to become the primary is randomly selected from 10 to 10 + 2^count, where count is the number of conflicts in becoming the primary. For example, if count = 4 and 2^4 = 16, a value is randomly selected from the range of 10 to (10 + 16) as the backoff interval for the next attempt to become the primary.

[0134] Step S504: randomly selecting a value from the second value selection range to obtain a backoff time interval, wherein at least a portion of the values in the second value selection range is greater than the maximum value in the first value selection range.

[0135] For example, if the preset number threshold is 9, if the number of master conflicts is higher than 9, the backoff time interval is randomly selected from 10 to 600.

[0136] Step S505: Set a timer according to the backoff time interval.

[0137] Step S506: After the timer times out, the current database node switches to the master node.

[0138] Step S507: Whether network traffic is cut off is detected. If so, step S508 is executed and then step S501 is executed; if not, step S509 is executed.

[0139] Step S508: The current database node exits the master node state.

[0140] Step S509: The number of master conflicts is set to 0, and the current data of the current database node is backed up.

[0141] The purpose of backing up the current data of the current database node is to facilitate data rollback during subsequent data recovery.

[0142] By randomly selecting the backoff time interval within the range of values, the frequency with which the current database node attempts to become the master node can be reduced, preventing both ends from becoming the master node at a high frequency, and reducing the impact on the database's external services. At the same time, if the original master node fails and exits the master node state, the current database node can promptly become the master node and provide external services.

[0143] Of course, in addition to randomly selecting the backoff time interval within the above value selection range, the backoff time interval can also be directly calculated based on the positive correlation between the backoff time interval and the number of conflicts, and / or the inverse correlation between the backoff time interval and the network communication quality.

[0144] By flexibly adjusting the backoff time interval to determine the time for the next attempt to become the master, changes in the network can be more flexibly responded to, thereby improving system performance and reliability.

[0145] It should be noted that if the solution for avoiding brain split in the database cluster in the above embodiment is adopted, when data recovery is performed after network communication is restored and data operation records of other database nodes are used to synchronize data of the current database node, since the current database node and other database nodes will not become master nodes at the same time, that is, the time corresponding to the data operation records of the current database node and other database nodes does not overlap, the data operation records can be directly spliced together in chronological order according to the time corresponding to each data operation record, and data recovery of the current database node is performed based on the spliced data operation records.

[0146] Specifically, in step S140, data operation records of other database nodes are used to synchronize data of the current database node, including:

[0147] Step S141: Based on the creation time records of the current database node and other database nodes, the data operation records of the current database node and other database nodes are sequentially spliced in chronological order to obtain the full data operation records during the network communication abnormality period.

[0148] Step S142: Synchronize data on the current database node using the full data operation record.

[0149] For example, see Figure 6 , Figure 6 is a schematic diagram of data storage shown in an exemplary embodiment of the present application, such as Figure 6As shown in the figure, the database cluster consists of Node 1 and Node 2. When the network communication between Node 1 and Node 2 is abnormal, Node 1 becomes the master node and records data A. However, due to the network communication anomaly, data A cannot be synchronized to Node 2. Subsequently, Node 1 abnormally exits the master node state, and Node 2 becomes the master node and records data B. However, data B cannot be synchronized to Node 1. After the network between the two nodes returns to normal, a new election is held, and Node 2 is selected as the master node after the network communication is restored.

[0150] If node 2 is directly converted to the master node, the slave node (node 1) will fully synchronize the current data of the master node (node 2), resulting in the loss of data A in node 1 and incomplete data information.

[0151] Therefore, Node 2 is initially configured as a slave node, while Node 1 is used as the proxy master node to maintain external services. Node 2 obtains the data operation records corresponding to Data A from Node 1 for data synchronization. Furthermore, the data operations received during Node 1's time as the proxy master node are synchronized to Node 2. Once synchronization is complete, Node 2 is switched to the master node, and Node 1 is switched to the slave node.

[0152] Take node 1 and node 2 as an example. Node 2 is selected as the master node after network communication is restored. Figure 7 , Figure 7 FIG. 1 is a flowchart of data synchronization after network communication is restored, as shown in FIG. 2 . Figure 7 Shown, including:

[0153] Step S701: Detect whether both node 1 and node 2 have network flow cutoff information in the master time record or network flow cutoff record file. If not, execute step S702; if so, execute steps S703 to S707.

[0154] Optionally, based on the above embodiment, since the database node will be in the master node state for a very short time even if it fails to become the master when trying to become the master, there may be data operation records at this time. Therefore, when any database node perceives that the container cloud is performing network flow cutting, it records the current time point to the network flow cutting record file, refreshes the data operation record, and writes subsequent addition, deletion and modification records into the new data operation record to ensure the data integrity of subsequent data synchronization.

[0155] Step S702: Node 2 is switched to the master node, and node 1 is switched to the slave node, and the slave node fully synchronizes the data of the master node.

[0156] Step S703: Node 2 switches to a slave node, and node 1 switches to a proxy master node. Node 2 obtains all data operation records after the time T1 when node 1 first becomes the master.

[0157] For example, the first master time T1 is Figure 6 The start time of data A is recorded in .

[0158] Step S704: Merge the data operation records of node 1 and node 2 in sequence according to the time sequence of the data operation records of node 1 and node 2 to obtain the full data operation records.

[0159] Step S705: The data of node 2 is rolled back to T1;

[0160] Step S706: Import data sequentially based on the data operation records. After the import is completed, the data operations received during the period when synchronization node 1 becomes the proxy master node are transferred to node 2.

[0161] For example, the sequence number and offset of the data operation record of the database of node 2 are set so that node 2 can continue to synchronize the subsequent data of node 1 as the proxy master node from the time point when data recovery is completed.

[0162] Step S707: Node 2 completes data synchronization, and node 2 switches to the master node, and node 1 switches to the slave node.

[0163] Of course, after data synchronization is completed, it also includes clearing related record information to avoid wasting storage resources due to invalid data.

[0164] The database cluster exception handling method provided by the present application records the time when the current database node becomes the master node during the network communication abnormality to obtain the master time record, and records the data operations received by the current database node after becoming the master node during the network communication abnormality to obtain the data operation record; if the network communication between the current database node and other database nodes is restored, and the current database node is selected as the master node after the network communication is restored, and other database nodes are selected as slave nodes after the network communication is restored, then detect whether the other database nodes have the master time record during the network communication abnormality; if the other database nodes have the master time record during the network communication abnormality, The current database node is switched to a slave node, and other database nodes are instructed to switch to proxy master nodes. The data operation records of other database nodes are obtained to synchronize data with the current database node, and the data operations received during the period when other database nodes become proxy master nodes are synchronized to the current database node. In response to the completion of synchronization, the current database node is switched to a master node, and other database nodes are instructed to switch to slave nodes. Other database nodes can be used as proxy master nodes to ensure that external services are available at any time, and the current database node first synchronizes the difference data as a slave node and then is converted to a master node, which can ensure the integrity of the data on the final master node.

[0165] Figure 8FIG. 1 is a block diagram of an exception handling device for a database cluster shown in an exemplary embodiment of the present application. Figure 8 As shown, the exemplary database cluster exception handling device 800 includes:

[0166] The data recording module 810 is configured to, in response to a network communication anomaly between the current database node and other database nodes, record the time when the current database node becomes the master node during the network communication anomaly to obtain a master time record, and record data operations received by the current database node after becoming the master node during the network communication anomaly to obtain a data operation record;

[0167] The master node detection module 820 is configured to detect whether there are master node time records for the other database nodes during the period of abnormal network communication if the network communication between the current database node and the other database nodes is restored and the current database node is selected as the master node after the network communication is restored and the other database nodes are selected as slave nodes after the network communication is restored;

[0168] The record acquisition module 830 is used to switch the current database node to a slave node if there is a master time record of other database nodes during the network communication abnormality, and instruct other database nodes to switch to acting master nodes, and obtain data operation records of other database nodes;

[0169] The data synchronization module 840 is used to synchronize data of the current database node using data operation records of other database nodes, and synchronize data operations received by other database nodes during the period when they became proxy master nodes to the current database node;

[0170] The master-slave switching module 850 is configured to switch the current database node to the master node in response to synchronization completion, and instruct other database nodes to switch to slave nodes.

[0171] It should be noted that the database cluster exception handling device provided in the above embodiment and the database cluster exception handling method provided in the above embodiment are based on the same concept. The specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In actual applications, the database cluster exception handling device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited here.

[0172] See also Figure 9 , Figure 9This is a schematic diagram of the structure of an embodiment of an electronic device of the present application. Electronic device 900 includes memory 901 and processor 902. Processor 902 is configured to execute program instructions stored in memory 901 to implement the steps of any of the aforementioned database cluster exception handling method embodiments. In a specific implementation scenario, electronic device 900 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 900 may also include mobile devices such as laptops and tablet computers, which are not limited herein.

[0173] Specifically, processor 902 is used to control itself and memory 901 to implement the steps of any of the aforementioned database cluster exception handling method embodiments. Processor 902 may also be referred to as a central processing unit (CPU). Processor 902 may be an integrated circuit chip with signal processing capabilities. Processor 902 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 902 may be implemented by an integrated circuit chip.

[0174] See also Figure 10 , Figure 10 The computer-readable storage medium 1000 stores program instructions 1010 that can be executed by a processor, and the program instructions 1010 are used to implement the steps of any of the above-mentioned database cluster exception handling method embodiments.

[0175] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0176] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0178] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in either hardware or software functional units. If the integrated units are implemented as software functional units and sold or used as standalone products, they may be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product, stored on a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

Claims

1. A database cluster exception handling method, characterized in that: The database cluster includes at least two database nodes, and the method includes: In response to a network communication anomaly between the current database node and other database nodes, recording a time when the current database node becomes a master node during the network communication anomaly to obtain a master becoming time record, and recording data operations received by the current database node after becoming the master node during the network communication anomaly to obtain a data operation record; If the network communication between the current database node and the other database nodes is restored, and the current database node is selected as the master node after the network communication is restored, and the other database nodes are selected as slave nodes after the network communication is restored, detecting whether the other database nodes have a master time record during the period of the network communication anomaly; If the other database node has a master time record during the network communication abnormality, switching the current database node to a slave node, instructing the other database node to switch to a proxy master node, and obtaining data operation records of the other database node; Performing data synchronization on the current database node using the data operation records of the other database nodes, and synchronizing data operations received during the period when the other database nodes serve as the proxy master node to the current database node; In response to synchronization completion, the current database node is switched to a master node, and the other database nodes are instructed to switch to slave nodes.

2. The method according to claim 1, characterized in that The method further comprises: If the other database nodes do not have a master time record during the network communication abnormality, the current database node is directly switched to the master node, and the other database nodes are instructed to switch to the slave nodes.

3. The method according to claim 1, characterized in that After detecting that the network communication between the current database node and the other database nodes is restored, the method further includes: Based on the time when the current database node and the other database nodes last became master nodes, and / or the database node that serves as the master node when the network communication is restored, and / or the Internet Protocol addresses of the current database node and the other database nodes, a master node election is performed for the current database node and the other database nodes.

4. The method according to claim 1, wherein The database nodes in the database cluster are communicatively connected to the container cloud, and the database nodes have corresponding node states, including a master node state corresponding to the database node being a master node and a slave node state corresponding to the database node being a slave node; After detecting that network communication between the current database node and other database nodes is abnormal, the method further includes: If it is detected that the current database node is selected as a master node in the slave node state, the current database node is switched from a slave node to a master node; Feedback the node status of the current database node to the container cloud, so that when the container cloud detects that two or more database nodes are master nodes based on the node status of each database node, it stops the database cluster from processing data operations; If an event that stops the database cluster from processing data operations is detected, the current database node exits the master node state and determines a time for a next attempt to become the master. In response to the arrival of the time for the next attempt to become the master, the current database node is switched back to the master node, and the node status of the current database node is sent to the container cloud to continue detecting whether there is an event that stops the database cluster from processing data operations. If it is detected that there is no event that stops the database cluster from processing data operations, the current database node is maintained as the master node.

5. The method according to claim 4, characterized in that The step of synchronizing data of the current database node by using the data operation records of the other database nodes includes: Based on the master time records of the current database node and the other database nodes, the data operation records of the current database node and the other database nodes are sequentially spliced in chronological order to obtain the full data operation records during the network communication abnormality period; The full data operation record is used to synchronize data on the current database node.

6. The method according to claim 4, characterized in that Determining the time for the next attempt to become the master includes: Counting the number of times the current database node switches from a master node to a slave node during the network communication abnormality to obtain the number of master conflicts; and / or detecting the network communication quality corresponding to the current database node; Calculating a backoff time interval based on the number of master conflicts and / or the network communication quality; The backoff time interval is used to calculate the time for the next attempt to become the master.

7. The method according to claim 6, characterized in that The calculating the backoff time interval based on the number of master conflicts and / or the network communication quality includes: generating a target value selection range based on the number of master conflicts and / or the network communication quality; A random value is selected within the target value selection range to obtain a backoff time interval.

8. The method according to claim 7, characterized in that The generating of a target value selection range based on the number of master conflicts and / or the network communication quality includes: Obtaining an initial value selection range, and calculating a value growth value based on the number of master conflicts and / or the network communication quality; wherein the number of master conflicts is proportional to the value growth value, and the network communication quality is inversely proportional to the value growth value; The initial value selection range is modified using the value increment value to obtain a target value selection range.

9. The method according to claim 7, characterized in that The step of randomly selecting a value within the target value selection range to obtain a backoff time interval includes: Randomly selecting a value within the target value selection range to obtain a value to be verified, and setting a maximum backoff time based on the network communication quality; If the value to be verified is greater than the maximum backoff time, the maximum backoff time is used as the backoff time interval; If the value to be verified is not greater than the maximum backoff time, the value to be verified is used as the backoff time interval.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 9.

Images