Cluster switching method and device for distributed database cluster, equipment and medium

Abstract

The embodiment of the invention provides a cluster switching method and device for a distributed database cluster, equipment and a medium. The method comprises the following steps: configuring parameters in a cluster switching process to generate configuration parameters; and then, carrying out backup on the Zookeeper metadata. And in the process of performing cluster switching on the first cluster and the second cluster according to the configuration parameters, if switching failure is detected, performing rollback according to the Zookeeper metadata. Wherein the configuration parameters can reduce the possibility of failure of table structure check in the cluster switching process and reduce the possibility of distributed transaction compensation of the standby cluster, the first cluster is the main cluster, and the second cluster is the standby cluster. According to the technical scheme, the reliability of cluster switching is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a cluster switching method, device, equipment and medium for a distributed database cluster. Background Art

[0002] Cluster switching between distributed database clusters refers to the process of switching database services from the primary cluster to the backup cluster in a distributed database system when a disaster occurs in the primary cluster (such as hardware failure, natural disaster, serious software failure, etc.).

[0003] Currently, the monitoring system continuously monitors various indicators of the main cluster. When it determines that the main cluster has a fault, it makes a switching decision based on the pre-set strategy. After that, it directs the client's request from the main cluster to the backup cluster, so that the client can normally access the database service provided by the backup cluster.

[0004] However, when cluster switching between instances fails, the database service cannot be restored to an available state in time, resulting in low reliability of cluster switching. Summary of the invention

[0005] The embodiments of the present application provide a cluster switching method, apparatus, device and medium for a distributed database cluster, so as to achieve the effect of improving the reliability of cluster switching.

[0006] In a first aspect, an embodiment of the present application provides a cluster switching method of a distributed database cluster, including:

[0007] Configuring parameters of the cluster switching process to generate configuration parameters, wherein the configuration parameters can reduce the possibility of failure of table structure check during the cluster switching process and reduce the possibility of distributed transaction compensation by the standby cluster;

[0008] Back up Zookeeper metadata;

[0009] In the process of performing cluster switching between the first cluster and the second cluster according to the configuration parameters, if switching failure is detected, rollback is performed according to the Zookeeper metadata, and the first cluster is the main cluster and the second cluster is the standby cluster.

[0010] In a second aspect, an embodiment of the present application provides a cluster switching device for a distributed database, including:

[0011] A configuration module is used to configure parameters of the cluster switching process and generate configuration parameters, wherein the configuration parameters can reduce the possibility of failure of table structure check during the cluster switching process and reduce the possibility of distributed transaction compensation by the standby cluster;

[0012] Backup module, used to back up Zookeeper metadata;

[0013] The rollback module is used to roll back according to the Zookeeper metadata if a switch failure is detected during the cluster switch between the first cluster and the second cluster according to the configuration parameters, wherein the first cluster is the main cluster and the second cluster is the standby cluster.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the implementation of the first aspect as described above.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the implementation method of the first aspect as above.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the implementation method of the first aspect as above.

[0019] The cluster switching method, device, equipment and medium of the distributed database cluster provided by the embodiment of the present application include: generating configuration parameters by configuring the parameters of the cluster switching process. Afterwards, backing up the Zookeeper metadata. In the process of cluster switching between the first cluster and the second cluster according to the configuration parameters, if the switching failure is detected, rollback is performed according to the Zookeeper metadata. Among them, the configuration parameters can reduce the possibility of failure of the table structure check during the cluster switching process, and reduce the possibility of distributed transaction compensation by the standby cluster, the first cluster is the main cluster, and the second cluster is the standby cluster. In this technical solution, by cluster switching between the first cluster and the second cluster according to the configuration parameters, the cluster switching process is simplified, sufficient time is provided for the transaction preparation process, and the success rate and reliability of the switching process are improved. Further, the Zookeeper metadata is backed up so that when there is a problem with the switching, the Zookeeper metadata can be rolled back so that the first cluster and the second cluster can be restored to the state before the switching, further improving the reliability of the cluster switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1 Schematic diagram of the cluster switching method of the distributed database cluster provided in this application Figure 1 ;

[0022] Figure 2 Schematic diagram of the cluster switching method of the distributed database cluster provided in this application Figure 2 ;

[0023] Figure 3 A schematic diagram of a process flow of a backup process before switching provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a cluster switching process provided in an embodiment of the present application;

[0025] Figure 5 A flowchart of a post-switching inspection process provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a scenario of a cluster switching method for a distributed database cluster provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a cluster switching device for a distributed database cluster provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0029] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] First, the terms used in this application are explained:

[0033] Distributed database: A distributed database is a database system that stores data in multiple physical nodes (which can be different servers or storage devices) instead of storing it in a single node like a traditional database. These nodes are connected through a network and work together to provide data storage and management functions. They can effectively solve online transaction processing (OLTP) scenarios with ultra-large concurrency, ultra-high performance, and ultra-large capacity. Its core features are strong data consistency, high performance and low cost, linear horizontal expansion, financial-grade high availability, enterprise-level security, and convenient operation and maintenance.

[0034] Primary cluster and standby cluster: If the business normally accesses a database instance in the cluster, and the database instance synchronizes data to a read-only database instance in another cluster, the instance that is normally accessed is called the primary instance, and the cluster it is in is the primary cluster (primary distributed database connection (DCN)). The read-only database instance is called the standby instance, and the cluster it is in is the standby cluster (standby DCN).

[0035] Cluster switching: Switching between database instances in different clusters, that is, switching between primary and standby instances. Specifically, the previously normally accessible primary instance is set to read-only and becomes the new standby instance; the original standby instance is set to be normally accessible and becomes the new primary instance.

[0036] Next, the application scenarios involved in this application are explained:

[0037] Cluster switching between distributed database clusters refers to the process of switching database services from the primary cluster to the backup cluster in a distributed database system when a disaster occurs in the primary cluster (such as hardware failure, natural disaster, serious software failure, etc.). This process is designed to ensure database availability, data integrity, and business continuity.

[0038] Currently, the monitoring system continuously monitors various indicators of the main cluster. When it is determined that the main cluster has a fault, it makes a switching decision based on the pre-set strategy. For example, if more than a certain proportion of nodes in the main cluster fail, it decides to start cluster switching. After that, the client's request is directed from the main cluster to the backup cluster, including updating domain name resolution, adjusting the configuration of the load balancer, etc., so that the client can normally access the database service provided by the backup cluster.

[0039] However, when the cluster switching between instances fails, the database service cannot be restored to an available state in time. For example, if a deadlock or infinite loop occurs in the data synchronization link during the switching process, the database cannot provide services normally, resulting in a long-term business interruption, and thus failing to meet the preset recovery time objective (RTO) requirements. In the modern business environment, many businesses have very high requirements for database availability, and long-term interruptions may cause huge economic losses and user churn.

[0040] That is to say, the reliability of the existing technology when performing cluster switching between distributed database clusters is low.

[0041] Based on the above technical problems, the invention of the present application is conceived as follows: Considering that the reasons for cluster switching failure mainly include too strict inspection of the table leading to inspection timeout, and the standby cluster performing distributed transaction compensation leading to inconsistency between the global transaction identifier (GTID) of the main cluster and the GTID of the cluster. Therefore, the parameters of the cluster switching process can be configured in advance to generate configuration parameters, which can reduce the possibility of failure of the table structure inspection during the cluster switching process and the possibility of distributed transaction compensation by the standby cluster. In addition, before switching, the zookeeper (Zookeeper, English abbreviation: zk) metadata is backed up. So that when the switch fails, it is rolled back according to the Zookeeper metadata and restored to the state before the switch, avoiding the normal use of the database service. In other words, parameter configuration is performed from multiple aspects before switching to ensure the smooth progress of the switching process, and it can be rolled back in time when there is a problem in the switching process to ensure the reliability of the switching process.

[0042] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0043] Figure 1Schematic diagram of the cluster switching method of the distributed database cluster provided in this application Figure 1 ,like Figure 1 As shown, the cluster switching method of the distributed database cluster includes the following steps:

[0044] S11. Configure parameters of the cluster switching process to generate configuration parameters.

[0045] Among them, the configuration parameters can reduce the possibility of failure in table structure check during cluster switching and reduce the possibility of distributed transaction compensation in the standby cluster.

[0046] Exemplarily, reducing the possibility of failure in checking the table structure may be reflected in reducing the frequency and scope of checking the table, reducing the strictness of the checking, and the like.

[0047] Exemplarily, the inspection scope includes a data range in a table that needs to be inspected and a table range that needs to be inspected.

[0048] Exemplarily, reducing the strictness of the check may be achieved by reducing an exact match to an approximate match, performing only a non-null check on the data, and not performing a structure check on the table.

[0049] In one possible implementation, parameters can be configured in the following two ways:

[0050] 1. Configure the inspection mode to target mode.

[0051] Among them, the table structure is not checked in the target mode.

[0052] By configuring the check mode as the target mode, the table structure is not checked during the cluster switching process, preventing the switching failure caused by the format check timeout due to too many tables, thereby ensuring the reliability of the subsequent switching process.

[0053] 2. Increase the timeout period of the extended architecture (XA).

[0054] Among them, the XA protocol is a commonly used standard in distributed transaction processing. XA transactions involve the coordinated operation of multiple resources (such as multiple databases or other transactional resources). When a distributed transaction starts, it goes through multiple stages, among which the "prepare" stage is a critical link. In this stage, the transaction manager sends a "prepare" request to each resource manager, and the resource manager executes part of the transaction and feeds back the results to the transaction manager.

[0055] GTID is used to uniquely identify a transaction. In a master-slave environment, the first cluster and the second cluster need to maintain transaction consistency, including transaction order, content, etc.

[0056] In the "prepare" phase, if a problem occurs (such as network failure, resource failure, etc.), the second cluster cannot complete the preparation operation and respond to the transaction coordinator (first cluster related component) within the XA timeout period. The second cluster will misjudge that there is a problem with the transaction in the first cluster (in fact, there is no problem), and then start the distributed transaction compensation mechanism, resulting in changes in the transaction execution order or the introduction of additional transactions, which in turn leads to inconsistent GTIDs between the master and the slave, causing cluster switching failure.

[0057] Therefore, by increasing the XA timeout period, sufficient time can be provided for the transaction preparation process, reducing the possibility of the second cluster performing distributed transaction compensation, thereby ensuring the accuracy of cluster switching.

[0058] Optionally, in order to ensure successful cluster switching, it is also necessary to pre-determine whether the first cluster and the second cluster meet the cluster switching condition, so that when the cluster switching condition is met, the parameters of the cluster switching process are configured to generate configuration parameters.

[0059] It should be understood that the cluster switching conditions will be explained in the following embodiments and will not be repeated here.

[0060] S12. Back up Zookeeper metadata.

[0061] Optionally, the Zookeeper metadata includes node information of the database cluster (the first cluster and the second cluster), node health status information, cluster size information, data sharding information, GTID-related information, and consistency protocol-related parameters.

[0062] It should be understood that the Zookeeper metadata may also include other information for indicating the current status of the first cluster and the second cluster, so that the first cluster and the second cluster can be restored to the current status according to the Zookeeper metadata. The data contained in the Zookeeper metadata can be determined according to actual conditions, and the embodiments of the present application do not impose specific restrictions on this.

[0063] Optionally, after backing up the Zookeeper metadata, the backed-up Zookeeper metadata may be stored in a preset path.

[0064] S13. In the process of performing cluster switching between the first cluster and the second cluster according to the configuration parameters, if a switching failure is detected, a rollback is performed according to the Zookeeper metadata.

[0065] The first cluster is a primary cluster, and the second cluster is a secondary cluster.

[0066] It should be understood that switching failure includes deadlock, infinite loop, interruption, error, etc.

[0067] Rollback refers to restoring the states of the first cluster and the second cluster to a previous point in time.

[0068] In a possible implementation, when a switching failure occurs during cluster switching between the first cluster and the second cluster, the stored Zookeeper metadata can be obtained from a preset path, and a rollback is performed based on the Zookeeper metadata to restore the first cluster and the second cluster to a state before the switching.

[0069] The cluster switching method of the distributed database cluster provided in the embodiment of the present application generates configuration parameters by configuring the parameters of the cluster switching process. Afterwards, the Zookeeper metadata is backed up. In the process of cluster switching between the first cluster and the second cluster according to the configuration parameters, if the switching failure is detected, the Zookeeper metadata is rolled back. Among them, the configuration parameters can reduce the possibility of failure of the table structure check during the cluster switching process, and reduce the possibility of distributed transaction compensation by the standby cluster, the first cluster is the main cluster, and the second cluster is the standby cluster. In this technical solution, by cluster switching between the first cluster and the second cluster according to the configuration parameters, the cluster switching process is simplified, sufficient time is provided for the transaction preparation process, and the success rate and reliability of the switching process are improved. Further, the Zookeeper metadata is backed up so that when there is a problem with the switching, the Zookeeper metadata can be rolled back so that the first cluster and the second cluster are restored to the state before the switching, further improving the reliability of the cluster switching.

[0070] Figure 2 Schematic diagram of the cluster switching method of the distributed database cluster provided in this application Figure 2 ,like Figure 2 As shown, the cluster switching method of the distributed database cluster includes the following steps:

[0071] S11. Configure parameters of the cluster switching process to generate configuration parameters.

[0072] It should be understood that the implementation method and process of this step have been Figure 1 The above explanation has been given in the illustrated embodiment and will not be described in detail here.

[0073] S12. Back up Zookeeper metadata.

[0074] It should be understood that the implementation method and process of this step have been Figure 1 The above explanation has been given in the illustrated embodiment and will not be described in detail here.

[0075] S21. Isolate the database instance of the first cluster from the database instance of the second cluster to prevent traffic from entering the first cluster and the second cluster.

[0076] It should be understood that the database instance of the first cluster can be isolated from the database instance of the second cluster at the network level, the database configuration level, and the application level.

[0077] By isolating the database instance of the first cluster from the database instance of the second cluster, it is possible to avoid traffic continuously entering the database, which may cause data inconsistency, reduce interference with cluster switching operations, and avoid data corruption caused by conflicts between switching operations and traffic operations. At the same time, when problems occur during cluster switching, it is easier to troubleshoot when the database instance of the first cluster is isolated from the database instance of the second cluster, and it is easier to restore to the original state if the switching fails.

[0078] S22: After performing cluster switching on the first cluster and the second cluster, verify the switched first cluster and the switched second cluster to determine whether there is a problem in the disaster recovery switching process.

[0079] Exemplarily, assuming that the first cluster is cluster A and the second cluster is cluster B, after cluster switching is performed on the first cluster and the second cluster, the first cluster after switching is cluster A and the second cluster after switching is cluster B.

[0080] When performing cluster switching between the first cluster and the second cluster, the purpose is to change the first cluster to a standby cluster and read-only, and change the second cluster to a primary cluster and writeable.

[0081] The verification includes at least one of the following:

[0082] 1. Check whether the master-slave relationship of the first cluster after switching is the standby cluster, and whether the master-slave relationship of the second cluster after switching is the master cluster.

[0083] 2. Check whether the delay of the first cluster after switching and the delay of the second cluster after switching are both less than the preset delay.

[0084] 3. Check whether the GTID of the first cluster after switching is consistent with the GTID of the second cluster after switching.

[0085] 4. Verify whether the read / write status of the first cluster after switching is read-only, and whether the read / write status of the second cluster after switching is writable.

[0086] 5. Verify whether the slave database status of the first cluster after switching is a slave database, and whether the read / write status of the second cluster after switching is a master database.

[0087] Check the data consistency, master-slave relationship, latency, read-write status, etc. of the switched first cluster and the switched second cluster to ensure that the database has complete data and normal functions before providing external services.

[0088] S23. If the verification fails, roll back according to Zookeeper metadata.

[0089] Optionally, the stored Zookeeper metadata may be obtained from a preset path, and a rollback may be performed according to the Zookeeper metadata to restore the first cluster and the second cluster to a state before the switch.

[0090] S24: If the verification passes, the isolation of the switched first cluster and the switched second cluster is released.

[0091] After the verification is passed, it means that external services can be provided, so it is necessary to release the isolation of the switched first cluster and the switched second cluster to ensure the smooth implementation of the external services.

[0092] Optionally, the attributes of the model in the CMDB may be modified to achieve the purpose of checking whether a specific distributed database connection (DCN) resource exists in the configuration management database (CMDB), creating a new resource if it does not exist, and updating the attribute information of the resource if it does exist.

[0093] In the above embodiment, before cluster switching, the database instance of the first cluster is isolated from the database instance of the second cluster to prevent traffic from entering the first cluster and the second cluster, thereby ensuring smooth cluster switching. At the same time, after cluster switching, data consistency, master-slave relationship, latency, read-write status, etc. are checked to ensure that the database has complete data and normal functions before providing external services.

[0094] Optionally, in some embodiments, the cluster switching condition includes at least one of the following:

[0095] 1. The tables of the first cluster and the second cluster have the same structure and are available.

[0096] Optionally, it may also be required that the table of the first cluster and the table of the second cluster have the same name.

[0097] 2. The master-slave relationship of the first cluster is the master cluster, and the master-slave relationship of the second cluster is the standby cluster.

[0098] 3. The delay of the first cluster and the delay of the second cluster are both less than the preset delay.

[0099] 4. The GTID of the first cluster is consistent with the GTID of the second cluster.

[0100] 5. The read-write status of the first cluster is writable, and the read-write status of the second cluster is read-only.

[0101] 6. The instance alias of the first cluster is consistent with the instance alias of the second cluster.

[0102] 7. The synchronization relationship between the first cluster and the second cluster is in normal mode.

[0103] Among them, the normal mode is a relatively basic and simple synchronization mode. It mainly refers to the primary cluster passing data changes (such as insert, update, delete operations, etc.) to the standby cluster in a certain way to achieve data backup and failover capabilities in some cases. Compared with other more advanced or special synchronization modes (such as strong consistency synchronization mode, asynchronous optimization mode, etc.), the normal mode may have different characteristics in terms of data consistency guarantee, synchronization frequency, and occupation of system resources.

[0104] 8. The first cluster and the second cluster have no ongoing large transactions, and the existing large transactions have been completely synchronized, and the resources related to the large transactions have been released.

[0105] A large transaction refers to a transaction that includes multiple database operations (such as insert, update, delete, etc.) in the database and has a large amount of operation data or a long execution time.

[0106] 9. The database instances of the first cluster and the database instances of the second cluster turn off the transaction marking mode.

[0107] 10. The first cluster has no long transactions

[0108] A long transaction refers to a transaction that lasts for a long time in the database.

[0109] 11. The heartbeats of the master node and the standby node in the second cluster are normal.

[0110] 12. The content format of the / etc / hosts file of the control node and data node of the first cluster meets the preset format.

[0111] The preset format includes whether there are no duplicate lines in the / etc / hosts file, whether the number of lines between #BEGIN tdsql_zk_hosts and #END tdsql_zk_hosts in the file meets the requirement (for example, 1 line, which can be set according to the actual situation), and whether the number of lines between #BEGIN tdsql_zk_hosts and #END tdsql_zk_hosts meets the requirement (for example, 5 or 7, which can be set according to the actual situation).

[0112] 13. The database parameters of the first cluster are consistent with the database parameters of the second cluster.

[0113] 14. The monitoring library information on zk in the first cluster and the second cluster is correct.

[0114] 15. The database instances of the first cluster and the database instances of the second cluster do not use the event scheduler (English: event_scheduler) function.

[0115] 16. Database bridge (English: DBbridge) is used.

[0116] 17. There are read-only nodes on zk.

[0117] 18. The heartbeat between the first cluster and the second cluster is normal.

[0118] Next, the disaster recovery switching process is explained in terms of pre-switching backup, cluster switching, and post-switching inspection through several specific embodiments, and the overall disaster recovery switching process is explained through an overall flow.

[0119] Figure 3 The flowchart of the pre-switching backup process provided by the embodiment of the present application is as follows. Figure 3 As shown, the backup process before switching may include the following steps:

[0120] S311 , query all databases (Database, DB) of the first cluster.

[0121] S312: Query all DBs of the second cluster.

[0122] S321. Set not to check the table structure of the first cluster.

[0123] S322: Set not to check the table structure of the second cluster.

[0124] S331. Increase the XA timeout duration of the first cluster.

[0125] S332. Increase the XA timeout duration of the second cluster.

[0126] S34. Back up Zookeeper metadata.

[0127] Figure 4 The schematic diagram of the cluster switching process provided in the embodiment of the present application is as follows. Figure 3 As shown, the cluster switching process may include the following steps:

[0128] S41. Cluster switching.

[0129] If successful, execute S42; if the switch is stuck, execute S44.

[0130] S42: Query the DB of the second cluster after switching.

[0131] S43: Restore the XA timeout duration of the second cluster after switching.

[0132] That is, the XA timeout duration of the second cluster after switching is restored to the value before being increased in S332.

[0133] S44. Roll back according to Zookeeper metadata.

[0134] S45: Query the DB of the first cluster after switching.

[0135] S46: Restore the XA timeout duration of the first cluster after the switch.

[0136] That is, the XA timeout duration of the first cluster after the switch is restored to the value before the increase in S331.

[0137] Figure 5 The following is a flow chart of the post-switching inspection process provided in the embodiment of the present application. Figure 5 As shown, the post-switching inspection process includes the following steps:

[0138] S51 : Checking the master-slave relationship between the switched first cluster and the switched second cluster and their respective delays.

[0139] S52. Verify data consistency.

[0140] S531 , query the DB master database of the first cluster after switching.

[0141] S532: Check whether the first cluster after switching is a standby cluster, and whether the read-write state is read-only.

[0142] S533: Check whether the slave database status of the first cluster after switching is a slave database.

[0143] S541. Query the DB master database of the second cluster after switching.

[0144] S542: Check whether the switched second cluster is the main cluster and whether the read / write status is writable.

[0145] S543: Check whether the slave database status of the second cluster after switching is the master database.

[0146] Figure 6 A schematic diagram of a cluster switching method for a distributed database cluster provided in an embodiment of the present application. Figure 6 As shown, when performing cluster switching, if the switching reports an error or times out, the switching is rolled back; if the switching is successful, it ends directly.

[0147] The cluster switching process involves the following seven aspects:

[0148] Aspect 1: Check before switching: 18 checks are performed, including table structure check, master-slave delay check, synchronization mode check, data consistency check, master-slave cluster read and write and large transaction status check.

[0149] Aspect 2: Backup metadata: Metadata includes the primary-backup replication relationship and cluster architecture metadata.

[0150] Aspect 3: Isolate instances: Disconnect application and database links to ensure database quiescence.

[0151] Aspect 4: Database switching: Modify the master-slave relationship between the first cluster and the second cluster, switch the first cluster from read-write to read-only, and switch the second cluster from read-only to read-write.

[0152] Aspect 5: Check after switching: data consistency check and read-write status check.

[0153] Aspect 6: Lift isolation: allow business traffic to enter.

[0154] Aspect 7: Configuration update: Update the disaster recovery relationship of the CMDB configuration model.

[0155] It should be understood that Figure 3-Figure 6 The illustrated embodiment has been explained in the above embodiments and will not be described again in detail here.

[0156] It should be understood that this technical solution is an atomic multi-shard high-reliability switching solution for distributed databases. Starting from the entire inter-cluster switching process before, during, and after the switch, the switching solution is developed based on the distributed database and the integrated operation and maintenance platform, creating a more stable and more available one-click switching and automatic rollback process:

[0157] For the pre-switching stage, we continue to improve the inspection mechanism and conduct various checks on the table structure, data consistency, latency, etc. between the primary and standby clusters to ensure that the database is ready to be switched at any time; at the same time, we back up the database instance metadata so that it can be used for rollback after a subsequent switching exception.

[0158] For the switching process, we conduct in-depth research on the switching principle and implement abnormal rollback of switching. When an error occurs, the rollback is automatically triggered based on the backed-up metadata.

[0159] For the post-switching inspection, data consistency, master-slave cluster relationship, latency, and read-write status are checked to ensure that the database has complete data and normal functions before providing external services; and the configuration information after the switch is updated.

[0160] It should be understood that the entire switching process solves problems such as high risk of switching multiple nodes at the same time, many potential failure points in numerous components, many bottlenecks, and non-atomic switching, and achieves the effect of being able to switch the database, daring to switch, and easy to switch.

[0161] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0162] Figure 7 This is a schematic diagram of the structure of a cluster switching device for a distributed database cluster provided in an embodiment of the present application. Figure 7 As shown, the cluster switching device 70 of the distributed database cluster includes:

[0163] The configuration module 71 is used to configure the parameters of the cluster switching process and generate configuration parameters. The configuration parameters can reduce the possibility of failure of table structure check during the cluster switching process and reduce the possibility of distributed transaction compensation by the standby cluster.

[0164] The backup module 72 is used to back up the Zookeeper metadata.

[0165] The rollback module 73 is used to roll back according to the Zookeeper metadata if a switch failure is detected during the cluster switch between the first cluster and the second cluster according to the configuration parameters, with the first cluster being the main cluster and the second cluster being the standby cluster.

[0166] In a possible implementation, the configuration module 71 is specifically configured to:

[0167] Configure the check mode to the target mode, in which the table structure is not checked.

[0168] Increase the XA timeout period.

[0169] In a possible implementation, after performing cluster switching on the first cluster and the second cluster, the cluster switching device 70 of the distributed database cluster further includes:

[0170] The verification module is used to verify the switched first cluster and the switched second cluster to determine whether there is a problem in the disaster recovery switching process.

[0171] The rollback module 73 is also used to roll back according to the Zookeeper metadata if the verification fails.

[0172] In a possible implementation, the verification includes at least one of the following:

[0173] It is checked whether the master-slave relationship of the first cluster after the switch is a standby cluster, and whether the master-slave relationship of the second cluster after the switch is a master cluster.

[0174] It is checked whether the delay of the first cluster after switching and the delay of the second cluster after switching are both less than the preset delay.

[0175] Verify whether the GTID of the first cluster after switching is consistent with the GTID of the second cluster after switching.

[0176] It is checked whether the read / write status of the first cluster after switching is read-only, and whether the read / write status of the second cluster after switching is writable.

[0177] Verify whether the slave database status of the first cluster after switching is a slave database, and whether the read / write status of the second cluster after switching is a master database.

[0178] In a possible implementation, after backing up the Zookeeper metadata, the cluster switching device 70 of the distributed database cluster further includes:

[0179] The isolation module is used to isolate the database instance of the first cluster from the database instance of the second cluster to prevent traffic from entering the first cluster and the second cluster.

[0180] Correspondingly, after verifying the switched first cluster and the switched second cluster, the isolation module is further used to:

[0181] If the verification passes, the isolation of the switched first cluster and the switched second cluster is released.

[0182] In a possible implementation, the configuration module 71 is specifically configured to:

[0183] It is determined whether the first cluster and the second cluster meet a cluster switching condition.

[0184] If satisfied, the parameters of the cluster switching process are configured to generate configuration parameters.

[0185] In a possible implementation, the cluster switching condition includes at least one of the following:

[0186] The table structure of the first cluster and the table structure of the second cluster are identical and available.

[0187] The master-slave relationship of the first cluster is the master cluster, and the master-slave relationship of the second cluster is the standby cluster.

[0188] The delay of the first cluster and the delay of the second cluster are both less than the preset delay.

[0189] The GTID of the first cluster is consistent with the GTID of the second cluster.

[0190] The read-write status of the first cluster is writable, and the read-write status of the second cluster is read-only.

[0191] The instance alias of the first cluster is consistent with the instance alias of the second cluster.

[0192] The cluster switching device for a distributed database cluster provided in an embodiment of the present application may be used to execute the cluster switching method for a distributed database cluster in any of the above embodiments, and its implementation principle and technical effects are similar and will not be described in detail herein.

[0193] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0194] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 81 and a memory 82. Optionally, the electronic device 80 also includes a communication component 83. The processor 81, the memory 82 and the communication component 83 are connected via a bus 84.

[0195] In the specific implementation process, at least one processor 81 executes the computer execution instructions stored in the memory 82, so that at least one processor 81 executes the above method.

[0196] The specific implementation process of the processor 81 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0197] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0198] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0199] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0200] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0201] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0202] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0203] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0204] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0208] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0209] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A cluster switching method for a distributed database cluster, characterized in that: include: Configuring parameters of the cluster switching process to generate configuration parameters, wherein the configuration parameters can reduce the possibility of failure of table structure check during the cluster switching process and reduce the possibility of distributed transaction compensation by the standby cluster; Back up Zookeeper metadata; In the process of performing cluster switching between the first cluster and the second cluster according to the configuration parameters, if switching failure is detected, rollback is performed according to the Zookeeper metadata, and the first cluster is the main cluster and the second cluster is the standby cluster.

2. The method according to claim 1, characterized in that The configuring of the parameters of the cluster switching process includes: Configure the check mode to be a target mode, in which the table structure is not checked; Increase the extended architecture XA timeout period.

3. The method according to claim 1 or 2, characterized in that: After performing cluster switching on the first cluster and the second cluster, the method further includes: Verify the first cluster after switching and the second cluster after switching to determine whether there are any problems in the disaster recovery switching process; If the verification fails, rollback is performed according to the Zookeeper metadata.

4. The method according to claim 3, characterized in that The verification includes at least one of the following: Verifying whether the master-slave relationship of the first cluster after the switch is a standby cluster, and whether the master-slave relationship of the second cluster after the switch is a master cluster; Verifying whether the delay of the first cluster after the switching and the delay of the second cluster after the switching are both less than a preset delay; Check whether the global transaction identifier GTID of the first cluster after the switch is consistent with the GTID of the second cluster after the switch; Verify whether the read / write status of the first cluster after the switch is read-only, and whether the read / write status of the second cluster after the switch is writable; It is checked whether the slave database state of the first cluster after the switching is a slave database, and whether the read / write state of the second cluster after the switching is a master database.

5. The method according to claim 3, characterized in that: After backing up the Zookeeper metadata, the method further includes: Isolating the database instance of the first cluster from the database instance of the second cluster to prevent traffic from entering the first cluster and the second cluster; Accordingly, after verifying the switched first cluster and the switched second cluster, the method further includes: If the verification passes, the isolation of the switched first cluster and the switched second cluster is released.

6. The method according to claim 1 or 2, characterized in that: The step of configuring the parameters of the cluster switching process to generate the configuration parameters includes: Determining whether the first cluster and the second cluster meet a cluster switching condition; If the conditions are met, the parameters of the cluster switching process are configured to generate the configuration parameters.

7. The method according to claim 6, characterized in that The cluster switching condition includes at least one of the following: The table structure of the first cluster is the same as the table structure of the second cluster and is available; The master-slave relationship of the first cluster is a master cluster, and the master-slave relationship of the second cluster is a standby cluster; The delay of the first cluster and the delay of the second cluster are both less than a preset delay; The GTID of the first cluster is consistent with the GTID of the second cluster; The read / write status of the first cluster is writable, and the read / write status of the second cluster is read-only; The instance alias of the first cluster is consistent with the instance alias of the second cluster.

8. A cluster switching device for a distributed database, characterized in that: include: A configuration module is used to configure parameters of the cluster switching process and generate configuration parameters, wherein the configuration parameters can reduce the possibility of failure of table structure check during the cluster switching process and reduce the possibility of distributed transaction compensation by the standby cluster; Backup module, used to back up Zookeeper metadata; The rollback module is used to roll back according to the Zookeeper metadata if a switch failure is detected during the cluster switch between the first cluster and the second cluster according to the configuration parameters, wherein the first cluster is the main cluster and the second cluster is the standby cluster.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images