Distributed database cluster splitting and merging method and device

Abstract

The invention discloses a method and a device for splitting and merging a distributed database cluster. The method comprises the following steps: acquiring information of a distributed database cluster to be split; wherein the information comprises database access times, time delay and the number of nodes; the nodes comprise a CN, a GTM and a DN; receiving a cluster splitting request; distributing splitting nodes based on the splitting request and the information of the distributed database cluster; and splitting the distributed database cluster based on the allocated splitting nodes. Through the processing scheme disclosed by the invention, splitting and merging commands are completed by one key, the user operation is simple, and node redundancy is avoided.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method and device for splitting and merging a distributed database cluster. Background Art

[0002] Distributed databases generally include three types of nodes: CN, GTM, and DN (such as Figure 1 shown): CN: Responsible for receiving client requests, performing SQL parsing and optimization, generating execution plans, and coordinating data nodes for data query and writing; DN: Completes data storage and local data query and writing; GTM: Generates and maintains global timestamps to ensure strong consistency of cluster data.

[0003] DN has multiple high availability groups (shards). A high availability group includes multiple replicas, including a primary node and multiple standby nodes. The primary node will synchronize logs to the standby node to keep data synchronized. If the primary DN fails, the primary-standby switchover can be triggered to upgrade an optimal standby DN to the primary DN to ensure the continuity of cluster services.

[0004] To ensure high availability, GTM also includes multiple replicas, including a master node and multiple standby nodes. The master GTM synchronizes data to the standby GTM. If the master GTM fails, it can also trigger a master-standby switchover and upgrade an optimal standby GTM to the master GTM.

[0005] Multiple CN nodes are deployed in a cluster, and the data (DDL data) between each CN is synchronized in real time. Users can choose any CN to conduct business. When a CN fails, users can choose to access from another CN.

[0006] From the above introduction, we can see that in most distributed clusters, each node has a certain degree of redundancy to ensure high availability. In some cases, users want to split a distributed cluster into multiple clusters for different business uses, and also want to merge multiple clusters into one cluster to provide external services.

[0007] The existing cluster splitting has the following problems: 1. The user will only give instructions on how to split a cluster, and how to split a cluster into one or more optimal clusters needs to be automatically determined.

[0008] 2. During the split process, some users are allowed to stop the business, while others are not. If stopping the business is not allowed, how to ensure the data consistency of the split cluster?

[0009] 3. There are many specific splitting and merging steps, and a one-click command needs to be provided to facilitate user operation. Summary of the invention

[0010] In view of this, an embodiment of the present disclosure provides a distributed database cluster splitting method, which at least partially solves the problems existing in the prior art.

[0011] In a first aspect, an embodiment of the present disclosure provides a distributed database cluster splitting method, the method comprising the following steps: Obtain information about the distributed database cluster to be split; wherein the information includes the number of database accesses, latency, and number of nodes; the nodes include CN, GTM, and DN; Receive a cluster split request; Allocating a split node based on the split request and information of the distributed database cluster; The distributed database cluster is split based on the allocated split nodes.

[0012] According to a specific implementation of an embodiment of the present disclosure, the allocating a split node based on the split request and the information of the distributed database cluster includes: Sort the CNs from small to large based on the number of database accesses; wherein the CNs are obtained based on the number of concurrent connections of the first cluster in the split request × 120; Sort the DN shard copies according to the data synchronization delay from large to small. The larger the delay, the higher the DN split weight; Analyze the network latency of the CN and DN shard copies obtained by the first cluster. The smaller the latency, the higher the DN split weight. Calculate the DN shard copy split priority based on the DN split weight; Sort the GTM shard copies by data synchronization latency from large to small. The greater the latency, the higher the GTM split weight. Analyze the network latency between the CN and GTM shard copies obtained by the first cluster. The smaller the latency, the higher the GTM split weight. The GTM shard copy split priority is calculated based on the GTM split weight.

[0013] According to a specific implementation manner of an embodiment of the present disclosure, the distributed database cluster is split based on the allocated splitting nodes, including: a service-stopping splitting method and a service-non-stopping splitting method.

[0014] According to a specific implementation of an embodiment of the present disclosure, the service shutdown and splitting method includes the following steps: Resolve the number of CNs, GTM copies, and DN copies required for the second cluster based on the allocated split nodes, and keep the remaining CNs, GTM copies, and DN copies in the first cluster; Stop all nodes included in the second cluster; In the first cluster, remove the nodes of the second cluster and retain the node data; Modify the connection string information of the GTM and DN nodes of the first cluster, and delete the replication slot from the primary node to the corresponding node of the second cluster; Deleting the routing information from all CNs in the first cluster to the CNs in the second cluster; Modify the connection string information of the GTM and DN nodes of the second cluster; Start the master node and standby node of the second cluster based on the allocated split nodes, establish master-standby connections respectively, and start synchronizing data; Start the CN of the second cluster and modify the routes of CN and GTM to DN.

[0015] According to a specific implementation of the embodiment of the present disclosure, the non-stop service splitting method includes the following steps: Increase the number of log files retained; Record the wal log position of each shard; Remove the nodes of the second cluster from the first cluster and retain the node data; Modify the connection string information of the GTM and DN nodes of the first cluster, and delete the replication slot from the primary node to the corresponding node of the second cluster; Deleting the routing information from all CNs in the first cluster to the CNs in the second cluster; Splitting the nodes corresponding to the second cluster from the first cluster, and after the splitting is completed, triggering the first cluster to create a data consistency point; Collect log files of each shard and send the log files to the second cluster; Start the second cluster master node and set the startup log target to lsn_end; Start the master node and standby node of the second cluster based on the allocated split nodes, establish master-standby connections respectively, and start synchronizing data; Start the CN of the second cluster and modify the routes of CN and GTM to DN.

[0016] According to a specific implementation of the embodiment of the present disclosure, the method further includes a distributed database cluster merging method, and the method includes the following steps: Obtaining node information of a distributed database cluster to be merged: the distributed database cluster to be merged includes a first cluster and a second cluster; Stop all nodes of the second cluster; Set the node connection string information of the second cluster to connect to the master node of the first cluster; Trigger the full build data of all GTM and DN of the second cluster, with the source end being the corresponding master node of the first cluster; Trigger the full build data of the CN of the second cluster, the source end is any CN of the first cluster; Modify the GTM on the first cluster and the backup machine information on the DN master node, and establish a data replication slot to the corresponding node of the second cluster; Restart all GTM and DN nodes on the second cluster as standby nodes, and start synchronizing data based on the data replication slot; Modify the routing information of all CNs in the first cluster and add routing to the CNs in the second cluster; Restart the routing of the second cluster CN.

[0017] In a second aspect, an embodiment of the present disclosure further provides a distributed database cluster splitting device, the device comprising: A data acquisition module is configured to acquire information of the distributed database cluster to be split; wherein the information includes the number of database accesses, latency, and number of nodes; the nodes include CN, GTM, and DN; A rule module is configured to receive a cluster split request; allocate a split node based on the split request and information of the distributed database cluster; The node splitting module is configured to split the distributed database cluster based on the allocated splitting nodes.

[0018] According to a specific implementation of the embodiment of the present disclosure, the device further includes: A node merging module is configured to obtain node information of a distributed database cluster to be merged: the distributed database cluster to be merged includes a first cluster and a second cluster; Stop all nodes of the second cluster; Set the node connection string information of the second cluster to connect to the master node of the first cluster; Trigger the full build data of all GTM and DN of the second cluster, with the source end being the corresponding master node of the first cluster; Trigger the full build data of the CN of the second cluster, the source end is any CN of the first cluster; Modify the GTM on the first cluster and the backup machine information on the DN master node, and establish a data replication slot to the corresponding node of the second cluster; Restart all GTM and DN nodes on the second cluster as standby nodes, and start synchronizing data based on the data replication slot; Modify the routing information of all CNs in the first cluster and add routing to the CNs in the second cluster; and, Restart the routing of the second cluster CN.

[0019] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor implements the distributed database cluster splitting method described in any one of the first aspect or any one of the implementations of the first aspect.

[0020] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the distributed database cluster splitting method in the aforementioned first aspect or any implementation of the first aspect.

[0021] The distributed database cluster splitting method in the embodiment of the present disclosure can complete the splitting and merging commands with one click, which is simple for users to operate and avoids node redundancy; and can simultaneously support both shutdown splitting and non-shutdown splitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0023] Figure 1 This is a schematic diagram of a distributed database architecture; Figure 2 A flowchart of a distributed database cluster splitting method provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of a distributed database cluster splitting method provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a distributed database cluster provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of a distributed database cluster split result provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of a distributed database cluster split result provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of the structure of a distributed database cluster splitting device provided in an embodiment of the present disclosure; Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0025] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0026] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0027] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0028] The embodiment of the present invention provides a distributed database cluster splitting method, in which splitting and merging commands are completed with one key, user operation is simple, and node redundancy is avoided.

[0029] The present invention solves the problems existing in the prior art through three modules, such as Figure 2 As shown, there are cluster splitting algorithm module, cluster splitting execution module and cluster merging execution module. The main responsibilities of each module are as follows: Cluster splitting algorithm module: collects information about the database cluster, including the number of database accesses, shard copy latency, etc.; receives cluster splitting requests from users, refers to the database cluster information, and automatically allocates nodes to each cluster after splitting according to a certain algorithm. Input the node information of each cluster after splitting to the cluster splitting module.

[0030] Cluster splitting execution module: receives the cluster splitting request from the cluster algorithm module and executes the cluster splitting operation; it is responsible for ensuring data consistency during the cluster splitting process.

[0031] Cluster merging module: receives user cluster splitting and merging requests and performs cluster merging operations.

[0032] Take the following cluster as an example. The cluster has three CNs, namely CN1, CN2, and CN3. GTM has four replicas. There are three high availability groups, DN1, DN2, and DN3, and each high availability group has four replicas. For GTM and high availability groups, since both have four replicas, two synchronous standby machines and one asynchronous standby machine are set to meet the majority principle.

[0033] like Figure 3 As shown, at step S310, information of the distributed database cluster to be split is obtained; wherein the information includes the number of database accesses, latency and number of nodes; the nodes include CN, GTM and DN.

[0034] More specifically, the process proceeds to step S320.

[0035] At step S320 , a cluster splitting request is received.

[0036] More specifically, the process proceeds to step S330.

[0037] In step S330 , a split node is allocated based on the split request and the information of the distributed database cluster.

[0038] More specifically, the factors that affect cluster split node allocation mainly include the following data: the number of concurrent connections to the CN, the synchronization delay of shard replica data, the communication delay between the CN and DN, etc. At the same time, the calculation needs to meet the following basic principles: 1) Minimum requirements for running a single distributed cluster: single GTM copy, 1 CN, and single copy for each DN shard.

[0039] 2) GTM is recommended for high-availability operation of distributed clusters. Each DN shard has three copies. There are two backup machines, one synchronous backup and one asynchronous backup. When splitting, priority is given to ensuring that the old cluster has three copies, and the redundant nodes are removed, but the removed cluster must meet the minimum operation requirements.

[0040] In an embodiment of the present invention, the splitting node is allocated based on the splitting request and the information of the distributed database cluster, including: sorting CN from small to large based on the number of database accesses; wherein, CN is obtained based on the concurrent number of the first cluster × 120 in the splitting request; sorting DN shard copies from large to small according to the data synchronization delay, and the larger the delay, the higher the DN splitting weight; analyzing the network delay between the CN obtained by the first cluster and the DN shard copy, and the smaller the delay, the higher the DN splitting weight; calculating the DN shard copy splitting priority based on the DN splitting weight; sorting the GTM shard copy from large to small according to the data synchronization delay, and the larger the delay, the higher the GTM splitting weight; analyzing the network delay between the CN obtained by the first cluster and the GTM shard copy, and the smaller the delay, the higher the GTM splitting weight; calculating the GTM shard copy splitting priority based on the GTM splitting weight.

[0041] More specifically, the splitting steps are as follows: 1. Continuously collect data such as the number of concurrent CN connections, shard replica data synchronization delay, and CN and DN communication delay from the database cluster.

[0042] 2. Receive the user's cluster split request.

[0043] 3. First, split the CN. The CN splitting refers to the historical connection number, and the historical connection number is sorted from small to large. The smaller the number of connections, the higher the weight of the split. Keep the number of CNs required by cluster 1 (or the first cluster) × 120, and split the remaining CNs to other clusters.

[0044] 4. Analyze the synchronization delay data of the DN shard replica data and sort them from large to small according to the delay. The larger the delay, the higher the weight of the split, and the first weight score is obtained.

[0045] 5. Analyze the network delay between the CN and the DN shard copies of the split CN calculated in step 3. The smaller the delay, the higher the split weight, and the second weight score is obtained.

[0046] 6. Comprehensively consider steps 4 and 5. According to the weights set by the user for 4 and 5, calculate the DN shard replica splitting priority (that is, the sum of the first weight score and the second weight score), and split the replicas to the new cluster (the second cluster) according to the priority and the basic principles.

[0047] The GTM sharding and DN sharding algorithms are the same.

[0048] More specifically, the process proceeds to step S340.

[0049] At step S340, the distributed database cluster is split based on the allocated split nodes.

[0050] More specifically, each DN shard and GTM are used as a multi-copy high availability group, and the data of the host will be synchronized to all standby machines through the replication slot. Therefore, the information of other replicas will be saved on each replica. The standby node will try to connect to the master node based on the saved information of other replicas. After the master node receives the connection request from the standby node, it will establish a replication slot and start the data transmission between the master and standby nodes. Because some replicas are removed to other clusters, it is necessary to traverse all replicas retained in this cluster and delete the saved replica information that needs to be removed.

[0051] As a cluster, each GTM stores the routing information of other CNs and DNs. Each CN retains the routing information of GTMs and DNs. When executing SQL statements, if it is DDL, the execution plan is distributed to all CNs and DNs. If it is DML, the appropriate DN is selected for distribution based on the DN's routing information.

[0052] For each DN shard, CN will save the route to the primary node and the routes to other backup nodes. When the plan is executed, the primary node route is selected first. If the connection to the primary node fails, the backup node will be selected for execution. The advantage of retaining all replica routes is that if the replica switches between the primary and backup nodes, the execution plan can quickly adapt without being notified of the primary and backup switch operation, reducing the SQL execution failure rate. Because some CN and DN copies of the cluster are removed to other clusters, the routing information of the transplanted nodes must be deleted from all the remaining nodes in the cluster. For example, GTM originally retained the routes to CN1, CN2, and CN3. After CN3 is transplanted to other clusters, GTM only retains the routes to CN1 and CN2, and DDL only needs to be distributed to CN1 and CN2.

[0053] Because there are multiple shards in a distributed database, during the splitting process, the speed at which each shard is split varies. This causes a time difference in the time at which data is received by different shards, leading to data inconsistency.

[0054] To solve this problem, the present invention provides a solution to ensure data consistency. Taking cluster 1 (or the first cluster) as an example, cluster 1 and cluster 2 (or the second cluster) are split into cluster 1 and cluster 2. 1) To ensure that the logs of cluster 1 are not recycled during cluster splitting, increase the number of log retention files.

[0055] 2) Before splitting the cluster, the split module records the wal log position of each shard, such as the log position of shard DN1 is lsn_start. The log is recorded in the log file xlog_file_start.

[0056] 3) Perform the splitting work and split the nodes corresponding to cluster 2 from cluster 1.

[0057] 4) After all shard nodes are split from cluster 1, the split module triggers cluster 1 to create a data consistency point and records the location of the data consistency point for each shard. For example, the log location of shard DN1 is lsn_end. The log is recorded in the log file xlog_file_end.

[0058] 5) The splitting module collects the log files of each shard, such as shard DN1, that is, the logs between xlog_file_start and xlog_file_end, and sends this part of the log files to cluster 2.

[0059] 6) Start cluster 2 node and set the startup log target to lsn_end. Each shard log is replayed to lsn_end, and the data consistency can be guaranteed at this point.

[0060] Taking the following cluster as an example, the detailed splitting steps are described as follows: like Figure 4 As shown in the figure, there are three CNs in the cluster, namely CN1, CN2, and CN3. GTM has four replicas, and there are three high availability groups DN1, DN2, and DN3, each with four replicas. For GTM and high availability groups, since both have four replicas, two synchronous standby machines and one asynchronous standby machine are set to meet the majority principle.

[0061] The user wants to split the cluster into two clusters. The cluster splitting algorithm module automatically calculates the node information assigned to the two clusters as follows: Cluster 1: CN: CN1, CN2; GTM: GTM1, GTM2; DN1:DN1_1,DN1_2; DN2:DN2_1,DN2_2; DN3:DN3_1,DN3_2 Cluster 2: CN: CN3; GTM: GTM3, GTM4; DN1:DN1_3,DN1_4; DN2:DN2_3,DN2_4; DN3:DN3_3,DN3_4 Users can choose between two types of splitting: splitting without stopping the service and splitting without stopping the service.

[0062] In the embodiment of the present invention, the distributed database cluster is split based on the allocated splitting nodes, including: a service-stopping splitting method and a service-non-stopping splitting method.

[0063] In an embodiment of the present invention, the service shutdown and splitting method includes the following steps: based on the assigned split node, the number of CNs, the number of GTM copies and the number of DN copies required for the second cluster are parsed, and the remaining CNs, GTM copies and DN copies are retained in the first cluster; all nodes contained in the second cluster are stopped; the nodes of the second cluster are removed in the first cluster, and the node data is retained; the connection string information of the GTM and DN nodes of the first cluster is modified, and the replication slot from the primary node to the corresponding node of the second cluster is deleted; the routing information of all CNs on the first cluster to the CNs of the second cluster is deleted; the connection string information of the GTM and DN nodes of the second cluster is modified; based on the assigned split node, the primary node and the backup node of the second cluster are started, and the primary and backup connections are established respectively to start synchronizing data; the CN of the second cluster is started, and the routing of CN, GTM and DN is modified.

[0064] More specifically, the service shutdown and splitting steps are as follows: 1) The split module first sets the state of cluster 1 to read-only state. The state database can only be read but not written. 2) After waiting for all 2pc transactions to be committed, stop all nodes belonging to cluster 2; 3) Remove the nodes of cluster 2 from cluster 1, but do not delete the node data; 4) Modify the connection string information of cluster 1 GTM and DN nodes, and delete the replication slot from the primary node to the corresponding node in cluster 2.

[0065] 5) Delete all routing information from CN to CN3 in cluster 1. At this point, cluster 1 has completed the processing and can provide services to the outside world.

[0066] 6) Change the status of cluster 1 to normal mode, which is readable and writable 7) Modify the connection string information of the GTM and DN nodes of the new cluster; 8) Start GTM3, DN1_3, DN2_3, and DN3_3 as master nodes respectively; 9) Start GTM4, DN1_4, DN2_4, and DN3_4 as standby nodes, and establish master-slave connections with GTM3, DN1_3, DN2_3, and DN3_3 respectively to start synchronizing data; 10) Start CN3 and modify the routes of GTM and three high availability groups on CN3. The new cluster is started and can provide services to the outside world.

[0067] At this point, the original cluster splits into two independent clusters, such as Figure 5 , Figure 6 As shown, they provide services to the outside world respectively.

[0068] In an embodiment of the present invention, the non-stop splitting method includes the following steps: increasing the number of log retention files; recording the wal log position of each shard; removing the nodes of the second cluster in the first cluster and retaining the node data; modifying the connection string information of the GTM and DN nodes of the first cluster, and deleting the replication slot from the primary node to the corresponding node of the second cluster; deleting the routing information of all CNs on the first cluster to the CN of the second cluster; splitting the nodes corresponding to the second cluster from the first cluster, and after the splitting is completed, triggering the first cluster to create a data consistency point; collecting the log files of each shard and sending the log files to the second cluster; starting the master node of the second cluster and setting the startup log target to lsn_end; starting the master node and standby node of the second cluster based on the allocated split nodes, and establishing master-standby connections respectively, and starting data synchronization; starting the CN of the second cluster and modifying the routing of CN and GTM with DN.

[0069] More specifically, the steps for splitting without stopping the service are: To ensure that the logs of cluster 1 are not recycled during cluster splitting, increase the number of log retention files.

[0070] The split module records the current log position of each shard's wal. For example, the log position of shard DN1 is lsn_start. The log is recorded in the log file xlog_file_start.

[0071] In cluster 1, remove the nodes of cluster 2, but do not delete the node data. Modify the connection string information of cluster 1 GTM and DN nodes, and delete the replication slot from the primary node to the corresponding node in cluster 2.

[0072] Delete all routing information from CN1 to CN3 in cluster 1.

[0073] After all shard nodes are split from cluster 1, the split module triggers cluster 1 to create a data consistency point and records the location of the data consistency point for each shard. For example, the log location of shard DN1 is lsn_end. The log is recorded in the log file xlog_file_end.

[0074] The splitting module collects the log files of each shard, such as shard DN1, that is, the logs between xlog_file_start and xlog_file_end, and sends this part of the log files to cluster 2.

[0075] Modify the connection string information of the GTM and DN nodes of the new cluster; Start GTM3, DN1_3, DN2_3, and DN3_3 as master nodes respectively, and set the startup log target to lsn_end. Play back each shard log to lsn_end, and the data at this point can be guaranteed to be consistent; Start GTM4, DN1_4, DN2_4, and DN3_4 as standby nodes, and establish master-slave connections with GTM3, DN1_3, DN2_3, and DN3_3 respectively to start synchronizing data; Start CN3 and modify the GTM on CN3 and the routes of the three high availability groups. The new cluster is started and can provide services to the outside world.

[0076] At this point, the original cluster is split into the following two independent clusters, each providing services to the outside world.

[0077] In an embodiment of the present invention, the method also includes: determining whether the distributed database cluster after splitting can be split again, including: determining whether the number of nodes in the distributed cluster to be split meets a preset standard; wherein, when the preset standard is met, determining that the distributed cluster can be split; when the preset standard is not met, determining that the distributed cluster cannot be split.

[0078] In the embodiment of the present invention, the preset standard is that the distributed cluster is no less than a single copy of GTM, 1 CN, and a single copy of each DN shard.

[0079] In an embodiment of the present invention, the method also includes a distributed database cluster merging method, which includes the following steps: obtaining node information of the distributed database clusters to be merged: the distributed database clusters to be merged include a first cluster and a second cluster; stopping all nodes of the second cluster; setting the node connection string information of the second cluster, connecting to the master node of the first cluster; triggering the full build data of all GTMs and DNs of the second cluster, the source end is the corresponding master node of the first cluster; triggering the full build data of the CN of the second cluster, the source end is any CN of the first cluster; modifying the GTM on the first cluster and the standby machine information on the DN master node, and establishing a data replication slot to the corresponding node of the second cluster; restarting all GTMs and DN nodes on the second cluster as standby nodes, and starting to synchronize data between the master and standby based on the data replication slot; modifying the routing information of all CNs on the first cluster, adding routes to the CNs of the second cluster; restarting the routes of the CNs of the second cluster.

[0080] More specifically, the node connection string information of the second cluster is set, that is, all GTM and DN nodes are traversed, and the copy information stored therein is modified to be the collection of the copies of the first cluster and the second cluster.

[0081] All GTM and DN nodes of the first cluster need to be added with the replica information of the second cluster. When the backup machine of the second cluster connects to the primary node of the first cluster, the primary node will establish a replication slot to the replica and start data synchronization.

[0082] The CN of the second cluster also needs to be modified to add the routing information of the first cluster. After the nodes of the second cluster are built from the data of the first cluster, the data of the second cluster has been synchronized with the first cluster. Because the second cluster is added, the GTM and DN shards of the first cluster already have a master node, so the GTM and DN nodes of the second cluster are pulled up as backups. After pulling up, the master node will establish a replication slot with the backup node and start master-backup data synchronization.

[0083] After the new service starts, because the routing information of the cluster has been merged with the second cluster, both DML and DDL will be distributed to the corresponding nodes.

[0084] For example, to merge two independent clusters into one cluster, the tool command format is as follows: merge_cluster cluster1 cluster2 Merge cluster2 into cluster1. If cluster2 is merged into cluster1, all data in cluster2 will be discarded and the data on cluster1 will prevail.

[0085] The processing steps of merge_cluster are: 1: Get the node information of the two clusters: 2: First, stop all nodes of cluster2; 3: Set the node connection string information of cluster2 cluster and connect to the master node of cluster1; 4: Trigger the full build data of all GTM and DN of cluster2, with the source end being the corresponding master node of cluster1; 5: Trigger the full build data of CN of cluster2, the source end is any CN of cluster1; 6: Modify the GTM on cluster1 and the standby information on the DN master node, and establish a data replication slot to the corresponding node of cluster2; 7: Restart all GTM and DN nodes on cluster2 as standby nodes. Since the replication slot has been established, the master and standby nodes start synchronizing data. 8. Modify the routing information of all CNs on cluster1 and add routing to CN3; 9. Restart CN3.

[0086] At this point, the two independent clusters are merged into one cluster to provide external services together.

[0087] Figure 7 The distributed database cluster splitting device 700 provided by the present invention is shown, including a data acquisition module 710 , a rule module 720 and a node splitting module 730 .

[0088] The data acquisition module 710 acquires information of the distributed database cluster to be split; wherein the information includes the number of database accesses, latency, and number of nodes; the nodes include CN, GTM, and DN; The rule module 720 is used to receive a cluster split request; allocate a split node based on the split request and information of the distributed database cluster; The node splitting module 730 is used to split the distributed database cluster based on the allocated splitting nodes.

[0089] In an embodiment of the present invention, the device also includes: a node merging module, which is configured to obtain node information of the distributed database clusters to be merged: the distributed database clusters to be merged include a first cluster and a second cluster; stop all nodes of the second cluster; set the node connection string information of the second cluster to connect to the master node of the first cluster; trigger the full build data of all GTMs and DNs of the second cluster, the source end is the corresponding master node of the first cluster; trigger the full build data of the CN of the second cluster, the source end is any CN of the first cluster; modify the standby machine information on the GTM and DN master nodes on the first cluster, and establish a data replication slot to the corresponding node of the second cluster; restart all GTM and DN nodes on the second cluster as standby nodes, and the master and standby start synchronizing data based on the data replication slot; modify the routing information of all CNs on the first cluster, add routes to the CNs of the second cluster; and restart the routes of the CNs of the second cluster.

[0090] See also Figure 8 The embodiment of the present disclosure further provides an electronic device 80, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed database cluster splitting method in the aforementioned method embodiment.

[0091] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the distributed database cluster splitting method in the aforementioned method embodiment.

[0092] The embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the distributed database cluster splitting method in the aforementioned method embodiment.

[0093] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 80 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0094] like Figure 8 As shown, the electronic device 80 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 80 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0095] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 80 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 80 having various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0096] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0097] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0098] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0099] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including the at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0100] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0101] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0104] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0105] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A distributed database cluster splitting method, characterized in that: The method comprises the following steps: Obtain information about the distributed database cluster to be split; wherein the information includes the number of database accesses, latency, and number of nodes; the nodes include CN, GTM, and DN; Receive a cluster split request; Allocating a split node based on the split request and information of the distributed database cluster; The distributed database cluster is split based on the allocated split nodes.

2. The distributed database cluster splitting method according to claim 1, characterized in that: The allocating a split node based on the split request and information of the distributed database cluster includes: Sort the CNs from small to large based on the number of database accesses; wherein the CNs are obtained based on the number of concurrent connections of the first cluster in the split request × 120; Sort the DN shard copies by data synchronization delay from large to small. The larger the delay, the higher the DN split weight. Analyze the network latency of the CN and DN shard copies obtained by the first cluster. The smaller the latency, the higher the DN split weight. Calculate the DN shard copy split priority based on the DN split weight; Sort the GTM shard copies by data synchronization latency from large to small. The greater the latency, the higher the GTM split weight. Analyze the network latency between the CN and GTM shard copies obtained by the first cluster. The smaller the latency, the higher the GTM split weight. The GTM shard copy split priority is calculated based on the GTM split weight.

3. The distributed database cluster splitting method according to claim 1, characterized in that: The distributed database cluster is split based on the allocated splitting nodes, including: a service-stopping splitting method and a service-non-stopping splitting method.

4. The distributed database cluster splitting method according to claim 3, characterized in that: The service suspension and splitting method comprises the following steps: Resolve the number of CNs, GTM copies, and DN copies required for the second cluster based on the allocated split nodes, and keep the remaining CNs, GTM copies, and DN copies in the first cluster; Stop all nodes included in the second cluster; In the first cluster, remove the nodes of the second cluster and retain the node data; Modify the connection string information of the GTM and DN nodes of the first cluster, and delete the replication slot from the primary node to the corresponding node of the second cluster; Deleting the routing information from all CNs in the first cluster to the CNs in the second cluster; Modify the connection string information of the GTM and DN nodes of the second cluster; Start the master node and standby node of the second cluster based on the allocated split nodes, establish master-standby connections respectively, and start synchronizing data; Start the CN of the second cluster and modify the routes of CN and GTM to DN.

5. The distributed database cluster splitting method according to claim 3, characterized in that: The non-stop service splitting method comprises the following steps: Increase the number of log files retained; Record the wal log position of each shard; Remove the nodes of the second cluster from the first cluster and retain the node data; Modify the connection string information of the GTM and DN nodes of the first cluster, and delete the replication slot from the primary node to the corresponding node of the second cluster; Deleting the routing information from all CNs in the first cluster to the CNs in the second cluster; Splitting the nodes corresponding to the second cluster from the first cluster, and after the splitting is completed, triggering the first cluster to create a data consistency point; Collect log files of each shard and send the log files to the second cluster; Start the second cluster master node and set the startup log target to lsn_end; Start the master node and standby node of the second cluster based on the allocated split nodes, establish master-standby connections respectively, and start synchronizing data; Start the CN of the second cluster and modify the routes of CN and GTM to DN.

6. The distributed database cluster splitting method according to any one of claims 1 to 5, characterized in that: The method also includes a distributed database cluster merging method, the method comprising the following steps: Obtaining node information of a distributed database cluster to be merged: the distributed database cluster to be merged includes a first cluster and a second cluster; Stop all nodes of the second cluster; Set the node connection string information of the second cluster to connect to the master node of the first cluster; Trigger the full build data of all GTM and DN of the second cluster, with the source end being the corresponding master node of the first cluster; Trigger the full build data of the CN of the second cluster, the source end is any CN of the first cluster; Modify the GTM on the first cluster and the backup machine information on the DN master node, and establish a data replication slot to the corresponding node of the second cluster; Restart all GTM and DN nodes on the second cluster as standby nodes, and start synchronizing data based on the data replication slot; Modify the routing information of all CNs in the first cluster and add routing to the CNs in the second cluster; Restart the routing of the second cluster CN.

7. A distributed database cluster splitting device, characterized in that: The device comprises: A data acquisition module is configured to acquire information of a distributed database cluster to be split; wherein the information includes the number of database accesses, latency, and number of nodes; the nodes include CN, GTM, and DN; A rule module is configured to receive a cluster split request; allocate a split node based on the split request and information of the distributed database cluster; The node splitting module is configured to split the distributed database cluster based on the allocated splitting nodes.

8. The distributed database cluster splitting device according to claim 7, characterized in that: The device also includes: A node merging module is configured to obtain node information of a distributed database cluster to be merged: the distributed database cluster to be merged includes a first cluster and a second cluster; Stop all nodes of the second cluster; Set the node connection string information of the second cluster to connect to the master node of the first cluster; Trigger the full build data of all GTM and DN of the second cluster, with the source end being the corresponding master node of the first cluster; Trigger the full build data of the CN of the second cluster, the source end is any CN of the first cluster; Modify the backup information of the GTM and DN master nodes on the first cluster, and establish a data replication slot to the corresponding node of the second cluster; Restart all GTM and DN nodes on the second cluster as standby nodes, and start synchronizing data based on the data replication slot; Modify the routing information of all CNs in the first cluster and add routing to the CNs in the second cluster; and, Restart the routing of the second cluster CN.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the distributed database cluster splitting method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the distributed database cluster splitting method as claimed in any one of claims 1 to 6.

Images