Method and device for automatically upgrading distributed cluster by stand-alone database cluster

Abstract

The invention provides a method and device for automatically upgrading a distributed cluster by a stand-alone database cluster, and the method comprises the steps: continuously carrying out cluster upgrading judgment, and triggering cluster upgrading if the judgment is successful; analyzing data node information of the original stand-alone database cluster, and configuring a cluster upgrading strategy; calculating a plan of each node of the upgraded distributed database according to the upgrading strategy, and executing an upgrading process; carrying out distribution key planning on the database tables after upgrading, and writing distribution key information by traversing all the database tables; and after the distribution key information is written in, carrying out data relocation and redistribution. According to the method, the stand-alone database can be automatically upgraded into the distributed data cluster, the workload is reduced, the working cost is reduced, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of database technology, and in particular relates to a method and device for automatically upgrading a distributed cluster from a single-machine database cluster. Background Technology

[0002] In a single-machine database cluster, DN usually refers to the node responsible for storing and managing data. DN completes data storage and local data query and writing. Multiple copies can be used to solve high availability problems. The architecture is as follows Figure 1 As shown in the figure, a primary DN can have multiple backup DNs.

[0003] The general architecture of a distributed database cluster is as follows Figure 2 As shown in the figure, it includes three components: coordinator CN, global transaction manager GTM, and data node DN: Coordinator CN: responsible for receiving client requests, performing SQL parsing and optimization, generating execution plans, and coordinating data nodes for data query and writing; in a cluster, multiple CNs will be deployed, and the data (DDL data) between each CN will be synchronized in real time. Users can choose any CN to conduct business. When a CN fails, users can choose to access from another CN.

[0004] Global Transaction Manager GTM: Generates and maintains global timestamps to ensure strong consistency of cluster data; Data node DN: completes data storage and local data query and writing. DN has multiple high availability groups (shards), and a high availability group includes multiple copies, including the primary node and multiple standby nodes. The primary node will synchronize logs to the standby node to keep data synchronized. If the primary node fails, it can trigger a primary-standby switchover and upgrade an optimal standby node to the primary node to ensure the continuity of cluster services.

[0005] From the architecture of the stand-alone database cluster and the distributed database cluster, it can be seen that the architecture of the stand-alone database cluster is simple and the maintenance cost is low. However, the amount of data supported for storage is limited. The architecture of the distributed database cluster is complex and the maintenance cost is relatively high, but it can support larger data and support expansion.

[0006] When operators initially install data, they may choose to install a stand-alone database because the amount of data is not large. However, as the business expands, the amount of data continues to increase, and the original cluster can no longer meet the data storage needs. However, the stand-alone database cluster cannot be expanded. The traditional way to upgrade a distributed database requires backing up the stand-alone data, erasing the database, reinstalling the distributed database, and then importing the data. The operation is complicated and the business needs to be stopped for a long time during the entire process. In addition, since the distributed database database table needs to specify the distribution key, the user needs to pre-plan the distribution column of the database table before manually importing the data, which is a huge workload. SUMMARY OF THE INVENTION

[0007] The present invention proposes a method and device for automatically upgrading a single-machine database cluster to a distributed cluster, which can automatically upgrade a single-machine database to a distributed data cluster, reduce workload, lower work costs, and improve work efficiency.

[0008] To achieve the above purpose, the technical solution of the present invention is implemented as follows: A method for automatically upgrading a distributed cluster from a single-machine database cluster, comprising: S1, continue to make cluster upgrade decisions, and a successful decision will trigger a cluster upgrade; S2. Analyze the data node information of the original single-machine database cluster and configure the cluster upgrade strategy; calculate the planning of each node of the upgraded distributed database according to the upgrade strategy and execute the upgrade process; S3. After the upgrade, the distribution key of the database table is planned, and the distribution key information is written by traversing all database tables; S4. After the distribution key information is written, data is moved and redistributed.

[0009] Furthermore, the cluster upgrade strategy in step S2 is that all data node copies are upgraded to a shard of a distributed data node, and the node planning method includes: S201, the server where each replica is located is changed from the DN of the original stand-alone database cluster to the primary DN of the shard of the distributed database cluster; S202, find the optimally configured server and deploy the global transaction manager GTM on it; S203, traverse the servers that have been changed to primary DN and have not deployed GTM, and subtract the resource requirements of the primary DN from the resources of the server. If the remaining resources are greater than the resources required by the coordinator CN, the server deploys the coordinator CN.

[0010] Furthermore, the cluster upgrade strategy in step S2 is to add a shard to the original cluster, and the node planning method includes: S211, find the backup DN with the largest network delay to the primary DN in the original single-machine database cluster, and change it to the primary DN of the newly added shard of the added distributed database cluster; S212, find the server with the best configuration and deploy the global transaction manager GTM on it; S213, obtain the remaining backup DNs in the original stand-alone database cluster, sort the network delays of the backup DNs and the original stand-alone database cluster primary DNs, change half of the backup DNs with smaller network delays and the original stand-alone database cluster primary DNs into a shard of the distributed database cluster; the others are used as backup DNs of the new shards; S214, traverse the servers that have been changed to primary DN and have not deployed GTM, and subtract the resource requirements of the primary DN from the resources of the server. If the remaining resources are greater than the resources required by the coordinator CN, the server deploys the coordinator CN.

[0011] Furthermore, the method for planning the distribution key of the database table in step S3 includes: S301, continuously collect all query SQL executed by the database; S302, perform distribution key analysis on the collected query SQL: traverse the query SQL, and cache information when encountering the attribute field of the table connection. The cache structure includes four fields: namespace, table name, attribute name, and cost. The value of the cost is represented by the number of times the attribute name appears. After each query SQL is analyzed, the value of the cost field is accumulated; S303, taking the attribute field with the highest cost value as the recommended distribution key; S304, recommend the distribution key for each table, traverse all database tables, and write the distribution key information.

[0012] Furthermore, the method of data migration and redistribution in step S4 includes: S401, traverse the primary DN of each shard after the upgrade; S402, traverse all database tables, calculate the data that needs to be retained for each primary DN according to the hash algorithm, and delete the redundant data; S403, according to the node planning of step S2, set the primary and backup connection information of each shard; S404, build all backup DNs in full and completely copy the data of the primary DN.

[0013] On the other hand, the present invention also proposes a device for automatically upgrading a distributed cluster from a single-machine database cluster, comprising: Upgrade judgment module: continuously make cluster upgrade judgments, and trigger cluster upgrades if the judgment is successful; Upgrade execution module: analyze the data node information of the original stand-alone database cluster and configure the cluster upgrade strategy; calculate the planning of each node of the upgraded distributed database according to the upgrade strategy and execute the upgrade process; Distribution key planning module: After upgrading, the distribution key planning of the database table is carried out, and the distribution key information is written by traversing all database tables; Data redistribution module: After the distribution key information is written, data is moved and redistributed.

[0014] Furthermore, a first policy submodule is set in the upgrade execution module to upgrade all data node copies to a shard of a distributed data node, including: Primary DN change unit: The server where each replica is located is changed from the DN of the original stand-alone database cluster to the primary DN of the shard of the distributed database cluster; Global management unit: find the optimally configured server and deploy the global transaction manager GTM on it; Coordinator unit: traverse the servers that have been changed to primary DN and have not deployed GTM, and use the server resources to subtract the resource requirements of the primary DN. If the remaining resources are greater than the resources required by the coordinator CN, the server will deploy the coordinator CN.

[0015] Furthermore, the second strategy submodule is set in the upgrade execution module, and a shard is added on the basis of the original cluster, including: Primary DN unit: Find the backup DN with the largest network delay to the primary DN in the original single-machine database cluster, and change it to the primary DN of the newly added shard of the added distributed database cluster; Global management unit: find the best configured server and deploy the global transaction manager GTM on it; Standby DN unit: obtain the remaining standby DNs in the original stand-alone database cluster, sort the network delays of the standby DNs and the original stand-alone database cluster master DNs, change the half of the standby DNs with the smaller network delay and the original stand-alone database cluster master DN into a shard of the distributed database cluster; the others are used as standby DNs of the new shards; Coordinator unit: traverse the servers that have been changed to primary DN and have not deployed GTM, and use the server resources to subtract the resource requirements of the primary DN. If the remaining resources are greater than the resources required by the coordinator CN, the server will deploy the coordinator CN.

[0016] Furthermore, the distribution key planning module includes: Collection unit: continuously collects all query SQL executed by the database; Analysis unit: Perform distribution key analysis on the collected query SQL: traverse the query SQL, and cache the information when encountering the attribute field of the table connection. The cache structure includes four fields: namespace, table name, attribute name, and cost. The value of the cost is represented by the number of times the attribute name appears. After each query SQL is analyzed, the value of the cost field is accumulated; Recommendation unit: the attribute field with the highest cost value is used as the recommended distribution key; Traversal unit: recommends the distribution key of each table, traverses all database tables, and writes the distribution key information.

[0017] Furthermore, the data redistribution module includes: Primary DN traversal unit: traverse the primary DN of each shard after the upgrade; Calculation unit: traverses all database tables, calculates the data that needs to be retained for each primary DN according to the hash algorithm, and deletes redundant data; Setting unit: according to the node planning of the upgrade execution module, set the master and backup connection information of each shard; Copy unit: build all backup DNs in full and completely copy the data of the primary DN.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1) The present invention provides a method for planning the distribution of each node of a distributed database on the original cluster server when a single-machine database cluster is upgraded to a distributed database cluster; 2) The present invention provides a method for planning the distribution key of data distribution and how to redistribute the data when the data is distributed from a single database to multiple shards; 3) The method of the present invention can automatically upgrade a single-machine database to a distributed data cluster without the need for manual operation by the user, thus reducing the workload, lowering the work cost, and improving work efficiency. Brief Description of the Figures

[0019] Figure 1 This is a schematic diagram of the architecture of a single-machine database cluster; Figure 2 Schematic diagram of the distributed database cluster; Figure 3 is a schematic diagram of the method of Example 1 of the present invention; Figure 4 This is a schematic diagram of data redistribution of a four-copy stand-alone database cluster upgraded to a four-shard distributed database cluster in Example 1 of the present invention. Specific implementation method

[0020] It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict. ​

[0021] In order to make the purpose and features of the present invention more obvious and easy to understand, the following is a further description in conjunction with the accompanying drawings and specific embodiments.

[0022] The main design idea of ​​the present invention is to solve the following technical difficulties: 1) When upgrading a single-machine database cluster to a distributed database cluster, how to plan the distribution of each node of the distributed database on the servers of the original cluster; 2) Distributed database data nodes have multiple shards. Data needs to be distributed from a single data point on a single machine to multiple shards. How to plan the distribution key for data distribution and how to redistribute it; 3) Upgrading a single-machine database to a distributed data cluster requires many steps and needs to be done automatically rather than manually by the user.

[0023] In order to solve the above problems, the present invention is as follows Figure 3 As shown in the figure, the automatic upgrade is carried out within a time limit through the four steps of upgrade judgment, upgrade execution, distribution key planning, and data redistribution. The specific workflow includes the following: 1. Upgrade judgment: 1. You can set up an upgrade decision module to continuously collect the DN data size of the data nodes of the original stand-alone database cluster; 2. When the data size exceeds the user-configured threshold, the cluster is upgraded to a distributed database cluster. The upgrade can be triggered by the upgrade decision module sending an upgrade request to the module responsible for the upgrade (such as the upgrade described below).

[0024] 2. Upgrade execution: 1. You can set up an upgrade execution module to collect the configuration information of each server in the original stand-alone database cluster, including: memory, CPU, hard disk and other data, as well as network delay information between servers; 2. The upgrade execution module receives the upgrade request triggered by the upgrade judgment module. It analyzes the node information of the original cluster. If the original cluster has only one node, it cannot be upgraded. It directly returns the upgrade failure information to the upgrade judgment module, prompting the user to add a new copy to trigger the upgrade.

[0025] 3. If the original single-machine database cluster has multiple copies, including multiple data node DNs. The user can configure the upgrade strategy, which has two options: 1) All data copies are upgraded to a shard of the distributed database cluster; 2) Add a shard based on the original cluster. During the automatic upgrade process, the user can choose which specific strategy to use, or the strategy can be automatically selected according to the number of data nodes, such as the number of backup DNs.

[0026] 3.1. If you choose the first upgrade strategy, all replicas are upgraded to a shard of the distributed database cluster. You can plan the nodes according to the following algorithm: (1) The server where each copy of the original stand-alone database cluster is located is changed from a DN node to the primary DN of a shard of the distributed database cluster; (2) Find the optimally configured server and deploy the global transaction manager GTM on it; (3) Traverse the servers that have been changed to primary DN and have not deployed GTM. Since they have been deployed as primary DN, subtract the primary DN resource requirements from the server resources. If the remaining resources of the server are greater than the resources required to deploy the coordinator CN, the coordinator CN is deployed on the server.

[0027] 3.2 If you choose the second upgrade strategy, add a shard based on the original cluster, that is, only upgrade one B as a shard master DN of the distributed database cluster, and plan the nodes according to the following algorithm: (1) Find the backup DN with the longest network delay to the primary DN among the backup DNs of the original stand-alone database cluster, and change it to the primary DN of a shard of the distributed database cluster; this is because the backup DN with the longest network delay to the original primary DN is the least suitable as the backup DN of the original primary DN, so it is taken out as the primary DN of the new shard; (2) Find the server with the best configuration and deploy the global transaction manager GTM on it; (3) Obtain the remaining backup DNs of the original stand-alone database cluster, and sort the network delays of the backup DNs and the original stand-alone database cluster primary DNs. For example, the network delays are sorted from small to large and then divided into two halves. The first half is the backup DN with small network delay, which is still retained as the backup DN of the original primary DN and is changed to a shard of the distributed database cluster together with the original primary DN; the other half of the backup DN is the backup DN with large network delay, which is used as the backup DN of the new shard primary DN; (4) Traverse the servers that have been changed to primary DN and have not deployed GTM. Since they have been deployed as primary DN, subtract the primary DN resource requirements from the server resources. If the remaining resources of the server are greater than the resources required to deploy the coordinator CN, the coordinator CN is deployed on the server.

[0028] 4. According to the above process, the upgrade execution module calculates the distribution of each node of the upgraded distributed database. Next, the upgrade process of the distributed database will be executed: (1) First, lock the single-machine database cluster, which can only be read but not written; (2) Wait for all backup DN synchronization data to complete; (3) Restart the original backup DN selected as the new shard master DN and change it to the master DN; ​(4) Deploy the global transaction manager GTM and coordinator CN, and create connection information to the shards.

[0029] 3. Distribution key planning: After all nodes are deployed, the next step is to redistribute the data in the distributed database. Before data migration, the distribution key problem of the database table must be solved. In a distributed database, the distribution key setting of the data table is particularly critical. If it is not set properly, it will cause frequent data exchange between data nodes when the business SQL is executed, resulting in an increase in the system network load and a decrease in database performance.

[0030] You can set up a distribution key planning module to be responsible for the planning of distribution keys. The workflow of this module is as follows: 1. Continuously collect all query SQL executed by the database; 2. Perform distribution key analysis on query SQL, including: traversing query SQL, caching information when encountering attribute fields of table connection, the cache structure includes four fields: namespace, table name, attribute name, and cost. The cost can be expressed by the number of times the attribute name appears. After each query SQL is analyzed, the cost value is accumulated; 3. The higher the cost value of an attribute field in the database table, the more reasonable it is to use the attribute field as a distribution key; 4. Recommend the distribution key for each table, traverse all database tables, and write the distribution key information.

[0031] Fourth, data redistribution: After the distribution key recommendation is completed, you can set up the data redistribution module to migrate the data. The data distribution method of the distributed database is to shard according to the hash rule, that is, hash(key)% number of storage nodes (key is the distribution key of the data table), and store the data on each node. Because before creating the distributed database node during the upgrade process, each backup DN of the stand-alone database cluster completes data synchronization with the primary DN, that is, the data of each DN node is the full set. The data migration work only needs to delete the redundant data of each shard primary DN. Taking the upgrade of a four-copy stand-alone database cluster to a four-shard distributed database cluster as an example, the data distribution changes are as follows Figure 4 As shown, data is redistributed from one node to four shards of the distributed database.

[0032] The specific workflow of the data redistribution module includes: 1. Traverse the primary DN of each shard; 2. Traverse all database tables, calculate the data that needs to be retained for this primary DN according to the hash algorithm, and delete the redundant data; 3. According to the node planning given during the upgrade, set the primary and backup connection information of each shard; 4. Perform a full build on all backup servers, that is, completely copy the data of the primary DN of your own shard.

[0033] At this point, the distributed database has been upgraded, the cluster status is set to normal, and the cluster can provide services externally.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically upgrading a distributed cluster from a single-machine database cluster, characterized in that: include: S1. Continue to make cluster upgrade decisions. If the decision is successful, the cluster upgrade is triggered. S2. Analyze the data node information of the original stand-alone database cluster and configure the cluster upgrade strategy; Calculate the plan of each node of the upgraded distributed database according to the upgrade strategy and execute the upgrade process; S3. After the upgrade, the distribution key of the database table is planned, and the distribution key information is written by traversing all database tables; S4. After the distribution key information is written, data is migrated and redistributed.

2. The method for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 1, characterized in that: The cluster upgrade strategy in step S2 is that all data node replicas are upgraded to a shard of a distributed data node, and the node planning method includes: S201, the server where each replica is located is changed from the DN of the original stand-alone database cluster to the primary DN of the shard of the distributed database cluster; S202, finding a server with the best configuration and deploying a global transaction manager GTM on it; S203. Traverse the servers that have been changed to primary DN and have not deployed GTM, and subtract the resource requirements of the primary DN from the resources of the server. If the remaining resources are greater than the resources required by the coordinator CN, the server deploys the coordinator CN.

3. The method for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 1, characterized in that: The cluster upgrade strategy in step S2 is to add a shard to the original cluster. The node planning method includes: S211, find the backup DN with the largest network delay to the primary DN in the original single-machine database cluster, and change it to the primary DN of the newly added shard of the added distributed database cluster; S212, finding a server with the best configuration, and deploying a global transaction manager GTM on it; S213, obtain the remaining backup DNs in the original stand-alone database cluster, sort the network delays of the backup DNs and the original stand-alone database cluster primary DN, change half of the backup DNs with smaller network delay and the original stand-alone database cluster primary DN into a shard of the distributed database cluster; the rest are used as backup DNs of the new shards; S214. Traverse the servers that have been changed to primary DN and have not deployed GTM, and subtract the resource requirements of the primary DN from the resources of the server. If the remaining resources are greater than the resources required by the coordinator CN, the server deploys the coordinator CN.

4. The method for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 1, characterized in that: The method for planning the distribution key of the database table in step S3 includes: S301, continuously collect all query SQL executed by the database; S302, performing distribution key analysis on the collected query SQL: traversing the query SQL, caching information when encountering the attribute field of the table connection, the cache structure includes four fields: namespace, table name, attribute name, and cost, where the value of the cost is represented by the number of times the attribute name appears; after each query SQL is analyzed, the value of the cost field is accumulated; S303: Use the attribute field with the highest cost value as the recommended distribution key; S304. Recommend a distribution key for each table, traverse all database tables, and write the distribution key information.

5. The method for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 1, characterized in that: The method of data migration and redistribution in step S4 includes: S401, traverse the primary DN of each shard after the upgrade; S402, traverse all database tables, calculate the data that needs to be retained for each primary DN according to the hash algorithm, and delete the redundant data; S403: According to the node planning of step S2, set the primary and backup connection information of each shard; S404. Fully build all backup DNs and completely copy the data of the primary DN.

6. A device for automatically upgrading a distributed cluster from a single-machine database cluster, characterized in that: include: Upgrade judgment module: continuously makes cluster upgrade judgments, and triggers cluster upgrades if the judgment is successful; Upgrade execution module: analyzes the data node information of the original stand-alone database cluster and configures the cluster upgrade strategy; Calculate the plan of each node of the upgraded distributed database according to the upgrade strategy and execute the upgrade process; Distribution key planning module: After the upgrade, the distribution key planning of the database table is carried out, and the distribution key information is written by traversing all database tables; Data redistribution module: After the distribution key information is written, data is migrated and redistributed.

7. The device for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 6, characterized in that: The first strategy submodule is set in the upgrade execution module to upgrade all data node copies to a shard of the distributed data node, including: Primary DN change unit: The server where each replica is located is changed from the DN of the original stand-alone database cluster to the primary DN of the shard of the distributed database cluster; Global management unit: Find the optimally configured server and deploy the global transaction manager GTM on it; Coordinator unit: traverse the servers that have been changed to primary DN and have not deployed GTM, subtract the resource requirements of the primary DN from the server's resources, and if the remaining resources are greater than the resources required by the coordinator CN, the server deploys the coordinator CN.

8. The device for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 6, characterized in that: Set up the second strategy submodule in the upgrade execution module and add a shard based on the original cluster, including: Primary DN unit: Find the backup DN with the longest network delay to the primary DN in the original single-machine database cluster, and change it to the primary DN of the newly added shard of the added distributed database cluster; Global management unit: Find the server with the best configuration and deploy the global transaction manager GTM on it; Backup DN unit: obtains the remaining backup DNs in the original stand-alone database cluster, sorts the network delays of the backup DNs and the original stand-alone database cluster primary DNs, and changes the half of the backup DNs with the smaller network delay and the original stand-alone database cluster primary DN into a shard of the distributed database cluster; the others are used as backup DNs of the new shards; Coordinator unit: traverse the servers that have been changed to primary DN and have not deployed GTM, subtract the resource requirements of the primary DN from the server's resources, and if the remaining resources are greater than the resources required by the coordinator CN, the server deploys the coordinator CN.

9. The device for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 6, characterized in that: The distribution key planning module includes: Collection unit: continuously collects all query SQL executed by the database; Analysis unit: Perform distribution key analysis on the collected query SQL: traverse the query SQL, and cache information when encountering the attribute field of the table connection. The cache structure includes four fields: namespace, table name, attribute name, and cost. The value of the cost is represented by the number of times the attribute name appears. After each query SQL is analyzed, the value of the cost field is accumulated. Recommendation unit: The attribute field with the highest cost value is used as the recommended distribution key; Traversal unit: recommends the distribution key of each table, traverses all database tables, and writes the distribution key information.

10. The device for automatically upgrading a distributed cluster from a single-machine database cluster according to claim 6, characterized in that: The data redistribution module includes: Primary DN traversal unit: traverses the primary DN of each shard after the upgrade; Calculation unit: traverses all database tables, calculates the data that needs to be retained for each primary DN according to the hash algorithm, and deletes redundant data; Setting unit: according to the node planning of the upgrade execution module, set the master and backup connection information of each shard; Copy unit: Fully build all backup DNs and completely copy the data of the primary DN.

Images