Distributed migration method and device for database and medium

Abstract

The embodiment of the invention discloses a distributed migration method and device for a database and a medium, is applied to the technical field of data migration and is used for solving the problem that existing data migration is low in efficiency and unreliable. The method comprises the steps of collecting a DDL script of a to-be-migrated database object in a source end database, and performing grammar analysis on the DDL script to obtain a compatibility evaluation result of the current to-be-migrated database object; determining a data scale of the to-be-migrated database object based on a compatibility evaluation result, executing a task allocation strategy corresponding to the to-be-migrated database object according to the data scale, and allocating a migration task of the to-be-migrated database object to a task queue of a working cluster; and monitoring the working state when the working node executes the migration task of the task queue, and if the working node is determined to be abnormal based on the working state, determining the uncompleted migration task of the task queue based on the management node, and performing redistribution.

Description

Technical Field

[0001] The present invention relates to the technical field of database migration, and in particular to a distributed migration method, device and medium of a database. Background Art

[0002] With the continuous development of Internet technology and the deepening of digital transformation, the amount of data is growing exponentially at an alarming rate. This trend not only promotes technological innovation, but also poses unprecedented challenges to enterprises' data storage, processing and query capabilities. In order to meet these challenges, many companies have to consider upgrading or replacing existing database systems to improve performance, reduce costs and meet the growing demand for data processing. For those companies that have accumulated complex and large systems and large-scale data, efficient data migration is more difficult.

[0003] The current database migration is due to the incompatibility characteristics that may exist between the source database and the target database, such as: data type differences, different index structures, storage mechanism differences, transaction processing mechanism differences, etc. Therefore, when migrating based on traditional migration tools, incompatibility issues can easily lead to data loss, data errors, or performance degradation after data migration, making it difficult to completely migrate the data of the source database to the target database. In addition, traditional stand-alone migration tools are difficult to meet the migration efficiency requirements of large-scale data companies. When processing large amounts of data, the migration speed may be slow due to the limited processing power of a single node, or even migration failure may occur due to resource exhaustion. In addition, stand-alone migration tools also have the risk of single point failure. Once a failure occurs during the migration process, it may cause the failure of the entire migration task and easily encounter resource bottleneck problems. Summary of the invention

[0004] In order to solve the above technical problems, one or more embodiments of this specification provide a distributed migration method, device and medium for a database.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of the present specification provide a distributed migration method of a database, which is applied to a distributed migration tool composed of a management cluster and a working cluster, wherein the management cluster includes one or more management nodes, and the working cluster includes one or more working nodes. The method includes:

[0007] Based on the configuration information of the source database migration task, DDL scripts of the database objects to be migrated in the source database are collected, and syntax analysis is performed on the DDL scripts to obtain compatibility assessment results of the database objects to be migrated;

[0008] Determine the data scale of the database object to be migrated based on the compatibility assessment result, execute the task allocation strategy corresponding to the database object to be migrated according to the data scale, and allocate the migration task of the database object to be migrated to the task queue of the working cluster; wherein the task queue is generated based on the list structure of the preset cache middleware;

[0009] The working state of the working node when executing the migration task of the task queue is monitored. If the working node is determined to be abnormal based on the working state, the unfinished migration tasks of the task queue are determined based on the management node and reallocated.

[0010] Optionally, in one or more embodiments of the present specification, the collecting of DDL scripts of database objects to be migrated in the source database based on the configuration information of the source database migration task specifically includes:

[0011] The front end determines the configuration phase of the source database migration task based on the data structure of the source database, determines the fine-grained order of the configuration phases, and determines the configuration order of each configuration phase;

[0012] The front end sequentially selects configuration information of each configuration stage based on the configuration order and the migration requirements, and sequentially filters the database objects of the source-end database migration task according to the configuration information of each configuration stage to obtain the database objects to be migrated;

[0013] The front end generates parameter data in a specified format according to the range corresponding to the database object to be migrated, and sends it to the management node of the management cluster, so as to collect data from the source database through the management node to obtain the DDL script of the database object to be migrated; wherein the specified format is JSON format.

[0014] Optionally, in one or more embodiments of the present specification, parsing the DDL script to obtain a compatibility assessment result of the database object to be migrated specifically includes:

[0015] Parsing the DDL script based on a preset syntax parser to convert the DDL script into a syntax tree;

[0016] Traversing the syntax tree according to preset rules to determine whether each node in the syntax tree contains a specific syntax format of the source database;

[0017] If so, evaluating whether the specific syntax format corresponding to the node can be replaced with the syntax format of the target database corresponding to the database object to be migrated;

[0018] If it cannot be replaced, the node is marked as an incompatible item, and the incompatibility information corresponding to the incompatible item is recorded;

[0019] If it can be replaced, the node is marked as a compatible item, the specific syntax format is converted and modified according to the syntax format of the target database to modify the DDL script, and the compatibility information corresponding to the compatible item is recorded;

[0020] The incompatible information and the compatible information are summarized to determine a compatibility evaluation result of the database object to be migrated.

[0021] Optionally, in one or more embodiments of the present specification, determining the data scale of the database object to be migrated based on the compatibility assessment result, executing the task allocation strategy corresponding to the database object to be migrated according to the data scale, and allocating the database object to be migrated to the corresponding task queue specifically includes:

[0022] If it is determined that the data size of the database object to be migrated is greater than a preset threshold value according to the compatibility assessment result of the database object to be migrated, single-table sharding is performed on the database object to be migrated according to the number of available working nodes in the working cluster to obtain multiple data blocks;

[0023] If the data size of the data block is greater than the preset threshold, the data block is iteratively divided into equal parts to obtain the logical sub-tables corresponding to the database objects to be migrated, and a task object is created for each of the logical sub-tables; wherein the task object has a unique label;

[0024] A hash value is obtained by performing a hash calculation on the unique label of each task object, and the task object is added to a task queue corresponding to the hash value range according to the hash value.

[0025] Optionally, in one or more embodiments of the present specification, determining the data scale of the database object to be migrated based on the compatibility assessment result, executing the task allocation strategy corresponding to the database object to be migrated according to the data scale, and allocating the database object to be migrated to the corresponding task queue specifically includes:

[0026] If it is determined that the data size of the database object to be migrated is less than a preset threshold value according to the compatibility evaluation result of the database object to be migrated, the database object to be migrated is used as a logical sub-table, and a task object is created for the logical sub-table; wherein the task object has a unique label;

[0027] A hash value is obtained by performing a hash calculation on the unique label of each task object, and the task object is added to a task queue corresponding to the hash value range according to the hash value.

[0028] Optionally, in one or more embodiments of the present specification, before monitoring the working state of the working node when executing the migration task of the task queue, the method further includes:

[0029] Each working node in the working cluster sends service information to a preset service registration center for registration to determine the registered working nodes in the working cluster;

[0030] The management node sends a service discovery request to the provisioned service registration center to obtain available instance information corresponding to the service discovery request, and enables the service registration center to screen the registered working nodes according to the available instance information, determine the available working nodes, and establish a communication connection between the available working nodes and the management node according to the available instance information;

[0031] The management node sends a data migration request to the available working node, so that the available working node actively obtains the migration task in the corresponding task queue in response to the data migration request.

[0032] Optionally, in one or more embodiments of the present specification, after sending a data migration request to the available working node through the management node so that the available working node actively obtains the migration task in the corresponding task queue in response to the data migration request, the method further includes:

[0033] The available working node records the task progress of processing the migration task in real time to obtain a progress log;

[0034] The available working node obtains the current progress log of the available working node based on a preset period, so as to compare the current progress log with the progress log of the previous period, and obtain the change data of the migration progress;

[0035] The available working node synchronizes the current progress log and the change data of the migration progress to the management node based on the available working node, so as to summarize each migration task through the management node and return the summary result to the front end for display.

[0036] Optionally, in one or more embodiments of the present specification, monitoring the working state of the working node when executing the migration task of the task queue, if the working node is determined to be abnormal based on the working state, determining the unfinished migration tasks of the task queue based on the management node and reallocating them, specifically includes:

[0037] Performing heartbeat detection on the available working nodes based on the management node to monitor the working status of the working nodes when executing the migration tasks of the task queue; wherein the working status includes: normal operation and downtime;

[0038] If it is determined that the working state is down, the management node obtains the unfinished migration tasks of the task queue corresponding to the working node;

[0039] Resending the service discovery request to the provisioned service registration center through the management node to obtain currently available instance information;

[0040] The management node screens the registered working nodes according to the currently available instance information, determines the currently available working nodes, and establishes a communication connection between the currently available working nodes and the management node according to the currently available instance information;

[0041] A hash value is obtained by performing a hash calculation on the unique label corresponding to the unfinished migration task according to the number of currently available working nodes, and the unfinished migration task is added to a task queue corresponding to the hash value range according to the hash value.

[0042] One or more embodiments of the present specification provide a distributed migration device for a database, the device comprising:

[0043] at least one processor; and,

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0046] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be able to: execute any of the above-described methods.

[0047] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0048] By first parsing the syntax and evaluating the compatibility of the DDL script, the compatibility issues of the source and target sides are fully considered, so that various database objects of the source database can be completely and accurately migrated. In addition, the migration tasks are assigned to the working nodes through the distributed architecture, and when the working nodes are abnormal, the management node can reallocate the migration tasks, avoiding the problem of cost waste caused by the entire migration task having to be stopped when the traditional single-architecture migration tool server goes down. The task allocation strategy can be dynamically adjusted according to the data scale, so that the migration of large-scale data can be processed in parallel, thereby increasing the migration speed. For small-scale data, a more lightweight migration method can be adopted to avoid unnecessary resource overhead, that is, through the task allocation strategy based on the data scale, the system can cope with data migration needs of different scales and improve the adaptability of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0050] Figure 1 A schematic diagram of a distributed migration method for a database provided in an embodiment of this specification;

[0051] Figure 2 A schematic diagram of a logical architecture of a distributed database migration provided in an embodiment of this specification;

[0052] Figure 3 A schematic diagram of a task allocation strategy for distributed migration of a database proposed in an embodiment of this specification;

[0053] Figure 4 A schematic diagram of the structure of a distributed migration device for a database provided in an embodiment of this specification;

[0054] Figure 5 A schematic diagram of the structure of a non-volatile storage medium provided in an embodiment of this specification. DETAILED DESCRIPTION

[0055] The embodiments of this specification provide a distributed migration method, device and medium for a database.

[0056] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0057] like Figure 1 As shown, the embodiment of this specification provides a schematic diagram of a distributed migration method for a database. Figure 1 It can be seen that in one or more embodiments of this specification, a distributed migration method of a database includes:

[0058] S101: Based on the configuration information of the source-side database migration task, DDL scripts of the database objects to be migrated in the source-side database are collected, and syntax analysis is performed on the DDL scripts to obtain compatibility assessment results of the database objects to be migrated.

[0059] The current database migration is due to the incompatible characteristics that may exist between the source database and the target database, such as: data type differences, different index structures, storage mechanism differences, transaction processing mechanism differences, etc. Therefore, when migrating based on traditional migration tools, it is easy to cause data loss, data errors or performance degradation after data migration due to incompatibility issues, making it difficult to completely migrate the data of the source database to the target database. Therefore, it is necessary to perform pre-migration analysis before migration to identify the incompatible characteristics between heterogeneous databases and implement corresponding solutions. Specifically, in one or more embodiments of this specification, in order to consider the compatibility issues between the source and target ends so that various database objects on the source end can be migrated completely and accurately, the configuration information of the source database migration task will be determined in the front-end interface in the embodiment of this specification, so that the management node can collect the DDL scripts of the database objects to be migrated in the source database according to the configuration information of the source database migration task, and perform syntax parsing on the DDL script to obtain the compatibility evaluation results of the current database objects to be migrated. Among them, it should be noted that syntax parsing can be based on SQL parser for parsing evaluation to identify or solve potential compatibility issues.

[0060] Specifically, in one or more embodiments of the present specification, based on the configuration information of the source database migration task, collecting the DDL scripts of the database objects to be migrated in the source database specifically includes the following process:

[0061] First, it is necessary to evaluate the task configuration at the front-end node to define the scope of objects that need to be migrated, that is, first determine the configuration stage of the source database migration task based on the data structure of the source database, and determine the fine-grainedness of the configuration stage for sorting, and determine the configuration order of each configuration stage. For example, in a certain application scenario, the configuration stage can be divided into four stages: database user filtering configuration, database mode filtering configuration, database object type filtering configuration, and database object filtering configuration. Since each stage is a more fine-grained division of the previous stage, and each completed stage configuration can be directly selected to complete, all database objects under the current stage configuration are filtered out. You can also choose to perform a more fine-grained configuration in the next step until the target object to be migrated is filtered out. Therefore, in the embodiment of this specification, the configuration information of each configuration stage will be selected in turn according to the above-determined configuration order and migration requirements, so that the database objects of the source database migration task are filtered in turn according to the configuration information of each configuration stage, and the database objects to be migrated are obtained. Then, the parameter data of the specified format is generated according to the range corresponding to the database object to be migrated, and sent to the management node of the management cluster, so that the source database can be collected through the management node to obtain the DDL script of the database object to be migrated. The specified format is JSON format. That is to say, after generating parameter data in the specified format, that is, JSON parameters, according to the range corresponding to the database object to be migrated, a request will be sent to the management cluster through the front-end client to collect data from the source database and collect the DDL scripts of each database object.

[0062] Further, after obtaining the DDL script of the database object based on the above process, it is necessary to perform syntax parsing on the collected DDL script to achieve compatibility analysis, that is, in one or more embodiments of the present specification, the DDL script is parsed to obtain the compatibility evaluation result of the current database object to be migrated, which specifically includes the following process:

[0063] First, the DDL script is parsed based on a preset syntax parser to convert the DDL script into a syntax tree. Then, in order to convert the incompatible syntax into the syntax format of the target database, the syntax tree is traversed according to the preset rules to determine whether each node in the syntax tree contains a specific syntax format of the source database. If there is a specific syntax format, the specific syntax format corresponding to the node will be evaluated to see whether it can be replaced with the syntax format of the target database corresponding to the current database object to be migrated. If the replacement cannot be performed, the node needs to be marked as an incompatible item, and the incompatible information corresponding to the incompatible item is recorded to facilitate subsequent manual modification. If the replacement can be performed, the node is marked as a compatible item, and the specific syntax format is converted and modified according to the syntax format of the target database, and the compatibility information corresponding to the compatible item is recorded. Then the management node will summarize the incompatible information and the compatible information to determine the compatibility evaluation result of the current database object to be migrated. In a certain application scenario, a compatibility assessment report will be generated based on the compatibility assessment results. The main contents of the report include indicators such as the total number of database objects, table size, number of directly compatible objects, number of compatible objects after automatic conversion, and number of incompatible objects. Finally, the report is returned to the front end to facilitate the migration personnel to propose migration implementation technical solutions based on the content of this report, effectively improving the user experience and the efficiency of database object structure migration. Through the above process, incompatible nodes can be quickly found or converted DDL scripts can be obtained, thus avoiding the problem of encountering complex database systems when migrating between heterogeneous databases, and requiring manual identification of syntax differences, which is often easy to miss, wasting both time and manpower.

[0064] S102: Determine the data scale of the database object to be migrated based on the compatibility assessment result, and execute the task allocation strategy corresponding to the database object to be migrated according to the data scale, so as to allocate the migration task of the database object to be migrated to the task queue of the working cluster; wherein the task queue is generated based on the list structure of the preset cache middleware;.

[0065] Because the current traditional single-node migration tool often requires a lot of time in front of massive data, and the single-node machine resources are prone to bottlenecks. Therefore, in order to overcome this bottleneck problem, this specification embodiment adopts the process of multi-work node migration, and in order to further improve the migration efficiency and balance the task load between each work node, in this specification embodiment, the management node will determine the data scale of the data object to be migrated according to the compatibility assessment result obtained in the above step S101, so as to execute the task allocation strategy corresponding to the database object to be migrated according to the data scale, so as to allocate the migration task of the database object to be migrated to the task queue of the working cluster. Among them, it should also be noted that the task queue uses the list data structure in the cache middleware such as redis for the management node to generate a task queue corresponding to the available nodes. In this process, the task allocation strategy can be dynamically adjusted according to the data scale, so that the migration of large-scale data can be processed in parallel, thereby improving the migration speed. For small-scale data, a more lightweight migration method can be adopted to avoid unnecessary resource overhead, that is, the system can cope with data migration needs of different scales through the task allocation strategy based on the data scale to improve the adaptability of data migration.

[0066] Specifically, in one or more embodiments of the present specification, the data scale of the database object to be migrated is determined based on the compatibility assessment result, so as to execute the task allocation strategy corresponding to the database object to be migrated according to the data scale, so as to allocate the database object to be migrated to the corresponding task queue, which specifically includes the following process:

[0067] If the compatibility assessment result of the database object to be migrated determines that the data size of the database object to be migrated is greater than the preset threshold, then Figure 3 As shown, the current database object to be migrated is sharded by a single table according to the number of available working nodes in the working cluster, thereby obtaining multiple data blocks. It should be noted that single table sharding is to divide the large table into data blocks of equal data size according to the number of available working nodes. If the data size of the data block after the division is still greater than the preset threshold, it is necessary to continue to iteratively divide the data block to obtain the logical sub-table corresponding to the current database object to be migrated, and create a task object for each logical sub-table, wherein the task object has a unique label and records the current table and data range. That is, if the size of each data block still exceeds the threshold, it is further divided into smaller data blocks. Finally, until the size of each data block is less than the threshold, the table is range-sharded according to the total number of blocks to obtain the divided logical sub-table, and then the divided logical sub-table is placed in the task object. Then the unique label of each task object is hashed to obtain a hash value, and the task object is added to the task queue corresponding to the hash value range according to the hash value.

[0068] In this process, data scale assessment and single-table sharding can reasonably allocate resources in the work cluster, while iterative equal division ensures that the size of each data block is moderate, avoiding resource waste or overload problems. In addition, this method can evenly distribute task objects in the task queue, avoiding task accumulation and bottlenecks in the task queue.

[0069] In another embodiment, the data scale of the database object to be migrated is determined based on the compatibility assessment result, so as to execute the task allocation strategy corresponding to the database object to be migrated according to the data scale, so as to allocate the database object to be migrated to the corresponding task queue, which specifically includes the following process:

[0070] If the data size of the database object to be migrated is determined to be less than the preset threshold value based on the compatibility assessment results of the current database object to be migrated, then the table with a small data size that does not reach the preset threshold value, that is, the database object to be migrated, is directly placed into the task object as a logical sub-table, and then the unique label of each task object is hashed to obtain a hash value, and the task object is added to the task queue corresponding to the hash value range according to the hash value. In this process, the additional computing and storage overhead introduced by the sharding operation for database objects with small data sizes is avoided, so that migration tasks with small data volumes can be processed more quickly, and the entire small table is migrated as a logical sub-table, ensuring the integrity and consistency of the data and reducing the risk of potential data loss or damage caused by sharding.

[0071] S103: monitoring the working status of the working node when executing the migration task of the task queue, and if the working node is determined to be abnormal based on the working status, determining the unfinished migration tasks of the task queue based on the management node and reallocating them.

[0072] In order to avoid the problem that the entire migration task has to be stopped when the traditional single-architecture migration tool server goes down, resulting in cost waste, in the embodiment of this specification, the management node monitors the working status of the working node when executing the migration task of the task queue, and determines whether there is an abnormality in the working node based on the working status. If so, the management node determines the unfinished migration tasks in the task queue and reallocates them.

[0073] Furthermore, if Figure 2 In one or more embodiments of this specification, before monitoring the working status of the working node when executing the migration task of the task queue, the method also includes:

[0074] First, all the working nodes in the working cluster will send the service information to the preset service registration center for registration, so as to obtain the registered working nodes in the working cluster. It should be noted that the service information includes: IP address, port number, service name, etc. When the management node of the management cluster wants to communicate with the working node in the working cluster, the management node will send a service discovery request to the preset service registration center to obtain the available instance information corresponding to the service discovery request, that is, all the available instance information of the specific service name. Then the service registration center will screen the registered working nodes according to the available instance information, determine the available working nodes, and establish a communication connection between the available working nodes and the management node according to the available instance information. Then the management node sends a data migration request to the available working node, so that the available working node actively obtains the migration task in the corresponding task queue in response to the data migration request. That is, the data migration request can be sent to the working node using the RPC remote communication method to notify the working node to actively obtain the corresponding task from the corresponding task queue, and then the working node starts to execute the data migration demand.

[0075] Furthermore, in order to enable the front-end node, that is, the client node, to better monitor the progress of the entire migration work, in one or more embodiments of the present specification, after the management node sends a data migration request to all working nodes so that the available working nodes actively obtain the migration tasks in the corresponding task queue in response to the data migration request, the method further includes the following process:

[0076] The available working nodes will record the task progress of the migration task in real time and obtain the progress log. At the same time, the available working nodes will obtain the current progress log of the available working nodes according to the preset period, so as to compare the current progress log with the progress log of the previous period and obtain the change data of the migration progress. Then, the available working nodes will synchronize the current progress log and the change data of the migration progress to the management node according to the available working nodes, so as to summarize the migration tasks through the management node and return the summary results to the front end for display.

[0077] Specifically, in one or more embodiments of the present specification, the working state of the working node when executing the migration task of the task queue is monitored. If the working node is determined to be abnormal based on the working state, the unfinished migration task of the task queue is determined based on the management node and reallocated, which specifically includes the following process:

[0078] First, Figure 2As shown, a heartbeat detection will be performed on the available working nodes based on the management node to monitor the working status of the working nodes when executing the migration tasks of the task queue. Among them, it can be understood that the working status includes: normal operation and downtime. If the working status is determined to be downtime, it means that the working node is abnormal at this time, and the management node will obtain the unfinished migration tasks of the task queue corresponding to the working node for reallocation. Specifically, the management node will resend the service discovery request to the preset service registration center to obtain all currently available instance information corresponding to the specific service name of the unfinished migration task. Then, the management node will screen the registered working nodes according to the currently available instance information, determine the currently available working nodes, and establish a communication connection between the currently available working nodes and the management node according to the currently available instance information. At this time, the unique label corresponding to the unfinished migration task is hashed according to the number of currently available working nodes to obtain a hash value, and the unfinished migration task is added to the task queue of the corresponding hash value range according to the hash value. That is, the management node will re-hash the unique label of the task according to the number of currently available working nodes, so as to map the unfinished migration task to the task queue according to the hash value range. It should be noted that each task queue is uniquely bound to a working node. If the task queue already exists, the corresponding unfinished migration tasks will be directly assigned to the task queue. If the task queue does not exist, a new task queue needs to be created and a migration request will be sent to the new working node to execute the unfinished migration tasks.

[0079] In this process, the abnormal status of the working node is discovered in time through heartbeat detection, and measures are taken quickly to reallocate tasks, ensuring the continuity and high availability of the migration work and avoiding the interruption or delay of the entire migration task due to the failure of a single working node. In addition, the management node can dynamically adjust the task allocation strategy according to the number of currently available working nodes, and evenly distribute tasks to different task queues through hash calculation, improving resource utilization and task execution efficiency. Even if some working nodes fail, the system can quickly recover and continue to execute the migration task, enhancing the fault tolerance and robustness of the system and improving the reliability of the migration work.

[0080] like Figure 4 As shown, the present specification embodiment provides a schematic diagram of the structure of a distributed migration device for a database. Figure 4 It can be seen that in one or more embodiments of this specification, a distributed migration device for a database includes:

[0081] at least one processor; and,

[0082] a memory communicatively connected to the at least one processor; wherein,

[0083] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0084] like Figure 5 The embodiment of this specification provides a structural schematic diagram of a non-volatile storage medium. Figure 5 It can be seen that in one or more embodiments of this specification, a non-volatile storage medium stores computer executable instructions 501, and the computer executable instructions 501 can: execute any of the methods described above.

[0085] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0086] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A distributed migration method for a database, characterized in that: Applied to a distributed migration tool consisting of a management cluster and a working cluster, wherein the management cluster includes one or more management nodes, the working cluster includes one or more working nodes, and the method includes: Based on the configuration information of the source-side database migration task, DDL scripts of the database objects to be migrated in the source-side database are collected, and syntax analysis is performed on the DDL scripts to obtain compatibility assessment results of the database objects to be migrated; Determine the data scale of the database object to be migrated based on the compatibility assessment result, execute the task allocation strategy corresponding to the database object to be migrated according to the data scale, and allocate the migration task of the database object to be migrated to the task queue of the working cluster; wherein the task queue is generated based on the list structure of the preset cache middleware; The working state of the working node when executing the migration task of the task queue is monitored. If the working node is determined to be abnormal based on the working state, the unfinished migration tasks of the task queue are determined based on the management node and reallocated.

2. A distributed migration method for a database according to claim 1, characterized in that: The step of collecting the DDL scripts of the database objects to be migrated in the source database based on the configuration information of the source database migration task specifically includes: Determine the configuration phase of the source database migration task based on the data structure of the source database, determine the fine-grained order of the configuration phases, and determine the configuration order of each configuration phase; Selecting configuration information of each configuration stage in sequence based on the configuration order and the migration requirement, filtering database objects of the source-end database migration task in sequence according to the configuration information of each configuration stage, and obtaining database objects to be migrated; Generate parameter data in a specified format according to the range corresponding to the database object to be migrated, and send it to the management node of the management cluster, so as to collect data from the source database through the management node to obtain the DDL script of the database object to be migrated; wherein the specified format is JSON format.

3. The distributed migration method of a database according to claim 1, characterized in that: Parsing the DDL script to obtain the compatibility assessment result of the database object to be migrated specifically includes: Parsing the DDL script based on a preset syntax parser to convert the DDL script into a syntax tree; Traversing the syntax tree according to preset rules to determine whether each node in the syntax tree contains a specific syntax format of the source database; If so, evaluating whether the specific syntax format corresponding to the node can be replaced with the syntax format of the target database corresponding to the database object to be migrated; If it cannot be replaced, the node is marked as an incompatible item, and the incompatibility information corresponding to the incompatible item is recorded; If it can be replaced, the node is marked as a compatible item, the specific syntax format is converted and modified according to the syntax format of the target database to modify the DDL script, and the compatibility information corresponding to the compatible item is recorded; The incompatible information and the compatible information are summarized to determine a compatibility evaluation result of the database object to be migrated.

4. The distributed migration method of a database according to claim 1, characterized in that: Determining the data scale of the database object to be migrated based on the compatibility assessment result, executing the task allocation strategy corresponding to the database object to be migrated according to the data scale, and allocating the database object to be migrated to the corresponding task queue, specifically includes: If it is determined that the data size of the database object to be migrated is greater than a preset threshold value according to the compatibility assessment result of the database object to be migrated, single-table sharding is performed on the database object to be migrated according to the number of available working nodes in the working cluster to obtain multiple data blocks; If the data size of the data block is greater than the preset threshold, the data block is iteratively divided into equal parts to obtain the logical sub-tables corresponding to the database objects to be migrated, and a task object is created for each of the logical sub-tables; wherein the task object has a unique label; A hash value is obtained by performing a hash calculation on the unique label of each task object, and the task object is added to a task queue corresponding to the hash value range according to the hash value.

5. The distributed migration method of a database according to claim 1, characterized in that: Determining the data scale of the database object to be migrated based on the compatibility assessment result, executing the task allocation strategy corresponding to the database object to be migrated according to the data scale, and allocating the database object to be migrated to the corresponding task queue, specifically includes: If it is determined that the data size of the database object to be migrated is less than a preset threshold value according to the compatibility evaluation result of the database object to be migrated, the database object to be migrated is used as a logical sub-table, and a task object is created for the logical sub-table; wherein the task object has a unique label; A hash value is obtained by performing a hash calculation on the unique label of each task object, and the task object is added to a task queue corresponding to the hash value range according to the hash value.

6. A distributed migration method for a database according to claim 4, characterized in that: Before monitoring the working state of the working node when executing the migration task of the task queue, the method further includes: Each working node in the working cluster sends service information to a preset service registration center for registration to determine the registered working nodes in the working cluster; The management node sends a service discovery request to the provisioned service registration center to obtain available instance information corresponding to the service discovery request, and enables the service registration center to screen the registered working nodes according to the available instance information, determine the available working nodes, and establish a communication connection between the available working nodes and the management node according to the available instance information; A data migration request is sent to the available working node through the management node, so that the available working node actively obtains the migration task in the corresponding task queue in response to the data migration request.

7. A distributed migration method for a database according to claim 6, characterized in that: After sending a data migration request to the available working node through the management node so that the available working node actively obtains the migration task in the corresponding task queue in response to the data migration request, the method further includes: The available working node records the task progress of processing the migration task in real time to obtain a progress log; The available working node obtains the current progress log of the available working node based on a preset period, so as to compare the current progress log with the progress log of the previous period, and obtain the change data of the migration progress; The available working node synchronizes the current progress log and the change data of the migration progress to the management node based on the available working node, so as to summarize each migration task through the management node and return the summary result to the front end for display.

8. A distributed migration method for a database according to claim 4, characterized in that: Monitoring the working state of the working node when executing the migration task of the task queue, if the working node is determined to be abnormal based on the working state, determining the unfinished migration tasks of the task queue based on the management node and reallocating them, specifically including: Performing heartbeat detection on the available working nodes based on the management node to monitor the working status of the working nodes when executing the migration tasks of the task queue; wherein the working status includes: normal operation and downtime; If it is determined that the working state is down, the management node obtains the unfinished migration tasks of the task queue corresponding to the working node; Resending the service discovery request to the provisioned service registration center through the management node to obtain currently available instance information; The management node screens the registered working nodes according to the currently available instance information, determines the currently available working nodes, and establishes a communication connection between the currently available working nodes and the management node according to the currently available instance information; A hash value is obtained by performing a hash calculation on the unique label corresponding to the unfinished migration task according to the number of currently available working nodes, and the unfinished migration task is added to a task queue corresponding to the hash value range according to the hash value.

9. A distributed migration device for a database, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: execute any of the methods described in claims 1-8.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions can: execute the method described in any one of claims 1 to 8.

Images