Data migration methods, systems, equipment, and storage media

Abstract

This specification provides one or more embodiments of a data migration method, database system, electronic device, and computer-readable storage medium. A migration link is established between a source database and a target database for running a first migration task, which migrates at least one data table from the source database to the target database. The method includes: during the execution of the first migration task, in response to a table addition instruction, determining a target data table to be added for migration in the source database; creating a second migration task to at least perform incremental migration of the target data table; during the parallel execution of the first and second migration tasks in the migration link, polling to detect the synchronization between the incremental migration time points in the first and second migration tasks; if they are synchronized, merging the second migration task into the first migration task. This achieves the addition of a target data table for migration based on the existing migration link.

Description

Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data migration method, database system, electronic device, and computer-readable storage medium. Background Technology

[0002] In database management scenarios, to ensure data storage security, it's often necessary to migrate data from one database to another for backup, or copy portions of the data. In practical applications, data migration devices can migrate data tables selected by the user in the source database to the target database. During the migration process, based on actual business needs, users may require modifications to synchronization objects (e.g., adding new data tables); therefore, it's necessary to provide a data migration method to meet these needs. Summary of the Invention

[0003] In view of the above, one or more embodiments of this specification provide a data migration method, a database system, an electronic device, and a computer-readable storage medium.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a data migration method is proposed, wherein a migration link is established between a source database and a target database for running a first migration task, the first migration task being used to migrate at least one data table from the source database to the target database; the method includes:

[0006] During the execution of the first migration task, in response to the table addition instruction, the target data table to be added and migrated in the source database is determined;

[0007] Create a second migration task to perform at least an incremental migration of the target data table;

[0008] During the parallel execution of the first migration task and the second migration task in the migration link, the synchronization status between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task is detected by polling.

[0009] If the two are synchronized, the second migration task is merged into the first migration task, so that the merged first migration task can perform incremental migration of the at least one data table and the target data table based on the synchronized time point.

[0010] Optionally, the polling detection of the synchronization between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task includes:

[0011] Polling and detecting the first delay between the incremental migration time point in the first migration task and the current time, and the second delay between the incremental migration time point in the second migration task and the current time, until both are less than the first threshold;

[0012] Alternatively, poll the time points of incremental migration in the first migration task and the time points of incremental migration in the second migration task until the difference between the two is less than a second threshold.

[0013] Optionally, if the two are synchronized, the second migration task is merged into the first migration task so that the merged first migration task can perform incremental migration of the at least one data table and the target data table, including:

[0014] If both are synchronized, the second migration task ends, the target data table is added to the task information of the first migration task, and the first migration task is restarted; wherein, the restarted first migration task can perform incremental migration of all data tables indicated by the modified task information based on the incremental migration time points recorded before the restart.

[0015] Optional, also includes:

[0016] During the execution of the first migration task, in response to the table modification instruction, the task information of the first migration task is modified according to the modification information carried by the table modification instruction, and the first migration task is restarted.

[0017] The modification information is used to indicate at least one of the following modification events: deleting a data table, modifying the name of a data table, modifying the name of a column in a data table, adding a column to a data table, and modifying the data type.

[0018] Optional, also includes:

[0019] Upon receiving a table addition instruction or a table modification instruction, check whether the first migration task is in a normal execution state;

[0020] If so, the first migration task is changed from the normal execution state to the synchronized object modification state, wherein the synchronized object modification state indicates that no other table add instructions and / or other table modification instructions will be responded to before the received table add instruction or table modify instruction is completed; and after restarting the first migration task, the first migration task is changed from the synchronized object modification state to the normal execution state.

[0021] If not, do not respond to received table add or modify commands, and output a message indicating that the synchronized object cannot be modified.

[0022] Optional, also includes:

[0023] After restarting the first migration task, the incremental migration time points recorded before the restart are rolled back by a preset time to obtain the target time point;

[0024] Incremental migration of all data tables in the modified task information is performed based on the target time point.

[0025] Optionally, the source database may include a distributed database;

[0026] The first migration task includes a first parent task and a first child task. The number of the first child tasks is determined according to the number of partitions corresponding to the data tables in the first migration task. The first parent task is used at least to perform a global pre-check between the source database and the target database. Different first child tasks are used at least to perform partition pre-checks between the source database and the target database, as well as incremental migration of the data tables in the first migration task in different partitions.

[0027] The second migration task includes a second parent task and a second sub-task. The number of the second sub-tasks is determined according to the number of partitions corresponding to the target data table. The second parent task is used at least to perform a global pre-check between the source database and the target database. Different second sub-tasks are used at least to perform partition pre-checks between the source database and the target database, as well as incremental migration of the target data table in different partitions.

[0028] Optionally, the first parent task is also used to implement the structure migration of the data table, and / or, different first sub-tasks are also used to implement the full migration of the data table in different partitions in the migration task;

[0029] Among them, the global pre-check in the first parent task and the partition pre-check in all first sub-tasks have higher running priority than the structure migration in the first parent task, and the structure migration in the first parent task has higher running priority than the full migration and incremental migration in the first sub-tasks.

[0030] The second parent task is also used to implement the structural migration of the target data table; and / or, different second sub-tasks are also used to implement the full migration of the target data table in different partitions;

[0031] Among them, the global pre-check in the second parent task and the partition pre-check in all second sub-tasks have a higher execution priority than the structure migration in the second parent task, and the structure migration in the second parent task has a higher execution priority than the full migration and incremental migration in the second sub-tasks.

[0032] Optionally, the partition where the target data table is located includes a first partition and / or a second partition; wherein, the first partition indicates that the data table in the first migration task is also stored in this partition, and the second partition indicates that the data table in the first migration task is not stored in this partition.

[0033] The creation of the second migration task includes:

[0034] Create a second parent task under the first parent task in the form of parent and child tasks;

[0035] A second subtask is created under the first subtask corresponding to the first partition in the form of parent and child tasks, and / or a second subtask corresponding to the second partition is created; wherein the second subtask corresponding to the second partition runs in the migration chain until the data migration is completed.

[0036] Optionally, the polling detection of the synchronization between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task includes:

[0037] For any first subtask and second subtask corresponding to the first partition, poll and check the first delay duration between the incremental migration time point in the first subtask and the current time, and the second delay duration between the incremental migration time point in the second subtask and the current time, until both are less than the first threshold.

[0038] If the two are synchronized, the second migration task is merged into the first migration task so that the merged first migration task can perform incremental migration of the at least one data table and the target data table, including:

[0039] For any first subtask and second subtask corresponding to a first partition, if both the first delay duration and the second delay duration are less than the second threshold, the second subtask is terminated, the target data table in the first partition is migrated to the task information of the first subtask, and the first subtask is restarted; wherein, the restarted first subtask can perform incremental migration of all subtables in the first partition as indicated by the modified task information based on the incremental migration time point recorded before the restart;

[0040] If all second subtasks in the first partition are completed, the target data table is added to the task information of the first parent task for migration, and the second parent task is terminated.

[0041] Optional, also includes:

[0042] In response to the table modification instruction, the task information of the first parent task and each first subtask in the first migration task is modified according to the modification information carried by the table modification instruction, and each first subtask in the first migration task is restarted.

[0043] According to a second aspect of one or more embodiments of this specification, a database system is provided, including a data migration device, a source database, and a target database; the data migration device is connected to the source database and the target database respectively.

[0044] The data migration apparatus is used to execute the data migration method described in any one of the first aspects to migrate data from the source database to the target database.

[0045] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0046] processor;

[0047] Memory used to store processor-executable instructions;

[0048] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0049] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described above.

[0050] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0051] In this embodiment, during the execution of the first migration task, a second migration task is created in response to a table addition instruction to at least incrementally migrate the target data table to be added to the source database. During the parallel execution of the first and second migration tasks in the migration chain, if the time points of incremental migration in the first and second migration tasks are determined to be synchronized, the second migration task is merged into the first migration task. This allows the merged first migration task to incrementally migrate the at least one data table and the target data table based on the synchronized time points. This allows for the addition of a target data table to be migrated without interrupting or adding a new migration chain, and the merging of the second migration task into the first migration task facilitates later maintenance and management.

[0052] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the structure of a database system provided in an exemplary embodiment.

[0054] Figure 2 This is a schematic diagram of a first migration task provided in an exemplary embodiment.

[0055] Figure 3 This is a schematic diagram of another first migration task provided in an exemplary embodiment.

[0056] Figure 4 This is a flowchart illustrating a data migration method provided in an exemplary embodiment.

[0057] Figure 5 This is a schematic diagram of a first migration task and a second migration task provided in an exemplary embodiment.

[0058] Figure 6 This is a schematic diagram of a second type of first migration task and second migration task provided in an exemplary embodiment.

[0059] Figure 7 This is a schematic diagram of a third type of first migration task and second migration task provided in an exemplary embodiment.

[0060] Figure 8 This is a flowchart illustrating another data migration method provided in an exemplary embodiment.

[0061] Figure 9 This is a flowchart illustrating another data migration method provided in an exemplary embodiment.

[0062] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0064] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0065] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0066] A database is a repository for organizing, storing, and managing data according to a data structure. It is a large collection of organized, shareable, and centrally managed data that is stored permanently in a computer.

[0067] Pre-check: Before data synchronization, it is necessary to perform checks on the source database and target database, such as connectivity checks, account permission checks, and database configuration checks. These are collectively referred to as pre-checks.

[0068] Full synchronization / migration: The process of synchronizing existing data before the current time point at the start of data synchronization is called the full step, which pulls data by directly reading existing data.

[0069] Incremental synchronization / migration: After the data synchronization starts, incrementally written data is synchronized by reading and parsing the source database write logs. This process is called incremental synchronization.

[0070] Partitioning: In distributed databases, data is typically stored in a distributed manner to improve data read and write performance. The distributed storage units are called partitions.

[0071] Migration link: refers to the communication path between the source database and the target database used to run migration tasks.

[0072] Synchronization objects: The mapping relationship between the source database and the target database, including information such as databases, tables, fields, and partitions.

[0073] This specification addresses problems in the related art by providing a data migration method. It also relates to a data migration apparatus, a database system, an electronic device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0074] Figure 1 This is a schematic structural diagram of a database system provided in an exemplary embodiment. The database system includes a data migration device 10, a source database 20, and a target database 30. The data migration device 10 is connected to both the source database 20 and the target database 30. The data migration method provided in this embodiment can be executed by the data migration device.

[0075] In some embodiments, the data migration apparatus may be applied to an electronic device. Exemplary examples include, but are not limited to, physical servers, virtual servers, cloud servers, personal computers, laptop computers, cellular phones, camera phones, smartphones, personal digital assistants, media players, navigation devices, email sending and receiving devices, game consoles, tablet computers, wearable devices, or any combination of these devices.

[0076] The creation and execution process of the first migration task is illustrated here: The data migration device receives a migration instruction, which specifies a source database, a target database, and at least one data table in the source database to be migrated. Please refer to [link to relevant documentation]. Figure 2 The data migration device can determine the source database and the target database based on the migration instruction, establish a migration link between the source database and the target database, and create a first migration task based on the migration instruction to migrate at least one data table from the source database to the target database via the migration link. The migration link refers to the communication path between the source database and the target database.

[0077] For example, the first migration task is used to perform at least one of pre-check, structure migration, full migration, and incremental migration. Pre-check is used to perform checks on the source and target databases, including connectivity checks, account permission checks, and database configuration checks. Structure migration is used to migrate the data structure of the data table to be migrated to the target database. Full migration is used to migrate all the data in the data table. Incremental migration is used to migrate the incremental data in the data table.

[0078] In one possible implementation, the source database is a distributed database, meaning the data tables in the source database are stored in different partitions of the source database. For example, such as... Figure 3 Data table A includes partitions A1 and A2, with partition A1 stored in partition 1 and partition A2 stored in partition 2. Therefore, the first migration task can include a first parent task and a first child task.

[0079] The number of the first subtasks is determined based on the number of partitions corresponding to the data tables in the first migration task. In one example, the data tables to be migrated include data table A, which is stored in two partitions, so the number of the first subtasks is 2. In another example, the data tables to be migrated include data table A and data table C. Data table A is stored in partitions 1 and 2, and data table C is stored in partitions 1, 3, and 4. Based on one partition corresponding to one first subtask, the number of the first subtasks is 4.

[0080] The first parent task is used to perform a global pre-check between the source and target databases, such as global connectivity checks, global permission checks, and / or global configuration information checks. For example, global connectivity checks check whether the connections between all partitions in the source and target databases are normal and whether subsequent data migration is possible. Global permission checks check the access permissions and read / write permissions of all partitions in both the source and target databases. Global configuration information checks check parameters such as port numbers, partition information, and storage space sizes of all partitions in both the source and target databases.

[0081] In some possible implementations, the first parent task is also used to implement the structure migration of the data table; the structure of the data table includes, but is not limited to, the table name and the fields in the table (such as field name, field type and field length).

[0082] Different first subtasks are used at least to perform partition pre-checks between the source and target databases, and incremental migration of data tables in the first migration task across different partitions. The partition pre-checks include, but are not limited to, connection detection, permission detection, and / or configuration information detection between partitions of the source database and the target database (or partitions of the target database). Additionally, in some possible implementations, different first subtasks are also used to perform full migration of data tables in the migration task across different partitions.

[0083] Continuing with the previous example, suppose that in the source database, table A1 in partition 1 will be migrated to partition 11 of the target database, and table A2 in partition 2 will be migrated to partition 12 of the target database. Figure 3 The first subtask, Subtask 1, is used to perform a partition pre-check between partition 1 and partition 11 of the target database, including but not limited to connection checks, permission checks, and / or configuration information checks. Connection checks involve verifying the connectivity between partition 1 and partition 11, checking whether the source and target partitions can connect normally and whether a migration link can be established to achieve data migration. Permission checks involve checking access permissions and read / write permissions between partition 1 and partition 11. Configuration information checks involve checking parameters such as port numbers, partition information, and storage space size between partition 1 and partition 11. After the partition pre-check, Subtask 1 is also used to perform the existing and incremental migration of table A1. The second subtask, Subtask 2, is used to perform a partition pre-check between partition 2 and partition 12 of the target database, and the existing and incremental migration of table A2.

[0084] In this process, the global pre-check in the first parent task and the partition pre-checks in all first sub-tasks have a higher execution priority than the structure migration in the first parent task. The structure migration in the first parent task has a higher execution priority than the full migration and incremental migration in the first sub-tasks. For example, the global pre-check in the first parent task and the partition pre-checks in all first sub-tasks run simultaneously (or the global pre-check in the first parent task can run first, and the partition pre-checks in each first sub-task can be performed only after the pre-checks are successful). After the pre-checks pass, the structure migration in the first parent task runs first. After the structure migration is complete, the incremental migration and full migration in each first sub-task run sequentially. It is understood that different first sub-tasks are independent of each other and do not interfere with each other; different first sub-tasks can run in parallel to improve migration efficiency.

[0085] The data migration method provided in this manual will be illustrated by example below: Figure 4 This is a schematic flowchart illustrating a data migration method provided in an exemplary embodiment. A migration link is established between a source database and a target database for running a first migration task, which is used to migrate at least one data table from the source database to the target database. The method can be executed by a data migration apparatus, and the method includes:

[0086] In S101, during the execution of the first migration task, in response to the table addition instruction, the target data table to be added for migration in the source database is determined.

[0087] In S102, a second migration task is created to perform at least an incremental migration of the target data table.

[0088] In S103, during the parallel execution of the first migration task and the second migration task in the migration link, the synchronization status between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task is polled and detected.

[0089] In S104, if the two are synchronized, the second migration task is merged into the first migration task, so that the merged first migration task can realize the incremental migration of the at least one data table and the target data table based on the synchronized time point.

[0090] This embodiment enables the addition of a migration target data table on the basis of the existing migration link without interrupting the migration link or adding a new migration link, and merges the second migration task into the first migration task, which facilitates later maintenance and management.

[0091] In some embodiments, upon receiving a table add instruction, the data migration device can detect whether the first migration task is in a normal execution state. If so, the first migration task is changed from a normal execution state to a synchronization object modification state. The synchronization object modification state indicates that no other table add instructions and / or other table modification instructions will be responded to before the received table add instruction is completed, thus avoiding table add errors. If not, the data migration device does not respond to the received table add instruction or table modification instruction and outputs a prompt message indicating that the synchronization object cannot be modified. This embodiment, by checking the status of the first migration task to determine whether to respond to the table add instruction, helps to ensure the accuracy of data migration.

[0092] For example, the table addition instruction can carry relevant information about the target data table to be migrated (such as the table name or the storage location of the target data table, etc.), so that the data migration device can determine the target data table to be migrated in the source database based on the table addition instruction. The number of target data tables to be migrated can be one or more; this embodiment does not impose any limitation on this. Furthermore, the data migration device can create a second migration task based on the table addition instruction, and the second migration task is used at least to implement the incremental migration of the target data table.

[0093] For example, in addition to implementing the incremental migration of the target data table, the second migration task is also used to implement at least one of pre-checking, target data table structure migration, and existing data migration. Pre-checking involves performing checks on the source and target databases, including connectivity checks, account permission checks, and database configuration checks. Structure migration involves migrating the data structure of the target data table to the target database. Full migration involves migrating all data in the target data table. Incremental migration involves migrating incremental data in the target data table.

[0094] After constructing the second migration task, the data migration device can run the first and second migration tasks in parallel within the existing migration chain. During the parallel execution of the first and second migration tasks within the migration chain, the data migration device can poll and detect the synchronization between the incremental migration time points in the first and second migration tasks to determine the timing for merging the first and second migration tasks. The incremental migration time point indicates the write time of the incremental data to be migrated in the source database. For example, the data migration device can merge the second migration task into the first migration task if the incremental migration time points in the first and second migration tasks are synchronized, so that the merged first migration task can perform incremental migration of at least one data table and the target data table based on the synchronized time points. This embodiment achieves the addition of a target data table for migration on the basis of the existing migration chain through task merging without interrupting the migration chain or adding a new migration chain.

[0095] This section provides an example illustrating the synchronization between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task:

[0096] In one possible implementation, the data migration device can poll and detect a first delay between the incremental migration time point in the first migration task and the current time, and a second delay between the incremental migration time point in the second migration task and the current time, and determine whether both the first delay and the second delay are less than a first threshold; if both the first delay and the second delay are less than the first threshold, it is determined that the incremental migration time points in the first migration task and the incremental migration time points in the second migration task are synchronized; otherwise, the polling and detection step continues.

[0097] In another possible implementation, the data migration device can poll the incremental migration time points in the first migration task and the incremental migration time points in the second migration task to determine whether the difference between the two time points is less than a second threshold. If the difference between the two time points is less than the second threshold, it is determined that the incremental migration time points in the first migration task and the incremental migration time points in the second migration task are synchronized. If not, the polling and detection steps continue.

[0098] The specific values ​​of the first and second thresholds can be set according to the actual application scenario, and this embodiment does not impose any restrictions on them. For example, the first and second thresholds can be any values ​​within 10 seconds.

[0099] Here is an illustrative description of the process of merging the second migration task and the first migration task: when the incremental migration time points in the first migration task and the incremental migration time points in the second migration task are synchronized, the data migration device can end the second migration task, add the target data table to be migrated in the task information of the first migration task, and restart the first migration task; wherein, the restarted first migration task can perform incremental migration of all data tables indicated by the modified task information based on the incremental migration time points recorded before the restart.

[0100] Another special case is that during the process of polling and checking the synchronization between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task, if the two time points are not yet synchronized, but the incremental migration in the second migration task has been completed, the second migration task can also be terminated, the target data table can be added to the task information of the first migration task, and the first migration task can be restarted.

[0101] Furthermore, to prevent incremental data loss, after restarting the first migration task, the data migration device can roll back the incremental migration time points recorded before the restart by a preset time interval to obtain the target time point. Then, based on the target time point, incremental migration is performed on all data tables in the modified task information, i.e., incremental data within the preset time interval is resynchronized, thereby effectively preventing incremental data loss. It is understood that the specific size of the preset time interval can be set according to the actual application scenario, such as a preset time interval of 8 seconds, 10 seconds, or 15 seconds, etc.

[0102] For example, during the execution of a table add command, if the status of the first migration task is changed to the synchronized object modification status, then after restarting the first migration task, the data migration device can change the first migration task from the synchronized object modification status to the normal execution status, so that the user can continue to trigger other table add commands or table modification commands, etc., according to the actual situation.

[0103] In other embodiments, besides adding data tables, users may have other modification needs, such as deleting data tables, changing data table names, changing column names in data tables, etc. Therefore, to meet the above needs, the embodiments of this specification also support the ability to perform the above modifications.

[0104] During the execution of the first migration task, the data migration device receives a table modification instruction and checks whether the first migration task is in a normal execution state. If so, the first migration task is changed from the normal execution state to the synchronization object modification state. The synchronization object modification state indicates that no other table addition instructions and / or other table modification instructions will be responded to before the received table modification instruction is completed. If not, the received table addition instructions or table modification instructions will not be responded to, and a prompt message indicating that the synchronization object cannot be modified will be output.

[0105] For example, the data migration device can respond to a table modification instruction, modify the task information of the first migration task according to the modification information carried by the table modification instruction, and restart the first migration task; wherein, the modification information is used to indicate at least one of the following modification events: deleting a data table, modifying the name of a data table, modifying the name of a column in a data table, adding a column to a data table, and modifying the data type, but is not limited thereto.

[0106] For example, during the execution of a table modification instruction, the status of the first migration task is changed to the synchronized object modification status; then after restarting the first migration task, the data migration device can change the first migration task from the synchronized object modification status to the normal execution status, so that the user can continue to trigger other table addition instructions or table modification instructions, etc., according to the actual situation.

[0107] Furthermore, to prevent incremental data loss, after restarting the first migration task, the data migration device can roll back the incremental migration time point recorded before the restart by a preset time to obtain the target time point; then, based on the target time point, incremental migration of all data tables in the modified task information is performed, that is, the incremental data within the preset time is resynchronized, thereby effectively preventing incremental data loss.

[0108] In some embodiments, the source database includes a distributed database. The first migration task includes a first parent task and first sub-tasks, the number of which is determined based on the number of partitions corresponding to the data tables in the first migration task. The first parent task is used at least to perform a global pre-check between the source and target databases. Different first sub-tasks are used at least to perform partition pre-checks between the source and target databases, and incremental migration of the data tables in the first migration task across different partitions. In some possible implementations, the first parent task is also used to perform a structural migration of the data tables, and / or, different first sub-tasks are also used to perform a full migration of the data tables in the migration task across different partitions.

[0109] The second migration task includes a second parent task and a second child task. The number of the second child tasks is determined based on the number of partitions corresponding to the target data table. For example, assuming a new migration data table B is added and stored in 3 partitions, there are 3 corresponding second child tasks.

[0110] The second parent task is used at least to perform a global pre-check between the source and target databases. Different second sub-tasks are used at least to perform partition pre-checks between the source and target databases, and incremental migration of the target data table across different partitions. In some possible implementations, the second parent task is also used to perform structural migration of the target data table; and / or, different second sub-tasks are also used to perform full migration of the target data table across different partitions. The global pre-check in the second parent task and the partition pre-checks in all second sub-tasks have a higher execution priority than the structural migration in the second parent task. The structural migration in the second parent task has a higher execution priority than the full migration and incremental migration in the second sub-tasks. For the specific meanings of global pre-check, partition pre-check, structural migration, full migration, and incremental migration, please refer to the above description, which will not be repeated here.

[0111] For example, in the scenario of adding a response table instruction, the partition where the target data table is located includes a first partition and / or a second partition; wherein, the first partition indicates that the partition of the data table in the first migration task is also stored in this partition, and the second partition indicates that the partition of the data table in the first migration task is not stored in this partition. Specifically, creating the second migration task in S102 includes: creating a second parent task under the first parent task in the form of parent-child tasks; creating a second subtask under the first subtask corresponding to the first partition in the form of parent-child tasks, and / or creating a second subtask corresponding to the second partition; wherein, the second subtask corresponding to the second partition runs in the migration chain until the data migration is complete.

[0112] exist Figure 3 Based on the aforementioned embodiment, assume that the table addition instruction indicates the addition of data table B:

[0113] In the first possible implementation, please refer to Figure 5 Data table B includes sub-tables B1 and B2. Sub-table B1 is stored in partition 1, and sub-table B2 is stored in partition 2. During the creation of the second migration task, the data migration device can create the second parent task under the first parent task in the form of parent-child tasks, create the second subtask 1 under the first subtask 1 corresponding to partition 1 in the form of parent-child tasks, and create the second subtask 2 under the first subtask 2 corresponding to partition 2 in the form of parent-child tasks.

[0114] In the second possible implementation, please refer to Figure 6 Data table B includes partitions B1, B2, and B3. Partition B1 is stored in partition 1, partition B2 in partition 2, and partition B3 in partition 3. During the creation of the second migration task, the data migration device can create the second parent task under the first parent task in a parent-child task format, create the second subtask 1 under the first subtask 1 corresponding to partition 1 in a parent-child task format, create the second subtask 2 under the first subtask 2 corresponding to partition 2 in a parent-child task format, and directly create the second subtask 3 corresponding to partition 3. Based on the principle of one partition corresponding to one subtask, second subtask 1 needs to be merged into first subtask 1, second subtask 2 needs to be merged into first subtask 2, and second subtask 3 does not need to be merged.

[0115] For the third possible implementation, please refer to Figure 7 Data table B includes sub-tables B1 and B2. Sub-table B1 is stored in partition 3, and sub-table B2 is stored in partition 4. During the creation of the second migration task, the data migration device can create the second parent task under the first parent task in the form of parent and child tasks, and directly create the second sub-task 1 corresponding to partition 3 and the second sub-task 2 corresponding to partition 4.

[0116] Understandable, Figure 5 , Figure 6 as well as Figure 7 In this context, the pre-checks in the first and second parent tasks refer to global pre-checks, while the pre-checks in the first and second sub-tasks refer to partition pre-checks.

[0117] In one example, the first and second migration tasks are executed in parallel within the migration chain. For the second migration task, the second parent task within the second migration task is run first. If the global pre-check in the second parent task passes, the second subtask within the second migration task is run. If the partition pre-check in the second subtask within the second migration task passes, the structure migration in the second parent task is run. After the structure migration is completed, the full migration and incremental migration in the second subtask within the second migration task are run. Of course, the structure migration and full migration in the second migration task are optional, and it can be decided whether to migrate these two items based on the actual situation. This specification implements the merging of any second subtask corresponding to a first partition into the first subtask corresponding to that first partition, thereby achieving one subtask running per partition, which facilitates later maintenance and management.

[0118] In one possible implementation, for any first subtask and second subtask corresponding to a first partition, the data migration device can poll and check the first delay duration between the incremental migration time point in the first subtask and the current time, and the second delay duration between the incremental migration time point in the second subtask and the current time, until both are less than a first threshold, then it can be determined that the incremental migration time point in the first subtask and the incremental migration time point in the second subtask are synchronized.

[0119] In another possible implementation, for any first subtask and second subtask corresponding to a first partition, the data migration device can poll and check the incremental migration time points in the first subtask and the second subtask until the difference between the two is less than a second threshold, then it can be determined that the incremental migration time points in the first subtask and the second subtask are synchronized.

[0120] For any first subtask and second subtask corresponding to a first partition, if the incremental migration time points in the first subtask and the second subtask are synchronized, the data migration device can terminate the second subtask and add a new partition of the target data table in the first partition to the task information of the first subtask. For example, using... Figure 5 For example, the second subtask 1 and the second subtask 2 can be terminated. Then, the task information of the first subtask 1 corresponding to partition 1 can be modified from A1 to A1 and B1, and the task information of the first subtask 2 corresponding to partition 2 can be modified from A2 to A2 and B2. Then, the first subtask can be restarted. The first migration task after restarting can perform incremental migration of all tables in the first partition based on the incremental migration time point recorded before restarting, as indicated by the modified task information.

[0121] With all second subtasks in the first partition completed, the data migration device can add the target data table to be migrated in the task information of the first parent task, for example, using... Figure 5 For example, the task information of the first parent task is modified from migrating data table A to migrating data tables A and B, and then the second parent task is terminated. The second subtask corresponding to the second partition can be designated as the first subtask in the first migration task and does not need to be terminated until the table migration in the second partition is complete.

[0122] In some embodiments, when the source database is a distributed database, if a table modification instruction is received during the execution of the first migration task, the data migration device can respond to the table modification instruction by modifying the task information of the first parent task and each first sub-task in the first migration task according to the modification information carried by the table modification instruction, and restart each first sub-task in the first migration task. The modification information indicates at least one of the following modification events: deleting a data table, modifying the data table name, modifying the name of a column in the data table, adding a column to the data table, and modifying the data type. Of course, if the modification information indicates deleting a data table, and there are no other data tables to be migrated after the deletion of the data table, then the first migration task ends.

[0123] The following is in conjunction with the appendix Figure 8 and Figure 9 Taking the application of the data migration method provided in this specification in a distributed database as an example, the data migration method will be further explained.

[0124] Figure 8 This specification illustrates a flowchart of a data migration method according to an embodiment, which specifically includes:

[0125] In S201, during the execution of the first migration task, if a table addition instruction is received, it is checked whether the first migration task is in a normal execution state; if yes, S203 is executed; otherwise, S202 is executed. The first migration task includes a first parent task and a first child task.

[0126] In S202, if not, do not respond to the received table addition command, and output a prompt message that the synchronization object cannot be modified.

[0127] In S203, if so, the first parent task in the first migration task is changed from the normal execution state to the synchronization object modification state. The synchronization object modification state indicates that no other table addition instructions and / or other table modification instructions will be responded to before the received table addition instruction or table modification instruction is completed.

[0128] In S204, in response to the table addition instruction, the target data table to be added and migrated in the source database is determined; and a second migration task is constructed; wherein, constructing the second migration task includes: creating a second parent task under a first parent task in the form of parent-child tasks; creating a second subtask under a first subtask corresponding to the first partition in the form of parent-child tasks; and / or creating a second subtask corresponding to the second partition, so as to realize the migration of the target data table. See also... Figure 5 , Figure 6 as well as Figure 7 .

[0129] The partition containing the target data table includes a first partition and / or a second partition; wherein the first partition indicates that the data table in the first migration task is also stored in this partition, and the second partition indicates that the data table in the first migration task is not stored in this partition.

[0130] In S205, the first migration task and the second migration task are executed in parallel in the migration chain. For the second migration task, the second parent task in the second migration task is run first. If the global pre-check in the second parent task passes, the second sub-task in the second migration task is run. If the partition pre-check in the second sub-task in the second migration task passes, the structure migration in the second parent task is run. After the structure migration is completed, the full migration and incremental migration in the second sub-task in the second migration task are run.

[0131] In S206, for any first subtask and second subtask corresponding to the first partition, the first delay duration between the incremental migration time point in the first subtask and the current time, and the second delay duration between the incremental migration time point in the second subtask and the current time are checked in a round-robin fashion to determine whether both are less than a first threshold. If yes, proceed to S207; otherwise, proceed to S206.

[0132] In S207, if both the first delay duration and the second delay duration are less than the second threshold, the second subtask is terminated, the target data table in the first partition is migrated to the task information of the first subtask, and the first subtask is restarted; after restarting the first subtask, the incremental migration time point recorded before the restart is rolled back by a preset duration to obtain the target time point; based on the target time point, the incremental migration of all subtables in the first partition indicated by the modified task information is performed.

[0133] In S208, if all second subtasks in the first partition are completed, the target data table is added to the task information of the first parent task for migration, and the second parent task is terminated. Furthermore, the second subtask corresponding to the second partition can be designated as the first subtask in the first migration task.

[0134] In S209, the first parent task is changed from the synchronization object modification state to the normal execution state.

[0135] This embodiment enables the addition of a migration target data table based on the existing migration link without interrupting the migration link or adding a new migration link.

[0136] Figure 9 This specification illustrates a flowchart of another data migration method provided in one embodiment, which specifically includes:

[0137] In S301, during the execution of the first migration task, if a table modification instruction is received, it is checked whether the first migration task is in a normal execution state; if yes, S303 is executed; if no, S302 is executed.

[0138] In S302, if not, the received table modification command is not responded to, and a prompt message indicating that the synchronization object cannot be modified is output.

[0139] In S303, if so, the first parent task in the first migration task is changed from the normal execution state to the synchronization object modification state. The synchronization object modification state indicates that before the received table addition instruction or table modification instruction is completed, other table addition instructions and / or other table modification instructions will no longer be responded to.

[0140] In S304, in response to the table modification instruction, the task information of the first parent task and each first subtask in the first migration task is modified according to the modification information carried by the table modification instruction, and each first subtask in the first migration task is restarted; for each first subtask, after restarting the first subtask, the incremental migration time point recorded before the restart is rolled back by a preset time to obtain the target time point; based on the target time point, the incremental migration of all sub-tables in the first partition indicated by the modified task information is performed.

[0141] In S305, the first parent task is changed from the synchronization object modification state to the normal execution state.

[0142] This embodiment enables modification of information related to at least one data table in the first migration task without interrupting the migration link or adding a new migration link, based on the existing migration link.

[0143] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0144] For a corresponding method embodiment, please refer to [link / reference]. Figure 1 This specification also provides a database system, including a data migration device, a source database, and a target database; the data migration device is connected to the source database and the target database respectively; the data migration device is used to execute the data migration method described in any one of the above descriptions to migrate data from the source database to the target database.

[0145] Corresponding to the above method embodiments, this specification also provides an electronic device. Figure 10This is a schematic structural diagram of an electronic device 100 provided in an exemplary embodiment. Please refer to... Figure 10 At the hardware level, the electronic device 100 includes a processor 110, an internal bus 120, a network interface 130, memory 140, and non-volatile memory 150, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, for example, the processor 110 reads the corresponding computer program from the non-volatile memory 150 into memory 140 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices. The network interface 130 enables the electronic device 100 to communicate via one or more networks. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Network interface 130 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0146] For example, electronic device 100 can interface with one or more databases 200 (such as source databases and target databases) via network interface 160.

[0147] For example, Figure 1 The data migration device shown can be applied to, for example... Figure 10 The electronic device shown is used to implement the technical solution described in this specification.

[0148] In some embodiments, this specification also provides a resource scheduling system, which includes: a target resource node-side scheduler corresponding to any target resource node in a resource node cluster, for executing the steps of any of the methods described above; and a central scheduler corresponding to the resource node cluster, for executing the steps of any of the methods described above.

[0149] In some embodiments, this specification also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor implements the method described in any one of the above embodiments by executing the executable instructions.

[0150] In some embodiments, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in any of the preceding embodiments.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0152] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0153] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0155] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0158] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0159] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0160] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data migration method, comprising establishing a migration link between a source database and a target database for running a first migration task, the first migration task being used to migrate at least one data table from the source database to the target database; the method comprising: During the execution of the first migration task, in response to the table addition instruction, the target data table to be added and migrated in the source database is determined; Create a second migration task to perform at least an incremental migration of the target data table; During the parallel execution of the first migration task and the second migration task in the migration link, the synchronization status between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task is detected by polling; the incremental migration time point indicates the write time of the incremental data to be migrated in the source database; If the two are synchronized, the second migration task is merged into the first migration task so that the merged first migration task can realize the incremental migration of the at least one data table and the target data table based on the synchronized time point. The polling detection of the synchronization between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task includes: Polling and detecting the first delay between the incremental migration time point in the first migration task and the current time, and the second delay between the incremental migration time point in the second migration task and the current time, until both are less than the first threshold; Alternatively, poll the time points of incremental migration in the first migration task and the time points of incremental migration in the second migration task until the difference between the two is less than a second threshold.

2. The method according to claim 1, wherein if the two are synchronized, the second migration task is merged into the first migration task so that the merged first migration task can realize the incremental migration of the at least one data table and the target data table, comprising: If both are synchronized, the second migration task ends, the target data table is added to the task information of the first migration task, and the first migration task is restarted; wherein, the restarted first migration task can perform incremental migration of all data tables indicated by the modified task information based on the incremental migration time points recorded before the restart.

3. The method according to claim 1, further comprising: During the execution of the first migration task, in response to the table modification instruction, the task information of the first migration task is modified according to the modification information carried by the table modification instruction, and the first migration task is restarted. The modification information is used to indicate at least one of the following modification events: deleting a data table, modifying the name of a data table, modifying the name of a column in a data table, adding a column to a data table, and modifying the data type.

4. The method according to any one of claims 1 to 3, further comprising: Upon receiving a table addition instruction or a table modification instruction, check whether the first migration task is in a normal execution state; If so, the first migration task is changed from the normal execution state to the synchronized object modification state, wherein the synchronized object modification state indicates that no other table add instructions and / or other table modification instructions will be responded to before the received table add instruction or table modify instruction is completed; and after restarting the first migration task, the first migration task is changed from the synchronized object modification state to the normal execution state. If not, do not respond to received table add or modify commands, and output a message indicating that the synchronized object cannot be modified.

5. The method according to claim 2 or 3, further comprising: After restarting the first migration task, the incremental migration time points recorded before the restart are rolled back by a preset time to obtain the target time point; Incremental migration of all data tables in the modified task information is performed based on the target time point.

6. The method according to any one of claims 1 to 3, wherein the source database comprises a distributed database; The first migration task includes a first parent task and a first child task, the number of which is determined based on the number of partitions corresponding to the data tables in the first migration task; wherein, The first parent task is used at least to perform a global pre-check between the source database and the target database; The different first subtasks are used at least to perform partition pre-check between the source database and the target database, as well as incremental migration of the data tables in the first migration task in different partitions; The second migration task includes a second parent task and a second sub-task. The number of the second sub-tasks is determined according to the number of partitions corresponding to the target data table. The second parent task is used at least to perform a global pre-check between the source database and the target database. Different second sub-tasks are used at least to perform partition pre-checks between the source database and the target database, as well as incremental migration of the target data table in different partitions.

7. The method according to claim 6, wherein the first parent task is further configured to implement the structural migration of the data table, and / or, different first sub-tasks are further configured to implement the full migration of the data table in the migration task in different partitions; in, The global pre-check in the first parent task and the partition pre-check in all first child tasks have a higher execution priority than the structure migration in the first parent task. The structure migration in the first parent task has a higher execution priority than the full migration and incremental migration in the first child tasks. The second parent task is also used to implement the structural migration of the target data table; and / or, different second sub-tasks are also used to implement the full migration of the target data table in different partitions; Among them, the global pre-check in the second parent task and the partition pre-check in all second sub-tasks have a higher execution priority than the structure migration in the second parent task, and the structure migration in the second parent task has a higher execution priority than the full migration and incremental migration in the second sub-tasks.

8. The method according to claim 6, wherein the partition in which the target data table is located includes a first partition and / or a second partition; wherein, The first partition indicates that the partition also stores the sub-tables of the data tables in the first migration task, and the second partition indicates that the partition does not store the sub-tables of the data tables in the first migration task. The creation of the second migration task includes: Create a second parent task under the first parent task in the form of parent and child tasks; A second subtask is created under the first subtask corresponding to the first partition in the form of parent and child tasks, and / or a second subtask corresponding to the second partition is created; wherein the second subtask corresponding to the second partition runs in the migration chain until the data migration is completed.

9. The method according to claim 8, wherein the polling detection of the synchronization between the incremental migration time points in the first migration task and the incremental migration time points in the second migration task comprises: For any first subtask and second subtask corresponding to the first partition, poll and check the first delay duration between the incremental migration time point in the first subtask and the current time, and the second delay duration between the incremental migration time point in the second subtask and the current time, until both are less than the first threshold. If the two are synchronized, the second migration task is merged into the first migration task so that the merged first migration task can perform incremental migration of the at least one data table and the target data table, including: For any first subtask and second subtask corresponding to a first partition, if both the first delay duration and the second delay duration are less than the second threshold, the second subtask is terminated, the target data table in the first partition is migrated to the task information of the first subtask, and the first subtask is restarted; wherein, the restarted first subtask can perform incremental migration of all subtables in the first partition as indicated by the modified task information based on the incremental migration time point recorded before the restart; If all second subtasks in the first partition are completed, the target data table is added to the task information of the first parent task for migration, and the second parent task is terminated.

10. The method of claim 6, further comprising: In response to the table modification instruction, the task information of the first parent task and each first subtask in the first migration task is modified according to the modification information carried by the table modification instruction, and each first subtask in the first migration task is restarted.

11. A database system, comprising a data migration device, a source database, and a target database; wherein the data migration device is connected to the source database and the target database respectively; The data migration device is used to execute the data migration method according to any one of claims 1 to 10, so as to migrate data from the source database to the target database.

12. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 10.

Images