Data migration method and device, equipment and medium

By dividing the data into balanced groups and binding the migration server, the problems of low data migration efficiency and poor configuration flexibility are solved, and an efficient and flexible data migration process is achieved.

CN118939633APending Publication Date: 2024-11-12CHINA CONSTRUCTION BANK +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411077233.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing data migration technology is inefficient and has poor migration configuration flexibility. Especially when facing large data tables, full migration takes time and is prone to failure, and only part of the data may be required during the testing phase.

Method used

By dividing the data to be migrated in the source database into multiple data balanced groups, and binding each group and server according to the processing capabilities of the migration server, using data balanced grouping and migration duration strategies, dynamically adjusting the migration server to match the processing Capability and data volume to achieve flexible data migration.

Benefits of technology

It improves the efficiency, scalability and flexibility of data migration, and can adjust the concurrency amount in real time according to needs, ensuring the reliability and efficiency of the migration process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939633A_ABST
    Figure CN118939633A_ABST
Patent Text Reader

Abstract

The invention relates to the field of data processing, in particular to a data migration method and device, equipment and a medium, and is used for solving the problems of low data migration efficiency and poor migration configuration flexibility in the existing data migration technology. The method comprises the steps that data to be migrated in a source database is divided into a plurality of groups based on the migration processing capacity of each migration server, and the data amount of each group is distributed in a balanced mode; each group is bound with each migration server based on the migration duration of each migration server for migrating the data in each group, and each migration server is bound with one or more groups; and in response to a data migration starting instruction, sending a migration configuration reading notification to each migration server, so that each migration server migrates the data in the group bound with the migration server from the source database to the target database. The data migration efficiency can be improved, and the expandability and flexibility of data migration can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data migration method, device, equipment and medium. Background Art

[0002] In the process of database replacement, we usually face the problem of how to efficiently and accurately migrate the massive data in the source database to the target database.

[0003] Migration tools in related technologies usually use a configuration file-based approach to achieve data migration between data sources, or use online migration technology to complete data migration tasks. However, both types of technologies face certain limitations. On the one hand, migration tools based on configuration files rely on configuration files to manage jobs, and often require restarting services when adjusting configurations or adding new jobs. This not only affects the flexibility and efficiency of operations, but also makes the configuration process extremely complex and error-prone when faced with a large number of tables to be migrated. On the other hand, although online migration technology can perform data migration without interrupting source-side business, it usually only provides table-level full migration strategies. For large data tables, full migration is not only time-consuming, but once an error occurs during the migration process, it may cause the migration of the entire table to fail. In addition, large data tables often cover a range of data for many years, and only data from a certain period of time may be required during the testing and verification phase.

[0004] Therefore, existing data migration technologies have the problems of low data migration efficiency and poor migration configuration flexibility. Summary of the invention

[0005] In view of the above problems, the present application provides a data migration method, apparatus, device and medium, which are used to solve the problems of low data migration efficiency and poor migration configuration flexibility in the prior art.

[0006] In a first aspect, the present application provides a data migration method, the method comprising:

[0007] Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups, wherein the amount of data among the groups is evenly distributed;

[0008] Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups;

[0009] In response to the data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database.

[0010] In a possible implementation manner, the data to be migrated in the source database is determined in the following manner:

[0011] In response to the received data migration scope configuration instruction, determine the data migration scope;

[0012] Based on the determined data migration scope, determining the data to be migrated in the source database;

[0013] The data to be migrated in the source database belongs to one or more target tables in the source database.

[0014] In a possible implementation, the method further includes:

[0015] Based on the migration progress reported by each of the migration servers, after determining that the data migration of the target table is completed and the migration progress of each of the groups is normal, sending a verification instruction to each of the migration servers, so that each of the migration servers performs consistency verification on the data within the specified range according to the verification rule corresponding to each specified range in the target table where the migration is completed, wherein the verification instruction includes instruction information for the target table where the migration is completed;

[0016] If there is a group with abnormal migration progress or the received verification result indicates that the consistency check has failed, the abnormal information is recorded and an alarm is triggered;

[0017] The migration progress is the migration progress of each migration server migrating the data in the group bound to itself to the target database.

[0018] In a possible implementation, the data to be migrated in the source database is divided into multiple groups based on the migration processing capability of each migration server, including:

[0019] Determining a migration threshold corresponding to each migration server based on the migration processing capability of each migration server, wherein the migration threshold of the migration server is a maximum amount of data that can be processed in parallel when the migration server migrates data;

[0020] The data to be migrated in the source database is grouped based on a minimum value among the migration thresholds so that the total amount of data included in each group does not exceed the minimum value.

[0021] In a possible implementation, binding each of the groups to each of the migration servers based on a migration duration of each of the migration servers migrating data in each of the groups includes:

[0022] For any migration server, based on the migration processing capability of the any migration server, determining a migration duration for the any migration server to migrate the data in each of the groups;

[0023] combining the determined migration durations of the migration servers for migrating the data in each group, and binding the groups to the migration servers based on the principle of shortest total migration duration;

[0024] The total migration duration is the sum of the migration durations for each migration server to migrate the data in the group bound to itself from the source database to the target database.

[0025] In a second aspect, the present application provides a data migration method, the method comprising:

[0026] After receiving the migration configuration reading notification, determine the group to which it is bound based on the binding information, wherein the binding information is obtained by dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution based on the migration processing capability of each migration server, and binding each group with each migration server based on the migration time of each migration server to migrate the data in each group;

[0027] Based on the data migration mode, the data in the bound group is migrated from the source database to the target database.

[0028] In a possible implementation manner, the data migration method includes a mapping relationship between the first data table in the source database and the second data table in the target database;

[0029] The step of migrating the data in the bound group from the source database to the target database based on the data migration method includes:

[0030] Based on the source data table identifier of the data in the bound group, determining from the first data table the target table to which the data in the bound group belongs;

[0031] Based on the mapping relationship, determining a second data table corresponding to the target table;

[0032] The data in the bound group is migrated from the target table in the source database to the corresponding second data table in the target database.

[0033] In a possible implementation, the method further includes:

[0034] If the data in the bound group belongs to multiple target tables, and the number of groups currently being processed in parallel is less than the number of bound groups, then determine the ratio of the amount of data included in the target table with the highest migration level in each bound group to the total amount of data in the group, where the target table with the highest migration level in each group is the target table with the highest migration priority among the target tables to which the data in each group belongs;

[0035] Determine the migration order between bound groups according to the determined ratio, and determine the migration order of data within each bound group according to the migration priority;

[0036] The data in the bound groups are migrated from the source database to the target database according to the number of groups currently being processed in parallel, the determined migration order between the groups, and the migration order of the data within the groups.

[0037] In a possible implementation, the method further includes:

[0038] According to a preset reporting rule, reporting the migration progress of the data in the bound group, wherein the reporting rule includes reporting when a set migration progress is reached and / or reporting when a set reporting time is reached;

[0039] After receiving the verification instruction including the indication information of the target table after the migration is completed, the consistency verification is performed on the data in the specified range according to the verification rule corresponding to each specified range in the target table after the migration is completed, and the verification result is reported.

[0040] In a possible implementation, the method further includes:

[0041] Performing consistency check on the data in each specified range of the migrated target table according to the check priority configured for the data in each specified range of the specified target table;

[0042] After determining that the verification result indicates that the consistency check has failed or the consistency check of the data in each specified range of the target table after the migration is completed, the verification result is reported.

[0043] In a third aspect, the present application provides a data migration method, the method comprising:

[0044] Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups, wherein the amount of data among the groups is evenly distributed;

[0045] Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups;

[0046] In response to the data migration start instruction, each migration server determines the group bound to itself based on the binding information between each migration server and the group, and migrates the data in the group bound to itself from the source database to the target database based on the data migration mode.

[0047] In a fourth aspect, the present application provides a data migration device, the device comprising:

[0048] A data partitioning module, used to partition the data to be migrated in the source database into a plurality of groups based on the migration processing capability of each migration server, wherein the amount of data between the groups is evenly distributed;

[0049] A data binding module, configured to bind each of the groups to each of the migration servers based on a migration duration of each of the migration servers for migrating data in each of the groups, wherein each of the migration servers is bound to one or more of the groups;

[0050] The data migration start module is used to send a migration configuration read notification to each migration server in response to the data migration start instruction, so that each migration server migrates the data in the group bound to itself from the source database to the target database.

[0051] In a possible implementation manner, the device further includes a module for determining data to be migrated, and the module for determining data to be migrated is configured to determine the data to be migrated in the source database in the following manner:

[0052] In response to the received data migration scope configuration instruction, determine the data migration scope;

[0053] Based on the determined data migration scope, determining the data to be migrated in the source database;

[0054] The data to be migrated in the source database belongs to one or more target tables in the source database.

[0055] In a possible implementation, the device further includes a verification indication and alarm module, which is used to send a verification instruction to each migration server based on the migration progress reported by each migration server, each time it is determined that the data migration of the target table is completed and the migration progress of each group is normal, so that each migration server performs consistency verification on the data within the specified range according to the verification rule corresponding to each specified range in the target table where the migration is completed, wherein the verification instruction includes indication information for the target table where the migration is completed;

[0056] If there is a group with abnormal migration progress or the received verification result indicates that the consistency check has failed, the abnormal information is recorded and an alarm is triggered;

[0057] The migration progress is the migration progress of each migration server migrating the data in the group bound to itself to the target database.

[0058] In a possible implementation manner, the data partitioning module is specifically used to:

[0059] Determining a migration threshold corresponding to each migration server based on the migration processing capability of each migration server, wherein the migration threshold of the migration server is a maximum amount of data that can be processed in parallel when the migration server migrates data;

[0060] The data to be migrated in the source database is grouped based on a minimum value among the migration thresholds so that the total amount of data included in each group does not exceed the minimum value.

[0061] In a possible implementation, the data binding module is specifically used to:

[0062] For any migration server, based on the migration processing capability of the any migration server, determining a migration duration for the any migration server to migrate the data in each of the groups;

[0063] combining the determined migration durations of the migration servers for migrating the data in each group, and binding the groups to the migration servers based on the principle of shortest total migration duration;

[0064] The total migration duration is the sum of the migration durations for each migration server to migrate the data in the group bound to itself from the source database to the target database.

[0065] In a fifth aspect, the present application provides a data migration device, the device comprising:

[0066] a binding information determination module, configured to determine the group to which it is bound based on the binding information after receiving the migration configuration reading notification, wherein the binding information is obtained by binding each of the groups to each of the migration servers based on the migration processing capability of each migration server after dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution and based on the migration time of each migration server migrating the data in each of the groups;

[0067] The data migration module is used to migrate the data in the bound group from the source database to the target database based on the data migration method.

[0068] In a possible implementation manner, the data migration method includes a mapping relationship between the first data table in the source database and the second data table in the target database; the data migration module is specifically used to:

[0069] Based on the source data table identifier of the data in the bound group, determining from the first data table the target table to which the data in the bound group belongs;

[0070] Based on the mapping relationship, determining a second data table corresponding to the target table;

[0071] The data in the bound group is migrated from the target table in the source database to the corresponding second data table in the target database.

[0072] In a possible implementation, the device further includes a sequential migration module, which is used to determine the proportion of the amount of data included in the highest migration-level target table in each bound group to the total amount of data in the group if the data in the bound group belongs to multiple target tables and the number of groups currently being processed in parallel is less than the number of bound groups, wherein the highest migration-level target table in each group is the target table with the highest migration priority among the target tables to which the data in each group belongs;

[0073] Determine the migration order between bound groups according to the determined ratio, and determine the migration order of data within each bound group according to the migration priority;

[0074] The data in the bound groups are migrated from the source database to the target database according to the number of groups currently being processed in parallel, the determined migration order between the groups, and the migration order of the data within the groups.

[0075] In a possible implementation, the device further includes a reporting and verification module, configured to report the migration progress of the data in the bound group according to a preset reporting rule, wherein the reporting rule includes reporting when a set migration progress is reached and / or reporting when a set reporting time is reached;

[0076] After receiving the verification instruction including the indication information of the target table after the migration is completed, the consistency verification is performed on the data in the specified range according to the verification rule corresponding to each specified range in the target table after the migration is completed, and the verification result is reported.

[0077] In a possible implementation, the device further includes a sequential verification module, configured to perform consistency verification on the data in each specified range of the target table after the migration according to the verification priority configured for the data in each specified range of the specified target table;

[0078] After determining that the verification result indicates that the consistency check has failed or the consistency check of the data in each specified range of the target table after the migration is completed, the verification result is reported.

[0079] In a sixth aspect, the present application provides an electronic device, including:

[0080] A memory for storing program instructions;

[0081] A processor is used to call the program instructions stored in the memory, and execute the steps included in the method of any one of the first aspect, or execute the steps included in the method of any one of the second aspect, or execute the steps included in the method of any one of the third aspect according to the obtained program instructions.

[0082] In the seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods in the first aspect, or executes any one of the methods in the second aspect, or executes any one of the methods in the third aspect.

[0083] In an eighth aspect, the present application provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method described in any one of the first aspect, or the method described in any one of the second aspect, or the method described in any one of the third aspect.

[0084] The beneficial effects of this application are as follows:

[0085] In an embodiment of the present application, first, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups with evenly distributed data volumes; then, based on the migration time of each migration server migrating the data in each group, each group is bound to each migration server, wherein each migration server is bound to one or more groups; finally, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database, thereby reasonably matching the migration processing capability of the migration server with the amount of migration data assigned to it for processing through balanced data grouping and a binding strategy based on migration time, thereby improving the overall data migration efficiency, and at the same time, by dynamically adding migration servers, the concurrency of migration can be adjusted in real time to cope with different migration requirements, thereby improving the scalability and flexibility of data migration.

[0086] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0088] Figure 1A schematic diagram of an application scenario provided for an embodiment of the present application;

[0089] Figure 2 A schematic diagram of a data migration method provided in an embodiment of the present application;

[0090] Figure 3 A schematic diagram of a data partitioning method flow provided in an embodiment of the present application;

[0091] Figure 4 A schematic diagram of a data binding method flow provided in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of a data verification method flow provided in an embodiment of the present application;

[0093] Figure 6 A schematic diagram of another data migration method provided in an embodiment of the present application;

[0094] Figure 7 A schematic diagram of a method flow for performing data migration according to migration priority provided in an embodiment of the present application;

[0095] Figure 8 A schematic diagram of a migration reporting and verification method flow provided in an embodiment of the present application;

[0096] Fig. 9 A schematic diagram of a method flow for performing data verification according to verification priority provided in an embodiment of the present application;

[0097] Fig.10 A schematic diagram of a data migration process based on five stages provided in an embodiment of the present application;

[0098] Fig.11 A schematic diagram of a data migration device provided in an embodiment of the present application;

[0099] Fig.12 A schematic diagram of another data migration device provided in an embodiment of the present application;

[0100] Fig.13 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0102] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0103] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0104] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0105] Before introducing the data migration method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0106] In the process of database replacement, we usually face the problem of how to efficiently and accurately migrate the massive data in the source database to the target database.

[0107] Migration tools in related technologies usually use a configuration file-based approach to achieve data migration between data sources, or use online migration technology to complete data migration tasks. However, both types of technologies face certain limitations. On the one hand, migration tools based on configuration files rely on configuration files to manage jobs, and often require restarting services when adjusting configurations or adding new jobs. This not only affects the flexibility and efficiency of operations, but also makes the configuration process extremely complex and error-prone when faced with a large number of tables to be migrated. On the other hand, although online migration technology can perform data migration without interrupting source-side business, it usually only provides table-level full migration strategies. For large data tables, full migration is not only time-consuming, but once an error occurs during the migration process, it may cause the migration of the entire table to fail. In addition, large data tables often cover a range of data for many years, and only data from a certain period of time may be required during the testing and verification phase.

[0108] Therefore, existing data migration technologies have the problems of low data migration efficiency and poor migration configuration flexibility.

[0109] In view of this, in order to improve the efficiency of data migration and improve the scalability and flexibility of data migration, the embodiments of the present application provide a data migration method, device, equipment and medium. Some preferred embodiments of the present application are described below in conjunction with the drawings of the specification.

[0110] Embodiment 1:

[0111] In this application, data migration refers to the process of migrating a large amount of data from one storage system (such as a source database) to another storage system (such as a target database). The migration server is a server dedicated to performing data migration tasks, responsible for reading the data to be migrated from the source database and transferring it to the target database. The application scenario of this application is described below.

[0112] like Figure 1 The figure shows a schematic diagram of an application scenario provided in an embodiment of the present application.

[0113] The data migration application scenario provided in this application includes a source database, a target database, a migration configuration server and various migration servers.

[0114] The source database is used to store the data that needs to be migrated, and the target database is the destination of the data migration.

[0115] The migration configuration server is responsible for planning, coordinating, and monitoring the entire migration process, such as determining which data needs to be migrated from the source database to the target database, determining the migration tasks assigned to each migration server, and monitoring the migration process. Each migration server reads data from the source database based on the migration tasks assigned by the migration configuration server and writes the data to the target database.

[0116] Figure 2 A flow chart of a data migration method provided in an embodiment of the present application is provided, wherein the method is applied to migrate a configuration server.

[0117] like Figure 2 As shown, the process includes the following steps:

[0118] S201: Divide the data to be migrated in the source database into multiple groups based on the migration processing capability of each migration server;

[0119] Among them, the amount of data among each group is evenly distributed.

[0120] In the prior art, when performing database migration, a table-level full migration is usually performed, that is, if a table migration is configured, the entire table is migrated. However, in actual data migration scenarios, different project cycles have different requirements for data volume. For example, for a large bank account system, there are many tables with large data volumes. If only the full migration strategy is used, not only will the full migration take a long time, and if the migration process fails, the full table migration will fail. In addition, a large table may cover a data range of several years. In the testing and verification phase, only one year or a few months of data may be needed.

[0121] Therefore, in order to improve the efficiency of data migration and improve the scalability and flexibility of data migration, the embodiments of the present application support configuring the data migration range, and then automatically determine the data to be migrated according to the configured data migration range.

[0122] As a feasible implementation method, the embodiment of the present application adopts the following method to determine the data to be migrated in the source database:

[0123] In response to the received data migration scope configuration instruction, determine the data migration scope;

[0124] Based on the determined data migration scope, determine the data to be migrated in the source database;

[0125] The data to be migrated in the source database belongs to one or more target tables in the source database.

[0126] Specifically, by including the identifier of the target table to be migrated and the migration range for the target table in the data migration range configuration instruction, the data to be migrated within the data migration range is determined. Based on the configurable data migration range, the embodiment of the present application can support migration at the table plus partition level, that is, support migration of all or part of the data in the target table. For example, if the data migration range configuration instruction can indicate that the data to be migrated includes all the data in Table 1 and the data in Table 2 for the most recent year.

[0127] In the embodiment of the present application, the migration processing capability of the migration server refers to the ability of the migration server to concurrently process migration tasks. The stronger the migration processing capability of the migration server, the larger the amount of data that can be migrated simultaneously (i.e., the maximum amount of data that can be processed in parallel in the same time).

[0128] In the embodiments of the present application, the migration processing capabilities of each migration server may be the same or different. Therefore, in order to improve the efficiency of data migration and improve the scalability and flexibility of data migration, the present application divides the data to be migrated, especially a large amount of data to be migrated, into multiple groups with balanced data volume distribution based on the migration processing capabilities of each migration server, which helps to flexibly schedule migration tasks and reasonably match the migration processing capabilities of the migration server with the amount of migration data allocated to it for processing, and flexibly adapt to different migration needs and scenarios.

[0129] S202: Binding each group to each migration server based on the migration time of each migration server migrating data in each group;

[0130] Each migration server is bound to one or more groups.

[0131] In order to improve the efficiency of data migration and improve the scalability and flexibility of data migration, in the embodiment of the present application, after the data to be migrated is grouped, each group can be bound to each migration server based on the migration time of each migration server migrating the data in each group. Specifically, as described in the aforementioned embodiment, the migration processing capabilities of each migration server may be different. Therefore, in order to improve the efficiency of data migration, by estimating the migration time of each migration server migrating each group of data, and binding each group to each migration server based on the estimated migration time, the binding goal is to avoid overloading some migration servers, while also preventing some migration servers from being idle and causing waste of resources.

[0132] It should be noted that in the embodiments of the present application, the migration duration can be calculated based on the data transmission speed and data volume, or other existing migration duration estimation methods can be adopted according to actual scenarios and needs. The present application does not limit this.

[0133] S203: In response to the data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database.

[0134] In an embodiment of the present application, after the migration configuration server groups the data to be migrated and determines the binding relationship between each group and each migration server, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server so that each migration server migrates the data in the group bound to itself from the source database to the target database.

[0135] In some embodiments, the migration configuration server of the present application monitors the resource configuration of the current migration server. When the number of migration servers changes, the groups bound to each migration server are dynamically adjusted based on the data to be migrated that has not yet been migrated, so that the number of channels for data migration can be flexibly reduced or expanded.

[0136] In some embodiments, the present application configures a migration configuration file on the migration configuration server, and stores the binding information of each group and each migration server and the data migration method in the migration configuration file, so that each migration server determines the group bound to itself by reading the migration configuration file, and migrates the data in the group bound to itself from the source database to the target database according to the configured data migration method.

[0137] As mentioned above, the data to be migrated in the source database belongs to one or more target tables in the source database.

[0138] In some embodiments, the above-mentioned data migration method includes a mapping relationship between a first data table in a source database and a second data table in a target database, so that each migration server determines the target table where the data in its own bound group is located from the first data table in the source database by reading the migration configuration file, and after exporting the data in the bound group from the determined target table, determines the second data table corresponding to the target table based on the above-mentioned mapping relationship, and imports the exported data into the position corresponding to the determined second data table in the target database.

[0139] In an embodiment of the present application, first, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups with evenly distributed data volumes; then, based on the migration time of each migration server migrating the data in each group, each group is bound to each migration server, wherein each migration server is bound to one or more groups; finally, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database, thereby reasonably matching the migration processing capability of the migration server with the amount of migration data assigned to it for processing through balanced data grouping and a binding strategy based on migration time, thereby improving the overall data migration efficiency, and at the same time, by dynamically adding migration servers, the concurrency of migration can be adjusted in real time to cope with different migration requirements, thereby improving the scalability and flexibility of data migration.

[0140] Embodiment 2:

[0141] refer to Figure 3In order to further improve the data migration efficiency and improve the scalability and flexibility of data migration, based on the above embodiment, in the embodiment of the present application, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into multiple groups, including the following steps:

[0142] S301: Determine a migration threshold corresponding to each migration server based on the migration processing capability of each migration server.

[0143] The migration threshold of the migration server is the maximum amount of data that can be processed in parallel by the migration server when migrating data.

[0144] S302: based on the minimum value of each migration threshold, group the data to be migrated in the source database so that the total amount of data included in each group does not exceed the minimum value.

[0145] From the above steps, it can be seen that when the data to be migrated in the source database is grouped based on the minimum value of each migration threshold, the data size of each group depends on the migration processor with the weakest migration processing capability (the migration server with the smallest migration threshold).

[0146] Exemplarily, when the amount of data to be migrated is 2000 GB, the following two grouping situations may be included.

[0147] Case 1:

[0148] If the migration processing capabilities of migration servers 1 and 2 are the same and the migration thresholds are both 100 GB / hour, the amount of data to be migrated can be divided into 20 groups according to the migration processing capabilities of migration servers 1 and 2, with each group including 100 GB of data. The allocation plan for subsequent migration tasks can be that the two migration servers each process 10 groups.

[0149] Case 2:

[0150] If the migration processing capabilities of migration servers 1 and 2 are different, and the migration thresholds are 200GB / hour and 800GB / hour respectively, the amount of data to be migrated can be divided into 10 groups according to the migration processing capability of migration server 1, with each group including 200GB of data. The allocation plan for subsequent migration tasks can be that migration server 1 processes 2 groups, and the other 8 groups are processed by migration server 2.

[0151] In some embodiments, in addition to grouping the data to be migrated based on the minimum value of each migration threshold, the data to be migrated can also be grouped according to a preset partitioning threshold, so that the data size corresponding to each group obtained by the partitioning does not exceed the partitioning threshold, and the data to be migrated is divided into multiple groups with balanced data volume distribution. The specific partitioning threshold can be set according to actual needs on the premise of facilitating the flexible scheduling of migration tasks. For example, the partitioning threshold can be taken as the common divisor of each migration threshold. In the above case 2, 20 groups each including 100 GB of data can be divided.

[0152] Embodiment 3:

[0153] refer to Figure 4 In order to further improve the data migration efficiency and improve the scalability and flexibility of data migration, based on the above embodiments, in the embodiments of the present application, the binding of each group to each migration server based on the migration time of each migration server migrating the data in each group includes:

[0154] S401 : for any migration server, determining a migration duration for the migration server to migrate data in each group based on the migration processing capability of the migration server.

[0155] S402: Combine the determined migration durations of each migration server for migrating data in each group, and bind each group to each migration server based on the principle of shortest total migration duration.

[0156] The total migration duration is the sum of the migration durations for each migration server to migrate the data in the group bound to itself from the source database to the target database.

[0157] For example, if the amount of data to be migrated is 1000 GB, there are three migration servers, marked as A, B and C respectively, and the migration processing capabilities of servers A, B and C are as follows.

[0158] Migration server A: 100 GB / hour.

[0159] Migration server B: 400 GB / hour.

[0160] Migration server C: 200 GB / hour.

[0161] The minimum migration threshold of each migration server is 100. After 1000 GB of data to be migrated is divided into 10 groups of 100 GB in size, the binding relationship determined based on the principle of shortest total migration time is as follows.

[0162] Migration server A: bind 1 group (the migration data volume is 100 GB, and the migration time is 100 GB / 100 GB / h = 1 hour).

[0163] Migration server B: bind 6 groups (the migration data volume is 600 GB, and the migration time is 600 GB / 400 GB / h = 1.5 hours).

[0164] Migration server C: binds 4 groups (the migration data volume is 300 GB, and the migration time is 300 GB / 200 GB / h=1.5 hours).

[0165] Embodiment 4:

[0166] In order to further improve the efficiency of data migration and improve the scalability and flexibility of data migration, based on the above embodiments, in the embodiments of the present application, the above migration configuration file also includes a migration priority configured for a specified target table, so that each migration server can migrate the data in the bound group from the source database to the target database by reading the migration priority configured in the migration configuration file.

[0167] Specifically, for each migration server, first, determine whether the data in the bound group belongs to multiple target tables, and whether the number of groups currently processed in parallel is less than the number of bound groups; if it is determined that the data in the bound group belongs to multiple target tables, and the number of groups currently processed in parallel is less than the number of bound groups, then determine the proportion of the data amount included in the highest migration level target table in each bound group to the total amount of data in the group;

[0168] Among them, the target table with the highest migration level in each group is the target table with the highest migration priority among the target tables to which the data in each group belongs.

[0169] Afterwards, the migration order between the bound groups is determined according to the determined ratio, and the migration order of the data within each bound group is determined according to the migration priority.

[0170] Finally, according to the number of groups currently being processed in parallel, the determined migration order between groups, and the migration order of data within a group, the data in the bound groups is migrated from the source database to the target database.

[0171] For example, if a migration server is bound to two groups (Group 1 and Group 2), the data in the bound groups come from two target tables (Table 1 and Table 2) and the number of groups that the migration server can process in parallel is 1, and Table 1 has a higher priority than Table 2, then the group with a higher proportion of data from Table 1 will be migrated first among Group 1 and Group 2, and the data from Table 1 will be migrated first among Group 1 and Group 2.

[0172] Embodiment 5:

[0173] In order to further improve the efficiency of data migration and improve the scalability and flexibility of data migration, on the basis of the above embodiments, in the embodiments of the present application, the above migration configuration file also includes reporting rules, so that each migration server determines the reporting rules by reading the migration configuration file, and reports the migration progress of the data in the bound group according to the determined reporting rules. Specifically, the reporting rules include reporting when the set migration progress is reached and / or reporting when the set reporting time is reached.

[0174] In some embodiments, in order to further improve the efficiency of data migration and improve the scalability and flexibility of data migration, based on the above embodiments, in the embodiments of the present application, the above migration configuration file also includes field conversion rules, so that each migration server converts the content of the specified field according to the field conversion rules before migrating the data to the target database. For example, for characters in half-width fields in the source database, the fields are converted to half-width before being migrated to the target database.

[0175] Embodiment 6:

[0176] refer to Figure 5 In order to further improve the efficiency of data migration and improve the scalability and flexibility of data migration, based on the above embodiment, in the embodiment of the present application, the above migration configuration file also includes verification rules configured for each specified range in the specified target table. Based on this, the method also includes the following steps S501 and S502.

[0177] S501 , based on the migration progress reported by each migration server, after it is determined that the data migration of the target table is completed and the migration progress of each group is normal, a verification instruction is sent to each migration server.

[0178] The verification instruction includes instruction information for the target table after the migration is completed.

[0179] S502: If there is a group with abnormal migration progress or the received verification result indicates that the consistency verification has failed, the abnormal information is recorded and an alarm is triggered.

[0180] The migration progress is the migration progress of each migration server migrating the data in the group bound to itself to the target database.

[0181] Optionally, the migration configuration server determines whether the migration progress is normal based on whether the time taken by each migration server to migrate each group of tasks determined in the aforementioned embodiment is consistent with the migration progress of each group reported by each migration server.

[0182] Therefore, in the embodiment of the present application, each time the migration configuration server determines the target table for migration completion, if it finds a group with abnormal migration progress, it will not send a verification instruction to each migration server, but record the abnormal information and trigger an alarm to avoid affecting the entire migration process.

[0183] In some embodiments, after receiving the data migration start instruction, the migration configuration server of the present application collects statistics on the status information of each migration server during the migration process (including migration progress and verification results), and determines whether the data migration is completed based on the statistical information, so that the data can be used after migration, thereby improving the efficiency of data migration.

[0184] In S501, the present application flexibly configures the verification range of the data of the specified target table and the verification methods corresponding to different ranges. Based on this verification mechanism, a verification instruction including the indication information for the target table after migration is completed is sent to each migration server, so that each migration server performs consistency verification on the data within the specified range according to the verification rules corresponding to each specified range in the target table after migration is completed. Therefore, the present application implements a flexible verification mechanism, which can optimize resource utilization and improve the efficiency of verification work. The specific reasons are as follows.

[0185] First, not all data requires the same verification strength. For unimportant data, sampling can save a lot of time and computing resources, allowing verification resources to be more concentrated on the verification of key data.

[0186] Secondly, different data types and formats may require different validation strategies. For example, for numerical data, you may need to check the range, precision, and outliers; for text data, you may need to check the length, format, and specific pattern. Using targeted validation methods can significantly improve validation efficiency and reduce unnecessary calculations and comparisons.

[0187] Finally, by flexibly configuring the verification range, you can also implement partition verification of the target table. When it is determined that a partition has not passed the consistency check, you can directly locate the data that failed to migrate without having to check the entire table.

[0188] In some embodiments, the granularity of verification in the present application is each target table:

[0189] After the migration configuration server determines that the migration has been completed and the migration progress of each group is normal, it sends a verification instruction including instruction information for the migration has been completed to each migration server, so that each migration server reads the migration configuration file to determine the verification rules, and performs consistency verification on the data in the specified range of the migration has been completed according to the determined verification rules, wherein the migration server that has migrated the data of the target table verifies the data migrated by itself.

[0190] In some embodiments, if part of the data in the target table belongs to the data migration scope, this part of the data corresponds to one or more partitions in the target table. The granularity of verification in this application can also be each partition in each target table:

[0191] After the migration configuration server determines the partition of the target table for which migration has been completed and the migration progress of each group is normal, the migration configuration server sends a verification instruction including instruction information for the partition to each migration server, so that each migration server reads the migration configuration file to determine the verification rule, and performs consistency verification on the data in the specified range corresponding to the partition according to the determined verification rule;

[0192] The migration server that has migrated the data of the partition verifies the data migrated by itself.

[0193] In some embodiments, the above-mentioned migration configuration file also includes a verification priority configured for the data in each specified range of the specified target table, so that each migration server performs consistency verification on the data in the specified range of the migrated target table according to the verification priority configured for the data in each specified range of the specified target table and according to the determined verification rules.

[0194] Specifically, the verification priority configured for the data in each specified range of the specified target table is based on the consideration that for databases with large data volumes and complex structures, prioritizing the verification of key data can make more efficient use of verification resources. If problems arise with key data during the migration process, prioritizing the verification of these data can quickly identify the problems and take immediate remedial measures, such as immediately stopping the entire migration process, to avoid further data damage or loss, thereby helping to reduce the risk of data migration.

[0195] In some embodiments, in S502, if the migration configuration server of the present application determines that there is a group with abnormal migration progress or the received verification result indicates a consistency check failure, it will re-instruct the migration servers with abnormal progress or consistency check failure to migrate again after troubleshooting, and will instruct these migration servers to clean up the data with abnormal progress or consistency check failure in the target database before migration.

[0196] In some embodiments, the above-mentioned migration configuration file also includes a migration execution window set for data within a specified range of the data to be migrated. Based on the migration execution window, the present application supports the migration configuration server to group and bind the data within the specified range, and then the migration server to migrate the data within the specified range within a specified time window.

[0197] Embodiment 7:

[0198] refer to Figure 6 Based on the same technical concept, on the basis of the above embodiments, in an embodiment of the present application, a data migration method is provided, which is applied to a migration server and includes the following steps:

[0199] S601, after receiving a migration configuration reading notification, determine the group to which the group is bound based on the binding information.

[0200] The binding information is obtained by dividing the data to be migrated in the source database into multiple groups with balanced data volume distribution based on the migration processing capacity of each migration server, and binding each group to each migration server based on the migration time of each migration server to migrate the data in each group.

[0201] In the embodiment of the present application, after receiving the migration configuration reading notification, each migration server will read the migration configuration file obtained in the above embodiment from the migration configuration server, and then complete the migration of data in the group bound to itself according to the read content.

[0202] The specific contents of the migration configuration file have been described in the above embodiments and will not be repeated here.

[0203] S602: Migrate the data in the bound group from the source database to the target database based on the data migration method.

[0204] In some embodiments, the above-mentioned data migration method includes a mapping relationship between a first data table in a source database and a second data table in a target database. As a feasible implementation method, based on the data migration method, the data in the bound group is migrated from the source database to the target database, including: based on the source data table identifier of the data in the bound group, determining from the first data table the target table to which the data in the bound group belongs; based on the above-mentioned mapping relationship, determining the second data table corresponding to the target table; and migrating the data in the bound group from the target table in the source database to the corresponding second data table in the target database.

[0205] refer to Figure 7In order to further improve the efficiency of data migration and improve the scalability and flexibility of data migration, based on the above embodiment, if the data in the group currently bound to the migration server belongs to multiple target tables, and the number of groups currently processed in parallel is less than the number of bound groups, the method further includes:

[0206] S701, determining the proportion of the amount of data included in the highest migration level target table in each bound group to the total amount of data in the group.

[0207] The target table with the highest migration level in each group is the target table with the highest migration priority among the target tables to which the data in each group belongs;

[0208] S702, determining a migration order between bound groups according to the determined ratio, and determining a migration order of data within each bound group according to the migration priority;

[0209] S703 , migrating the data in the bound groups from the source database to the target database according to the number of groups currently being processed in parallel, the determined migration order between the groups, and the migration order of the data within the group.

[0210] The specific implementation of the above steps refers to the aforementioned Example 4 and will not be described in detail here.

[0211] refer to Figure 8 In order to further improve the data migration efficiency and improve the scalability and flexibility of data migration, based on the above embodiment, the method further includes:

[0212] S801: Report the migration progress of data in the bound group according to a preset reporting rule.

[0213] The reporting rules include reporting when a set migration progress is reached and / or reporting when a set reporting time is reached.

[0214] S802, after receiving the verification instruction including the indication information of the target table after the migration, according to the verification rule corresponding to each specified range in the target table after the migration, perform consistency verification on the data in the specified range, and report the verification result.

[0215] refer to Fig. 9 In order to further improve the data migration efficiency and improve the scalability and flexibility of data migration, based on the above embodiment, the method further includes:

[0216] S901, performing consistency check on the data in each specified range of the migrated target table according to the check priority configured for the data in each specified range of the specified target table;

[0217] S902: After determining that the verification result indicates that the consistency check has failed or the consistency check of the data in each specified range of the target table after the migration is completed, the verification result is reported.

[0218] The specific implementation of the above steps refers to the aforementioned embodiments 5 and 6 and will not be described in detail here.

[0219] Embodiment 8:

[0220] In some embodiments, the migration configuration server and each migration server in the aforementioned embodiments are concentrated in the same migration cluster, and the data migration method in the aforementioned embodiments is performed in the migration cluster. Based on the same technical concept, on the basis of the aforementioned embodiments, in an embodiment of the present application, a data migration method applied to a migration cluster is provided, and the method comprises the following steps:

[0221] Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into multiple groups; wherein the amount of data between the groups is evenly distributed; based on the migration time of each migration server to migrate the data in each group, each group is bound to each migration server; wherein each migration server is bound to one or more groups; in response to the data migration start instruction, each migration server determines the group to which it is bound based on the binding information between each migration server and the group, and migrates the data in the group to which it is bound from the source database to the target database based on the data migration method.

[0222] In order to more clearly illustrate the technical solution of the embodiment of the present application, the entire data migration process is divided into five stages: data binding, data unloading, data cleaning and loading, data verification and statistical information collection. On the premise of setting a migration operation window, the data migration solution of the present application is illustrated by taking the source database as an Oracle database and the target database as a TiDB database as an example.

[0223] It should be noted that the following content is only an example of the data migration solution provided by the present application. The embodiment of the present application can be adjusted accordingly according to actual conditions. For example, the type of database, the replacement of the loading tool, the size of the multiple files into which each group of data is split when uninstalling, etc., are not limited here.

[0224] refer to Fig.10 , the specific migration process is as follows.

[0225] The following S1001-S1003 illustrate the data binding phase from the perspective of the migration configuration server.

[0226] S1001, determining the data to be migrated in the Oracle database according to the received data migration range configuration instruction;

[0227] S1002: based on the migration processing capability of each migration server, divide the data to be migrated into a plurality of groups with balanced data distribution.

[0228] S1003: Bind each group to each migration server based on the migration time of each migration server migrating the data in each group.

[0229] The following S1004-S1006 illustrate the data unloading phase from the perspective of each migration server.

[0230] S1004, periodically read the migration configuration file, and execute S1005 when it is determined that the current migration is within the migration operation window period.

[0231] S1005: Determine the group to which the group is bound based on the binding information configured in the migration configuration file.

[0232] S1006, based on the data migration mode configured in the migration configuration file, split the data in each group bound to itself into multiple files, unload them from the Oracle database to the specified shared folder, and update the data unloading status of each file.

[0233] Specifically, the data migration method includes an export program (pre-configured export SQL) for executing data unloading. Each migration server executes the export SQL on the Oracle database, splits and unloads the data in each group bound to itself into multiple files (such as one file for every 500,000 records to avoid excessive unloading pressure caused by a single file being too large), and stores the files in the table / segment directory of the specified directory.

[0234] After each file is uninstalled, the data uninstallation status of the file is updated to success, so that other subsequent stages can process the successfully uninstalled file.

[0235] The following S1007-S1009 describe the data cleaning and loading phase and the data verification phase from the perspective of each migration server.

[0236] S1007, based on the data migration method configured in the migration configuration file, clean up the content stored in the corresponding location in the TiDB database and update the data cleaning status of each file.

[0237] Specifically, the data migration method includes an export program (pre-configured cleanup SQL) for performing data cleanup. The corresponding position in the above TiDB database refers to the location where the data in the file successfully unloaded in S1006 should be migrated. For the file successfully unloaded in S1006, each migration server cleans up the content stored in the corresponding position in the TiDB database according to the pre-configured cleanup SQL to facilitate subsequent data loading.

[0238] S1008, based on the data migration method configured in the migration configuration file, import the files that have completed data unloading and data cleaning into the corresponding locations in the TiDB database, and update the data loading status of each file.

[0239] Specifically, the data migration method includes an export program (pre-configured loading SQL) for executing data loading. In an embodiment of the present application, the pre-configured loading SQL can be implemented based on the lightning tool of TiDB. Each migration server uses the lightning tool of TiDB to load the files that are successfully unloaded in S1006 and successfully cleaned in S1007 into the library, and updates the data loading status to success after the loading is successful, so that the subsequent links can process the files successfully loaded by the device.

[0240] S1009: Based on the verification rules configured in the migration configuration file, a consistency check is performed on the target table where the file that has completed data loading is located, and the verification results of each target table are updated.

[0241] Specifically, the verification rules include a verification program for executing data verification (pre-configured source-side verification SQL and target-side verification SQL). Optionally, the source-side verification SQL and the target-side verification SQL can be verified based on whether the number of data items is consistent or whether the CRC32 values ​​of key fields are consistent. The specific verification process refers to the aforementioned embodiment and will not be described in detail here.

[0242] The following S1010 describes the statistical information collection phase from the perspective of the migration configuration server.

[0243] S1010, recording migration status information of the to-be-migrated data corresponding to each target table.

[0244] In some embodiments, in order to facilitate data migration monitoring and scheduling, the present application views the migration status information (unloading status, cleaning status, loading status, verification status, and statistical information collection status) of each target table through the interface, and the background resident process initiates the following five processes respectively: data binding process, data unloading process, data cleaning and loading process, data verification process, and statistical information collection process.

[0245] Among them, the data binding process is used to execute the above S1001-S1003, the data unloading process is used to execute the above S1004-S1006, the data cleaning and loading process is used to execute the above S1007-S1008, the data verification process is used to execute the above S1009, and the statistical information collection process is used to execute the above S1010.

[0246] Through the above five processes, the migration of each target table can be divided into five standardized steps. The five steps are serially dependent. When the previous step is completed, the subsequent step is automatically called up to achieve streaming processing. If there is no step to be processed, the process will be dormant. At the same time, each group migrates in parallel without waiting for each other. Through this streaming processing, the waiting time can be greatly reduced, the existing migration resources can be maximized, and the migration of massive data can be completed.

[0247] Embodiment 9:

[0248] Based on the same technical concept, on the basis of the above embodiments, in an embodiment of the present application, a data migration device is provided. Fig.11 A schematic diagram of the structure of a data migration device provided in an embodiment of the present application. Fig.11 As shown, the device comprises:

[0249] The data partitioning module 1101 is used to partition the data to be migrated in the source database into a plurality of groups based on the migration processing capability of each migration server, wherein the amount of data between the groups is evenly distributed;

[0250] The data binding module 1102 is used to bind each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups;

[0251] The data migration initiation module 1103 is configured to send a migration configuration read notification to each migration server in response to the data migration initiation instruction, so that each migration server migrates the data in the group bound to itself from the source database to the target database.

[0252] In a possible implementation manner, the device further includes a module for determining data to be migrated, and the module for determining data to be migrated is configured to determine the data to be migrated in the source database in the following manner:

[0253] In response to the received data migration scope configuration instruction, determine the data migration scope;

[0254] Based on the determined data migration scope, determining the data to be migrated in the source database;

[0255] The data to be migrated in the source database belongs to one or more target tables in the source database.

[0256] In a possible implementation, the device further includes a verification indication and alarm module, which is used to send a verification instruction to each migration server based on the migration progress reported by each migration server, each time it is determined that the data migration of the target table is completed and the migration progress of each group is normal, so that each migration server performs consistency verification on the data within the specified range according to the verification rule corresponding to each specified range in the target table where the migration is completed, wherein the verification instruction includes indication information for the target table where the migration is completed;

[0257] If there is a group with abnormal migration progress or the received verification result indicates that the consistency check has failed, the abnormal information is recorded and an alarm is triggered;

[0258] The migration progress is the migration progress of each migration server migrating the data in the group bound to itself to the target database.

[0259] In a possible implementation, the data partitioning module 1101 is specifically used to:

[0260] Determining a migration threshold corresponding to each migration server based on the migration processing capability of each migration server, wherein the migration threshold of the migration server is a maximum amount of data that can be processed in parallel when the migration server migrates data;

[0261] The data to be migrated in the source database is grouped based on a minimum value among the migration thresholds so that the total amount of data included in each group does not exceed the minimum value.

[0262] In a possible implementation, the data binding module 1102 is specifically used for:

[0263] For any migration server, based on the migration processing capability of the any migration server, determining a migration duration for the any migration server to migrate the data in each of the groups;

[0264] combining the determined migration durations of the migration servers for migrating the data in each group, and binding the groups to the migration servers based on the principle of shortest total migration duration;

[0265] The total migration duration is the sum of the migration durations for each migration server to migrate the data in the group bound to itself from the source database to the target database.

[0266] Embodiment 10:

[0267] Based on the same technical concept, on the basis of the above embodiments, in an embodiment of the present application, a data migration device is provided. Fig.12 A schematic diagram of the structure of a data migration device provided in an embodiment of the present application. Fig.12 As shown, the device comprises:

[0268] The binding information determination module 1201 is used to determine the group to which it is bound based on the binding information after receiving the migration configuration reading notification;

[0269] The binding information is obtained by dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution based on the migration processing capability of each migration server, and binding each group with each migration server based on the migration time of each migration server migrating the data in each group;

[0270] The data migration module 1202 is used to migrate the data in the bound group from the source database to the target database based on the data migration method.

[0271] In a possible implementation manner, the data migration method includes a mapping relationship between the first data table in the source database and the second data table in the target database; the data migration module 1202 is specifically used to:

[0272] Based on the source data table identifier of the data in the bound group, determining from the first data table the target table to which the data in the bound group belongs;

[0273] Based on the mapping relationship, determining a second data table corresponding to the target table;

[0274] The data in the bound group is migrated from the target table in the source database to the corresponding second data table in the target database.

[0275] In a possible implementation, the device further includes a sequential migration module, which is used to determine the proportion of the amount of data included in the highest migration-level target table in each bound group to the total amount of data in the group if the data in the bound group belongs to multiple target tables and the number of groups currently being processed in parallel is less than the number of bound groups, wherein the highest migration-level target table in each group is the target table with the highest migration priority among the target tables to which the data in each group belongs;

[0276] Determine the migration order between bound groups according to the determined ratio, and determine the migration order of data within each bound group according to the migration priority;

[0277] The data in the bound groups are migrated from the source database to the target database according to the number of groups currently being processed in parallel, the determined migration order between the groups, and the migration order of the data within the groups.

[0278] In a possible implementation, the device further includes a reporting and verification module, configured to report the migration progress of the data in the bound group according to a preset reporting rule, wherein the reporting rule includes reporting when a set migration progress is reached and / or reporting when a set reporting time is reached;

[0279] After receiving the verification instruction including the indication information of the target table after the migration is completed, the consistency verification is performed on the data in the specified range according to the verification rule corresponding to each specified range in the target table after the migration is completed, and the verification result is reported.

[0280] In a possible implementation, the device further includes a sequential verification module, configured to perform consistency verification on the data in each specified range of the target table after the migration according to the verification priority configured for the data in each specified range of the specified target table;

[0281] After determining that the verification result indicates that the consistency check has failed or the consistency check of the data in each specified range of the target table after the migration is completed, the verification result is reported.

[0282] In an embodiment of the present application, first, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups with evenly distributed data volumes; then, based on the migration time of each migration server migrating the data in each group, each group is bound to each migration server, wherein each migration server is bound to one or more groups; finally, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database, thereby reasonably matching the migration processing capability of the migration server with the amount of migration data assigned to it for processing through balanced data grouping and a binding strategy based on migration time, thereby improving the overall data migration efficiency, and at the same time, by dynamically adding migration servers, the concurrency of migration can be adjusted in real time to cope with different migration requirements, thereby improving the scalability and flexibility of data migration.

[0283] Embodiment 11:

[0284] Based on the same technical concept, on the basis of the above embodiments, in the embodiment of the present application, an electronic device 1300 is further provided, such as Fig.13 As shown, it includes at least one processor 1302; and a memory 1301 that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the following steps:

[0285] Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups, wherein the amount of data among the groups is evenly distributed;

[0286] Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups;

[0287] In response to the data migration start instruction, sending a migration configuration read notification to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database;

[0288] Or follow these steps:

[0289] After receiving the migration configuration reading notification, determine the group to which it is bound based on the binding information, wherein the binding information is obtained by dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution based on the migration processing capability of each migration server, and binding each group with each migration server based on the migration time of each migration server to migrate the data in each group;

[0290] Based on the data migration mode, the data in the bound group is migrated from the source database to the target database.

[0291] The above-mentioned memory 1301 is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory 1301 may be a volatile memory (volatile memory), such as a random-access memory (RAM); it may also be a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); it may also be a combination of any one or more of the above-mentioned volatile memories and non-volatile memories.

[0292] The processor 1302 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP. It may also be a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0293] In an embodiment of the present application, first, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups with evenly distributed data volumes; then, based on the migration time of each migration server migrating the data in each group, each group is bound to each migration server, wherein each migration server is bound to one or more groups; finally, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database, thereby reasonably matching the migration processing capability of the migration server with the amount of migration data assigned to it for processing through balanced data grouping and a binding strategy based on migration time, thereby improving the overall data migration efficiency, and at the same time, by dynamically adding migration servers, the concurrency of migration can be adjusted in real time to cope with different migration requirements, thereby improving the scalability and flexibility of data migration.

[0294] Embodiment 12:

[0295] Based on the same technical concept, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the following steps:

[0296] Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups, wherein the amount of data among the groups is evenly distributed;

[0297] Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups;

[0298] In response to the data migration start instruction, sending a migration configuration read notification to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database;

[0299] Or implement the following steps:

[0300] After receiving the migration configuration reading notification, determine the group to which it is bound based on the binding information, wherein the binding information is obtained by dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution based on the migration processing capability of each migration server, and binding each group with each migration server based on the migration time of each migration server to migrate the data in each group;

[0301] Based on the data migration mode, the data in the bound group is migrated from the source database to the target database.

[0302] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical storage such as CD, DVD, BD, HVD, etc., and semiconductor storage such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD), etc.

[0303] In an embodiment of the present application, first, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups with evenly distributed data volumes; then, based on the migration time of each migration server migrating the data in each group, each group is bound to each migration server, wherein each migration server is bound to one or more groups; finally, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database, thereby reasonably matching the migration processing capability of the migration server with the amount of migration data assigned to it for processing through balanced data grouping and a binding strategy based on migration time, thereby improving the overall data migration efficiency, and at the same time, by dynamically adding migration servers, the concurrency of migration can be adjusted in real time to cope with different migration requirements, thereby improving the scalability and flexibility of data migration.

[0304] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data migration methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data migration method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0305] In an embodiment of the present application, first, based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups with evenly distributed data volumes; then, based on the migration time of each migration server migrating the data in each group, each group is bound to each migration server, wherein each migration server is bound to one or more groups; finally, in response to a data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database, thereby reasonably matching the migration processing capability of the migration server with the amount of migration data assigned to it for processing through balanced data grouping and a binding strategy based on migration time, thereby improving the overall data migration efficiency, and at the same time, by dynamically adding migration servers, the concurrency of migration can be adjusted in real time to cope with different migration requirements, thereby improving the scalability and flexibility of data migration.

[0306] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0307] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0308] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0309] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0310] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0311] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0312] The technical solution provided by the present application is introduced in detail above. The principles and implementation methods of the present application are explained by using specific examples in the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0313] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0314] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0315] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0316] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0317] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data migration method, characterized in that: include: Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups, wherein the amount of data among the groups is evenly distributed; Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups; In response to the data migration start instruction, a migration configuration read notification is sent to each migration server, so that each migration server migrates the data in the group bound to itself from the source database to the target database.

2. The method according to claim 1, characterized in that The data to be migrated in the source database is determined in the following manner: In response to the received data migration scope configuration instruction, determine the data migration scope; Determining the data to be migrated in the source database based on the determined data migration scope; The data to be migrated in the source database belongs to one or more target tables in the source database.

3. The method according to claim 2, characterized in that The method further comprises: Based on the migration progress reported by each of the migration servers, after determining that the data migration of the target table is completed and the migration progress of each of the groups is normal, sending a verification instruction to each of the migration servers, so that each of the migration servers performs consistency verification on the data within the specified range according to the verification rule corresponding to each specified range in the target table where the migration is completed, wherein the verification instruction includes instruction information for the target table where the migration is completed; If there is a group with abnormal migration progress or the received verification result indicates that the consistency check has failed, the abnormal information is recorded and an alarm is triggered; The migration progress is the migration progress of each migration server migrating the data in the group bound to itself to the target database.

4. The method according to claim 1, characterized in that The data to be migrated in the source database is divided into multiple groups based on the migration processing capability of each migration server, including: Determining a migration threshold corresponding to each migration server based on the migration processing capability of each migration server, wherein the migration threshold of the migration server is a maximum amount of data that can be processed in parallel when the migration server migrates data; The data to be migrated in the source database is grouped based on a minimum value among the migration thresholds so that the total amount of data included in each group does not exceed the minimum value.

5. The method according to any one of claims 1 to 4, characterized in that: Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups includes: For any migration server, based on the migration processing capability of the any migration server, determining a migration duration for the any migration server to migrate the data in each of the groups; combining the determined migration durations of the migration servers for migrating the data in each group, and binding the groups to the migration servers based on the principle of shortest total migration duration; The total migration duration is the sum of the migration durations for each migration server to migrate the data in the group bound to itself from the source database to the target database.

6. A data migration method, characterized in that: include: After receiving the migration configuration reading notification, determine the group to which it is bound based on the binding information, wherein the binding information is obtained by dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution based on the migration processing capability of each migration server, and binding each group with each migration server based on the migration time of each migration server to migrate the data in each group; Based on the data migration mode, the data in the bound group is migrated from the source database to the target database.

7. The method according to claim 6, characterized in that The data migration method includes a mapping relationship between a first data table in the source database and a second data table in the target database; The step of migrating the data in the bound group from the source database to the target database based on the data migration method includes: Based on the source data table identifier of the data in the bound group, determining from the first data table the target table to which the data in the bound group belongs; Based on the mapping relationship, determining a second data table corresponding to the target table; The data in the bound group is migrated from the target table in the source database to the corresponding second data table in the target database.

8. The method according to claim 7, characterized in that The method further comprises: If the data in the bound group belongs to multiple target tables, and the number of groups currently being processed in parallel is less than the number of bound groups, then determine the ratio of the amount of data included in the target table with the highest migration level in each bound group to the total amount of data in the group, where the target table with the highest migration level in each group is the target table with the highest migration priority among the target tables to which the data in each group belongs; Determine the migration order between bound groups according to the determined ratio, and determine the migration order of data within each bound group according to the migration priority; The data in the bound groups are migrated from the source database to the target database according to the number of groups currently being processed in parallel, the determined migration order between the groups, and the migration order of the data within the groups.

9. The method according to claim 7, characterized in that The method further comprises: According to a preset reporting rule, reporting the migration progress of the data in the bound group, wherein the reporting rule includes reporting when a set migration progress is reached and / or reporting when a set reporting time is reached; After receiving the verification instruction including the indication information of the target table after the migration is completed, the consistency verification is performed on the data in the specified range according to the verification rule corresponding to each specified range in the target table after the migration is completed, and the verification result is reported.

10. The method according to claim 9, characterized in that The method further comprises: Performing consistency check on the data in each specified range of the migrated target table according to the check priority configured for the data in each specified range of the specified target table; After determining that the verification result indicates that the consistency check has failed or the consistency check of the data in each specified range of the target table after the migration is completed, the verification result is reported.

11. A data migration method, characterized in that: include: Based on the migration processing capability of each migration server, the data to be migrated in the source database is divided into a plurality of groups, wherein the amount of data among the groups is evenly distributed; Binding each of the groups to each of the migration servers based on the migration duration of each of the migration servers migrating the data in each of the groups, wherein each of the migration servers is bound to one or more of the groups; In response to the data migration start instruction, each migration server determines the group bound to itself based on the binding information between each migration server and the group, and migrates the data in the group bound to itself from the source database to the target database based on the data migration mode.

12. A data migration device, characterized in that: include: A data partitioning module, used to partition the data to be migrated in the source database into a plurality of groups based on the migration processing capability of each migration server, wherein the amount of data between the groups is evenly distributed; A data binding module, configured to bind each of the groups to each of the migration servers based on a migration duration of each of the migration servers for migrating data in each of the groups, wherein each of the migration servers is bound to one or more of the groups; The data migration start module is used to send a migration configuration read notification to each migration server in response to the data migration start instruction, so that each migration server migrates the data in the group bound to itself from the source database to the target database.

13. A data migration device, characterized in that: include: a binding information determination module, configured to determine the group to which it is bound based on the binding information after receiving the migration configuration reading notification, wherein the binding information is obtained by binding each of the groups to each of the migration servers based on the migration processing capability of each migration server after dividing the data to be migrated in the source database into a plurality of groups with a balanced data volume distribution and based on the migration time of each migration server migrating the data in each of the groups; The data migration module is used to migrate the data in the bound group from the source database to the target database based on the data migration method.

14. An electronic device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 5, or execute the method according to any one of claims 6 to 10, or execute the method according to claim 11 according to the obtained program instructions.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer executes the method as described in any one of claims 1 to 5, or the method as described in any one of claims 6 to 10, or the method as described in claim 11.

16. A computer program product, characterized in that The computer program product comprises: a computer program code, which, when executed on a computer, enables the computer to execute the method as claimed in any one of claims 1 to 5, or the method as claimed in any one of claims 6 to 10, or the method as claimed in claim 11.