A method, apparatus and storage medium for database migration

Abstract

This invention discloses a method, apparatus, and storage medium for database migration, aiming to solve the problems of slow speed and low efficiency in database migration. The method includes: migrating all data from the source database to the target database, and calculating the first hash value of each batch of data read from the source database during the full data migration process; synchronizing incremental data from the source database to the target database, and recording the location of the incremental data during the synchronization process; then, checking whether the data in the source database and the data in the target database are consistent; when checking whether the data in the same batch is consistent, for each batch of data in the source database without incremental data, the corresponding first hash value is used for verification; for each batch of data in the source database with incremental data, the second hash value of the corresponding batch of data read again from the source database is used for verification; if the data in the source table is consistent with the data in the target table, the database migration is determined to be successful; otherwise, the database migration is determined to be unsuccessful.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to a method, apparatus and storage medium for database migration. Background Technology

[0002] As the backend supporting a project or application, databases often require migration as data grows.

[0003] Database migration is the process of moving data and objects from one database to another. Data consistency auditing before and after migration is an indicator of whether the migration was successful.

[0004] In existing technologies, most database audits are conducted after data migration is completed. This results in a long overall audit time, low audit efficiency, and the need to repeatedly access the source data, which increases the pressure on the source.

[0005] Therefore, how to migrate databases quickly, efficiently, and accurately has become a pressing technical problem that needs to be solved. Summary of the Invention

[0006] This invention provides a method, apparatus, and storage medium for database migration, in order to solve the technical problems of slow speed, low efficiency, and insufficient accuracy in database migration in the prior art.

[0007] Firstly, to solve the above-mentioned technical problems, the technical solution of a database migration method provided by the embodiments of the present invention is as follows:

[0008] The data in the source database is fully migrated to the target database, and during the full data migration process, the first hash value of each batch of data read from the source database is calculated.

[0009] The incremental data in the source database is synchronized to the target database, and during the synchronization process, the location of the incremental data where the data changes is recorded.

[0010] After completing the incremental data synchronization, the data in the source database is checked to see if it is consistent with the data in the target database. Specifically, when checking if the data in the same batch in the source database and the target database are consistent, for each batch of data in the source database that does not contain the incremental data, the corresponding first hash value is used for the check; for each batch of data in the source database that contains the incremental data, the second hash value of the corresponding batch of data is used for the check.

[0011] If the data in the source table matches the data in the target table, the database migration is considered successful; otherwise, the database migration is considered to have failed.

[0012] One possible implementation involves migrating all data from the source database to the target database, including:

[0013] Record the site information that begins full migration in the source database and obtain the total number of rows in the corresponding source data table;

[0014] Based on the total number of rows and the batch size, the data in the source data table is divided into multiple batches and migrated to the target database in sequence. After the full data migration is completed, the largest primary key in the source data table at the end of the full data migration is recorded.

[0015] One possible implementation involves synchronizing incremental data from the source database to the target database, including:

[0016] Starting from the position corresponding to the location information, incremental data from the source data table is obtained until the position corresponding to the maximum primary key is reached;

[0017] When acquiring the incremental data, determine whether the change type corresponding to the incremental data is a data manipulation language type; if so, synchronize the incremental data to the corresponding position in the target data table.

[0018] One possible implementation involves recording the location of incremental data where changes have occurred during the incremental data synchronization process, including:

[0019] When it is determined that the change type corresponding to the incremental data is the data operation language type, it is further determined whether the change operation corresponding to the incremental data is an update operation;

[0020] If it is determined that the change operation corresponding to the incremental data is an update operation, then record the batch numbers corresponding to the incremental data before and after the change.

[0021] If it is determined that the change operation corresponding to the incremental data is an insertion operation or a deletion operation, then the corresponding batch number and row number are calculated and recorded according to the primary key corresponding to the incremental data;

[0022] The batch numbers in the records are deduplicated to obtain the final batch number where the data has changed.

[0023] One possible implementation involves verifying whether the data in the source database is consistent with the data in the target database, including:

[0024] Determine whether the total number of rows in the source data table is the same as the total number of rows in the target data table;

[0025] If the total number of rows in the source data table is different from the total number of rows in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database.

[0026] The total number of rows in the source data table is the same as the total number of rows in the target data table. Further, it is determined whether the data in the same batch in the source data table and the target data table are the same.

[0027] If the source data table differs from any data in the same batch in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database.

[0028] If all data in the same batch are the same in the source data table and the target data table, then it is further determined whether the incremental data at the same position recorded in the source data table and the target data table are the same. If all incremental data at the same position are the same, then it is determined that the data in the source database is consistent with the data in the target database; otherwise, it is determined that the data in the source database is inconsistent with the data in the target database.

[0029] One possible implementation involves determining whether the total number of rows in the source data table is the same as the total number of rows in the target data table, including:

[0030] Obtain the first total number of rows of migrated data during the full data migration process, and the second and third total numbers of rows of incremental data inserted and deleted during the incremental data synchronization process, respectively;

[0031] The total number of rows in the source data table is obtained by subtracting the sum of the first total number of rows and the second total number of rows from the third total number of rows.

[0032] Perform a difference operation between the total number of rows in the source data table and the total number of rows in the target data table to obtain the difference result; if the difference result is 0, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are the same; otherwise, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are different.

[0033] One possible implementation involves determining whether the data in the same batch in the source data table and the target data table are the same, including:

[0034] Calculate the third hash value for each batch of data in the target database;

[0035] Retrieve a batch of data corresponding to each final batch number from the source database and perform hash calculation to obtain the corresponding second hash value;

[0036] For each batch of data in the source database that does not contain the incremental data, determine whether the first hash value of the corresponding batch of data is the same as the third hash value of the same batch in the target database. If they are the same, determine that the corresponding batch of data in the source database and the target database are the same; otherwise, determine that they are different.

[0037] For each batch of data corresponding to the final batch number in the source database, determine whether the corresponding second hash value is the same as the third hash value of the same batch in the target database. If they are the same, determine that the data in the same batch in the source database and the target database are the same; otherwise, determine that they are different.

[0038] Secondly, embodiments of the present invention provide a database migration apparatus, comprising:

[0039] The full migration and auditing unit is used to migrate all data from the source database to the target database, and during the full data migration process, calculates the first hash value of each batch of data read from the source database.

[0040] The incremental synchronization unit is used to synchronize incremental data from the source database to the target database, and during the incremental data synchronization process, it records the location of the incremental data where the data has changed.

[0041] The verification unit is used to verify whether the data in the source database and the data in the target database are consistent after the incremental data synchronization is completed. Specifically, when verifying the consistency of data in the same batch between the source database and the target database, for each batch of data in the source database that does not contain the incremental data, the corresponding first hash value is used for verification; for each batch of data in the source database that contains the incremental data, the second hash value of the corresponding batch of data reread from the source database is used for verification. If the data in the source table is consistent with the data in the target table, the database migration is determined to be successful; otherwise, the database migration is determined to have failed.

[0042] In one possible implementation, the full migration and auditing unit is further used for:

[0043] Record the site information that begins full migration in the source database and obtain the total number of rows in the corresponding source data table;

[0044] Based on the total number of rows and the batch size, the data in the source data table is divided into multiple batches and migrated to the target database in sequence. After the full data migration is completed, the largest primary key in the source data table at the end of the full data migration is recorded.

[0045] In one possible implementation, the incremental synchronization unit is further configured to:

[0046] Starting from the position corresponding to the location information, incremental data from the source data table is obtained until the position corresponding to the maximum primary key is reached;

[0047] When acquiring the incremental data, determine whether the change type corresponding to the incremental data is a data manipulation language type; if so, synchronize the incremental data to the corresponding position in the target data table.

[0048] In one possible implementation, the incremental synchronization unit is further configured to:

[0049] When it is determined that the change type corresponding to the incremental data is the data operation language type, it is further determined whether the change operation corresponding to the incremental data is an update operation;

[0050] If it is determined that the change operation corresponding to the incremental data is an update operation, then record the batch numbers corresponding to the incremental data before and after the change.

[0051] If it is determined that the change operation corresponding to the incremental data is an insertion operation or a deletion operation, then the corresponding batch number and row number are calculated and recorded according to the primary key corresponding to the incremental data;

[0052] The batch numbers in the records are deduplicated to obtain the final batch number where the data has changed.

[0053] In one possible implementation, the review unit is further configured to:

[0054] Determine whether the total number of rows in the source data table is the same as the total number of rows in the target data table;

[0055] If the total number of rows in the source data table is different from the total number of rows in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database.

[0056] The total number of rows in the source data table is the same as the total number of rows in the target data table. Further, it is determined whether the data in the same batch in the source data table and the target data table are the same.

[0057] If the source data table differs from any data in the same batch in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database.

[0058] If all data in the same batch are the same in the source data table and the target data table, then it is further determined whether the incremental data at the same position recorded in the source data table and the target data table are the same. If all incremental data at the same position are the same, then it is determined that the data in the source database is consistent with the data in the target database; otherwise, it is determined that the data in the source database is inconsistent with the data in the target database.

[0059] In one possible implementation, the review unit is further configured to:

[0060] Obtain the first total number of rows of migrated data during the full data migration process, and the second and third total numbers of rows of incremental data inserted and deleted during the incremental data synchronization process, respectively;

[0061] The total number of rows in the source data table is obtained by subtracting the sum of the first total number of rows and the second total number of rows from the third total number of rows.

[0062] Perform a difference operation between the total number of rows in the source data table and the total number of rows in the target data table to obtain the difference result; if the difference result is 0, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are the same; otherwise, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are different.

[0063] In one possible implementation, the review unit is used for:

[0064] Calculate the third hash value for each batch of data in the target database;

[0065] Retrieve a batch of data corresponding to each final batch number from the source database and perform hash calculation to obtain the corresponding second hash value;

[0066] For each batch of data in the source database that does not contain the incremental data, determine whether the first hash value of the corresponding batch of data is the same as the third hash value of the same batch in the target database. If they are the same, determine that the corresponding batch of data in the source database and the target database are the same; otherwise, determine that they are different.

[0067] For each batch of data corresponding to the final batch number in the source database, determine whether the corresponding second hash value is the same as the third hash value of the same batch in the target database. If they are the same, determine that the data in the same batch in the source database and the target database are the same; otherwise, determine that they are different.

[0068] Thirdly, embodiments of the present invention also provide a database migration apparatus, comprising:

[0069] At least one processor, and

[0070] Memory connected to the at least one processor;

[0071] The memory stores instructions that can be executed by the at least one processor, which executes the instructions stored in the memory to perform the method described in the first aspect above.

[0072] Fourthly, embodiments of the present invention also provide a readable storage medium, comprising:

[0073] memory,

[0074] The memory is used to store instructions that, when executed by a processor, cause the apparatus including the readable storage medium to perform the method described in the first aspect above.

[0075] Through the technical solutions in one or more of the above embodiments of the present invention, the embodiments of the present invention have at least the following technical effects:

[0076] In the embodiments provided by this invention, data is fully migrated from the source database to the target database, and during the full data migration process, a first hash value is calculated for each batch of data read from the source database; after the full migration is completed, incremental data in the source database is synchronized to the target database, and during the incremental data synchronization process, the location of the incremental data where data changes is recorded; after the incremental data synchronization is completed, the consistency between the data in the source database and the data in the target database is checked; wherein, when checking whether the data in the same batch in the source database and the target database are consistent, for each batch of data in the source database without incremental data, the corresponding first hash value is used for the check; for each batch of data in the source database with incremental data, the second hash value of the corresponding batch of data reread from the source database is used for the check; if the data in the source table and the data in the target table are consistent... If the data matches, the database migration is considered successful; otherwise, it is considered a failure. This allows for preliminary work during the migration process to facilitate subsequent review. After full migration and incremental synchronization, the source and target data tables are checked for data consistency. During the review of batch data, a batch of data in the source database that has changed is reread and recalculated, avoiding the need to recalculate all batches of data in the source database. If the review results are inconsistent, only the inconsistent data needs to be migrated again. This effectively shortens the overall migration and review time, improves migration accuracy, speed, and efficiency. Most data in the source database only needs to be accessed once. When there is hot data in the source database, it can greatly improve auditing and migration efficiency. Attached Figure Description

[0077] Figure 1 A flowchart of a database migration method provided in an embodiment of the present invention;

[0078] Figure 2 A flowchart of a full migration provided in an embodiment of the present invention;

[0079] Figure 3 A flowchart of incremental synchronization is provided for an embodiment of the present invention;

[0080] Figure 4 A flowchart for reviewing data before and after migration is provided as an embodiment of the present invention;

[0081] Figure 5 This is a schematic diagram of a database migration device provided in an embodiment of the present invention. Detailed Implementation

[0082] This invention provides a method, apparatus, and storage medium for database migration, in order to solve the technical problems of slow speed, low efficiency, and insufficient accuracy in database migration in the prior art.

[0083] To better understand the above technical solutions, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0084] Please refer to Figure 1 This invention provides a database migration method, the processing procedure of which is as follows.

[0085] Step 101: Migrate all data from the source database to the target database, and during the data migration process, calculate the first hash value of each batch of data read from the source database.

[0086] Migrating all data from the source database to the target database can be achieved in the following ways:

[0087] Record the location information at which the full migration begins in the source database and obtain the total number of rows in the corresponding source data table; based on the total number of rows and the batch size, divide the data in the source data table into multiple batches and migrate them to the target database in sequence; and after the full data migration is completed, record the largest primary key in the source data table at the end of the full data migration.

[0088] For example, if data starting from point 1 needs to be migrated from the source database to the target database, point 1 is recorded as the starting point for the full migration in the source database. The data table corresponding to point 1 in the source database is the source data table. Given the total number of rows in the source data table (assuming 5000) and a batch size of 1000, the data in the source data table can be divided into 5 batches: the primary key of the data in batch 1 is [1, 1000], the primary key of the data in batch 2 is [1001, 2000], the primary key of the data in batch 3 is [2001, 3000], the primary key of the data in batch 4 is [3001, 4000], and the primary key of the data in batch 5 is [4001, 5000]. Data is migrated batch by batch until the migration of the last batch is complete. After migrating the last row of the last batch from the source data table to the target data table, the primary key of the last row of the last batch is recorded as the maximum primary key.

[0089] It is important to understand that during the full migration of the source data table, there may be data insertions and deletions, which means that the maximum primary key at the end of the full migration of the source data table is not the total number of rows counted initially.

[0090] During the full data migration process, it is also necessary to calculate the first hash value of each batch of data read from the source database for subsequent review.

[0091] In some embodiments, during a full data migration, calculating the first hash value of each batch of data read from the source database can be achieved in the following ways:

[0092] During the migration of each batch of data, each batch of data read from the source data table is concatenated into a first string, and a hash calculation is performed on the first string to obtain the first hash value.

[0093] For example, the source data table has 3 fields (corresponding to 3 columns). The source data table is divided into 5 batches, each batch has 1000 rows of data. When reading 1000 rows of data from each batch from the source data table, these 1000 rows of data are concatenated into a first string, and a hash calculation is performed on this first string to obtain a first hash value.

[0094] Please see Figure 2 This is a flowchart of a full migration provided in an embodiment of the present invention.

[0095] Step 201: Record the location information of the migration start position in the source database and obtain the total number of rows in the source data table;

[0096] Step 202: Read data in batches, concatenate the read batches of data into the first string, and calculate the first hash value;

[0097] Step 203: Insert the read batch of data into the target data table;

[0098] Step 204: Determine if the full migration is complete;

[0099] The process can be completed by determining whether the batch number of the currently completed batch of data is the largest batch. If it is the largest batch number, the full migration is considered complete, and step 205 is executed. If the batch number of the currently completed batch of data is less than the largest batch number, the full migration is considered incomplete, and step 202 is executed.

[0100] Step 205: Record the largest primary key after the full migration is complete.

[0101] The process of migrating all other rows of data in the source data table can be repeated sequentially, and will not be elaborated here. After migrating the last row of data in the last batch of the source data table to the target data table, step 102 can be executed.

[0102] Step 102: Synchronize the incremental data from the source database to the target database, and record the location of the incremental data where the data changes during the synchronization process.

[0103] In some embodiments, incremental data in the source database can be synchronized to the target database in the following ways:

[0104] Starting from the position corresponding to the location information, incremental data is obtained from the source data table until the position corresponding to the largest primary key is reached. When obtaining incremental data, it is determined whether the change type corresponding to the incremental data is a Data Manipulation Language (DML) type. If so, the incremental data is synchronized to the corresponding position in the target data table.

[0105] Before determining whether the change type corresponding to the incremental data is a DML type, the incremental data can be parsed. The change type corresponding to the incremental data includes DML type and Data Definition Language (DDL) type. DDL type is not supported by default during data migration.

[0106] For example, the data in the source data table is divided into 5 batches. During the full migration of the data in batch 3, a row of data is inserted into the 3rd row of batch 1 in the source data table. During the full migration of the data in batch 5, the data in the 8th row of batch 3 is deleted and the data in the 6th row of batch 4 is changed.

[0107] After completing the full migration of the above 5 batches of data (corresponding to a maximum primary key of 5000), starting from the position corresponding to the location information, incremental data is retrieved from the source data table. This shows that the 3rd row of data in batch 1, the 8th row of data in batch 3, and the 6th row of data in batch 4 are incremental data. Retrieval stops when the maximum primary key value of 5000 is reached. Since the above insertion, deletion, and modification operations are DML types, the change type corresponding to the above incremental data is determined to be DML, and therefore the above incremental data can be synchronized to the target data table. If the change type of the above incremental data is determined to be not DML, such as DDL, then synchronization is abandoned.

[0108] In some embodiments, during incremental data synchronization, recording the location of incremental data where data changes can be achieved in the following ways:

[0109] When it is determined that the change type corresponding to the incremental data is a data manipulation language type, it is further determined whether the change operation corresponding to the incremental data is an update operation. If it is determined that the change operation corresponding to the incremental data is an update operation, the batch numbers corresponding to the incremental data before and after the change are recorded respectively. If it is determined that the change operation corresponding to the incremental data is an insert operation or a delete operation, the corresponding batch number and row number are calculated and recorded according to the primary key corresponding to the incremental data. The recorded batch numbers are deduplicated to obtain the final batch number of the data change.

[0110] For example, if it is determined that the data in the third row of batch 1 in the source data table is incremental data and belongs to the DML type, and it is further determined that this incremental data is an insert operation, then the primary key, batch number, and row number (i.e., the location of the incremental data) corresponding to this incremental data are recorded; and the incremental data is synchronized to the target data table according to the location of the incremental data.

[0111] It was determined that row 8 of batch 3 in the source data table was incremental data and belonged to the DML type. It was further determined that this incremental data was deleted. At this time, the primary key, batch number, and row number (i.e. the location of the incremental data) corresponding to this incremental data were recorded. Based on the location of the incremental data, the incremental data was synchronized to the target data table.

[0112] It was determined that the data in row 6 of batch 4 in the source data table was incremental data and belonged to the DML type. It was further determined that this incremental data was modified data, that is, it was changed through an update operation. At this time, it is necessary to record the batch number (i.e. the location of this incremental data) before and after the change of this incremental data, and synchronize the incremental data to the target data table according to the location of the incremental data.

[0113] Once the maximum primary key is reached, the incremental data synchronization is completed.

[0114] Finally, duplicate batch numbers corresponding to the recorded incremental data are deduplicated to obtain the final batch number where the data has changed.

[0115] If incremental data is generated after the largest primary key during the full migration process, it will not be synchronized for the time being, but will be synchronized in the subsequent review stage. See the following introduction for details.

[0116] Please see Figure 3 This is a flowchart of incremental synchronization provided for an embodiment of the present invention.

[0117] Step 301: Obtain incremental data;

[0118] Step 302: Determine if it is a DML type;

[0119] If yes, proceed to step 303.

[0120] Step 303: Determine whether the primary key corresponding to the incremental data is less than or equal to the maximum primary key;

[0121] If the primary key corresponding to the incremental data is less than or equal to the maximum primary key, it means that the incremental data was changed before the data corresponding to the maximum primary key. In other words, if the primary key corresponding to the incremental data is determined to be less than or equal to the maximum primary key, proceed to step 304. If the primary key corresponding to the incremental data is greater than the maximum primary key, it means that the incremental data was changed after the data corresponding to the maximum primary key. In other words, if the primary key corresponding to the incremental data is determined to be greater than the maximum primary key, do not process it for now, but process it in the subsequent review process.

[0122] Step 304: Determine if it is an update operation;

[0123] Determine whether the incremental data was generated by an update operation. If yes, proceed to step 305b; otherwise, proceed to step 305a.

[0124] Step 305a: Calculate and record the corresponding batch number based on the primary key of the incremental data, and record the number of rows inserted or deleted;

[0125] Step 305b: Calculate and record the two corresponding batch numbers based on the primary keys before and after the incremental data change;

[0126] After completing step 305a or step 305b, step 306 can be executed.

[0127] Step 306: Determine if incremental synchronization has ended;

[0128] If yes, proceed to step 307; otherwise, proceed to step 301.

[0129] Step 307: Remove duplicate batch numbers from the records.

[0130] After the incremental data synchronization is complete, step 103 can be executed.

[0131] Step 103: After completing incremental data synchronization, verify whether the data in the source database is consistent with the data in the target database. Specifically, when verifying the consistency of data in the same batch between the source and target databases, for each batch of data in the source database without incremental data, verify using the corresponding first hash value; for each batch of data in the source database with incremental data, verify using the second hash value of the corresponding batch of data reread from the source database. If the data in the source table is consistent with the data in the target table, the database migration is considered successful; otherwise, the database migration is considered to have failed.

[0132] After completing the full migration and incremental migration, obtain the total number of rows in the source data table and the total number of rows in the target data table, and determine whether their total number of rows is the same. If they are different, it is determined that the data in the source database is inconsistent with the data in the target database, and the migration fails.

[0133] In some embodiments, determining whether the total number of rows in the source data table is the same as the total number of rows in the target data table can be achieved in the following ways:

[0134] Obtain the first total number of rows of migrated data during the full data migration process, and the second and third total numbers of incremental data inserted and deleted during the incremental data synchronization process, respectively; perform a difference operation between the sum of the first and second total number of rows and the third total number of rows to obtain the total number of rows in the source data table; perform a difference operation between the total number of rows in the source data table and the total number of rows in the target data table to obtain the difference result; if the difference result is 0, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are the same; otherwise, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are different.

[0135] If a total of M rows of data are migrated during the full migration process, then the first total number of rows is M. During the incremental synchronization process, if a total of N rows of data are inserted and n rows of data are deleted, then the second total number of rows is N, and the third total number of rows is n. Therefore, the total number of rows in the source data table can be calculated as M + Nn. The total number of rows in the target data table can be directly read from the target data table. By calculating the difference between the total number of rows in the source data table and the total number of rows in the target data table, it can be determined whether their total number of rows is the same. That is, if the difference is 0, it means that their total number of rows is the same; otherwise, it means that their total number of rows is different.

[0136] If the total number of rows in the source data table is the same as the total number of rows in the target data table, then further determine whether the data in the same batch in the source data table and the target data table are the same. This can be achieved in the following ways:

[0137] Calculate the third hash value for each batch of data in the target database; retrieve the batch of data corresponding to each final batch number from the source database and perform hash calculation to obtain the corresponding second hash value; for each batch of data in the source database without incremental data, determine whether the first hash value of the corresponding batch of data is the same as the third hash value of the same batch in the target database. If they are the same, determine that the corresponding batch of data in the source database and the target database are the same; otherwise, determine that they are different. For each batch of data corresponding to each final batch number in the source database, determine whether the corresponding second hash value is the same as the third hash value of the same batch in the target database. If they are the same, determine that the corresponding batch of data in the source database and the target database are the same; otherwise, determine that they are different.

[0138] The methods for calculating the second and third hash values ​​are the same as those for calculating the first hash value, and will not be repeated here. In the process of determining whether data in the same batch is identical, if the result of the determination for any data in the same batch is different, it is determined that the data in the source database and the target database are inconsistent, and the migration fails.

[0139] If it is determined that all data in the same batch are the same in the source data table and the target data table, then it is further determined whether the incremental data at the same position in the source data table and the target data table are the same. If all the incremental data at the same position are the same, then it is determined that the data in the source database is consistent with the data in the target database, and the migration is successful; otherwise, it is determined that the migration has failed.

[0140] If there is incremental data after the maximum primary key, the data after the maximum primary key is divided into batches and migrated to the target data table. Each batch of these newly migrated data is reviewed, such as calculating the hash value of the data of the same batch in the source data table and the target data table. By judging whether their hash values ​​are the same, it is determined whether the corresponding batch of data has been migrated correctly.

[0141] Please see Figure 4 This is a flowchart for reviewing data before and after migration, provided as an embodiment of the present invention.

[0142] Step 401: Obtain the total number of rows in the source data table and the total number of rows in the target data table;

[0143] Step 402: Determine whether the total number of rows in the source data table is equal to the total number of rows in the target data table;

[0144] If they are equal, proceed to step 403; if they are not equal, proceed to step 410b.

[0145] Step 403: Sequentially retrieve data from the target data table for each batch and calculate the hash value of the corresponding batch;

[0146] The hash value obtained by performing hash calculations on the data in each batch of the target data table is the third hash value;

[0147] Step 404: Determine if the currently acquired batch of data has been changed during incremental synchronization;

[0148] If there are any changes, proceed to step 405; otherwise, proceed to step 406.

[0149] Step 405: Retrieve the batch of data from the source data table again and calculate the hash value;

[0150] The hash value calculated from the reacquired data is the second hash value; this is used as the hash value for comparison with the target data table.

[0151] Step 406: Determine whether the hash values ​​of this batch of data in the source data table and the target data table are equal;

[0152] If step 405 is executed, the second hash value is compared with the third hash value. If step 405 is not executed, the first hash value of the corresponding batch of data in the source data table calculated during the full data migration process is compared with the third hash value.

[0153] If they are equal, proceed to step 407; if they are not equal, proceed to step 410b.

[0154] Step 407: Determine whether all batches have completed the re-inspection;

[0155] Determine whether all batches have been reviewed based on whether the currently reviewed batch number is equal to the maximum batch number. If the currently reviewed batch number is equal to the maximum batch number, determine that all batches have been reviewed and proceed to step 408. If the currently reviewed batch number is less than the maximum batch number, determine that not all batches have been reviewed and proceed to step 403.

[0156] Step 408: Divide the incremental data with primary keys greater than the maximum primary key into batches and calculate the corresponding hash values;

[0157] Step 409: Determine whether the hash values ​​corresponding to the newly batched data in the source data table and the target data table are the same;

[0158] If they are the same, proceed to step 410a; if they are different, proceed to step 410b.

[0159] Step 410a: Consistent;

[0160] Step 410b: Inconsistent.

[0161] In the embodiments provided by this invention, data from the source database is fully migrated to the target database, and during the full data migration process, a first hash value is calculated for each batch of data read from the source database; after the full migration is completed, incremental data from the source database is synchronized to the target database, and during the incremental data synchronization process, the location of the incremental data where data changes is recorded; after the incremental data synchronization is completed, the consistency between the data in the source database and the data in the target database is checked; wherein, when checking whether the data in the same batch in the source database and the target database are consistent, for each batch of data in the source database without incremental data, the corresponding first hash value is used for checking; for each batch of data in the source database with incremental data, the second hash value of the corresponding batch of data reread from the source database is used for checking; if the data in the source table and the data in the target table are consistent... If the data is consistent, the database migration is considered successful; otherwise, it is considered a failure. This allows for preliminary work during the migration process to facilitate subsequent review. After full migration and incremental synchronization, the source and target data tables are checked for data consistency. During the review of batch data, a batch of data in the source database that has changed is reread and recalculated, avoiding the need to recalculate all batches of data in the source database. If the review results are inconsistent, only the inconsistent data needs to be migrated again. This effectively shortens the overall migration and review time, improves migration accuracy, speed, and efficiency. Most data in the source database only needs to be accessed once. When there is hot data in the source database, this greatly improves auditing and migration efficiency.

[0162] Based on the same inventive concept, one embodiment of the present invention provides a database migration apparatus. Specific implementation details of the database migration method of this apparatus can be found in the description of the method embodiments section; repeated details will not be repeated here. Figure 5 The device includes:

[0163] The full migration unit 501 is used to migrate all data from the source database to the target database, and during the full data migration process, calculates the first hash value of each batch of data read from the source database.

[0164] The incremental synchronization unit 502 is used to synchronize incremental data from the source database to the target database, and during the incremental data synchronization process, record the location of the incremental data where the data has changed.

[0165] The review unit 503 is used to review whether the data in the source database is consistent with the data in the target database after the incremental data synchronization is completed. Specifically, when reviewing whether the data in the same batch in the source database and the target database are consistent, for each batch of data in the source database that does not contain the incremental data, the corresponding first hash value is used for review; for each batch of data in the source database that contains the incremental data, the second hash value of the corresponding batch of data reread from the source database is used for review. If the data in the source table is consistent with the data in the target table, the database migration is determined to be successful; otherwise, the database migration is determined to have failed.

[0166] In one possible implementation, the full migration unit 501 is further configured to:

[0167] Record the site information that begins full migration in the source database and obtain the total number of rows in the corresponding source data table;

[0168] Based on the total number of rows and the batch size, the data in the source data table is divided into multiple batches and migrated to the target database in sequence. After the full data migration is completed, the largest primary key in the source data table at the end of the full data migration is recorded.

[0169] In one possible implementation, the incremental synchronization unit 502 is further configured to:

[0170] Starting from the position corresponding to the location information, incremental data from the source data table is obtained until the position corresponding to the maximum primary key is reached;

[0171] When acquiring the incremental data, determine whether the change type corresponding to the incremental data is a data manipulation language type; if so, synchronize the incremental data to the corresponding position in the target data table.

[0172] In one possible implementation, the incremental synchronization unit 502 is further configured to:

[0173] When it is determined that the change type corresponding to the incremental data is the data operation language type, it is further determined whether the change operation corresponding to the incremental data is an update operation;

[0174] If it is determined that the change operation corresponding to the incremental data is an update operation, then record the batch numbers corresponding to the incremental data before and after the change.

[0175] If it is determined that the change operation corresponding to the incremental data is an insertion operation or a deletion operation, then the corresponding batch number and row number are calculated and recorded according to the primary key corresponding to the incremental data;

[0176] The batch numbers in the records are deduplicated to obtain the final batch number where the data has changed.

[0177] In one possible implementation, the review unit 503 is further configured to:

[0178] Determine whether the total number of rows in the source data table is the same as the total number of rows in the target data table;

[0179] If the total number of rows in the source data table is different from the total number of rows in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database.

[0180] The total number of rows in the source data table is the same as the total number of rows in the target data table. Further, it is determined whether the data in the same batch in the source data table and the target data table are the same.

[0181] If the source data table differs from any data in the same batch in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database.

[0182] If all data in the same batch are the same in the source data table and the target data table, then it is further determined whether the incremental data at the same position recorded in the source data table and the target data table are the same. If all incremental data at the same position are the same, then it is determined that the data in the source database is consistent with the data in the target database; otherwise, it is determined that the data in the source database is inconsistent with the data in the target database.

[0183] In one possible implementation, the review unit 503 is further configured to:

[0184] Obtain the first total number of rows of migrated data during the full data migration process, and the second and third total numbers of rows of incremental data inserted and deleted during the incremental data synchronization process, respectively;

[0185] The total number of rows in the source data table is obtained by subtracting the sum of the first total number of rows and the second total number of rows from the third total number of rows.

[0186] Perform a difference operation between the total number of rows in the source data table and the total number of rows in the target data table to obtain the difference result; if the difference result is 0, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are the same; otherwise, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are different.

[0187] In one possible implementation, the review unit 503 is used for:

[0188] Calculate the third hash value for each batch of data in the target database;

[0189] Retrieve a batch of data corresponding to each final batch number from the source database and perform hash calculation to obtain the corresponding second hash value;

[0190] For each batch of data in the source database that does not contain the incremental data, determine whether the first hash value of the corresponding batch of data is the same as the third hash value of the same batch in the target database. If they are the same, determine that the corresponding batch of data in the source database and the target database are the same; otherwise, determine that they are different.

[0191] For each batch of data corresponding to the final batch number in the source database, determine whether the corresponding second hash value is the same as the third hash value of the same batch in the target database. If they are the same, determine that the data in the same batch in the source database and the target database are the same; otherwise, determine that they are different.

[0192] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0195] Based on the same inventive concept, this invention provides a database migration apparatus, comprising: at least one processor, and

[0196] Memory connected to the at least one processor;

[0197] The memory stores instructions that can be executed by the at least one processor, which executes the database migration method described above by executing the instructions stored in the memory.

[0198] Based on the same inventive concept, embodiments of the present invention also provide a readable storage medium, comprising:

[0199] memory,

[0200] The memory is used to store instructions that, when executed by a processor, cause the apparatus including the readable storage medium to perform the database migration method as described above.

[0201] The readable storage medium can be any available medium or data storage device accessible to the processor, including volatile memory or non-volatile memory, or both. By way of example, and not limitation, non-volatile memory can include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state disk (SSD), magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disc (MO), etc.), and optical storage (e.g., CD, DVD, BD, HVD, etc.). Volatile memory can include random access memory (RAM), which can act as an external cache memory. By way of example and not limitation, RAM can be obtained in various forms, such as Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random-Access Memory (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), and Sync Link DRAM (SLDRAM). The storage devices disclosed herein are intended to include, but are not limited to, these and other suitable types of memory.

[0202] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of machine program products implemented on one or more readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer / processor-usable program code.

[0203] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0204] These program instructions may also be stored in a readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a process implemented by the computer / processor, thereby providing instructions that execute on the computer / processor or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0206] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for database migration, characterized in that, include: The data in the source database is fully migrated to the target database. During the full data migration process, starting from the migration start position in the source database, multiple rows of data in the source database are read in batches. For each batch of data read, all rows of data in the batch are concatenated into a first string, and the first string is hashed to obtain the first hash value of the batch of data. The batch of data is then inserted into the target database until the last batch of data is inserted into the target database, thus completing the full data migration. The incremental data generated in the source database during the full data migration process and before the largest primary key at the end of the full data migration is synchronized to the target database, and the location of the incremental data where the data changes is recorded during the incremental data synchronization process; After completing the incremental data synchronization, the consistency between the data in the source database and the data in the target database is checked, and the incremental data generated in the source database during the full data migration process and after the maximum primary key is migrated to the target database. Specifically, when checking the consistency between the source database and the target database for the same batch of data, for each batch of data in the source database that does not contain the incremental data, the corresponding first hash value is used for the check; for each batch of data in the source database that contains the incremental data, the second hash value of the corresponding batch of data reread from the source database is used for the check. If the data in the source database is consistent with the data in the target database, the database migration is determined to be successful; otherwise, the database migration is determined to be unsuccessful.

2. The method as described in claim 1, characterized in that, Migrate all data from the source database to the target database, including: Record the site information in the source database corresponding to the migration start position, and obtain the total number of rows in the corresponding source data table; Based on the total number of rows and the batch size, the data in the source data table is divided into multiple batches and migrated to the target database in sequence. After the full data migration is completed, the largest primary key in the source data table at the end of the full data migration is recorded.

3. The method as described in claim 2, characterized in that, Synchronizing incremental data generated in the source database during the full data migration process, but before the completion of the full data migration and the largest primary key, to the target database includes: Starting from the position corresponding to the location information, incremental data from the source data table is obtained until the position corresponding to the maximum primary key is reached; When acquiring the incremental data, determine whether the change type corresponding to the incremental data is a data manipulation language type; if so, synchronize the incremental data to the corresponding position in the target data table.

4. The method as described in claim 3, characterized in that, During the incremental data synchronization process, the location of the incremental data where the data has changed is recorded, including: When it is determined that the change type corresponding to the incremental data is the data operation language type, it is further determined whether the change operation corresponding to the incremental data is an update operation; If it is determined that the change operation corresponding to the incremental data is an update operation, then record the batch numbers corresponding to the incremental data before and after the change. If it is determined that the change operation corresponding to the incremental data is an insertion operation or a deletion operation, then the corresponding batch number and row number are calculated and recorded according to the primary key corresponding to the incremental data; The batch numbers in the records are deduplicated to obtain the final batch number where the data has changed.

5. The method as described in claim 4, characterized in that, Verify the consistency between the data in the source database and the data in the target database, including: Determine whether the total number of rows in the source data table is the same as the total number of rows in the target data table; If the total number of rows in the source data table is different from the total number of rows in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database. The total number of rows in the source data table is the same as the total number of rows in the target data table. Further, it is determined whether the data in the same batch in the source data table and the target data table are the same. If the source data table differs from any data in the same batch in the target data table, then it is determined that the data in the source database is inconsistent with the data in the target database. If all data in the same batch are the same in the source data table and the target data table, then it is further determined whether the incremental data at the same position recorded in the source data table and the target data table are the same. If all incremental data at the same position are the same, then it is determined that the data in the source database is consistent with the data in the target database; otherwise, it is determined that the data in the source database is inconsistent with the data in the target database.

6. The method as described in claim 5, characterized in that, Determining whether the total number of rows in the source data table is the same as the total number of rows in the target data table includes: Obtain the first total number of rows of migrated data during the full data migration process, and the second and third total numbers of rows of incremental data inserted and deleted respectively during the incremental data synchronization process; The total number of rows in the source data table is obtained by performing a difference operation between the sum of the first total number of rows and the second total number of rows and the third total number of rows; Perform a difference operation between the total number of rows in the source data table and the total number of rows in the target data table to obtain the difference result; if the difference result is 0, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are the same; otherwise, it is determined that the total number of rows in the source data table and the total number of rows in the target data table are different.

7. The method as described in claim 5, characterized in that, Determining whether the data in the source data table and the target data table in the same batch are the same includes: Calculate the third hash value for each batch of data in the target database; Retrieve a batch of data corresponding to each final batch number from the source database and perform hash calculation to obtain the corresponding second hash value; For each batch of data in the source database that does not contain the incremental data, determine whether the first hash value of the corresponding batch of data is the same as the third hash value of the same batch in the target database. If they are the same, determine that the corresponding batch of data in the source database and the target database are the same; otherwise, determine that they are different. For each batch of data corresponding to the final batch number in the source database, determine whether the corresponding second hash value is the same as the third hash value of the same batch in the target database. If they are the same, determine that the data in the same batch in the source database and the target database are the same; otherwise, determine that they are different.

8. An apparatus for database migration, characterized in that, include: The full migration unit is used to migrate all data from the source database to the target database. During the full data migration process, starting from the migration start position in the source database, multiple rows of data in the source database are read in batches. After each batch of data is read, all rows of data in the batch are concatenated into a first string, and the first string is hashed to obtain the first hash value of the batch of data. The batch of data is then inserted into the target database until the last batch of data is inserted into the target database, thus completing the full data migration. The incremental synchronization unit is used to synchronize incremental data generated in the source database during the full data migration process and before the largest primary key at the end of the full data migration to the target database, and to record the location of incremental data where data changes during the incremental data synchronization process. The verification unit is used to verify whether the data in the source database is consistent with the data in the target database after the incremental data synchronization is completed, and to migrate the incremental data generated in the source database during the full data migration process and after the maximum primary key to the target database. Specifically, when verifying the consistency of the same batch of data between the source database and the target database, for each batch of data in the source database that does not contain the incremental data, the corresponding first hash value is used for verification; for each batch of data in the source database that contains the incremental data, the second hash value of the corresponding batch of data reread from the source database is used for verification. If the data in the source database is consistent with the data in the target database, the database migration is determined to be successful. Otherwise, the database migration is considered to have failed.

9. A database migration apparatus, characterized in that, include: At least one processor, and Memory connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the method as described in any one of claims 1-7 by executing the instructions stored in the memory.

10. A readable storage medium, characterized in that, Including memory, The memory is used to store instructions that, when executed by a processor, cause a device including the readable storage medium to perform the method as described in any one of claims 1 to 7.

Images