Data migration quantity verification method and computer-readable storage medium

Abstract

The present invention discloses a method for verifying the quantity of data migration and a computer-readable storage medium. The method includes: obtaining the primary key IDs of the migration data in the old database, sorting the primary key IDs, and writing them into the first set in the Redis database; obtaining each data table corresponding to the migration data in the new database according to a preset mapping rule; querying the primary key IDs within the corresponding ranges in each data table according to the first primary key ID and the last primary key ID in the first set, and writing them into the second set corresponding to each data table in the Redis database; calculating the missing sets corresponding to each data table respectively according to the first set and the second sets corresponding to each data table; if the missing sets corresponding to each data table are all empty, it is determined that the quantity verification of the data migration passes. The present invention can improve the efficiency of verification.

Description

Technical Field

[0001] The present invention relates to the technical field of data verification, and particularly to a method for verifying the quantity of data migration and a computer-readable storage medium. Background Art

[0002] In the prior art, many systems have a corresponding server, and each system also has a corresponding database. Due to the Internet, the amount of user data has increased significantly, so a large amount of data is stored in the database. Since Internet system applications are updated frequently, there may be a situation where new and old versions are used simultaneously. In this scenario, the new version system connects to the new database for use, and the old version system connects to the old database for use. In order to be compatible with new and old data, data migration may be required to migrate the information of the old database to the new database.

[0003] In the process of data migration, mainly one or more tables in the old database are migrated to one or more tables in the new database through a migration program. Generally, data is migrated in batches, regardless of the result of the migrated data. Finally, the verification of the data quantity is required. For example, a piece of data (corresponding to a primary key ID) in a certain table of the old database is migrated to the new database, and according to business requirements, 3 pieces of data in 3 corresponding tables are generated, where the primary key ID is the same. At this time, the verification of the quantity of data migration is required.

[0004] Currently, there are the following several ways to verify the quantity of data migration:

[0005] 1. Query the total number of records in the corresponding database table through count(*); however, this method can only know the total quantity and cannot clarify which IDs or records are missing, and a comparison operation for each piece of data needs to be performed.

[0006] 2. Batch query the information of the old database table, and query the record information in the new database one by one or in batches. If it exists, it means the record migration is successful; if it does not exist, it means the migration fails. However, using the one-by-one method will have the problem of low efficiency, and the batch method is limited by the query quantity supported by the database. For example, use the in method (used to query data within a certain range) to perform SQL queries. First, query a batch of data in the old database, and then use the in method to query which data exists in the new database. The data that does not exist is the data with failed migration. However, when using the in method, there is a quantity limit. The parameters following in in oracle cannot exceed 1000, and in mysql or other databases, there is also a quantity limit problem, and generally it cannot exceed several thousand. In this way, only about 1000 pieces of data can be queried each time, and the verification efficiency cannot be improved. If the data volume is very large, it will take a long time. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method for verifying the quantity of data migration and a computer-readable storage medium, which can improve the verification efficiency.

[0008] To solve the above technical problem, the technical solution adopted by the present invention is: A method for verifying the quantity of data migration, including:

[0009] Obtain the primary key IDs of the migration data in the old database, sort the primary key IDs, and write them into the first set in the Redis database;

[0010] According to the preset mapping rules, obtain the corresponding data tables of the migration data in the new database;

[0011] According to the first primary key ID and the last primary key ID in the first set, respectively query the primary key IDs within the corresponding range in each data table, and write them into the second set corresponding to each data table in the Redis database;

[0012] According to the first set and the second set corresponding to each data table, respectively calculate the missing sets corresponding to each data table;

[0013] If the missing sets corresponding to each data table are all empty, it is determined that the quantity verification of data migration passes.

[0014] The present invention also relates to a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented:

[0015] Obtain the primary key IDs of the migration data in the old database, sort the primary key IDs, and write them into the first set in the Redis database;

[0016] According to the preset mapping rules, obtain the corresponding data tables of the migration data in the new database;

[0017] According to the first primary key ID and the last primary key ID in the first set, respectively query the primary key IDs within the corresponding range in each data table, and write them into the second set corresponding to each data table in the Redis database;

[0018] According to the first set and the second set corresponding to each data table, respectively calculate the missing sets corresponding to each data table;

[0019] If the missing sets corresponding to each data table are all empty, it is determined that the quantity verification of data migration passes.

[0020] The beneficial effects of the present invention are as follows: By writing the corresponding primary key IDs in the old and new databases into different sets in the Redis database respectively, and then using the set operation method in Redis to calculate the missing set, it can be determined whether the data migration is complete by judging whether the missing set is empty; at the same time, since the missing set stores the primary key IDs that have not been successfully migrated, the specific missing data can also be quickly determined, which is convenient for subsequent re-migration. The present invention can quickly determine the total amount of missing data and the specific missing data, greatly improving the verification efficiency of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of a method for verifying the quantity of data migration of the present invention;

[0022] Figure 2 It is a flowchart of the method in the first embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To describe in detail the technical content, the achieved objectives and the effects of the present invention, the following is described in detail in conjunction with the embodiments and with reference to the accompanying drawings.

[0024] The most crucial concept of the present invention is: writing the corresponding primary key IDs in the old and new databases into different sets in the Redis database respectively, and then using the set operation method in Redis to quickly calculate the data that has not been successfully migrated.

[0025] Please refer to Figure 1 , a method for verifying the quantity of data migration, including:

[0026] Obtain the primary key IDs of the migrated data in the old database, and write the sorted primary key IDs into the first set in the Redis database;

[0027] Obtain the corresponding data tables of the migrated data in the new database according to the preset mapping rules;

[0028] According to the first primary key ID and the last primary key ID in the first set, query the primary key IDs within the corresponding range in each data table respectively, and write them into the second set corresponding to each data table in the Redis database;

[0029] Calculate the missing sets corresponding to each data table respectively according to the first set and the second sets corresponding to each data table;

[0030] If the missing sets corresponding to each data table are all empty, it is determined that the quantity verification of the data migration passes.

[0031] As can be seen from the above description, the beneficial effects of the present invention are as follows: the total amount of missing data and the specific missing data can be quickly determined, greatly improving the verification efficiency of data migration.

[0032] Further, the step of querying the primary key IDs within the corresponding ranges in each data table according to the first primary key ID and the last primary key ID in the first set and writing them into the second set corresponding to each data table in the Redis database is specifically as follows:

[0033] Taking the first primary key ID in the first set as the start ID and the last primary key ID in the first set as the end ID, querying the primary key IDs greater than or equal to the start ID and less than or equal to the end ID in a data table corresponding to the migrated data, and after sorting the queried primary key IDs, writing them into the second set corresponding to the data table in the Redis database.

[0034] As can be seen from the above description, since the primary key IDs in the first set are arranged in sequence, the range of the primary key IDs in the first set can be determined by obtaining the first primary key ID and the last primary key ID. Querying through this range in the new database can ensure that all the primary key IDs migrated from the old database and successfully migrated are obtained in the new database.

[0035] Further, the step of respectively calculating the missing sets corresponding to the data tables according to the first set and the second sets corresponding to the data tables is specifically as follows:

[0036] Performing an intersection calculation on the first set and the second sets corresponding to the data tables respectively to obtain the intersection sets corresponding to the data tables;

[0037] Performing a difference calculation on the first set and the intersection sets corresponding to the data tables respectively to obtain the missing sets corresponding to the data tables.

[0038] As can be seen from the above description, since the newly added primary key IDs in the new database may be obtained when obtaining the second set, the newly added primary key IDs are first screened out by calculating the intersection to obtain the primary key IDs migrated from the old database and successfully migrated in the new database, and then the first set obtained from the old database is subtracted from the intersection, that is, the successfully migrated primary key IDs are screened out from all the primary key IDs of the migrated data, and the remaining ones are the primary key IDs of the failed migrations.

[0039] Further, after respectively calculating the missing sets corresponding to the data tables according to the first set and the second sets corresponding to the data tables, it further includes:

[0040] If the missing sets corresponding to the respective data tables are not empty, re-migrate the missing primary key IDs in the respective data tables according to the missing sets corresponding to the respective data tables.

[0041] As can be seen from the above description, by re-migrating the missing primary key IDs, the integrity of data migration is ensured.

[0042] The present invention also proposes a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps described above are implemented.

[0043] Embodiment 1

[0044] Please refer to Figure 2 , Embodiment 1 of the present invention is: A method for verifying the quantity of data migration, which can verify the migration integrity of data migration between old and new databases, and includes the following steps:

[0045] S1: Obtain the primary key IDs of the migration data in the old database, and write the sorted primary key IDs into the first set in the Redis database.

[0046] Specifically, in the way of paging query of primary key IDs and in the ascending order of IDs, query the primary key IDs of the migration data in the old database, query 10,000 IDs in each batch, and write these IDs into the first set of Redis in a batch (batch) in the form of a pipeline in Redis (the data structure in Redis is a set collection). When a total of 1 million primary key IDs are written into the first set, the writing is paused.

[0047] S2: Obtain the respective data tables corresponding to the migration data in the new database according to a preset mapping rule. Among them, the mapping rule is determined according to different business scenarios and business requirements.

[0048] For example, assume that there is only organization-related data in the old database, and the primary key ID is the organization ID. When migrating these data to the new database, in addition to the organization-related data, institution-related data will be generated and the corresponding relationship between the organization and the institution will be recorded. That is to say, the organization data in the old database will be migrated to the new database and mapped to three data tables: the organization table, the institution table, and the node table (used to record the corresponding relationship between the organization and the institution). The organization ID field exists in all three data tables, and subsequent queries need to be performed in these three data tables.

[0049] S3: Query the primary key IDs within the corresponding range in the respective data tables according to the first primary key ID and the last primary key ID in the first set, and write them into the second set corresponding to the respective data tables in the Redis database.

[0050] Specifically, take the first primary key ID in the first set as the start ID, and the last primary key ID in the first set as the end ID. Then, using the start ID and the end ID as database query parameters, query the primary key IDs greater than the start ID and less than the end ID in each data table in the new database by means of paging query of primary key IDs and in ascending order of IDs. Query 10,000 IDs in each batch, and batch write these queried IDs into the second set corresponding to each data table in the Redis database in the way of pipeline in Redis.

[0051] Among them, if the sorting method in step S1 is descending order, then take the first primary key ID in the first set as the end ID, and the last primary key ID in the first set as the start ID.

[0052] For example, query the primary key IDs greater than the start ID and less than the end ID in the organization table, and then write the sorted queried primary key IDs into the second set corresponding to the organization table in the Redis database; and so on, to obtain the second set corresponding to the institution table and the second set corresponding to the node table.

[0053] S4: Calculate the missing sets corresponding to each data table respectively according to the first set and the second sets corresponding to each data table.

[0054] Specifically, perform intersection calculation on the first set and the second sets corresponding to each data table respectively to obtain the intersection sets corresponding to each data table; perform difference calculation on the first set and the intersection sets corresponding to each data table respectively to obtain the missing sets corresponding to each data table.

[0055] For example, assume that the first set is denoted as A, the second set corresponding to the organization table is denoted as B, the second set corresponding to the institution table is denoted as C, and the second set corresponding to the node table is denoted as D. Since A, B, C, and D are all sets in the Redis database, operations such as taking intersections, unions, and differences can be performed. Therefore, first perform the intersection operation of Redis on A and B, and denote the obtained intersection set as B1; perform the intersection operation of Redis on A and C, and denote the obtained intersection set as C1; perform the intersection operation of Redis on A and D, and denote the obtained intersection set as D1. Then perform the difference operation of Redis on A and B1, and denote the obtained difference set as B2, and B2 is the missing set corresponding to the organization table; perform the difference operation of Redis on A and C1, and denote the obtained difference set as C2, and C2 is the missing set corresponding to the institution table; perform the difference operation of Redis on A and D1, and denote the obtained difference set as D2, and D2 is the missing set corresponding to the node table.

[0056] S5: Determine whether the missing set corresponding to each data table is empty. If so, execute step S6; if not, execute step S7.

[0057] S6: Determine that the quantity verification of data migration has passed, that is, there is no missing migrated data.

[0058] S7: According to the missing sets corresponding to the respective data tables, re-migrate the missing primary key IDs in the respective data tables. Specifically, if the missing set corresponding to a data table is not empty, it means that some migrated data is not mapped to this table when migrated to the new database. Then, re-map the missing primary key IDs from the old database to this table, that is, supplement the missing primary key IDs in this table.

[0059] Furthermore, steps S1, S3, and S4 can be executed by multiple asynchronous tasks.

[0060] Furthermore, after the end of steps S6 and S7, it means that the primary key IDs of the current batch in the first set have been processed, and the operation of the next batch can be started, that is, re-execute step S1. Preferably, the first set can be cleared before re-executing step S1. Therefore, this embodiment can perform data verification on a data volume of millions, greatly improving the verification efficiency compared with the existing method.

[0061] This embodiment adopts the set operation method in a single-point Redis, loads the corresponding IDs of the old and new databases into the single-point Redis in batches, calculates the intersection of the two sets in the way of set operation in Redis, and then subtracts the intersection from the old data set. The remaining data is the IDs that have not been successfully migrated. Through this method, the number and specific data information of the data that have not been successfully migrated can be determined, which can be used for subsequent re-migration or other operations. Each set operation can be performed at the level of millions, which is hundreds of times faster than the previous calculation method using "in", greatly improving the verification efficiency.

[0062] Embodiment 2

[0063] This embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the program is executed by a processor, the following steps are implemented:

[0064] Obtain the primary key IDs of the migrated data in the old database, and write the sorted primary key IDs into the first set in the Redis database;

[0065] Obtain the respective data tables corresponding to the migrated data in the new database according to the preset mapping rules;

[0066] Query the primary key IDs within the corresponding ranges in each data table according to the first primary key ID and the last primary key ID in the first set, and write them into the second set corresponding to each data table in the Redis database;

[0067] Calculate the missing sets corresponding to each data table respectively according to the first set and the second set corresponding to each data table;

[0068] If the missing sets corresponding to each data table are all empty, it is determined that the quantity verification of data migration passes.

[0069] Further, the specific operation of querying the primary key IDs within the corresponding ranges in each data table according to the first primary key ID and the last primary key ID in the first set, and writing them into the second set corresponding to each data table in the Redis database is as follows:

[0070] Take the first primary key ID in the first set as the start ID, and the last primary key ID in the first set as the end ID. Query the primary key IDs greater than or equal to the start ID and less than or equal to the end ID in a data table corresponding to the migrated data, and write the queried primary key IDs into the second set corresponding to the data table in the Redis database after sorting.

[0071] Further, the specific operation of calculating the missing sets corresponding to each data table respectively according to the first set and the second set corresponding to each data table is as follows:

[0072] Perform an intersection calculation between the first set and the second set corresponding to each data table respectively to obtain the intersection sets corresponding to each data table;

[0073] Perform a difference calculation between the first set and the intersection sets corresponding to each data table respectively to obtain the missing sets corresponding to each data table.

[0074] Further, after calculating the missing sets corresponding to each data table respectively according to the first set and the second set corresponding to each data table, it further includes:

[0075] If the missing sets corresponding to each data table are not empty, re-migrate the missing primary key IDs in each data table respectively according to the missing sets corresponding to each data table.

[0076] In summary, a method for verifying the quantity of data migration and a computer-readable storage medium provided by the present invention write the corresponding primary key IDs in the old and new databases into different sets in the Redis database respectively, and then use the set operation method in Redis to calculate the missing set. By judging whether the missing set is empty, it can be determined whether the data migration is complete; at the same time, since the missing set stores the primary key IDs that have not been successfully migrated, the specific missing data can also be quickly determined, which is convenient for subsequent re-migration; by re-migrating the missing primary key IDs, the integrity of the data migration is ensured. The present invention can quickly determine the total missing quantity and the specific missing data, greatly improving the verification efficiency of data migration.

[0077] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in the relevant technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for verifying the quantity of data migration, characterized in that Including: Obtain the primary key IDs of the migration data in the old database, sort the primary key IDs, and write them into the first set in the Redis database; According to the preset mapping rules, obtain the corresponding data tables of the migration data in the new database; According to the first primary key ID and the last primary key ID in the first set, query the primary key IDs within the corresponding ranges in each of the data tables, and write them into the second set corresponding to each data table in the Redis database; According to the first set and the second sets corresponding to each of the data tables, calculate the missing sets corresponding to each of the data tables respectively; If the missing sets corresponding to each of the data tables are all empty, it is determined that the quantity verification of the data migration passes; After calculating the missing sets corresponding to each of the data tables according to the first set and the second sets corresponding to each of the data tables respectively, it further includes: If the missing sets corresponding to each of the data tables are not empty, re-migrate the missing primary key IDs in each of the data tables according to the missing sets corresponding to each of the data tables respectively; The calculating the missing sets corresponding to each of the data tables according to the first set and the second sets corresponding to each of the data tables specifically is: Perform an intersection calculation of the first set with the second sets corresponding to each of the data tables respectively to obtain the intersection sets corresponding to each of the data tables; Perform a difference calculation of the first set with the intersection sets corresponding to each of the data tables respectively to obtain the missing sets corresponding to each of the data tables.

2. The quantity verification method for data migration according to claim 1, characterized in that The querying the primary key IDs within the corresponding ranges in each of the data tables according to the first primary key ID and the last primary key ID in the first set, and writing them into the second set corresponding to each data table in the Redis database specifically is: Use the first primary key ID in the first set as the start ID, use the last primary key ID in the first set as the end ID, query the primary key IDs greater than or equal to the start ID and less than or equal to the end ID in a data table corresponding to the migration data, sort the queried primary key IDs, and write them into the second set corresponding to the data table in the Redis database.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the following steps are implemented: Obtain the primary key IDs of the migration data in the old database, sort the primary key IDs, and write them into the first set in the Redis database; According to the preset mapping rules, obtain the corresponding data tables of the migration data in the new database; According to the first primary key ID and the last primary key ID in the first set, query the primary key IDs within the corresponding ranges in each of the data tables, and write them into the second set corresponding to each data table in the Redis database; According to the first set and the second sets corresponding to each of the data tables, calculate the missing sets corresponding to each of the data tables respectively; If the missing sets corresponding to each of the data tables are all empty, it is determined that the quantity verification of the data migration passes; After calculating the missing sets corresponding to the respective data tables according to the first set and the second set corresponding to the respective data tables, it further includes: If the missing sets corresponding to the respective data tables are not empty, then respectively re-migrate the missing primary key IDs in the respective data tables according to the missing sets corresponding to the respective data tables; The specific method for calculating the missing sets corresponding to the respective data tables according to the first set and the second set corresponding to the respective data tables is as follows: Perform an intersection calculation of the first set with the second set corresponding to each respective data table to obtain an intersection set corresponding to each respective data table; Perform a difference calculation of the first set with the intersection set corresponding to each respective data table to obtain the missing sets corresponding to the respective data tables.

4. The computer-readable storage medium according to claim 3, wherein The specific method for querying the primary key IDs within the corresponding range in each data table according to the first primary key ID and the last primary key ID in the first set and writing them into the second set corresponding to each data table in the Redis database is as follows: Use the first primary key ID in the first set as the start ID and the last primary key ID in the first set as the end ID, query the primary key IDs greater than or equal to the start ID and less than or equal to the end ID in a data table corresponding to the migrated data, and write the queried primary key IDs into the second set corresponding to the data table in the Redis database after sorting.

Images