Data consistency verification method and device after heterogeneous database data migration

Abstract

The application relates to a data consistency checking method and device for a heterogeneous database after data migration, and particularly relates to the technical field of databases.The method comprises the following steps: acquiring comparison configuration information of the heterogeneous database after data migration; determining at least one data checking group according to the comparison configuration information of the heterogeneous database and data storage information of the heterogeneous database, and generating data query instructions corresponding to the data checking group; selecting a target query instruction in the data query instructions corresponding to the data checking group, extracting a first result set corresponding to the data checking group from a source database of the heterogeneous database according to the target query instruction, and extracting a second result set corresponding to the data checking group from a target database of the heterogeneous database; generating a concurrent checking task corresponding to the heterogeneous database based on the first result set and the second result set corresponding to the data checking group, checking the concurrent checking task, and generating a checking result of the concurrent checking task.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a method and apparatus for verifying data consistency after data migration from heterogeneous databases. Background Technology

[0002] Data migration is a common and very important task in the process of application upgrades and transformations. It can be used to verify data consistency between homogeneous and heterogeneous databases after data migration.

[0003] In related technologies, proprietary tools are typically used to perform data consistency verification on migrated databases. However, due to the differences in data types, storage methods, and access methods among heterogeneous databases, the use of proprietary tools for verification suffers from low execution efficiency and excessively long verification cycles, making it difficult to meet the needs of rapid consistency verification of massive heterogeneous data. Summary of the Invention

[0004] In view of this, this application provides a data consistency verification method and apparatus after data migration from heterogeneous databases. The main purpose is to improve the technical problems in related technologies where the verification is inefficient and has a long verification cycle when using dedicated tools due to differences in data types, storage methods, and access methods of heterogeneous databases, making it difficult to meet the needs of rapid consistency verification of massive heterogeneous data.

[0005] Firstly, this application provides a method for verifying data consistency after data migration from heterogeneous databases, the method comprising: Obtain the comparison configuration information of the heterogeneous databases after the data migration is completed; Based on the comparison configuration information and data storage information of the heterogeneous databases, at least one data verification group is determined, and a data query instruction corresponding to the data verification group is generated. Select the target query instruction from the data query instructions corresponding to the data validation group. According to the target query instruction, extract the first result set corresponding to the data validation group from the source database of the heterogeneous database, and extract the second result set corresponding to the data validation group from the target database of the heterogeneous database. Based on the first and second result sets corresponding to the data verification group, a concurrent verification task corresponding to the heterogeneous database is generated, the concurrent verification task is verified, and the verification result of the concurrent verification task is generated.

[0006] Secondly, this application provides a data consistency verification device after data migration from a heterogeneous database, the device comprising: The acquisition module is configured to acquire the comparison configuration information of heterogeneous databases after data migration is completed; The generation module is configured to determine at least one data verification group based on the comparison configuration information and data storage information of the heterogeneous database, and generate a data query instruction corresponding to the data verification group. The extraction module is configured to select a target query instruction from the data query instructions corresponding to the data validation group, extract the first result set corresponding to the data validation group from the source database of the heterogeneous database according to the target query instruction, and extract the second result set corresponding to the data validation group from the target database of the heterogeneous database. The verification module is configured to generate concurrent verification tasks corresponding to heterogeneous databases based on the first and second result sets corresponding to the data verification group, verify the concurrent verification tasks, and generate the verification results of the concurrent verification tasks.

[0007] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.

[0008] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0009] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of the first aspect.

[0010] By employing the above technical solution, this application provides a data consistency verification method and apparatus after heterogeneous database data migration. Compared with related technologies, this application obtains the comparison configuration information of the heterogeneous database after data migration; determines at least one data verification group based on the comparison configuration information and data storage information of the heterogeneous database, and generates a data query instruction corresponding to the data verification group; selects a target query instruction from the data query instructions corresponding to the data verification group, and extracts a first result set corresponding to the data verification group from the source database of the heterogeneous database and a second result set corresponding to the data verification group from the target database of the heterogeneous database according to the target query instruction; generates a concurrent verification task corresponding to the heterogeneous database based on the first and second result sets corresponding to the data verification group, verifies the concurrent verification task, and generates the verification result of the concurrent verification task. This is an embodiment.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating the data consistency verification method after heterogeneous database data migration provided in an embodiment of this application is shown. Figure 2 This document shows an overall flowchart of an example provided in an embodiment of this application; Figure 3 This illustration shows a schematic diagram of an example data extraction process provided in an embodiment of this application; Figure 4 This illustration shows a data comparison process diagram of an example provided in an embodiment of this application; Figure 5 This illustration shows a schematic diagram of an example data verification process provided in an embodiment of this application; Figure 6 A schematic diagram of the structure of the data consistency verification system after data migration of heterogeneous databases provided in this application embodiment is shown. Detailed Implementation

[0015] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0016] Specifically, performing data consistency comparison after data migration is a key task in measuring the success of the migration. When comparing data between two homogeneous databases, verification is typically done using proprietary tools from each database vendor. However, between heterogeneous databases, due to differences in data types, storage methods, and access methods, achieving consistent data comparison presents significant challenges. Currently, some data consistency comparison tools exist. For example, the open-source tool NineData employs a platform-based deployment, with data comparison being only a small part of its functionality, resulting in insufficient flexibility and limited functionality. Another type, represented by the open-source tool dataCompare, while achieving data comparison between heterogeneous databases, has significant shortcomings in areas such as data type (e.g., date) precision, concurrent comparison control, support for large fields (e.g., CLOB, RAW), database support types, and data comparison methods (value, hash, etc.).

[0017] To address the technical challenges of low execution efficiency, excessive memory consumption, and long verification cycles when using dedicated tools for consistency verification due to differences in data types, storage methods, and access methods in heterogeneous databases, which make it difficult to meet the needs of rapid consistency verification of massive heterogeneous data, this embodiment provides a data consistency verification method after data migration from a heterogeneous database. Figure 1 As shown, the method includes: Step 101: Obtain the comparison configuration information of the heterogeneous database after the data migration is completed.

[0018] Heterogeneous databases can include multiple independent databases with different data models and storage methods, such as the source database for data migration and the target database for receiving the migrated data.

[0019] In some embodiments, in order to verify the migrated heterogeneous database and ensure data consistency, the heterogeneous database can be configured first to obtain its comparison configuration information. The comparison configuration information may include verification data items, preset verification data volume, comparison content specification method, data normalization format configuration, etc., so as to automatically verify the heterogeneous database and improve verification efficiency.

[0020] Step 102: Based on the comparison configuration information and data storage information of the heterogeneous databases, determine at least one data verification group and generate the data query instruction corresponding to the data verification group.

[0021] The data storage information of heterogeneous databases may include the current storage information of the source database and the current storage information of the target database, such as table metadata. Data verification groups can be independent verification units divided according to business modules, table size, and verification type. For example, a data verification group may include the source table in the source data, the target table corresponding to the source table, and the mapping relationship between the two.

[0022] In some embodiments, before data verification, the data storage information in heterogeneous databases can be grouped according to the preset data volume in the comparison configuration information. For example, at least one database table with the same verification rules and capable of batch parallel verification can be packaged into an independent execution unit as a data verification group. Alternatively, each database table can be treated as a separate data verification group, using the same or different verification rules for parallel verification, thereby improving data verification efficiency. After grouping the current data storage information, corresponding data query instructions can be generated for each data group to sequentially read specific data for data consistency verification, achieving automated verification.

[0023] Step 103: Select the target query instruction in the data query instructions corresponding to the data validation group, and extract the first result set corresponding to the data validation group from the source database of the heterogeneous database according to the target database of the heterogeneous database, and extract the second result set corresponding to the data validation group from the target database of the heterogeneous database.

[0024] In some embodiments, various data query instructions corresponding to the data verification group can be generated according to the comparison configuration information. From the various data query instructions corresponding to the data verification group, the target query instruction to be executed for cross-database extraction for consistency verification can be selected, such as row count query instruction, field query instruction, record query instruction, table name query instruction, etc. After determining the target query instruction to be executed, the source database and the target database can be connected respectively. The first result set of the current verification group can be extracted from the source database in sequence using the result set cursor, and the second result set corresponding to the data verification group can be extracted from the target database to obtain the specific data to be verified, and then the verification is performed.

[0025] The source database can be the original database before the data migration, such as a MySQL business database. The target database can be the receiving database after the data migration is complete, and it belongs to a different database type than the source database, such as an Oracle business database.

[0026] The first result set includes the data set that matches the target query command found in the source database, and can be read in batches using a cursor. The second result set includes the data set that matches the target query command found in the target database, and can also be read in batches using a cursor. It adopts the same sorting rules as the source database to ensure that the order of the two datasets corresponds one-to-one and to prevent misalignment during comparison.

[0027] Step 104: Based on the first result set and the second result set corresponding to the data verification group, generate concurrent verification tasks corresponding to the heterogeneous database, verify the concurrent verification tasks, and generate the verification results of the concurrent verification tasks.

[0028] In some embodiments, multiple threads can be created for multiple data verification groups. The first result set and the second result set of each data verification group are used as multiple concurrent verification tasks. Multiple threads are used to perform data consistency verification on multiple concurrent verification tasks in parallel, so that multiple tasks perform data comparison operations simultaneously. During the execution of the verification process, these tasks can use different verification rules to perform verification. Finally, the verification results corresponding to each concurrent verification task are generated. The verification results may include data difference information (such as inconsistent data in a certain row), verification status (such as verification passed, verification failed), etc., to provide feedback on the consistency verification results of all migrated data in the heterogeneous database, thereby realizing an efficient and highly compatible heterogeneous database verification method.

[0029] Compared with related technologies, this embodiment first configures the heterogeneous database after data migration and obtains its comparison configuration information. Based on the configuration information and the data storage information in the current heterogeneous database, the data storage information is divided into multiple data verification groups, and corresponding data query instructions are generated. Then, the target query instruction is automatically selected from the data query instructions to extract the first result set and the second result set corresponding to each data verification group from the source database and the target database. The large amount of verification data in the heterogeneous database is divided into multiple concurrent verification tasks. Finally, the multiple concurrent verification tasks are verified to generate verification results. By grouping data in the heterogeneous database and automatically generating data query instructions, parallel verification of multiple data verification groups is achieved, effectively improving data verification efficiency, reducing the verification cycle, and meeting the needs of rapid consistency verification of massive heterogeneous data.

[0030] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the implementation of this embodiment, step 102 may optionally include: determining at least one data verification group based on the preset verification data volume in the comparison configuration information; determining the target comparison method corresponding to the data verification group according to the comparison method configuration item in the comparison configuration information; and generating a data query instruction corresponding to the data verification group according to the target comparison method and data storage information.

[0031] Specifically, the comparison configuration information can be used to pre-configure multiple verification configuration items based on business needs and system resources, such as comparison method configuration items, preset verification data volume, preset concurrency, and preset normalization format. The comparison method configuration item can be used to determine the target comparison method corresponding to the current data verification group. Multiple comparison methods can be selected, such as row count comparison, content comparison, table name comparison, etc., or multiple methods among these comparison items can be combined for comprehensive comparison to ensure the accuracy of data verification. For example, row count comparison and table name comparison can be used together to perform multiple verifications on a data verification group. Different verification methods can also be used for multiple data verification groups, such as one data verification group using the row count comparison method and another data verification group using the table name comparison method, and the two data verification groups use two threads for parallel verification.

[0032] In some embodiments, this embodiment may include a parameter parsing and task scheduling module, a data extraction module, and a data comparison module, such as... Figure 2 As shown, the comparison configuration information can first be read from the command line or configuration file of the heterogeneous database through the parameter parsing and task scheduling module. Then, the data to be compared and verified can be extracted through the data extraction module. For example, it can be configured to compare the query data corresponding to multiple database tables or multiple Structured Query Language (SQL) query statements at one time. Accordingly, for the sake of simplicity, a specific database table or an SQL query statement can be referred to as a table, and this abbreviation will be used throughout the text. After the comparison configuration information is parsed, each table can be defined as a concurrent verification task through the data comparison module. The system specifies a preset number of concurrent tasks through the comparison configuration information and automatically builds multiple threads to perform data consistency comparison on multiple concurrent verification tasks according to this number. Through multiple sets of comparisons, the system realizes fast data consistency comparison between tables in heterogeneous databases, improving verification efficiency.

[0033] Optionally, based on the target comparison method and data storage information, a data query instruction corresponding to the data verification group is generated, including: if the target comparison method is a row count comparison method, then a count query instruction corresponding to the data verification group is generated based on the data storage information; if the comparison result is that the number of rows in the source data is inconsistent with the number of rows in the target data, then a data query instruction corresponding to the data verification group is generated based on the comparison content specified in the comparison configuration information and the data storage information.

[0034] In some embodiments, the target comparison method for the current data verification can be determined by comparing configuration information. For example, if the row count comparison method is used, a corresponding counting query instruction is generated based on some fields in the data storage information to obtain the source data row count of the source verification data in the source database and the target data row count of the target verification data in the target database in the data verification group. The source data row count and the target data row count are compared to see if they are consistent. This simple and quick method is used for data verification. If the row count comparison passes, no other comparison method is needed to quickly verify whether the rough data is inconsistent, thus improving comparison efficiency. If the source data row count and the target data row count are inconsistent, i.e., the row count verification fails, a detailed comparison of the data content can be performed according to the specified comparison content. Based on some fields in the data storage information, a data query instruction corresponding to the data verification group is generated to obtain the detailed data to be compared.

[0035] Optionally, based on the comparison content specification method in the comparison configuration information, a data query instruction corresponding to the data verification group is generated according to the data storage information. Specifically, this may include: if the comparison content specification method is a specified table name comparison method, then the table metadata corresponding to the data verification group is obtained from the data storage information, and a full-field query instruction corresponding to the table metadata is generated; based on the unique index and / or primary key of the table metadata, an additional sorting clause is generated corresponding to the full-field query instruction, which is used to specify the query order; based on the full-field query instruction and the additional sorting clause, a data query instruction corresponding to the data verification group is generated.

[0036] In some embodiments, if the heterogeneous database is determined to be compared by row count through comparison configuration information, and the comparison content specification does not specify a table name comparison method, then the table names of each table can be read according to the data storage information, and then the table metadata of each table can be read according to the table names to generate a full-field query instruction for full-field comparison. The full-field query instruction can be an SQL query statement containing all fields. Furthermore, to ensure the accuracy of the heterogeneous database comparison and avoid comparison bias, a data sorting configuration item can be configured during verification based on the comparison configuration information on the basis of the full-field query instruction. An attachment sorting clause can be generated according to the data sorting configuration item and the data attributes of the table metadata. For example, if the attachment sorting clause is to sort in ascending order according to the unique index, the table metadata obtained by the data verification group from the heterogeneous database will be sorted in ascending order according to the unique index. Then, the sorted data is subjected to consistency verification to ensure the accuracy of data verification.

[0037] For example, such as Figure 3As shown, it can be first determined whether the current data verification group uses row count comparison. If it is determined that row count comparison is used, a counting SQL statement for the group is generated. Then, this counting SQL statement is selected as the SQL statement. The query is executed in the source database according to the selected SQL statement to obtain the first result set (result set L), and the query is executed in the target database to obtain the second result set (result set R). If it is determined that the comparison content specification method is used, it is determined whether the comparison content specification method is a direct comparison method or a table name only comparison method. If it is a direct comparison method, a complete SQL statement is directly generated, and the result set is read according to the statement. If it is a table name only comparison method (only table name only comparison method), the result set is read. If a table name is specified, the table metadata for that group is retrieved, and an SQL query statement containing all fields is generated. The system then checks if the table metadata has a primary key. If it does, a sort clause based on the primary key is appended to the SQL query statement containing all fields. If no primary key exists, but a unique index exists, a sort clause based on the primary key is appended. If neither a primary key nor a unique index exists, a sort clause based on all fields is appended. Finally, the sort clauses are combined to generate a complete SQL statement for dataset querying. This achieves intelligent sorting of comparison data and consistency comparison of table metadata for tables without primary keys or unique indexes, meeting consistency comparison requirements in various data migration scenarios.

[0038] Optionally, in all the generated or specified SQL statements above, the query order specified in the sort clause can be ascending.

[0039] Optionally, step 104 may specifically include: normalizing the first result set and the second result set corresponding to the data verification group; generating multiple concurrent verification tasks based on the normalized first result set and the second result set; determining the target verification method corresponding to the concurrent verification task based on the comparison configuration information; and comparing the first result set and the second result set one by one according to the target verification method corresponding to the concurrent verification task to generate the verification result corresponding to the concurrent verification task.

[0040] Correspondingly, during the data extraction process, specific SQL statements can be intelligently generated based on the parsed task parameters, and connections can be made to the source database and target database respectively to obtain the corresponding result set cursors. Database data can be accessed in the form of result set cursors, and the amount of data returned from the database at one time can be controlled by setting parameters, effectively reducing memory consumption.

[0041] In some embodiments, due to the inconsistency in field types and storage methods of heterogeneous databases, data with the same business meaning may result in inconsistent comparison results. Therefore, a normalization process can be added to the data in the first and second result sets. By formatting the data in specified fields, this situation can be prevented. Specifically, normalization processing can be performed according to a preset normalization format. Based on the normalized first and second result sets and the target query instruction, concurrent verification tasks corresponding to each data verification group are generated, i.e., multiple threads are used for concurrent verification. Based on the comparison configuration information, the target verification method corresponding to each concurrent verification task is read, such as hash verification method, value verification method, etc., to meet various verification requirements. Data consistency verification is performed on each record and field stored in the first and second result sets, generating the verification results of each verification task. Through normalization processing, data consistency comparison of different data types in the same field can be achieved, improving the accuracy and efficiency of verification.

[0042] Among them, data normalization processing can support a large number of formatting methods, such as setting the significant digits of floating-point numbers, date and time format and time zone conversion, character set conversion, clearing low-value data in strings, converting numbers stored in text form to numerical values, and removing extra whitespace characters from strings, and all formatting methods support parameterized configuration.

[0043] Optionally, according to the target verification method corresponding to the concurrent verification task, the first result set and the second result set are compared one by one to generate the verification result corresponding to the concurrent verification task. Specifically, this may include: if the target verification method is a hash verification method, the source record hash value of the first result set and the target record hash value of the second result set are calculated, and the verification result corresponding to the concurrent verification task is generated based on the hash comparison result between the source record hash value and the target record hash value; if the target verification method is a value verification method, the source record field of the first result set is compared with the target record field of the second result set, and the verification result corresponding to the concurrent verification task is generated based on the field comparison result between the source record field and the target record field.

[0044] For example, such as Figure 4 As shown, data validation can be performed on the cursor of the result set obtained in the previous step: (1) Row count comparison: Output the row count comparison result. If the row counts are equal, the verification result is determined to be consistent. If they are not equal, the verification result is determined to be inconsistent. (2) Check whether result set L is empty or result set R is empty, output the comparison result. If both result set L and result set R are empty, the verification result is determined to be consistent; if at least one of result set L and result set R is not empty, the verification result is determined to be inconsistent. (3) When it is determined that there is inconsistency according to step (2), perform comparison item by item, initialize Lr and Rr respectively as the first record in result set L and result set R, and compare Lr and Rr: If Lr < Rr, it indicates that there is an extra record in result set L, output Lr to the difference result, and assign Lr=null; If Lr = Rr, assign Lr=null and Rr=null; If Lr > Rr, it indicates that there is an extra record in result set R, output Rr to the difference result, and assign Rr=null; (4) After updating Lr and Rr according to step (3), determine whether Lr is null (Lr=null?); if Lr=null, further determine whether there are remaining records in result set L: If there are remaining records in result set L, assign Lr as the next record in L, and then determine whether Rr is null; if there are no remaining records in result set L but there are remaining records in result set R, output all data of R to the difference result, which indicates there is extra data in R; If Lr is not null, further determine whether Rr is null: if Rr=null, further determine whether there are remaining records in result set R; if there are remaining records in result set R, assign Rr as the next record in R; if there are no remaining records in result set R but there are remaining records in result set L, output all data of L to the difference result, which indicates there is extra data in L.

[0045] Specifically, as shown in Figure 5 , when comparing Lr and Rr, perform data normalization processing on Lr and Rr, unify them into a common format before comparison. During comparison, the target verification mode can be determined first. If the target verification mode is a hash verification mode, calculate the hash value of the entire record before comparing the hash values, calculate the hash value for each row of data, compare whether the hash values are consistent, and record the content of the record if they are inconsistent. Since the databases are heterogeneous, the hash functions provided by the databases themselves are inconsistent and cannot be used, and a custom hash function can be adopted for implementation, which is more efficient than value comparison; if the target verification mode is a value verification mode, perform cyclic comparison field by field, compare whether the specific value of each field in each row of data is consistent, if the current field is verified to be consistent, verify the next field, and if the current field is verified to be inconsistent, record this specific field.

[0046] In this way, this embodiment can extract multiple data verification groups from heterogeneous databases according to the comparison configuration information and generate multiple concurrent verification tasks for parallel verification. During the comparison process, a rough comparison can be performed by comparing the number of rows, or a detailed comparison can be performed by comparing the content. During the content comparison process, hash verification can be performed one by one or value verification can be performed one by one to ensure the accuracy of verification while improving the verification efficiency. Furthermore, the data in the first result set and the second result set can be normalized before verification, supporting data consistency verification in various data migration scenarios and meeting various verification requirements.

[0047] Based on the above Figure 1 The specific implementation of the method shown in this embodiment provides a data consistency verification device after data migration from heterogeneous databases, such as... Figure 6 As shown, the device includes: an acquisition module 31, a generation module 32, an extraction module 33, and a verification module 34; The acquisition module 31 is configured to acquire the comparison configuration information of the heterogeneous database after the data migration is completed; The generation module 32 is configured to determine at least one data verification group based on the comparison configuration information and data storage information of the heterogeneous database, and generate a data query instruction corresponding to the data verification group. The extraction module 33 is configured to select a target query instruction from the data query instructions corresponding to the data validation group, extract the first result set corresponding to the data validation group from the source database of the heterogeneous database according to the target query instruction, and extract the second result set corresponding to the data validation group from the target database of the heterogeneous database. The verification module 34 is configured to generate concurrent verification tasks corresponding to heterogeneous databases based on the first result set and the second result set corresponding to the data verification group, verify the concurrent verification tasks, and generate the verification results of the concurrent verification tasks.

[0048] In some examples, the generation module 32 is specifically configured to determine at least one data verification group based on the preset verification data volume in the comparison configuration information; determine the number of source data rows in the source database and the number of target data rows in the target database in the data verification group according to the data storage information of the heterogeneous database; and generate a data query instruction corresponding to the data verification group based on the comparison result of the number of source data rows and the number of target data rows in the data verification group.

[0049] In some examples, the generation module 32 is specifically configured to reuse the counting query instruction corresponding to the data verification group if the comparison result shows that the number of source data rows is consistent with the number of target data rows; and to generate the data query instruction corresponding to the data verification group based on the comparison content specified in the comparison configuration information and the data storage information if the comparison result shows that the number of source data rows is inconsistent with the number of target data rows.

[0050] In some examples, the generation module 32 is specifically configured to, if the comparison content specified in the comparison configuration information is a specified table name comparison method, retrieve the table metadata corresponding to the data verification group from the data storage information of the heterogeneous database and generate a full-field query instruction corresponding to the table metadata; generate an additional sort clause corresponding to the full-field query instruction based on the unique index and / or primary key of the table metadata, the additional sort clause being used to specify the query order; and generate a data query instruction corresponding to the data verification group based on the full-field query instruction and the additional sort clause.

[0051] In some examples, the extraction module 33 is specifically configured to select a target query instruction from the data query instruction or count query instruction corresponding to the data validation group; according to the target query instruction, access the source database and the target database in the form of a result set cursor, extract the first result set corresponding to the data validation group from the source database, and extract the second result set corresponding to the data validation group from the target database.

[0052] In some examples, the verification module 34 is specifically configured to normalize the first result set and the second result set corresponding to the data verification group, generate multiple concurrent verification tasks based on the normalized first result set and the second result set, determine the target verification method corresponding to the concurrent verification task based on the comparison configuration information, and compare the first result set and the second result set one by one according to the target verification method corresponding to the concurrent verification task to generate the verification result corresponding to the concurrent verification task.

[0053] In some examples, the verification module 34 is specifically configured to, if the target verification method is hash verification, calculate the hash value of the source record in the first result set and the hash value of the target record in the second result set, and generate the verification result corresponding to the concurrent verification task based on the hash comparison result of the source record hash value and the target record hash value; if the target verification method is value verification, compare the source record field in the first result set with the target record field in the second result set, and generate the verification result corresponding to the concurrent verification task based on the field comparison result of the source record field and the target record field.

[0054] Based on the above, Figure 1 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 The method shown.

[0055] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0056] Based on the above, Figure 1 The method shown, and Figure 6 To achieve the above objectives, the present application also provides an electronic device, comprising a storage medium and a processor; the storage medium for storing a computer program; and the processor for executing the computer program to implement the above-described virtual device embodiments. Figure 1 The method shown.

[0057] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0058] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0059] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented using hardware. This application can extract multiple data verification groups from a heterogeneous database based on the comparison configuration information and generate multiple concurrent verification tasks for parallel verification. During the comparison process, a rough comparison can be performed using row count comparison, or a detailed comparison can be performed using content comparison. During the content comparison process, hash verification can be performed row by row, or value verification can be performed field by field, to improve verification efficiency while ensuring verification accuracy. Furthermore, the data in the first result set and the second result set can be normalized before verification, supporting data consistency verification in various data migration scenarios and meeting diverse verification requirements.

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0062] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for verifying data consistency after data migration from a heterogeneous database, characterized in that, include: Obtain the comparison configuration information of the heterogeneous databases after the data migration is completed; Based on the comparison configuration information and data storage information of the heterogeneous database, at least one data verification group is determined, and a data query instruction corresponding to the data verification group is generated. Select a target query instruction from the data query instructions corresponding to the data verification group, and extract the first result set corresponding to the data verification group from the source database of the heterogeneous database according to the target query instruction, and extract the second result set corresponding to the data verification group from the target database of the heterogeneous database. Based on the first result set and the second result set corresponding to the data verification group, a concurrent verification task corresponding to the heterogeneous database is generated, the concurrent verification task is verified, and the verification result of the concurrent verification task is generated.

2. The method according to claim 1, characterized in that, The step of determining at least one data verification group based on the comparison configuration information and data storage information of the heterogeneous database, and generating a data query instruction corresponding to the data verification group, includes: Based on the preset verification data volume in the comparison configuration information, at least one data verification group is determined. Based on the comparison method configuration item in the comparison configuration information, determine the target comparison method corresponding to the data verification group; Based on the target comparison method and the data storage information, a data query instruction corresponding to the data verification group is generated.

3. The method according to claim 2, characterized in that, The step of generating a data query instruction corresponding to the data verification group based on the target comparison method and the data storage information includes: If the target comparison method is a row count comparison method, then a count query instruction corresponding to the data verification group is generated based on the data storage information; If the comparison result shows that the number of rows of source data is inconsistent with the number of rows of target data, then based on the comparison content specified in the comparison configuration information, a data query instruction corresponding to the data verification group is generated according to the data storage information.

4. The method according to claim 3, characterized in that, The step of generating a data query instruction corresponding to the data verification group based on the comparison content specification method in the comparison configuration information and the data storage information includes: If the comparison content is specified as a table name comparison method, then the table metadata corresponding to the data verification group is obtained from the data storage information, and a full-field query instruction corresponding to the table metadata is generated. Based on the unique index and / or primary key of the table metadata, generate an additional sort clause corresponding to the full-field query instruction, the additional sort clause being used to specify the query order; Based on the full-field query instruction and the additional sorting clause, a data query instruction corresponding to the data validation group is generated.

5. The method according to claim 3, characterized in that, The step of selecting a target query instruction from the data query instructions corresponding to the data verification group, extracting a first result set corresponding to the data verification group from the source database of the heterogeneous database according to the target query instruction, and extracting a second result set corresponding to the data verification group from the target database of the heterogeneous database includes: Select the target query instruction from the data query instruction corresponding to the data verification group or the count query instruction; According to the target query instruction, the source database and the target database are accessed in the form of a result set cursor. The first result set corresponding to the data verification group is extracted from the source database, and the second result set corresponding to the data verification group is extracted from the target database.

6. The method according to claim 1, characterized in that, The process of generating a concurrent verification task corresponding to the heterogeneous database based on the first result set and the second result set corresponding to the data verification group, verifying the concurrent verification task, and generating the verification result of the concurrent verification task includes: The first result set and the second result set corresponding to the data verification group are normalized, and multiple concurrent verification tasks are generated based on the normalized first result set and the second result set. Based on the comparison configuration information, the target verification method corresponding to the concurrent verification task is determined; According to the target verification method corresponding to the concurrent verification task, the first result set and the second result set are compared one by one to generate the verification result corresponding to the concurrent verification task.

7. The method according to claim 1, characterized in that, The step of comparing the first result set and the second result set one by one according to the target verification method corresponding to the concurrent verification task to generate the verification result corresponding to the concurrent verification task includes: If the target verification method is a hash verification method, then calculate the source record hash value of the first result set and the target record hash value of the second result set, and generate the verification result corresponding to the concurrent verification task based on the hash comparison result of the source record hash value and the target record hash value; If the target verification method is a value verification method, then the source record field of the first result set is compared with the target record field of the second result set, and the verification result corresponding to the concurrent verification task is generated based on the field comparison result of the source record field and the target record field.

8. A data consistency verification device after data migration from a heterogeneous database, characterized in that, include: The acquisition module is configured to acquire the comparison configuration information of heterogeneous databases after data migration is completed; The generation module is configured to determine at least one data verification group based on the comparison configuration information and data storage information of the heterogeneous database, and generate a data query instruction corresponding to the data verification group. The extraction module is configured to select a target query instruction from the data query instructions corresponding to the data verification group, extract a first result set corresponding to the data verification group from the source database of the heterogeneous database according to the target query instruction, and extract a second result set corresponding to the data verification group from the target database of the heterogeneous database. The verification module is configured to generate concurrent verification tasks corresponding to the heterogeneous database based on the first result set and the second result set corresponding to the data verification group, verify the concurrent verification tasks, and generate the verification results of the concurrent verification tasks.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images