Data verification method, device, electronic device and computer readable storage medium

Abstract

The embodiments of the present application provide a data verification method, device, electronic device and computer-readable storage medium, which relate to the field of data processing technology. The method includes: the embodiments of the present application determine the target transaction by migrating from the first database to the second database, determine the transaction identifier and the first execution result of the target transaction; obtain the target snapshot in the second database that has an association relationship with the transaction identifier of the target transaction, and determine the second execution result corresponding to the first execution result from the target snapshot; determine the first verification result obtained after consistency verification of the first execution result and the second execution result. The embodiments of the present application realize that in the process of incremental data migration, the data events of each incremental data that have been migrated to the downstream database are verified, which, on the one hand, enables accurate verification of the incremental data, and on the other hand, effectively shortens the time difference between the migration and verification of the incremental data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology. Specifically, the present application relates to a data verification method, device, electronic device and computer-readable storage medium. Background Art

[0002] Data migration refers to the process of migrating data stored in the first database to the second database; data verification refers to the process of performing consistency verification on the data in the first database and the data migrated to the second database after the data migration is completed. The data migration process can be divided into two processes: full migration and incremental migration. Among them, full data migration refers to suspending writing to the first database at a preset time point, migrating all current data in the first database to the second database, and the data migrated in the full migration process is called full data; incremental migration refers to the process of resuming writing to the data in the first database after all the full data has been migrated, and migrating the changed data in the first database. The data migrated in the incremental migration process is called incremental data.

[0003] After the full data migration is completed, the full data is verified, that is, the data in the first and second databases are compared to determine whether the data in the first and second databases are consistent. In this process, the first and second databases are both suspended from writing and the data does not change, so the full data can be accurately verified; during the migration of incremental data, because the data in the first database is constantly changing, and the migration process continuously synchronizes the incremental data in the first database to the second database, the data in the first and second databases is constantly changing, so it is very difficult to verify the incremental data during the migration of incremental data.

[0004] The existing solution verifies incremental data in the same way as it verifies full data. When verifying incremental data, the writing to the first and second databases is suspended so that the data in the first and second databases do not change. The incremental data migrated within the target time period is verified. Although this method can accurately verify incremental data, it is time-consuming and a waste of time. In addition, since the business maintenance cycle is generally long, the use of this verification method is likely to result in the inability to discover inconsistent data in a timely manner, which can easily affect the subsequent business. Summary of the invention

[0005] The embodiments of the present application provide a data verification method, device, electronic device, computer-readable storage medium, and computer program product, which can solve the problem that the incremental data verification process takes a long time. The technical solution is as follows:

[0006] According to a first aspect of an embodiment of the present application, a data verification method is provided, the method comprising:

[0007] Determine that a target transaction is migrated from the first database to the second database, wherein the target transaction is a database transaction that has been executed in the first database;

[0008] Determine a transaction identifier and a first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining a database transaction, and the first execution result is an execution result corresponding to the target transaction after it is executed in the first database;

[0009] Obtaining a target snapshot in the second database that is associated with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is executed, and the at least one database transaction includes the target transaction;

[0010] Determine a second execution result corresponding to the first execution result from at least one execution result;

[0011] A first verification result obtained after consistency verification of the first execution result and the second execution result is determined, where the first verification result is used to indicate whether the first execution result and the second execution result are consistent.

[0012] In a possible implementation, determining a transaction identifier and a first execution result of a target transaction includes:

[0013] The target transaction is stored in a cache, and the target transaction in the cache is parsed to obtain a transaction identifier and a first execution result of the target transaction.

[0014] In a possible implementation, the execution result includes the primary key of the corresponding data row;

[0015] Determining a second execution result corresponding to the first execution result from at least one execution result includes:

[0016] A second execution result having the same primary key as the first execution result is determined from the at least one execution result.

[0017] In a possible implementation, the target transaction includes at least one event; before determining the transaction identifier and the first execution result of the target transaction, the following step is further included:

[0018] If it is determined that the primary keys of the data rows of two events in the target transaction are the same, the two events are merged to obtain a new event.

[0019] In a possible implementation, two events are merged to obtain a new event, including:

[0020] Determine that two events are the first event and the second event; the first event precedes the second event;

[0021] Determine event types of the first event and the second event, where the transaction types include insert type, update type, and delete type;

[0022] If the event type of the first event and the type of the second event meet the preset merging rule, the second event is re-determined as a new event obtained by merging the first event and the second event, and the first event is deleted.

[0023] In a possible implementation, obtaining a target snapshot in the second database that is associated with the transaction identifier of the target transaction includes:

[0024] The second database generates a snapshot corresponding to at least one execution result and generates a snapshot identifier corresponding to the snapshot; establishes an association relationship between the transaction identifier and the snapshot identifier, wherein at least one transaction identifier includes a transaction identifier of the target transaction;

[0025] A target snapshot identifier associated with the transaction identifier of the target transaction is obtained, and a target snapshot is determined from the second database according to the target snapshot identifier, wherein the target snapshot identifier is a unique identifier for determining the target snapshot.

[0026] In a possible implementation, after determining the transaction identifier and the first execution result of the target transaction, the method further includes:

[0027] If it is determined that the target snapshot does not exist in the second database, determining a third execution result having the same primary key as the first execution result from a data table of the second database;

[0028] A second verification result obtained after consistency verification of the first execution result and the third execution result is determined, where the second verification result is used to indicate whether the first execution result and the third execution result are consistent.

[0029] In a possible implementation, after determining a first verification result obtained by performing consistency verification on the first execution result and the second execution result, the method further includes:

[0030] If the first verification result is consistent, the target transaction is deleted from the cache and the next target transaction is parsed;

[0031] If the first verification result is inconsistent, first report information is sent, and the first report information is used to report that the first execution result and the second execution result are inconsistent.

[0032] In a possible implementation, after determining a second verification result obtained by performing consistency verification on the first execution result and the third execution result, the method further includes:

[0033] If the second verification result is consistent, the target transaction is deleted from the cache and the next target transaction is parsed;

[0034] If the second verification result is inconsistent, the third execution result corresponding to the first execution result is re-determined in the data table of the second database after a preset period of time, and the second verification result obtained after consistency verification of the first execution result and the third execution result is determined until the number of consistency verifications reaches a preset number.

[0035] According to a second aspect of an embodiment of the present application, a data verification device is provided, the device comprising:

[0036] A migration determination module, used to determine whether a target transaction is to be migrated from the first database to the second database, wherein the target transaction is a transaction that has been executed in the first database;

[0037] A first execution result determination module, used to determine a transaction identifier and a first execution result of a target transaction;

[0038] a target snapshot determination module, configured to obtain a target snapshot in the second database that is associated with the transaction identifier of the target transaction, obtain a target snapshot in the second database that is associated with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is executed; and the at least one database transaction includes the target transaction;

[0039] A second execution result determination module, used to determine a second execution result corresponding to the first execution result from at least one execution result;

[0040] The verification module is used to determine a first verification result obtained after consistency verification of the first execution result and the second execution result, and the first verification result is used to indicate whether the first execution result and the second execution result are consistent.

[0041] According to a third aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and when the processor executes the program, the steps of the method provided in the first aspect are implemented.

[0042] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0043] According to the fifth aspect of the embodiment of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device executes the steps of implementing the method provided in the first aspect.

[0044] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0045] The embodiment of the present application determines that the target transaction is migrated from the first database to the second database, wherein the target transaction is a database transaction that has been executed in the first database; determines the transaction identifier and the first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is the execution result obtained after the target transaction is executed in the first database; obtains a target snapshot in the second database that has an association relationship with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is completed, and at least one database transaction includes the target transaction; determines the second execution result corresponding to the first execution result from the at least one execution result; determines the first verification result obtained after the consistency check of the first execution result and the second execution result, and the first verification result is used to characterize whether the first execution result and the second execution result are consistent. In the process of incremental data migration, the data events of each incremental data that have been migrated to the downstream database are verified, so that the incremental data can be accurately verified, and the migration and verification of the incremental data are carried out in real time and synchronously, effectively shortening the time difference between the migration and verification of the incremental data, so that the incremental data migration results can be verified in real time or quasi-real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0047] Figure 1 A schematic diagram of a system architecture for implementing a data verification method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a data verification method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the association relationship between the transaction identifier and the snapshot identifier of the target transaction provided in an embodiment of the present application;

[0050] Figure 4A flowchart of another data verification method provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a data verification device provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of another data verification device provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0055] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0057] First, several terms involved in this application are introduced and explained:

[0058] Events, namely data events (DML events), represent changes in data rows in the database, including data row insertion (insert), data row update (update) and data row deletion (delete). These events are recorded in the form of binary log events.

[0059] A database transaction is a set of data events in a database work unit. This set of data events is either fully executed or not executed at all, and has the characteristics of atomicity, consistency, isolation, and persistence. Among them, a database transaction includes the original value of the corresponding data row (original data row), the change operation, and the execution result obtained after the change. Each database transaction in the binary log can be identified by a unique transaction identifier. The transaction identifier can be composed of the binary log file where the database transaction is located and the end position in the binary log, that is, the transaction end position. The transaction end position indicates the binary log file where a transaction is located and the offset of the end of the transaction in the binary log.

[0060] Binary log: Before executing a database transaction, the database writes the transaction to the log. The binary log usually consists of multiple ordered binlog files.

[0061] Snapshot, that is, data snapshot, is a backup method, that is, backing up the data of a disk at a certain point in time. Some databases (such as TiDB) support reading historical data, that is, data snapshots, such as data from 10 minutes ago, but databases generally only save data snapshots for a period of time, and expired snapshots will be recycled and inaccessible. Snapshots generally have a unique identifier, through which the data corresponding to the snapshot can be accessed, and it is represented by "snapshot identifier" below.

[0062] In industries such as banking and securities, data migration is quite common. Data migration refers to the process of migrating data stored in the first database to the second database. The data migration process can be divided into two processes: full migration and incremental migration. Among them, full data migration refers to suspending the writing of the first database at a preset time point and migrating all the current data of the first database to the second database. The data migrated during the full migration process is called full data; incremental migration refers to the process of resuming the writing of data in the first database after all the full data has been migrated, and then migrating the changed data in the first database. The data migrated during the incremental migration process is called incremental data.

[0063] After data migration is completed, it is generally necessary to perform consistency check on the migrated data (hereinafter referred to as data verification) to ensure that the data in the first database is completely and accurately migrated to the second database. Like data migration, data verification can also be divided into two processes: verifying the full data and verifying the incremental data.

[0064] Verify the full data, that is, compare the data in the first database with the data in the second database to determine whether the data in the first and second databases are consistent. During this process, the first and second databases are suspended from writing, and there is no changed data, so the full data can be accurately verified. During the migration of incremental data, the data in the first database is constantly changing. The migration process continuously synchronizes the incremental data in the first database to the second database. The data in the first and second databases are constantly changing. Therefore, it is very difficult to verify the incremental data during the migration of incremental data.

[0065] The existing solution verifies incremental data in the same way as it verifies full data. When verifying incremental data, writing to the first and second databases is suspended so that the data in the first and second databases do not change. Then, the incremental data migrated within the target time period is verified. Although this method can accurately verify incremental data, it is time-consuming and a waste of time. In addition, since the business maintenance cycle is generally long, using this verification method is likely to result in the inability to discover inconsistent data in a timely manner, which can easily affect the subsequent business.

[0066] The data verification method, device, electronic device, computer-readable storage medium and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0067] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0068] Figure 1 The system architecture diagram for implementing the data verification method provided in the embodiment of the present application is as follows. The entire system 10 includes a first database 110, a second database 120, and a data migration and verification tool 130. Among them, the first database and the second database can be any databases that store data. After writing data to the first database 110, the data in the first database 110 can be migrated to the second database 120. The data in the second database 120 comes from the first database 110, that is, the data in the second database 120 depends on the first database 110; the migration and verification tool 130 is used to migrate the data in the first database to the second database, and perform consistency verification on the migrated data. The executor of the embodiment of the present application is the data migration and verification tool 130.

[0069] The present application provides a data verification method, such as Figure 2As shown, the method includes:

[0070] Step S201 : determining that a target transaction is to be migrated from a first database to a second database, wherein the target transaction is a database transaction that has been executed in the first database.

[0071] The first database and the second database in the embodiment of the present application are any databases that can store data. When performing data migration, the database in the upstream database is usually migrated to the downstream database. That is, the first database of the present application can also be called the upstream database, and the second database can also be called the downstream database.

[0072] The database transaction in the embodiment of the present application is a work unit executed in the database, which is composed of a set of data events. This set of events is either fully executed or not executed at all, and has the characteristics of atomicity, consistency, isolation, and persistence. Events are used to change data rows in the database. A database transaction includes a set of events, and thus, database transactions are also used to change data rows in the database.

[0073] It can be understood that the data in the first database is migrated to the second database, that is, the change of the data in the second database is changed along with the change of the data in the first database.

[0074] The data in the first database of the embodiment of the present application changes with the execution of database transactions. After the first database executes the database transaction, the changed data rows obtained are incremental data. Therefore, before migrating the data, it is necessary to ensure that the first database has executed the database transaction.

[0075] In the embodiment of the present application, the database transactions of the first database are recorded in the binary log of the first database. The migration of incremental data in the first database is actually the migration of corresponding database transactions. If the database transactions in the first database are migrated to the second database, the corresponding incremental data in the first database can be migrated to the second database.

[0076] The execution body of the embodiment of the present application is a migration and verification tool, which is used to migrate data and perform consistency verification after migration. Before verifying the target transaction in the first database, on the one hand, it is necessary to ensure that the target transaction is a database transaction that has been executed by the first database. The database transaction that has been executed by the first database can realize the writing of incremental data to the first database; on the other hand, it is necessary to ensure that the target transaction has been migrated from the first database to the second database, and data verification can be performed after migration.

[0077] The embodiment of the present application pulls one or more database transactions from the binary log of the first database as target transactions to the cache, where the target transaction is a database transaction that has been executed by the first database, and the migration and verification tool does not make changes to the database transactions in the binary log.

[0078] Step S202, determining a transaction identifier and a first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining a database transaction, and the first execution result is an execution result corresponding to the target transaction after execution is completed in the first database.

[0079] The database transactions of the first database in the embodiment of the present application are recorded in the binary log of the first database. Each database transaction has its corresponding unique transaction identifier, which can uniquely determine the unique identifier of the corresponding database transaction. The transaction identifier in the embodiment of the present application is the transaction end position of the transaction in the binary log.

[0080] The database transaction in the embodiment of the present application includes the original data row before being changed, the operation instruction to change the original data row, and the first execution result obtained after the change. The first execution result is the execution result obtained after the target transaction is executed in the first database.

[0081] The operation instructions for changing the embodiment of the present application include insert instructions, update instructions and delete instructions. For the insert instruction, the original data row corresponding to the target transaction does not exist in the original data in the first database. After the first database executes the target transaction to insert the corresponding new data row, the corresponding new data row inserted is obtained, and the first execution result is the inserted new data row; for the update instruction, the database transaction includes the corresponding original data row, the first database executes the target transaction to update the corresponding original data row, and obtains the updated data row, and the first execution result is the updated data row; for the delete instruction, the database transaction includes the corresponding original data row, after the corresponding original data row is deleted, the corresponding original data row will no longer exist in the first database, and the first execution result is that the corresponding original data row does not exist in the first database.

[0082] A database transaction includes a group of events, each of which is used to change a data row, which means that a database transaction may correspond to multiple data rows. Therefore, in the embodiment of the present application, the original data row and the data row corresponding to the first execution result are not necessarily separate data rows, but may be a combination of multiple data rows. For events that are all delete instructions, the first execution result corresponding to the database transaction does not include the corresponding original data row.

[0083] The embodiment of the present application pulls the target transaction from the binary log of the first database to the cache, parses the target transaction in the cache, and obtains the transaction identifier and the first execution result of the target transaction.

[0084] Step S203: obtaining a target snapshot in the second database that is associated with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed after the second database executes the migration, and the at least one database transaction includes the target transaction.

[0085] Snapshot, also known as data snapshot, is a means of backing up and restoring online data. A snapshot can be a copy of the data it represents or a replica of the data. Most databases have a snapshot mechanism, that is, most data stored in a database has its corresponding snapshot.

[0086] For the second database with a snapshot mechanism, after the second database in the embodiment of the present application executes the migrated database transaction, it will store the execution result of the database transaction in the snapshot and record the current snapshot identifier. In the subsequent process, the corresponding execution result can be found by querying the snapshot corresponding to the snapshot identifier.

[0087] It is worth noting that a snapshot corresponds to at least one execution result obtained after the second database executes at least one database transaction that completes the migration. It can be the execution result of database transactions over a period of time, or it can be the execution result of one or more batches of database transactions in a batch processing process. The embodiments of the present application do not limit this.

[0088] During batch processing, multiple target transactions can be determined at one time. A corresponding snapshot records the data rows corresponding to multiple database transactions. Multiple target transactions can be verified at the same time, which improves verification efficiency.

[0089] During the migration process, after the second database completes executing the target transaction and obtains the target snapshot, the migration and verification tool will establish an association relationship between the transaction identifier of the target transaction and the snapshot identifier of the target snapshot. The target snapshot records at least one execution result corresponding to the execution of at least one database transaction completed by the second database, and the at least one database transaction includes the target transaction currently being verified, so that during the verification process, according to the association relationship between the transaction identifier of the target transaction and the snapshot identifier of the target snapshot, the target snapshot associated with the target transaction can be found, and then the second execution result corresponding to the first execution result can be found from the at least one execution result in the target snapshot. The detailed process is shown in the subsequent part.

[0090] Step S204: determine a second execution result corresponding to the first execution result from at least one execution result.

[0091] In the embodiment of the present application, after determining at least one execution result from the target snapshot, a second execution result corresponding to the first execution result is determined from the at least one execution result, wherein the primary key of the second execution result is the same as the primary key of the first execution result, indicating that the first execution result and the second execution result are obtained after the first database and the second database execute the same target transaction, respectively.

[0092] Step S205 , determining a first verification result obtained after consistency verification of the first execution result and the second execution result, wherein the first verification result is used to indicate whether the first execution result and the second execution result are consistent.

[0093] After determining the first execution result and the second execution result, the embodiment of the present application performs a consistency check on the first execution result and the second execution result to obtain a first check result, and the first check result is used to indicate whether the first execution result and the second execution result are consistent.

[0094] The target transaction of the embodiment of the present application is composed of a group of events. This group of events is either fully executed or not executed at all. Each event has its corresponding data row. The event is used to change the data row. There are three event types. The event type is determined according to the operation instructions in the event, namely insert type, update type and delete type, which are used for data row insertion, data row update and data row deletion respectively.

[0095] For data row insertion, that is, inserting a new data row in the upstream database, the first execution result is the inserted new data row, and the second execution result is the data row in the target snapshot that has the same primary key as the first execution result. At this time, the values ​​of each column in the first execution result and the values ​​of each column in the second execution result can be compared column by column, or a checksum can be calculated to determine whether the first execution result and the second execution result are consistent. If the checksum is consistent, it indicates that the first execution result and the second execution result are consistent; otherwise, the first execution result and the second execution result are inconsistent.

[0096] For data row update, that is, updating an existing data row in the upstream database, the first execution result is the updated data row, and the second execution result is the data row in the target snapshot that has the same primary key as the first execution result. At this time, the values ​​of each column in the first execution result and the values ​​of each column in the second execution result can also be compared column by column, or a checksum can be calculated to determine whether the first execution result and the second execution result are consistent. If the checksum result is consistent, it indicates that the first execution result and the second execution result are consistent. Otherwise, the first execution result and the second execution result are inconsistent.

[0097] For data row deletion, that is, the upstream database deletes an existing data row, the first execution result is that the data row does not exist. Similarly, if there is no data row with the same primary key as the first execution result in the target snapshot, the second execution result is that the data row does not exist, indicating that the verification result is consistent; otherwise, the second verification result is inconsistent.

[0098] The embodiment of the present application determines that the target transaction is migrated from the first database to the second database, determines the transaction identifier and the first execution result of the target transaction; obtains the target snapshot in the second database that is associated with the transaction identifier of the target transaction, and determines the second execution result corresponding to the first execution result from at least one execution result; determines the first verification result obtained after consistency verification of the first execution result and the second execution result, the first verification result is used to characterize whether the first execution result and the second execution result are consistent. The embodiment of the present application implements consistency verification of the first execution result in the target transaction in the first database and the second execution result stored in the target snapshot of the second database, rather than directly comparing the first execution result in the first database with the second execution result in the second database, and implements verification of data events of each incremental data that has been migrated to the downstream database during the incremental data migration process, so that the incremental data can be accurately verified, and at the same time, the migration and verification of the incremental data are carried out in real time and synchronously, effectively shortening the time difference between the migration and verification of the incremental data, so that the incremental data migration result can be verified in real time or quasi-real time.

[0099] The embodiment of the present application provides a possible implementation method for determining a transaction identifier and a first execution result of a target transaction, including:

[0100] The target transaction is stored in a cache, and the target transaction in the cache is parsed to obtain a transaction identifier and a first execution result of the target transaction.

[0101] When verifying the data consistency between the first database and the second database, the embodiment of the present application first pulls one or more database transactions from the first database, determines the target transaction from the one or more database transactions, parses the target transaction in the cache, and obtains the transaction identifier and the first execution result of the target transaction.

[0102] Among them, the transaction identifier is used to uniquely identify a corresponding database transaction. The transaction identifier can be composed of the binary log file where the corresponding database transaction is located and the end position in the binary log file, that is, the transaction end position. The transaction end position indicates the binary log file where a database transaction is located and the offset of the end of the transaction in the binary log.

[0103] Any database transaction in the embodiments of the present application includes an original data row, a change operation performed on the original data row, and an execution result obtained after performing the change operation. The database executes the database transaction by essentially changing the original data row according to the original data row and the change operation to obtain the execution result.

[0104] The first execution result of the embodiment of the present application is the changed data row obtained after the first database executes the target transaction. The first execution result is also recorded in the database transaction. The embodiment of the present application parses the target transaction in the cache to obtain the transaction identifier and the first execution result of the target transaction.

[0105] The embodiment of the present application provides a possible implementation method, and the execution result includes the primary key of the corresponding data row;

[0106] Determining a second execution result corresponding to the first execution result from at least one execution result includes:

[0107] A second execution result having the same primary key as the first execution result is determined from the at least one execution result.

[0108] The execution result of the embodiment of the present application includes the primary key of the corresponding data row. The primary key is one or more fields in the data table, and the primary key is used to uniquely identify the data row in the data table.

[0109] In the embodiment of the present application, the primary key of the second execution result is the same as the primary key in the first execution result. The first execution result and the second execution result are respectively obtained after the first database and the second database execute the same target transaction. After determining the primary key of the first execution result, a data behavior with the same primary key as the first execution result is determined from at least one execution result as the second execution result.

[0110] The embodiment of the present application provides a possible implementation manner, where a database transaction includes at least one event; before determining a transaction identifier and a first execution result of a target transaction, the process further includes:

[0111] For any two events in a database transaction, if it is determined that the primary keys of the data rows of the two events are the same, the two events are merged to obtain a new event.

[0112] The transaction in the embodiment of the present application includes a group of events, each of which includes at least one event, and each event is used to change the corresponding data row, that is, the execution result included in the database transaction is a group of data rows, which are the data rows obtained after the changes of each event.

[0113] In the embodiment of the present application, before determining the transaction identifier and the first execution result of the target transaction, if it is determined that the primary keys of the data rows of two events in the target transaction are the same, it indicates that the two events can be merged to obtain a new event after the merger, as described in detail in the subsequent content.

[0114] The embodiment of the present application provides a possible implementation method, which combines two events to obtain a new event, including:

[0115] Determine that two events are the first event and the second event; the first event precedes the second event;

[0116] Determine event types of the first event and the second event, where the event types include an insert type, an update type, and a delete type;

[0117] If the event type of the first event and the type of the second event meet the preset merging rule, the second event is re-determined as a new event obtained by merging the first event and the second event, and the first event is deleted.

[0118] The embodiment of the present application determines that the two events that can be merged are the first event and the second event, and the order of the first event takes precedence over the second event, that is, the first event is executed by the first database first, and the second event is executed by the first database later. Similarly, the transaction end position of the first event in the first database is earlier than the transaction end position of the second event in the first database.

[0119] After determining the first event and the second event that can be merged, the embodiment of the present application determines the event types of the first event and the second event, where the event types include insert type, update type, and delete type.

[0120] If the event type of the first event and the type of the second event meet the preset merging rule, the second event is determined to be a new event obtained by merging the first event and the second event, and the first event is deleted.

[0121] Specifically, if the event type of the first event is insert, and the event type of the second event is update, the type of the new event after merging is update, that is, insert occurs first and then update, and they are merged into update;

[0122] If the event type of the first event is insert, and the event type of the second event is delete, the type of the new event after merging is delete, that is, insert occurs first and then delete, and they are merged into delete;

[0123] If the event type of the first event is update, and the event type of the second event is update, the type of the new event after merging is update, that is, update first and update second, merged into update;

[0124] If the event type of the first event is update, and the event type of the second event is delete, the type of the new event after merging is delete, that is, update occurs first and then delete, and the merged event is delete;

[0125] If the event type of the first event is delete, and the event type of the second event is insert, the type of the new event obtained after merging is insert, that is, delete occurs first and then insert, and they are merged into insert.

[0126] The embodiment of the present application provides a possible implementation method for obtaining a target snapshot in a second database that is associated with a transaction identifier of a target transaction, including:

[0127] The second database generates a snapshot corresponding to the at least one execution result and generates a snapshot identifier corresponding to the snapshot;

[0128] Establishing an association relationship between a transaction identifier and a snapshot identifier, wherein at least one transaction identifier includes a transaction identifier of a target transaction;

[0129] Obtaining a target snapshot identifier associated with the transaction identifier of the target transaction, wherein the target snapshot identifier is a unique identifier for determining the target snapshot;

[0130] The target snapshot is determined from the second database according to the target snapshot identifier.

[0131] In the embodiment of the present application, after determining that any database transaction is migrated from the first database to the second database, for the second database with a snapshot mechanism, after the second database executes the at least one database transaction, at least one execution result is obtained, and a snapshot corresponding to the at least one execution result and a snapshot identifier corresponding to the snapshot are generated accordingly, an association relationship between the transaction identifier and the snapshot identifier is established, and the association relationship is stored in the metadata, that is, the association relationship in the metadata indicates that the database transaction has been migrated to the second database, and the transaction identifier is usually the transaction end position.

[0132] like Figure 3The figure shows a schematic diagram of the association relationship between the transaction identifier and the snapshot identifier provided by an embodiment of the present application, which includes database transactions: transaction 1, transaction 2..., the transaction identifier is the transaction end position; there is an association relationship between the end position of transaction 1 and the snapshot identifier A, that is, after determining the transaction end position of transaction 1, the snapshot identifier A can be determined, and then the target snapshot A can be found; there is an association relationship between the end position of transaction 2 and the snapshot identifier B, that is, after determining the transaction end position of transaction 2, the snapshot identifier B can be determined, and then the target snapshot B can be found, and so on. The embodiment of the present application will not give examples one by one here.

[0133] For the target transaction, after executing the target transaction, the second database generates a target data snapshot containing the execution result of the target transaction. The target snapshot identifier corresponding to the target data snapshot is a unique identifier for determining the target snapshot. The migration and verification tool will establish an association relationship between the transaction identifier of the target transaction and the snapshot identifier of the target data snapshot. In the verification phase, the target snapshot can be determined from the second database according to the target snapshot identifier by obtaining the target snapshot identifier that is associated with the transaction identifier of the target transaction, and then the second execution result corresponding to the first execution result can be determined from the target snapshot.

[0134] The embodiment of the present application provides a possible implementation method, which, after determining the transaction identifier and the first execution result of the target transaction, further includes:

[0135] If it is determined that the target snapshot does not exist in the second database, determining a third execution result having the same primary key as the first execution result from a data table of the second database;

[0136] A second verification result obtained after consistency verification of the first execution result and the third execution result is determined, where the second verification result is used to indicate whether the first execution result and the third execution result are consistent.

[0137] For a second database with a snapshot mechanism, the target snapshot in the second database can be determined, and then the second execution result can be determined, and the consistency check between the first execution result and the second execution result can be performed. However, not all second databases have a snapshot mechanism, and some second databases do not have a snapshot mechanism. In addition, snapshots generally have a certain retention time, such as a retention time of 10 minutes, that is, the second database will keep snapshots for a period of time, and expired snapshots may be retracted and inaccessible.

[0138] When it is determined that the target snapshot does not exist in the second target database, that is, there is no snapshot mechanism in the second target database or the snapshot has expired and been recovered, the embodiment of the present application directly determines the third execution result with the same primary key as the first execution result from the data table of the second database, and obtains the second verification result after performing a consistency check on the first execution result and the third execution result. The second verification result is used to characterize whether the first execution result and the third execution result are consistent.

[0139] Similar to the consistency check of the first execution result and the second execution result, for data row insertion, that is, inserting a new data row in the upstream database, the first execution result is the inserted new data row, and the third execution result is the data row in the data table of the second database that has the same primary key as the first execution result. At this time, the values ​​of each column in the first execution result and the values ​​of each column in the third execution result can be compared column by column, or the checksum can be calculated to determine whether the first execution result and the third execution result are consistent. If the check result is consistent, it indicates that the first execution result and the third execution result are consistent; otherwise, the first execution result and the third execution result are inconsistent.

[0140] For data row update, that is, updating an existing data row in the upstream database, the first execution result is the updated data row, and the third execution result is the data row in the data table of the second database with the same primary key as the first execution result. At this time, the values ​​of each column in the first execution result and the values ​​of each column in the third execution result can also be compared column by column, or the checksum can be calculated to determine whether the first execution result and the third execution result are consistent. If the check result is consistent, it indicates that the first execution result and the third execution result are consistent, otherwise, the first execution result and the third execution result are inconsistent.

[0141] For data row deletion, that is, the upstream database deletes an existing data row, the first execution result is that the data row does not exist. Similarly, if there is no data row with the same primary key as the first execution result in the target snapshot, the third execution result is that the data row does not exist. In this case, the second verification result is consistent; otherwise, the second verification result is inconsistent.

[0142] The embodiment of the present application provides a possible implementation manner, after determining a first verification result obtained by performing consistency verification on the first execution result and the second execution result, further comprising:

[0143] If the first verification result is consistent, the target transaction is deleted from the cache and the next target transaction is parsed;

[0144] If the first verification result is inconsistent, first report information is sent, and the first report information is used to report that the first execution result and the second execution result are inconsistent.

[0145] In the embodiment of the present application, if the first verification result is consistent, there is no need to verify the first execution result corresponding to the target transaction and the second execution result in the snapshot of the second database again. The target transaction can be directly deleted from the cache and the next target transaction can be parsed.

[0146] If the first verification result is inconsistent, the information of the inconsistent data row is recorded and the first report information is sent. The first report information is used to include the inconsistency between the first execution result and the second execution result. After sending the information, the next target transaction is analyzed.

[0147] The embodiment of the present application provides a possible implementation manner, after determining a second verification result obtained by performing consistency verification on the first execution result and the third execution result, further comprising:

[0148] If the second verification result is consistent, the target transaction is deleted from the cache and the next target transaction is parsed;

[0149] If the second verification result is inconsistent, the third execution result corresponding to the first execution result is re-determined in the data table of the second database after a preset period of time, and the second verification result obtained after consistency verification of the first execution result and the third execution result is determined until the number of consistency verifications reaches a preset number.

[0150] When determining that the second verification result is consistent, the embodiment of the present application can also directly delete the target transaction in the cache and parse the next target transaction; if the second verification result is inconsistent, the third execution result corresponding to the first execution result is re-determined in the data table of the second database after a preset period of time, and the second verification result obtained after the consistency check of the first execution result and the third execution result is determined until the number of consistency checks reaches a preset number of times.

[0151] Since the first database and the second database are constantly changing, if the first execution result in the first database is constantly changing, then the second execution result in the second database is also constantly changing. If after the target transaction occurs, the first database changes the first execution result again, obtains a new first execution result, and synchronizes the new database transaction corresponding to the change to the second database, then the consistency check result of the first execution result and the third execution result is inconsistent, which is a "false positive" situation. At this time, the new first execution result and the second execution result should be checked for consistency. Therefore, multiple repeated checks are required to reduce the probability of "false positive" situations. Within a preset number of times, if the check results are consistent, the target transaction is deleted from the cache. If the check result is still inconsistent after exceeding the preset number of times, the target transaction is also deleted from the cache, and a second report is sent. The second report is used to characterize that after multiple checks, the second check result is still inconsistent, and manual judgment is required to determine whether it is really inconsistent.

[0152] like Figure 4 The figure is a flow chart of a data verification method provided by an embodiment of the present application, including:

[0153] Step S401, determining that a target transaction is migrated from a first database to a second database, wherein the target transaction is a transaction that has been executed in the first database;

[0154] Step S402, parsing the target transaction in the cache to obtain a transaction identifier and a first execution result of the target transaction, wherein the transaction identifier is a unique identifier for determining a database transaction, and the first execution result is an execution result obtained after the target transaction is executed in the first database;

[0155] Step S403, determining whether a target snapshot associated with the transaction identifier of the target transaction can be obtained in the second database, wherein the target snapshot records at least one execution result corresponding to at least one database transaction that has been migrated and the at least one database transaction includes the target transaction;

[0156] Step S404: if the judgment result of step S403 is yes, determining the second execution result corresponding to the first execution result from the target snapshot;

[0157] Step S405 , determining a first verification result obtained after consistency verification of the first execution result and the second execution result, where the first verification result is used to indicate whether the first execution result and the second execution result are consistent.

[0158] Step S406: If the first verification result is consistent, the target transaction is deleted from the cache and the next target transaction is parsed;

[0159] Step S407: If the first verification result is inconsistent, first report information is sent, where the first report information is used to report that the first execution result and the second execution result are inconsistent.

[0160] Step S404', if the judgment result of step S403 is no, determining a third execution result corresponding to the first execution result from the data table of the second database;

[0161] Step S405', determining a second verification result obtained after consistency verification of the first execution result and the third execution result, the second verification result is used to indicate whether the first execution result and the third execution result are consistent.

[0162] Step S406', if the second verification result is consistent, then the target transaction is deleted from the cache and the next target transaction is parsed;

[0163] Step S407', if the second verification result is inconsistent, then after a preset time, the third execution result corresponding to the first execution result is re-determined in the data table of the second database, and the second verification result obtained after consistency verification of the first execution result and the third execution result is determined until the number of consistency verifications reaches the preset number.

[0164] The specific implementation process of the above steps S401-S407' is consistent with the above-mentioned embodiment, and will not be repeated here in the embodiment of the present application.

[0165] The present application embodiment provides a data verification device 50, such as Figure 5 As shown, the data verification device 50 may include:

[0166] A migration determination module 510, configured to determine whether a target transaction is to be migrated from a first database to a second database, wherein the target transaction is a database transaction that has been executed in the first database;

[0167] A first execution result determination module 520 is used to determine a transaction identifier and a first execution result corresponding to a target transaction, wherein the transaction identifier is a unique identifier for determining a database transaction, and the first execution result is an execution result corresponding to the target transaction after it is executed in the first database;

[0168] The target snapshot determination module 530 is configured to obtain a target snapshot in the second database that is associated with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is executed, and the at least one database transaction includes the target transaction;

[0169] A second execution result determination module 540, configured to determine a second execution result corresponding to the first execution result from at least one execution result;

[0170] The verification module 550 is used to determine a first verification result obtained after consistency verification of the first execution result and the second execution result, where the first verification result is used to indicate whether the first execution result and the second execution result are consistent.

[0171] The embodiment of the present application determines that the target transaction is migrated from the first database to the second database, wherein the target transaction is a database transaction that has been executed in the first database; determines the transaction identifier and the first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is the execution result obtained after the target transaction is executed in the first database; obtains a target snapshot in the second database that has an association relationship with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is completed, and at least one database transaction includes the target transaction; determines the second execution result corresponding to the first execution result from the at least one execution result; determines the first verification result obtained after the consistency check of the first execution result and the second execution result, and the first verification result is used to characterize whether the first execution result and the second execution result are consistent. In the process of incremental data migration, the data events of each incremental data that have been migrated to the downstream database are verified, so that the incremental data can be accurately verified, and the migration and verification of the incremental data are carried out in real time and synchronously, effectively shortening the time difference between the migration and verification of the incremental data, so that the incremental data migration results can be verified in real time or quasi-real time.

[0172] A possible implementation method is provided in an embodiment of the present application, where the first execution result determination module is specifically used to store the target transaction in a cache, parse the target transaction in the cache, and obtain a transaction identifier and a first execution result of the target transaction.

[0173] A possible implementation method is provided in an embodiment of the present application, and the execution result includes the primary key of the corresponding data row;

[0174] The second execution result determination module is specifically configured to determine, from at least one execution result, a second execution result having the same primary key as the first execution result.

[0175] An embodiment of the present application provides a possible implementation method, wherein the target transaction includes at least one event; and the device further includes:

[0176] The merging module is used to merge the two events to obtain a new event if it is determined that the primary keys of the data rows of two events in the target transaction are the same.

[0177] A possible implementation method is provided in an embodiment of the present application, and the merging module includes:

[0178] A merge event determination unit, used to determine that two events are respectively a first event and a second event; the first event is prior to the second event;

[0179] An event type determination unit, used to determine event types of the first event and the second event, where the transaction types include insert type, update type and delete type;

[0180] The merging unit is used to redefine the second event as a new event obtained by merging the first event and the second event if the event type of the first event and the type of the second event meet the preset merging rule, and delete the first event.

[0181] A possible implementation method is provided in an embodiment of the present application. The target snapshot determination module includes:

[0182] A relationship establishment submodule, wherein the second database generates a snapshot corresponding to at least one execution result and generates a snapshot identifier corresponding to the snapshot; an association relationship is established between the target transaction identifier and the snapshot identifier, wherein at least one transaction identifier includes a transaction identifier of the target transaction;

[0183] The target snapshot determination submodule is specifically configured to obtain a target snapshot identifier associated with the transaction identifier of the target transaction, wherein the target snapshot identifier is a unique identifier for determining the target snapshot; and determine the target snapshot from the second database according to the target snapshot identifier.

[0184] A possible implementation method is provided in an embodiment of the present application, and the device further includes:

[0185] A third execution result determination module, configured to determine, if it is determined that the target snapshot does not exist in the second database, a third execution result having the same primary key as the first execution result from a data table in the second database;

[0186] The verification module is also used to determine a second verification result obtained after consistency verification of the first execution result and the third execution result, and the second verification result is used to indicate whether the first execution result and the third execution result are consistent.

[0187] A possible implementation method is provided in an embodiment of the present application, and the device further includes:

[0188] A deletion module, configured to delete the target transaction from the cache and parse the next target transaction if the first verification result is consistent;

[0189] The reporting module is used to send first report information if the first verification result is inconsistent, and the first report information is used to report that the first execution result and the second execution result are inconsistent.

[0190] A possible implementation method is provided in an embodiment of the present application, where the deletion module is further configured to delete the target transaction in the cache and parse the next target transaction if the second verification result is consistent;

[0191] The verification module is also used to re-determine the third execution result corresponding to the first execution result in the data table of the second database after a preset period of time if the second verification result is inconsistent, and determine the second verification result obtained after consistency verification of the first execution result and the third execution result, until the number of consistency verifications reaches a preset number of times.

[0192] like Figure 6 As shown, it exemplarily shows another data verification device 60 provided in an embodiment of the present application, and the data verification device 60 includes:

[0193] Migration determination module 6010 , merging module 6020 , first execution result determination module 6030 , target snapshot determination module 6040 , second execution result determination module 6050 , third execution result determination module 6060 , verification module 6070 , deletion module 6080 , and reporting module 6090 .

[0194] The migration determination module 6010 is used to determine whether the target transaction is migrated from the first database to the second database, wherein the target transaction is a transaction that has been executed in the first database; the migration determination module 6010 is used to determine whether the target transaction is migrated from the first database to the second database, wherein the target transaction is a transaction that has been executed in the first database; the merging module 6020 is used to merge the two events to obtain a new event if the primary keys of the data rows of two events in the target transaction are the same; the merging module 6020 includes a merging event determination unit 6021, an event type determination unit 6022 and a merging unit 6023; the merging event determination unit 6021 is used to determine whether the two events are the first event and the second event respectively; the first event The order of the event precedes the second event; an event type determination unit 6022 is used to determine the event types of the first event and the second event, and the transaction types include insert type, update type and delete type; a merging unit 6023 is used to redetermine the second event as a new event obtained by merging the first event and the second event if the event type of the first event and the type of the second event meet the preset merging rule, and delete the first event; a first execution result determination module 6030 is specifically used to store the target transaction in the cache, parse the target transaction in the cache, and obtain the transaction identifier and the first execution result of the target transaction; a target snapshot determination module 6040 is used to obtain a target snapshot identifier that is associated with the transaction identifier of the target transaction; according to The target snapshot identifier determines the target snapshot from the second database, and the target snapshot records at least one execution result corresponding to the at least one database transaction completed by the second database after the migration is completed; the at least one database transaction includes the target transaction; the target snapshot determination module 6040 includes: a relationship establishment submodule 6041 and a target snapshot determination submodule 6042, the relationship establishment submodule 6041 generates a snapshot corresponding to the at least one execution result for the second database and generates a snapshot identifier corresponding to the snapshot; establishes an association relationship between the transaction identifier and the snapshot identifier, wherein the at least one transaction identifier includes the transaction identifier of the target transaction; the target snapshot determination submodule 6042 is specifically used to obtain a transaction identifier associated with the target transaction. a target snapshot identifier of the system, wherein the target snapshot identifier is a unique identifier for determining the target snapshot; determining the target snapshot from the second database according to the target snapshot identifier; a second execution result determination module 6050, specifically used to determine a second execution result having the same primary key as the first execution result from at least one execution result; a third execution result determination module 6060, used to determine a third execution result having the same primary key as the first execution result from a data table of the second database if it is determined that the target snapshot does not exist in the second database; a verification module 6070, used to determine a first verification result obtained after consistency verification of the first execution result and the second execution result, the first verification result being used to indicate whether the first execution result and the second execution result are consistent;The verification module 6070 is also used to determine the second verification result obtained after the consistency verification of the first execution result and the third execution result, and the second verification result is used to indicate whether the first execution result and the third execution result are consistent; the deletion module 6080 is used to delete the target transaction in the cache and parse the next target transaction if the first verification result is consistent; the deletion module 6080 is also used to delete the target transaction in the cache and parse the next target transaction if the second verification result is consistent; the reporting module 6090 is used to send the first report information if the first verification result is inconsistent, and the first report information is used to report that the first execution result and the second execution result are inconsistent; the verification module 6070 is also used to re-determine the third execution result corresponding to the first execution result in the data table of the second database after a preset time period if the second verification result is inconsistent, and determine the second verification result obtained after the consistency verification of the first execution result and the third execution result, until the number of consistency verifications reaches the preset number. ;

[0195] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0196] In an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executes the above-mentioned computer program to implement the steps of the data verification method, and compared with the related art, the following can be achieved: in the embodiment of the present application, a target transaction is migrated from a first database to a second database by determining that the target transaction is migrated from a first database, wherein the target transaction is a database transaction that has been executed in the first database; a transaction identifier and a first execution result corresponding to the target transaction are determined, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is an execution result corresponding to the target transaction after the target transaction is executed in the first database; a target snapshot in the second database that is associated with the transaction identifier of the target transaction is obtained, wherein the target snapshot records at least one execution result corresponding to at least one database transaction that has been migrated to the second database, and the at least one database transaction includes the target transaction; a second execution result corresponding to the first execution result is determined from the at least one execution result; a first verification result obtained after consistency verification of the first execution result and the second execution result is determined, and the first verification result is used to characterize whether the first execution result and the second execution result are consistent. During the incremental data migration process, the data events of each incremental data that have been migrated to the downstream database are verified, so that the incremental data can be accurately verified. At the same time, the migration and verification of the incremental data are carried out in real time and synchronously, effectively shortening the time difference between the migration and verification of the incremental data, so that the incremental data migration results can be verified in real time or quasi-real time.

[0197] In an alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0198] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0199] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0200] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0201] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0202] Among them, the electronic equipment package may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0203] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented. Compared with the prior art, the embodiment of the present application can be implemented as follows: the embodiment of the present application determines that the target transaction is migrated from the first database to the second database, wherein the target transaction is a database transaction that has been executed in the first database; determines the transaction identifier and the first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is the execution result obtained after the target transaction is executed in the first database; obtains a target snapshot in the second database that has an association relationship with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result obtained after at least one database transaction completed by the second database execution migration, and at least one database transaction includes the target transaction; determines the second execution result corresponding to the first execution result from at least one execution result; determines the first verification result obtained after the consistency verification of the first execution result and the second execution result, and the first verification result is used to characterize whether the first execution result and the second execution result are consistent. During the incremental data migration process, the data events of each incremental data that have been migrated to the downstream database are verified, so that the incremental data can be accurately verified. At the same time, the migration and verification of the incremental data are carried out in real time and synchronously, effectively shortening the time difference between the migration and verification of the incremental data, so that the incremental data migration results can be verified in real time or quasi-real time.

[0204] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0205] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor. Compared with the prior art, it can be realized that: the embodiment of the present application determines that the target transaction is migrated from the first database to the second database, wherein the target transaction is a database transaction that has been executed in the first database; determines the transaction identifier and the first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is the execution result obtained after the target transaction is executed in the first database; obtains a target snapshot in the second database that has an association relationship with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result obtained after at least one database transaction completed by the second database execution migration, and at least one database transaction includes the target transaction; determines the second execution result corresponding to the first execution result from at least one execution result; determines the first verification result obtained after the consistency verification of the first execution result and the second execution result, and the first verification result is used to characterize whether the first execution result and the second execution result are consistent. During the incremental data migration process, the data events of each incremental data that have been migrated to the downstream database are verified, so that the incremental data can be accurately verified. At the same time, the migration and verification of the incremental data are carried out in real time and synchronously, effectively shortening the time difference between the migration and verification of the incremental data.

[0206] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0207] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0208] The above are only optional implementation methods for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A data verification method, characterized in that: include: Determining a target transaction to be migrated from the first database to the second database from a binary log of the first database, wherein the target transaction is a database transaction that has been executed in the first database; the database transaction includes an original data row before being changed, an operation instruction to change the original data row, and a first execution result obtained after the change; Determine a transaction identifier and a first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is an execution result corresponding to the target transaction after it is executed in the first database; Acquire a target snapshot in the second database that is associated with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is executed, and the at least one database transaction includes the target transaction; Determine a second execution result corresponding to the first execution result from the at least one execution result; the primary key of the second execution result is the same as the primary key of the first execution result; Determine a first verification result obtained after performing a consistency check on the first execution result and the second execution result, where the first verification result is used to indicate whether the first execution result and the second execution result are consistent; The step of obtaining a target snapshot in the second database that is associated with the transaction identifier of the target transaction includes: The second database generates a snapshot corresponding to the at least one execution result and generates a snapshot identifier corresponding to the snapshot; Establishing an association relationship between the transaction identifier and the snapshot identifier, wherein at least one of the transaction identifiers includes a transaction identifier of the target transaction; A target snapshot identifier that has the association relationship with the transaction identifier of the target transaction is obtained, and a target snapshot is determined from the second database according to the target snapshot identifier.

2. The method according to claim 1, characterized in that The determining of the transaction identifier and the first execution result of the target transaction includes: The target transaction is stored in a cache, and the target transaction in the cache is parsed to obtain a transaction identifier and a first execution result of the target transaction.

3. The method according to any one of claims 1 or 2, characterized in that: The execution result includes the primary key of the corresponding data row; The determining, from the at least one execution result, a second execution result corresponding to the first execution result includes: A second execution result having the same primary key as the first execution result is determined from the at least one execution result.

4. The method according to claim 3, characterized in that The target transaction includes at least one event; before determining the transaction identifier and the first execution result of the target transaction, the method further includes: If it is determined that the primary keys of the data rows of the two events in the target transaction are the same, the two events are merged to obtain a new event.

5. The method according to claim 4, characterized in that The two events are merged to obtain a new event, including: Determining that the two events are respectively a first event and a second event; the first event is prior to the second event; Determine event types of the first event and the second event, where the event types include an insert type, an update type, and a delete type; If the event type of the first event and the event type of the second event meet the preset merging rule, the second event is re-determined as a new event obtained by merging the first event and the second event, and the first event is deleted.

6. The method according to claim 3, characterized in that After determining the transaction identifier and the first execution result of the target transaction, the method further includes: If it is determined that the target snapshot does not exist in the second database, determining a third execution result corresponding to the first execution result from a data table of the second database; Determine a second verification result obtained after consistency verification of the first execution result and the third execution result, where the second verification result is used to indicate whether the first execution result and the third execution result are consistent.

7. The method according to claim 2, characterized in that After determining a first verification result obtained by performing consistency verification on the first execution result and the second execution result, the method further includes: If the first verification result is consistent, deleting the target transaction in the cache and parsing the next target transaction; If the first verification result is inconsistent, first report information is sent, where the first report information is used to report that the first execution result and the second execution result are inconsistent.

8. The method according to claim 6, characterized in that After determining a second verification result obtained by performing consistency verification on the first execution result and the third execution result, the method further includes: If the second verification result is consistent, deleting the target transaction in the cache and parsing the next target transaction; If the second verification result is inconsistent, the third execution result corresponding to the first execution result is re-determined in the data table of the second database after a preset period of time, and the second verification result obtained after consistency verification of the first execution result and the third execution result is determined until the number of consistency verifications reaches a preset number.

9. A data verification device, characterized in that: include: A migration determination module, used to determine a target transaction to be migrated from the first database to the second database from the binary log of the first database, wherein the target transaction is a database transaction that has been executed in the first database; the database transaction includes an original data row before being changed, an operation instruction to change the original data row, and a first execution result obtained after the change; A first execution result determination module, configured to determine a transaction identifier and a first execution result corresponding to the target transaction, wherein the transaction identifier is a unique identifier for determining the database transaction, and the first execution result is an execution result corresponding to the target transaction after it is executed in the first database; a target snapshot determination module, configured to obtain a target snapshot in the second database that is associated with the transaction identifier of the target transaction, wherein the target snapshot records at least one execution result corresponding to at least one database transaction completed by the second database after the migration is executed, and the at least one database transaction includes the target transaction; A second execution result determination module, used to determine a second execution result corresponding to the first execution result from the at least one execution result; the primary key of the second execution result is the same as the primary key of the first execution result; a verification module, used to determine a first verification result obtained after performing a consistency check on the first execution result and the second execution result, wherein the first verification result is used to indicate whether the first execution result and the second execution result are consistent; The target snapshot determination module is specifically used for: The second database generates a snapshot corresponding to the at least one execution result and generates a snapshot identifier corresponding to the snapshot; Establishing an association relationship between the transaction identifier and the snapshot identifier, wherein at least one of the transaction identifiers includes a transaction identifier of the target transaction; A target snapshot identifier that has the association relationship with the transaction identifier of the target transaction is obtained, and a target snapshot is determined from the second database according to the target snapshot identifier.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images