Data verification method, device and system

Abstract

The present application provides a data verification method, device and system. The method includes: generating a first Merkle tree according to a first data table in a first database, and generating a second Merkle tree according to a second data table in a second database, wherein the structure and generation method of the first Merkle tree are exactly the same as those of the second Merkle tree, so whether the first data table and the second data table are consistent can be determined by directly comparing the hash values ​​of the root nodes of the two Merkle trees. Therefore, the data verification method provided by the present application can meet the needs of accurate and fast verification of massive data.

Description

Technical Field

[0001] The present application relates to the field of storage, and more specifically, to a data verification method, device and system. Background Art

[0002] In the process of data synchronization or data migration of a database system (for example, a heterogeneous database or a homogeneous database), it is necessary to verify the data consistency of the synchronization table of the source database and the target database to verify the correctness of data synchronization or data migration. In practical applications, there is usually a problem of inconsistency between the data in the source database and the data in the target database. On the one hand, due to data loss and data errors caused by factors such as hardware failure, software defects, human errors, and environmental interference during data transmission and data storage, the data in the source database and the data in the target database are inconsistent. On the other hand, due to performance issues of the database system, there may be a certain delay when the changes in the source data table are synchronized to the target database, resulting in inconsistent verification of the data table in the source database and the data table in the target database at a certain moment.

[0003] Traditional data verification methods cannot meet the needs of accurate and fast verification of massive data. Summary of the invention

[0004] The present application provides a data verification method, device and system, which can be applied in data synchronization or data migration scenarios. The method can better meet the needs of accurate and fast verification of massive data.

[0005] In a first aspect, a data verification method is provided, the method comprising:

[0006] Processing a first data table in a first database to generate a first Merkle tree, wherein each row of the first data table includes a row identifier and row data, and the first Merkle tree includes N first leaf nodes, wherein the N first leaf nodes correspond one-to-one to N first hash buckets, and a hash value of each of the first leaf nodes is determined according to a corresponding first hash bucket, and the N first hash buckets are obtained by performing hash partitioning on the first data table according to the row identifier, and any two first hash buckets are different, and N is a positive integer greater than or equal to 2;

[0007] Processing a second data table in a second database to generate a second Merkle tree, the second data table being obtained by synchronizing or migrating the first data table to the second database, the second Merkle tree comprising N second leaf nodes, the N second leaf nodes corresponding one-to-one to N second hash buckets, the hash value of each second leaf node being determined according to the corresponding second hash bucket, the N second hash buckets being obtained by hash partitioning the second data table according to row identifiers, the hash rule for obtaining the N second hash buckets by hash partitioning the second data table according to row identifiers being the same as the hash rule for obtaining the N first hash buckets by hash partitioning the first data table according to row identifiers, and any two second hash buckets being different;

[0008] The first Merkle tree is compared with the second Merkle tree to determine whether the first data table is consistent with the second data table.

[0009] In the above technical scheme, the hash value of each first leaf node of the first Merkle tree corresponding to the first data table is determined according to the first hash bucket corresponding thereto, and the hash value of each second leaf node of the second Merkle tree corresponding to the second data table is determined according to the second hash bucket corresponding thereto. Since the hash rule of obtaining N second hash buckets by hash partitioning the second data table according to the row identifier is the same as the hash rule of obtaining N first hash buckets by hash partitioning the first data table according to the row identifier, it can be ensured that the data corresponding to the same row identifier in the second data table and the first data table are respectively mapped to the second hash bucket and the first hash bucket with the same sequence number, so that the generated first Merkle tree has the same structure as the second Merkle tree. Therefore, the consistency of the first data table and the second data table can be determined directly by comparing the hash values ​​of the root nodes of the above two Merkle trees. Therefore, the data verification method provided by the present application can meet the needs of accurate and fast verification of massive data.

[0010] In combination with the first aspect, in some implementations of the first aspect, the first data table includes M rows, where M is a positive integer greater than or equal to 1, and the first data table in the first database is processed to generate a first Merkle tree, including:

[0011] Performing hash processing on the first data table to obtain M first hash groups, the M first hash groups corresponding one-to-one to the M rows, each of the first hash groups including a row identifier in the first data table and a hash value of row data corresponding to the row identifier, and each of the first hash groups including a different row identifier;

[0012] Mapping the M first hash groups to the N first hash buckets;

[0013] Determine hash values ​​of the N first leaf nodes according to the N first hash buckets;

[0014] The first Merkle tree is generated according to the hash values ​​of the N first leaf nodes.

[0015] It should be understood that the method for processing the second data table in the second database to generate the second Merkle tree is the same as the above method. Specifically, when the second data table includes K rows, K is a positive integer less than or equal to N, the method for processing the second data table in the second database to generate the second Merkle tree includes:

[0016] Performing hash processing on the second data table to obtain K second hash groups, the K second hash groups corresponding one-to-one to the K rows, each of the second hash groups including a row identifier in the second data table and a hash value of row data corresponding to the row identifier, and each of the second hash groups including a different row identifier;

[0017] Mapping the K second hash groups to the N second hash buckets;

[0018] Determine hash values ​​of the N second leaf nodes according to the N second hash buckets;

[0019] The second Merkle tree is generated according to the hash values ​​of the N second leaf nodes.

[0020] Among them, when K is equal to M, it can be understood that the number of rows included in the second data table is the same as the number of rows included in the first data table, that is, if there is no data missing during the process of synchronizing or migrating the first data table to the second database. When K is less than M, it can be understood that the number of rows included in the second data table is less than the number of rows included in the first data table, that is, if there is data missing during the process of synchronizing or migrating the first data table to the second database.

[0021] It should also be understood that the mapping rule for mapping the K second hash groups to the N second hash buckets is the same as the mapping rule for mapping the M first hash groups to the N first hash buckets.

[0022] In the above technical solution, a data partitioning algorithm is used to map the row data in the data table to hash buckets, and the hash buckets correspond one-to-one to the Merkle tree child nodes. Since the mapping rule of mapping K second hash groups to N second hash buckets is the same as the mapping rule of mapping M first hash groups to N first hash buckets, the generated first Merkle tree and the second Merkle tree have the same structure.

[0023] In combination with the first aspect, in some implementations of the first aspect, the hash value of each first leaf node is a hash value obtained by performing an XOR operation on the hash value included in the first hash group included in the corresponding first hash bucket.

[0024] It should be understood that the hash value of each second leaf node is a hash value obtained by performing an XOR operation on the hash values ​​included in the second hash group included in the corresponding second hash bucket.

[0025] In the above technical solution, the hash value of the leaf node is obtained by performing an XOR operation on the data in the hash bucket. Therefore, when performing consistency check on the first data table and the second data table, sorting of the row data can be avoided, thereby further improving the efficiency of data verification.

[0026] In combination with the first aspect, in some implementations of the first aspect, comparing the first Merkle tree with the second Merkle tree to determine whether the first data table is consistent with the second data table includes:

[0027] Determine a hash value of the first Merkle tree root node and a hash value of the second Merkle tree root node;

[0028] If the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, determining that the first data table is consistent with the second data table;

[0029] If the hash value of the first Merkle tree root node is different from the hash value of the second Merkle tree root node, it is determined that the first data table is inconsistent with the second data table.

[0030] In the above technical solution, since the structures of the first Merkle tree and the second Merkle tree (for example, the tree height, the row identifier in the data table corresponding to the leaf node) are exactly the same, when performing consistency verification, it is possible to accurately and quickly determine whether the first data table is consistent with the second data table by judging whether the hash value of the root node of the first Merkle tree is the same as the hash value of the root node of the second Merkle tree. Specifically, when the hash value of the root node of the first Merkle tree is the same as the hash value of the root node of the second Merkle tree, it can be determined that the first data table is consistent with the second data table. When the hash value of the root node of the first Merkle tree is different from the hash value of the root node of the second Merkle tree, it can be determined that the first data table is inconsistent with the second data table.

[0031] In combination with the first aspect, in some implementations of the first aspect, after determining that the first data table is inconsistent with the second data table, the method further includes:

[0032] Determine that the hash value of the i-th first leaf node is different from the hash value of the i-th second leaf node, the hash value of the i-th first leaf node is determined according to the i-th first hash bucket, and the hash value of the i-th second leaf node is determined according to the i-th second hash bucket, where i is a positive integer and 1≤i≤N;

[0033] Compare the i-th first hash bucket with the i-th second hash bucket to determine the row identifiers of the first data table and the second data table that are inconsistent;

[0034] According to the inconsistent row identifier, the row data corresponding to the inconsistent row identifier is queried from the first database and the second database respectively.

[0035] In the above technical solution, after determining that the first data table and the second data table are inconsistent, the inconsistent row identifiers in the first database and the second database can be specifically determined based on the hash value of the leaf node and the hash bucket corresponding to the leaf node. In a further step, the row data corresponding to the inconsistent row identifiers can be queried from the first database and the second database based on the determined inconsistent row identifiers. Since the data set size of the leaf node is controllable, the time required for the above determination of inconsistent row identifiers is also controllable.

[0036] In combination with the first aspect, in certain implementations of the first aspect, the first data table includes all the data in at least one data table in the first database.

[0037] In combination with the first aspect, in some implementations of the first aspect, the first data table includes incremental data in at least one data table in the first database.

[0038] In the above technical solution, the full data and the incremental data can be separated and data verification can be performed separately as two stages, which can save computing overhead. Specifically, since the full data includes a large amount of data, a Merkle tree with a higher number of layers can be constructed when verifying the full data. Since the incremental data includes a smaller amount of data, a Merkle tree with a lower number of layers can be constructed when verifying the incremental data.

[0039] In combination with the first aspect, in some implementations of the first aspect, the height of the first Merkle tree is associated with the first data table.

[0040] In the above technical solution, the height of the first Merkle tree can be adaptively adjusted according to the size of the first data table to be verified.

[0041] In combination with the first aspect, in some implementations of the first aspect, the first database and the second database are heterogeneous databases or homogeneous databases.

[0042] In the above technical solution, the data verification method provided by the present application can be applied to data consistency verification of homogeneous databases and data consistency verification of heterogeneous databases.

[0043] In combination with the first aspect, in some implementations of the first aspect, the first database is a relational database or a non-relational database, and the second database is a relational database or a non-relational database.

[0044] In a second aspect, a data verification device is provided, which executes the method in the above-mentioned first aspect and any possible implementation manner of the first aspect.

[0045] It should be understood that the data verification device provided by the present application is independently decoupled from the database system, so the data verification device will not cause an intrusive impact on the database system, for example, affecting the function and performance of the database system or occupying database system resources.

[0046] In a third aspect, a data verification device is provided, which includes a memory and a processor, the memory is used to store instructions, and the processor is used to read the instructions stored in the memory, so that the data verification device executes the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0047] In a fourth aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that any aspect of the first aspect and the method in any possible implementation of the first aspect are implemented.

[0048] In the specific implementation process, the processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit can be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit can be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation methods of the processor and various circuits.

[0049] In a fifth aspect, a processing device is provided, comprising a processor and a memory. The processor is used to read instructions stored in the memory, and can receive signals through a receiver and transmit signals through a transmitter to execute the method in the first aspect and any possible implementation of the first aspect.

[0050] Optionally, the number of the processors is one or more, and the number of the memories is one or more.

[0051] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0052] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. The embodiments of the present application do not limit the type of memory and the setting method of the memory and the processor.

[0053] It should be understood that the relevant data interaction process, such as sending indication information, can be a process of outputting indication information from a processor, and receiving capability information can be a process of receiving input capability information from a processor. Specifically, the processed output data can be output to a transmitter, and the input data received by the processor can come from a receiver. Among them, the transmitter and the receiver can be collectively referred to as a transceiver.

[0054] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect and any possible implementation of the first aspect.

[0055] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method in the first aspect and any possible implementation of the first aspect.

[0056] In an eighth aspect, a system is provided, which includes the data verification device described in the second aspect.

[0057] In the ninth aspect, a chip is provided, comprising at least one processor and an interface; the at least one processor is used to call and run a computer program so that the chip executes the method in the above-mentioned first aspect and any possible implementation of the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of a system 100 suitable for the data verification method provided in the present application.

[0059] Figure 2 It is a schematic diagram of the data verification device 130 provided in the present application.

[0060] Figure 3 It is a schematic flow chart of the data verification method 100 provided in this application.

[0061] Figure 4 It is a schematic diagram of a Merkle tree determined according to the method provided in this application.

[0062] Figure 5 This is a schematic diagram of extracting data from the first data table provided by the present application.

[0063] Figure 6 This is a schematic diagram of hash partitioning of data extracted from the first data table provided by the present application.

[0064] Figure 7 It is a schematic diagram of a Merkle tree determined according to the method provided in this application.

[0065] Figure 8 It is a schematic flow chart of the data verification method 200 provided in this application.

[0066] Fig. 9 It is a schematic diagram of a Merkle tree determined according to the method provided in this application.

[0067] Fig.10 It is a schematic structural diagram of a data verification device 1000 provided in this application.

[0068] Fig.11 It is a schematic structural diagram of a data verification device 1000 provided in this application.

[0069] Fig.12 It is a structural diagram of a system 1200 provided in this application. DETAILED DESCRIPTION

[0070] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0071] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0072] In this application, the terms "first", "second", "third", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "third", and no limitation is imposed on the quantity and execution order.

[0073] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0074] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0075] In the embodiments of the present application, "corresponding (corresponding, relevant)" and "corresponding (corresponding)" can sometimes be used interchangeably. It should be pointed out that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0076] In the embodiments of the present application, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0077] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0078] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0079] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0080] Below, the relevant technologies of this application are introduced:

[0081] For ease of understanding, before describing the data verification method provided in the present application, a brief introduction to the relevant terms involved in the present application is first given.

[0082] 1. Data verification

[0083] Data verification is a verification operation performed to ensure the integrity of data. Usually, a verification value is calculated for the original data using a specified algorithm. The receiver calculates the verification value once using the same algorithm. If the verification values ​​calculated twice are the same, it means that the data is consistent.

[0084] 2. Data replication

[0085] Data replication is the technology of copying data from one location to another. The technology involves sharing information to ensure consistency between redundant resources (such as software or hardware components) to improve reliability, fault tolerance, or accessibility.

[0086] 3. Merkel tree

[0087] The Merkle tree can also be called a hash tree. The Merkle tree is a binary tree consisting of a root node, a set of intermediate nodes, and a set of leaf nodes. The bottom leaf node contains the stored data or its hash value, each intermediate node is the hash value of the content of its two child nodes, and the root node is also composed of the hash value of the content of its two child nodes. When the data extraction module generates new data, the respective Merkle tree will be dynamically updated.

[0088] 4. Hash

[0089] Hash is a function that maps data of any length to data of fixed length. A slight change in the input data will cause the hash operation result to be completely different, and it is generally considered impossible to infer the characteristics of the original input data based on the hash value.

[0090] 5. Heterogeneous database (HDB)

[0091] A heterogeneous database is a collection of multiple related database systems that can achieve data sharing and transparent access. Each database system already exists before joining the heterogeneous database system and has its own database management system (DBMS). Each component of a heterogeneous database has its own autonomy. While achieving data sharing, each database system still retains its own application characteristics, integrity control, and security control.

[0092] 6. Homogeneous database

[0093] A homogeneous database means that all sites use the same DBMS software, and each site understands each other and cooperates to handle user needs.

[0094] 7. Relational database (RD)

[0095] RD refers to a database that uses a relational model to organize data. It stores data in the form of rows and columns to facilitate user understanding. A series of rows and columns in a relational database is called a table, and a group of tables constitutes a database. Users retrieve data from the database through queries, and a query is an execution code used to limit certain areas in the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them.

[0096] 8. Non-relational database (not only sql, NoSQL)

[0097] NoSQL refers to a database that uses a non-relational model to organize data. Non-relational databases can include the following types: key-value storage databases (for example, Oracle BDB), column storage databases (for example, HBase), document databases (for example, CouchDB or MongoDb), and graph databases.

[0098] Usually, in the process of synchronizing data between the source database and the target database, it is necessary to rely on physical media or network to transmit the data to be synchronized, which causes a certain delay between the source database and the target database. In addition, there are many uncertain factors affecting the data to be synchronized during the transmission process, such as hardware failure, software defects, human errors, environmental interference, etc., which may affect the reliability of the data to be synchronized (for example, data loss or data errors, etc.). The above reasons may lead to inconsistency between the source database and the target database. Therefore, after the data synchronization is completed, it is necessary to perform consistency verification on the synchronized data stored in the target database to ensure the reliability of the data stored in the target database. At present, an offline data verification method is usually used to obtain the data generated by the application software from the production database of each independent data source (i.e., the source database) to a unified offline database (i.e., the target database), so as to use offline data to verify whether the data of each independent data source is consistent. However, since obtaining data from the production database requires occupying database resources, if the impact on database performance is to be reduced, it is necessary to reduce the frequency of obtaining data, or even only perform it during the time when the business volume is small, which will affect the timeliness of offline data verification. In addition, the above data verification process usually requires data sorting, which often takes up a lot of system resources. Therefore, the above offline data verification method is usually unable to meet business needs when verifying massive data (for example, TB-level data).

[0099] The present application provides a data verification method, device and system that can better meet the needs of accurate and rapid verification of massive data.

[0100] To facilitate understanding, first combine Figure 1 and Figure 2 , a system and a data verification device suitable for the data verification method provided by this application are introduced in detail.

[0101] Figure 1 Schematic diagram of a system 100 suitable for the data verification method provided in the present application.

[0102] like Figure 1 As shown, the system 100 can be used for but not limited to the following scenarios: database data migration scenarios or data synchronization scenarios. The system 100 may include at least one source database 110, at least one target database 120, and at least one data verification device 130. The data verification device 130 is a system on a third-party hardware device independent of the source database 110 and the target database 120. The source database 110 is a database before data migration or replication, and the target database 120 is a database after data migration or replication.

[0103] In the present application, the type of the source database 110 and the type of the target database 120 are not specifically limited.

[0104] In one example, the source database 110 or the target database 120 may be a relational database. For example, the source database 110 or the target database 120 may be any of the following relational databases: Oracle, DB2, Microsoft SQL Server, Microsoft Access, MySQL. It should be understood that the type of relational database here is only for illustration and does not constitute any limitation on the system 100. For example, the relational database may also be a relational database of other types than those listed above.

[0105] In another example, the source database 110 or the target database 120 may be a non-relational database. For example, the source database 110 or the target database 120 may be any one of the following non-relational databases: NoSQL, Cloudant, and MongoDB. It should be understood that the type of non-relational database here is only for illustration and does not constitute any limitation on the system 100. For example, the non-relational database may also be other types of non-relational databases other than those listed above.

[0106] In another example, the source database 110 may be a non-relational database, and the target database 120 may be a relational database. For example, the source database 110 may be a NoSQL database, and the target database 120 may be an Oracle database.

[0107] In the present application, the source database 110 and the target database 120 may be isomorphic databases. The source database 110 and the target database 120 may also be heterogeneous databases, which is not limited.

[0108] In the present application, there is no specific limitation on the deployment of the source database 110, the target database 120 and the data verification device 130 in the device, but it is necessary to ensure that the data verification device 130 is a system independent of the source database 110 and the target database 120.

[0109] In an example, the source-side database 110 may be a physical module or a virtual module deployed on physical device #1, the target-side database 120 may be a physical module or a virtual module deployed on physical device #2, the data verification device 130 may be a physical module or a virtual module deployed on physical device #3, and physical device #1, physical device #2 and physical device #3 are different devices.

[0110] In another example, the source database 110 and the target database 120 may be different physical modules or virtual modules deployed on physical device #1, the data verification device 130 may be a physical module or virtual module deployed on physical device #2, and physical device #1 and physical device #3 are different devices.

[0111] See also Figure 1 , the source database 110 and the target database 120 can interact with each other (for example, data migration or data synchronization, etc.), and the source database 110 and the target database 120 can also interact with the data verification device 130 respectively. After synchronizing or migrating the data in the source database 110 to the target database 120, the data verification device 130 can extract the source data to be verified from the source database 110 and the target data to be verified from the target database 120, and perform consistency verification on the two parts of the extracted data to verify whether the data after data migration or data synchronization is consistent in the source database 110 and the target database 120. When the data verification device 130 determines that the data to be verified extracted from the source database is inconsistent with the data to be verified extracted from the target database, it can further determine which specific data is inconsistent. When the data verification device 130 has a storage function, the result of the consistency verification can also be stored in the data verification device 130.

[0112] It should be understood that Figure 1 This is only for illustration and does not constitute any limitation on the system to which this application is applicable. For example, the system 100 may also include a larger number of source databases 110 and / or target databases 120 and / or data verification devices 130. For example, the data verification device 130 may also include other modules, such as a verification execution module, a source-side data management module for verification, a target-side data management module for verification, etc.

[0113] Next, combine Figure 2 , introduce the above-mentioned Figure 1 A structural diagram of the data verification device 130 in FIG.

[0114] Figure 2 It is a schematic diagram of the data verification device 130 provided in the present application.

[0115] like Figure 2As shown, the device 130 may include: a source-side data extraction module 131, a source-side processing module 132, a target-side data processing module 133, a target-side data extraction module 134, a comparison module 135, and a storage module 136. The above modules may be connected via an internal connection path. For example, the source-side processing module 132 may interact with the comparison module 135, the source-side data extraction module 131, and the target-side processing module 133.

[0116] The source data extraction module 131 is used to obtain data from a source database (for example, the source database 110). Figure 1 The data is obtained from the source database 110.

[0117] The source-side processing module 131 is used to obtain data from the source-side data extraction module 131 and perform hash processing and data partition processing on the obtained data.

[0118] The target-side processing module 133 is used to obtain data from the target-side data extraction module 134, and perform hash processing and data partition processing on the obtained data.

[0119] The target-side data extraction module 134 is used to obtain data from a target-side database (for example, the target-side database 120). For example, the target-side data extraction module 134 may obtain data from Figure 1 The data is obtained from the target database 120.

[0120] The comparison module 135 is used to obtain the Merkle tree corresponding to the data from the source-end processing module 132 and the target-end processing module 133, and to perform data consistency verification based on the obtained Merkle tree. The comparison module 135 is the core module of the above-mentioned data verification device 130. Specifically, the comparison module 135 may also include a data comparison submodule and a data reverse query submodule. Among them, the data comparison submodule can quickly compare and find inconsistent row data identification data sets through the Merkle tree, and the data reverse query submodule can reverse query detailed data from the database according to the inconsistent row data identification, and finally find the data value corresponding to the inconsistent row identification.

[0121] The storage module 136 is used to store data and instructions.

[0122] It should be understood that Figure 2This is only for illustration and does not constitute any limitation on the data verification device 130 provided in the present application. For example, the source-side processing module 132 and the target-side processing module 133 in the data verification device 130 may also be included in the same processing module. For example, the source-side data extraction module 131 and the target-side data extraction module 134 in the data verification device 130 may also be included in the same processing module. For example, when the comparison module 135 in the data verification device 130 has the function of the storage module 136, the data verification device 130 may also not include the storage module 136.

[0123] Next, combine Figures 3 to 8 The data verification method provided in this application is introduced in detail.

[0124] Figure 3 It is a schematic flow chart of the data verification method 100 provided in this application.

[0125] like Figure 3 As shown, the method 100 may include steps 110 to 130. Steps 110 to 130 are described in detail below. The execution body of steps 110 to 130 may be Figure 2 The data verification device 130 is shown.

[0126] Step 110: Process the first data table in the first database to generate a first Merkle tree.

[0127] The first database can be understood as a production database, that is, a source database. For example, in one example, the first database can be Figure 1 The source database 110 is shown.

[0128] In the present application, there is no limitation on the data source and data size included in the first data table.

[0129] In one example, the first data table may include all data in at least one data table in the first database.

[0130] Optionally, the first data table may also include two or even more data tables in the first database. In this case, the first data table may be understood as a data set consisting of two or more data tables in the first database.

[0131] For example, when the first database includes data table #1, data table #2, and data table #3, the first data table may include all the data in data table #1. The first data table may also include all the data in data table #1 and data table #3. The first data table may also include all the data in data table #1, data table #2, and data table #3.

[0132] In another example, the first data table may include incremental data in at least one data table in the first database. That is, the data verification method provided by the present application may also perform consistency verification only on the changed data in the data table.

[0133] For example, at time #1, data table #1 and data table #2 are consistent, where data table #2 is obtained by copying data table #1. After time #1, part of the data in data table #1 changes (for example, data is updated, data is increased, or data is decreased). In this case, the data in data table #1 that has changed can be considered to be the data included in the first data table.

[0134] In the present application, each row of the first data table may include a row identifier and row data, the first Merkle tree includes N first leaf nodes, the N first leaf nodes correspond one-to-one to N first hash buckets, the hash value of each first leaf node is determined according to the corresponding first hash bucket, the N first hash buckets are obtained by hash partitioning the first data table according to the row identifier, any two first hash buckets are not the same, and N is a positive integer greater than or equal to 2.

[0135] In the present application, the type of row identifier included in the first data table is not specifically limited.

[0136] In one example, the row identifier may be a numerical row identifier, such as “5”.

[0137] In another example, the row identifier may also be a string type row identifier, such as "Zhang San" or "Li Si".

[0138] When the above row identifier is a string type row identifier, before building the Merkle tree, it is necessary to process the string type row identifier to obtain a hash value corresponding to the string type row identifier.

[0139] In the present application, N first hash buckets are obtained by hash partitioning the first data table according to row identifiers. It can be understood that no matter whether the row identifier of the first data table is a numeric row identifier or a string row identifier, when mapping the data in the first data table to the hash bucket, the row identifier included in the first data table can be first hashed to obtain a hash value corresponding to the row identifier, and then a modulo operation is performed on the hash value corresponding to the row identifier, and then the hash bucket corresponding to the row where the row identifier is located is determined according to the result of the modulo operation.

[0140] The above-mentioned N first leaf nodes correspond one-to-one with the N first hash buckets, which can be understood as that the i-th first leaf node among the N first leaf nodes (i.e., the first leaf node with sequence number i) corresponds to the i-th first hash bucket among the N first hash buckets (i.e., the first hash bucket with sequence number i). In other words, the first leaf node with sequence number i corresponds to the first hash bucket with sequence number i. Among them, the sequence number of each first leaf node among the N first leaf nodes is different, and the sequence number of each first hash bucket among the N first hash buckets is different, and i is a positive integer greater than or equal to 1 and less than or equal to N.

[0141] The above-mentioned N first hash buckets are obtained by performing hash partitioning on the first data table according to row identifiers. In the present application, no specific limitation is made on the correspondence between the first hash buckets and the first data table.

[0142] In one example, each first hash bucket is determined according to a row in the first data table. At this time, each first hash bucket corresponds to a row in the first data table. In this case, the number of row identifiers in the first data table included in each first hash bucket is the same.

[0143] In another example, at least one of the N first hash buckets is determined based on two or more rows in the first data table. In this case, at least one first hash bucket corresponds to two or more rows in the first data table. In this case, the number of row identifiers in the first data table included in each first hash bucket may be different.

[0144] Optionally, at least one of the N first hash buckets may be empty. That is, the N-1 first hash buckets are determined based on all row data included in the first data table, and the remaining first hash bucket does not include any data in the first data table.

[0145] The fact that any two first hash buckets are different may be understood as that the serial numbers corresponding to the any two first hash buckets are different, and the row identifiers in the first data table included in the any two first hash buckets are different.

[0146] For example, the sequence number of hash bucket #1 is 1, and hash bucket #1 includes two row identifiers in the first data table, which are "5" and "6", respectively, and the sequence number of hash bucket #2 is 2, and hash bucket #2 includes one row identifier in the first data table, which is "1". In this case, hash bucket #1 can be considered different from hash bucket #2.

[0147] In one example, the first data table may include M rows, where M is a positive integer greater than or equal to N. In this case, the above processing of the first data table in the first database to generate the first Merkle tree may include the following steps:

[0148] Perform hash processing on the M rows to obtain M first hash groups, where the M first hash groups correspond to the M rows one by one, each first hash group includes a row identifier in the M rows and a hash value of row data corresponding to the row identifier, and each first hash group includes a different row identifier;

[0149] Mapping the M first hash groups to N first hash buckets;

[0150] Determine hash values ​​of N first leaf nodes according to the N first hash buckets;

[0151] Generate the first Merkle tree according to the hash values ​​of the N first leaf nodes.

[0152] In the present application, there is no specific limitation on the number of first hash groups included in each of the N first hash buckets.

[0153] For example, the number of first hash groups included in each of the N first hash buckets may be the same. For example, the number of first hash groups included in each of the N first hash buckets may also be different. For example, the number of first hash groups included in some of the N first hash buckets is the same, and the number of first hash groups included in the remaining first hash buckets is different.

[0154] In the present application, the hash value of each first leaf node may be a hash value obtained by performing an XOR operation on the hash value included in the first hash group included in the corresponding first hash bucket. It should be understood that when there is a first hash bucket corresponding to the first leaf node that does not include any of the M first hash groups, the hash value of the first leaf node may be empty.

[0155] The above-mentioned determining the hash values ​​of the N first leaf nodes according to the N first hash buckets may include:

[0156] When the first hash bucket corresponding to at least one first leaf node among N first leaf nodes includes at least one first hash group among M first hash groups, the hash value of at least one first leaf node is a hash value obtained by performing an XOR operation on the hash value included in at least one first hash group among the M first hash groups included in the corresponding first hash bucket.

[0157] Optionally, the first hash bucket corresponding to the at least one first leaf node may also include two or more first hash groups among the M first hash groups.

[0158] When the first hash bucket corresponding to at least one first leaf node among the N first leaf nodes does not include one first hash group among the M first hash groups, the hash value of the at least one first leaf node is equal to zero.

[0159] The height of the first Merkle tree is associated with the first data table. Before establishing the first Merkle tree, it is also necessary to determine the relevant parameters of the first Merkle tree according to the size of the first data table, such as the number of leaf nodes included in the first Merkle tree, the tree height, etc. Among them, the tree height of the first Merkle tree will change adaptively with the size of the data included in the first data table. The larger the amount of data included in the first data table, the higher the tree height of the first Merkle tree. In other words, when the amount of data included in the first data table is large (for example, 1GB), the height of the corresponding first Merkle tree is higher than when the amount of data included in the first data table is small (for example, 100MB).

[0160] Below, Figure 4 Taking the Merkle tree shown in FIG. 1 as an example, the method for generating the first Merkle tree according to the hash value of the first leaf node is introduced.

[0161] like Figure 4 The Merkle tree shown (i.e., an example of the Merkle tree described above) has a tree height of 3, a number of intermediate nodes of 2, and a number of leaf nodes of 4. The top layer is the root node, the second top layer is the intermediate node, the next layer is the leaf node, and the bottom layer is the hash bucket described above (i.e., an example of the first hash bucket described above).

[0162] For ease of description, the four leaf nodes of the Merkle tree (i.e., an example of the first leaf node mentioned above) can be labeled as leaf node 1, leaf node 2, leaf node 3, and leaf node 4 from left to right. The four hash buckets of the Merkle tree can be labeled as hash bucket 1, hash bucket 2, hash bucket 3, and hash bucket 4 from left to right. Among them, the four leaf nodes of the Merkle tree correspond one-to-one to the four hash buckets. Specifically, leaf node 1 corresponds to hash bucket 1, leaf node 2 corresponds to hash bucket 2, leaf node 3 corresponds to hash bucket 3, and leaf node 4 corresponds to hash bucket 4.

[0163] The hash value of each leaf node of the Merkle tree is obtained by performing an XOR operation on the hash value included in the hash bucket corresponding to each leaf node. For example, the hash value of leaf node 1 is obtained by performing an XOR operation on the hash value included in hash bucket 1, that is, the hash value of leaf node 1 can be expressed as N0 = XOR (1, 5), and XOR (1, 5) represents the result of performing an XOR operation on the hash value of the row identifier 1 (i.e. 0xeffe898) and the hash value of the row identifier 5 (i.e. 0xb8b8dd) included in hash bucket 1.

[0164] The hash value of each intermediate node of the Merkle tree is obtained by hashing the hash values ​​of its two child nodes. For example, N4=H(N0,N1) represents the hash value of an intermediate node of the Merkle tree, and H(N0,N1) represents the result of hashing the hash values ​​of the two leaf nodes of the intermediate node (i.e., N0 and N1).

[0165] The hash value of the Merkle tree root node is obtained by performing a hash operation on the hash values ​​of its two child nodes. For example, H(N4,N5) represents the hash value of the Merkle tree root node.

[0166] The above-mentioned i-th leaf node #1 can be understood as the leaf node #1 with sequence number i, and the i-th hash bucket #1 can be understood as the hash bucket #1 with sequence number i, i=1,2,3,4.

[0167] It should be understood that Figure 4 This is for illustration only and does not constitute any limitation to the present application. For example, in some implementations, Figure 4 The Merkle tree shown may also include a greater number of leaf nodes. For example, in some implementations, Figure 4 The Merkle tree shown can also have a higher tree height.

[0168] Before step 110, the following operation may also be included: obtaining a first data table from a first database.

[0169] For ease of description, the following Figure 5 and Figure 6 The method of obtaining the first data table from the first database is specifically introduced. It should be understood that Figure 5 and Figure 6 This is only for illustration and does not constitute any limitation on the method of obtaining the first data table in the present application.

[0170] Figure 5 is a schematic diagram of extracting data from the first data table provided by the present application. It should be understood that Figure 5 This is just an example. For example, data table #1 may also include more (for example, 100) or fewer (for example, 4) rows of data. For example, the data extraction module may also include more threads.

[0171] like Figure 5 As shown, the execution subject of extracting data from the first data table may be a data extraction module. Specifically, the data extraction module may be Figure 2 The source end data extraction module 131 and the target end data extraction module 134 are shown. That is, Figure 2 The source-side data extraction module 131 and the target-side data extraction module 134 have the data extraction functions described below.

[0172] In one example, extracting data from the first data table may include but is not limited to the following steps:

[0173] Divide the first data table into S batches of data by row, where S is a positive integer greater than or equal to 1;

[0174] Use S processing threads to process S batches of data, and the S processing threads correspond to the S batches of data one by one;

[0175] The processed S batches of data are queued into S queues, and the S batches of data correspond to the S queues one by one.

[0176] The number of the processing threads can be set according to the size of the first data table. For example, when the first data table is large, a larger number of processing threads can be set. For example, when the first data table is small, a smaller number of processing threads can be set.

[0177] In the above technical solution, row data can be extracted from the first data table in batches, each batch of data can be processed by a separate thread, these threads can be executed in parallel, and the extracted data is put into a corresponding data queue.

[0178] Optionally, the same processing thread may be used to process the data in the data table.

[0179] In one example, the data extraction module may be Figure 2 The source data extraction module 131 in the above data extraction module can be Figure 2 The target end data extraction module 134 in FIG. Figure 2 The source-side data extraction module 131 and the target-side data extraction module 134 have the functions of the above-mentioned data extraction modules.

[0180] See also Figure 5 , there are 8 data in data table #1 (i.e., an example of the first data table). The 1st to 4th data can be used as the first batch of data, and the 5th to 8th data can be used as the second batch of data. The data extraction module can include two threads responsible for extracting data, namely thread #1 and thread #2. Thread #1 can be responsible for extracting the first batch of data, and thread #2 can be responsible for extracting the second batch of data. Thread #1 puts the extracted data into queue #1, and thread #2 puts the extracted data into queue #2. The data extraction threads, i.e., the above-mentioned thread #1 and the above-mentioned thread #2 can be executed in parallel to improve the extraction efficiency.

[0181] Figure 6 This is a schematic diagram provided by the present application for performing hash partitioning on data extracted from the first data table according to row identifiers.

[0182] like Figure 6 As shown in FIG. 1 , the execution subject of performing hash partitioning of the data extracted from data table #1 (i.e., an example of the first data table) according to the row identifier may be a data processing module. Specifically, the data processing module may be Figure 2 The source end processing module 132 and the target end data processing module 133 are shown. That is, Figure 2 The source-side processing module 132 and the target-side data processing module 133 in the embodiment have the hash partitioning function described below. Figure 6 , data table #1 includes 8 data items, and the row identifiers corresponding to these 8 data items are 1, 2, 3, ..., 8 respectively.

[0183] In the embodiment of the present application, the process of performing hash partitioning on the data table #1 according to the row identifier and mapping the result after hash partitioning to N (N=4) hash buckets #1 can be as follows:

[0184] First, hash the row data of each row of data included in data table #1 to obtain the hash value corresponding to the row data, and queue the hash value corresponding to the row data and the corresponding row identifier into hash data queue #1 in order. For ease of description, each row in hash data queue #1 can be recorded as a hash group (i.e., an example of the first hash group mentioned above). For example, the row identifier of the first hash group in hash data queue #1 is 1, and the stored hash value is 0xffe898. The row identifier of the fifth hash group in hash data queue #1 is 1, and the stored hash value is 0xb8bdd. For details, see Figure 6 , I will not give examples one by one here.

[0185] Optionally, in some implementations, a hash operation may be performed on both the row identifier and the row data of each row of data to obtain a hash value corresponding to the row identifier and a hash value corresponding to the row data.

[0186] Then, a modulo operation is performed on the row identifier of the hash data queue #1, and the hash bucket corresponding to the row where the row identifier is located is determined according to the result of the modulo operation of the row identifier. Specifically, the hash data queue #1 includes 8 hash groups, and the result obtained by performing a modulo operation on the row identifier 1 of the first hash group is 1, and the result obtained by performing a modulo operation on the row identifier 5 of the fifth hash group is 1, so the first hash group and the fifth hash group can be mapped to the first hash bucket #1 (i.e., the hash bucket with a sequence number of 1). In other words, the row data with a row identifier of 1 included in the data table #1 is mapped to the hash bucket #1 with a sequence number of 1. Similarly, the above processing can be performed on other hash groups in the hash data queue #1, and it can be obtained that the second hash group and the sixth hash group are mapped to the second hash bucket #1, the third hash group and the seventh hash group are mapped to the third hash bucket #1, and the fourth hash group and the eighth hash group are mapped to the fourth hash bucket #1.

[0187] It should be understood that Figure 6 This is just an example. For example, a greater number (e.g., 8) or a smaller number (e.g., 2) of hash buckets #1 may be included. For example, data table #1 may also include a greater number (e.g., 100 rows) or a smaller number (e.g., 4 rows) of row data.

[0188] Step 120: Process the second data table in the second database to generate a second Merkle tree.

[0189] The second database can be understood as a target database. For example, in one example, the second database can be Figure 1 The target database 120 is shown.

[0190] In the present application, the second data table is obtained by synchronizing or migrating the first data table to the second database, the second Merkle tree includes N second leaf nodes, the N second leaf nodes correspond one-to-one to N second hash buckets, the hash value of each second leaf node is determined according to the corresponding second hash bucket, the N second hash buckets are obtained by hash partitioning the second data table according to row identifiers, the hash rule for hash partitioning the second data table according to row identifiers to obtain the N second hash buckets is the same as the hash rule for hash partitioning the first data table according to row identifiers to obtain the N first hash buckets, and any two second hash buckets are not the same.

[0191] In the present application, the second Merkle tree includes N first leaf nodes, and the first Merkle tree also includes N first leaf nodes. Since the two Merkle trees include the same number of leaf nodes, it can be considered that the first Merkle tree and the second Merkle tree have the same tree height. In other words, the first Merkle tree and the second Merkle tree provided in the present application have the same tree height.

[0192] The above-mentioned N second leaf nodes correspond one-to-one with the N second hash buckets, which can be understood as that the i-th second leaf node among the N second leaf nodes (i.e., the second leaf node with sequence number i) corresponds to the i-th second hash bucket among the N second hash buckets (i.e., the second hash bucket with sequence number i). In other words, the second leaf node with sequence number i corresponds to the second hash bucket with sequence number i. Among them, the sequence number of each second leaf node among the N second leaf nodes is different, and the sequence number of each second hash bucket among the N second hash buckets is different, and i is a positive integer greater than or equal to 1 and less than or equal to N.

[0193] The hash value of each second leaf node is determined according to the corresponding second hash bucket. For a specific determination method, refer to the method for determining the first leaf node according to the corresponding first hash bucket in step 110.

[0194] The above-mentioned N second hash buckets are obtained by performing hash partitioning on the second data table according to row identifiers. The corresponding relationship between the second hash buckets and the second data table is not specifically limited in the present application.

[0195] In one example, each second hash bucket is determined according to a row in the second data table.

[0196] In another example, at least one second hash bucket among the N second hash buckets is determined according to two or more rows in the second data table.

[0197] In yet another example, at least one of the N second hash buckets may also be empty.

[0198] The above-mentioned N second leaf nodes correspond one-to-one with the N second hash buckets, which can be understood as that the i-th second leaf node among the N second leaf nodes corresponds to the i-th hash bucket among the N second hash buckets, where i is a positive integer greater than or equal to 1 and less than or equal to N. It should also be understood that the sequence number of each second leaf node among the N second leaf nodes is different.

[0199] The second data table is obtained by synchronizing or migrating the first data table to the second database, which may include the following situations:

[0200] If there is no data missing during the process of synchronizing or migrating the first data table to the second database, the number of rows included in the second data table is the same as the number of rows included in the first data table.

[0201] For example, the first data table includes 10 rows, each row includes a row identifier and row data. If there is no data missing during synchronization or migration of a data table to a second data table, the second data table obtained after synchronization or migration also includes 10 rows.

[0202] If data is missing during synchronization or migration of the first data table to the second database, the number of rows included in the second data table is less than the number of rows included in the first data table.

[0203] For example, the first data table includes 10 rows, each row includes a row identifier and row data. If one row of data is missing during synchronization or migration of a data table to a second data table, the second data table obtained after synchronization or migration includes 9 rows.

[0204] That is to say, the number of rows included in the second data table in the present application may be the same as the number of rows included in the first data table, or the number of rows included in the second data table may also be less than the number of rows included in the first data table.

[0205] The above-mentioned hash rule for hash partitioning the second data table according to row identifiers to obtain N second hash buckets is the same as the hash rule for hash partitioning the first data table according to row identifiers to obtain N first hash buckets. It can be understood that since the hash rules for partitioning the second data table and the first data table according to row identifiers are the same, it can be guaranteed that the data corresponding to the same row identifier in the second data table and the first data table will be mapped to hash buckets with the same label.

[0206] The fact that any two second hash buckets are different may be understood as that the sequence numbers corresponding to any two second hash buckets are different, and the row identifiers included in any two non-empty second hash buckets are different.

[0207] In the present application, when the second data table includes K rows and the first data table includes M rows, K is a positive integer less than or equal to M, processing the second data table in the second database to generate a second Merkle tree may include:

[0208] Perform hash processing on the K rows to obtain K second hash groups, where the K second hash groups correspond to the K rows one by one, each second hash group includes a row identifier in the K rows and a hash value of row data corresponding to the row identifier, and each second hash group includes a different row identifier;

[0209] According to mapping the K second hash groups to N second hash buckets;

[0210] Determine hash values ​​of N second leaf nodes according to the N second hash buckets;

[0211] Generate a second Merkle tree based on the hash values ​​of the N second leaf nodes.

[0212] Among them, when K is equal to M, it can be understood that the number of rows included in the second data table is the same as the number of rows included in the first data table, that is, if there is no data missing during the process of synchronizing or migrating the first data table to the second database. When K is less than M, it can be understood that the number of rows included in the second data table is less than the number of rows included in the first data table, that is, if there is data missing during the process of synchronizing or migrating the first data table to the second database.

[0213] It should be understood that the mapping rule for mapping K second hash groups to N second hash buckets is the same as the mapping rule for mapping M first hash groups to N first hash buckets, that is, the hash rule for hash partitioning the second data table according to row identifiers to obtain N second hash buckets is the same as the hash rule for hash partitioning the first data table according to row identifiers to obtain N first hash buckets.

[0214] For example, the first hash group with a sequence number of 1 includes a row identifier of 1 and a hash value of the corresponding row data, and the first hash group with a sequence number of 1 is mapped to the first hash bucket with a sequence number of 5. When the second data table includes a row with a row identifier of 1, the row with a row identifier of 1 corresponds to a second hash group with a sequence number of 1, and the second hash group with a sequence number of 1 is mapped to a second hash bucket with a sequence number of 5.

[0215] It should also be understood that when the second hash bucket corresponding to the hash value of the second leaf node is empty, the hash value of the second leaf node is also empty.

[0216] Before step 120, the process may further include acquiring a second data table from a second database.

[0217] The present application does not specifically limit the method for obtaining the second data table.

[0218] For example, a verification mark may be entered into a first data table in a first database, and when the second database is copied from the first database, when the verification mark is detected in the second database, the data with the verification mark is used as the data in the second data table.

[0219] Among them, the content not described in detail in step 120 is the same as the content described in the above step 110. Please refer to the above step 110 for details and will not be repeated here.

[0220] Step 130: Compare the first Merkle tree with the second Merkle tree to determine whether the first data table is consistent with the second data table.

[0221] Whether the first data table is consistent with the second data table can be understood as that the number of row identifiers stored in the first data table and the second data table is the same, and the content of row data corresponding to the same row identifier is also the same.

[0222] In the present application, comparing the first Merkle tree with the second Merkle tree to determine whether the first data table is consistent with the second data table may include:

[0223] Determine a hash value of a first Merkle tree root node and a hash value of a second Merkle tree root node;

[0224] If the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, determining that the first data table is consistent with the second data table;

[0225] If the hash value of the first Merkle tree root node is different from the hash value of the second Merkle tree root node, it is determined that the first data table is inconsistent with the second data table.

[0226] In the above technical solution, since the hash value of each node of the Merkle tree is obtained by performing a hash operation on the child nodes of each node, for example, the hash value of the root node is determined based on the two intermediate nodes corresponding to the root node, and the hash value of the leaf node is determined based on the data in the hash bucket corresponding to the leaf node. Therefore, when the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, it can be considered that the first data table and the second data table are consistent. When the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, it can be considered that there is a difference between the first data table and the second data table, that is, inconsistency.

[0227] Optionally, after determining that the first data table is inconsistent with the second data table, the following operations may also be included:

[0228] Determine that the hash value of the i-th first leaf node is different from the hash value of the i-th second leaf node, the hash value of the i-th first leaf node is determined according to the i-th first hash bucket, and the hash value of the i-th second leaf node is determined according to the i-th second hash bucket, where i is a positive integer and 1≤i≤N;

[0229] Compare the i-th first hash bucket with the i-th second hash bucket to determine the row identifiers of the first data table and the second data table that are inconsistent;

[0230] According to the inconsistent row identifiers, the row data corresponding to the inconsistent row identifiers are queried from the first database and the second database respectively.

[0231] The above determination that the hash value of the i-th first leaf node is different from the hash value of the i-th second leaf node, for example, can first compare the hash value of the first leaf node with the sequence number 1 and the hash value of the second leaf node with the sequence number 1 to determine that the hash values ​​are the same, and then continue to compare the hash value of the leaf node with the sequence number 2, and so on, until it is determined that the hash value of the first leaf node with the sequence number i is different from the hash value of the second leaf node with the sequence number i. The first leaf node with the sequence number i can be understood as the i-th first leaf node, and the second leaf node with the sequence number i can be understood as the i-th second leaf node. For example, the hash value of the first leaf node with the sequence number N and the hash value of the second leaf node with the sequence number N can also be compared to determine that the hash values ​​are the same, and then continue to compare the hash value of the leaf node with the sequence number N-1, and so on, until it is determined that the hash value of the first leaf node with the sequence number i is different from the hash value of the second leaf node with the sequence number i. The above-mentioned comparing the i-th first hash bucket with the i-th second hash bucket to determine the row identifiers of the inconsistencies between the first data table and the second data table may include:

[0232] Determine that a row identifier included in a hash group in the i-th first hash bucket is the same as a row identifier included in a hash group in the i-th second hash bucket but the corresponding hash values ​​are different;

[0233] It is determined that the row identifier is a row identifier that is inconsistent between the first data table and the second data table.

[0234] In the above technical solution, after locating the inconsistent row identifier, it is necessary to further locate the inconsistent row data. In this embodiment, the process of locating the inconsistent row data can be as follows: firstly, extract the row data identifier from the inconsistent row data, and search the corresponding row data from the source database and the target database respectively through the row data identifier, and the row data contains each row data, and find the inconsistent row data by direct comparison.

[0235] It is understandable that the data comparison module performs a top-down comparison process when performing data consistency comparison. After finding specific inconsistent data, the data storage module stores this information in a non-volatile storage medium for query when needed.

[0236] As an example, the following combination Figure 7 The two Merkle trees shown introduce the process of performing consistency comparison on the two Merkle trees according to the method provided in the above step 130.

[0237] like Figure 7 The two Merkle trees shown can be obtained according to the above step 110 and the above step 120. For the convenience of description, these two Merkle trees are recorded as Merkle tree #1 (i.e., an example of the above first Merkle tree) and Merkle tree #2 (i.e., an example of the above second Merkle tree).

[0238] Below, the hash value of each node is introduced by taking Merkle tree #1 as an example. Similarly, the hash value of each node of Merkle tree #2 is also obtained by a similar method. The four leaf nodes #1 of Merkle tree #1 (i.e., an example of the first leaf node mentioned above) correspond to four hash buckets #1 (i.e., an example of the first hash bucket mentioned above). From left to right, these four leaf nodes #1 are respectively recorded as the first leaf node #1, the second leaf node #1, the third leaf node #1, and the fourth leaf node #1. From left to right, these four hash buckets #1 are respectively recorded as the first hash bucket #1, the second hash bucket #1, the third hash bucket #1, and the fourth hash bucket #1. Each hash bucket #1 includes two hash groups, and each hash group includes a row identifier and a hash value. The hash value of the first leaf node #1 is obtained by performing an XOR operation on the hash values ​​included in the two hash groups in the first hash bucket #1, that is, the hash value of the first leaf node #1 is N0=XOR(1,5), where XOR(1,5) represents an XOR operation on the hash value corresponding to the row identifier 1 and the hash value corresponding to the row identifier 5, that is, XOR(1,5) is equal to XOR(011,101)(011,101). The hash value of the intermediate node #1 is obtained by performing a hash operation on the hash values ​​of its two child nodes. For example, N4=H(N0,N1) represents the hash value of an intermediate node #1 of the Merkle tree #1, and H(N0,N1) represents the result of performing a hash operation on the hash values ​​of the two leaf nodes #1 of the intermediate node #1 (that is, N0 and N1). The hash value of the root node #1 is obtained by performing a hash operation on the hash values ​​of its two child nodes. For example, H(N4,N5) represents the hash value of the root node #1 of the Merkle tree #1. Figure 7 , the process of consistency check for Merkle tree #1 and Merkle tree #2 can be as follows: first compare whether the hash values ​​of the root nodes are the same. Since the 4th hash bucket #1 of Merkle tree #1 is different from the 4th hash bucket #2 of Merkle tree #2 (i.e., the hash values ​​corresponding to the hash groups with row identifier "6" are different), the hash value of the root node #1 of Merkle tree #1 (i.e., H(N4, N5) in Merkle tree #1) and the hash value of the root node #2 of Merkle tree #2 (i.e., H(N4, N5) in Merkle tree #2) are different. Next, search from top to bottom to determine the leaf nodes that are inconsistent between Merkle tree #1 and Merkle tree #2. According to Figure 7 It can be seen that it can be determined that the 4th leaf node #1 is different from the 4th leaf node #2. Then, according to the hash buckets corresponding to the 4th leaf node #1 and the 4th leaf node #2, the inconsistent row identifier is determined to be "6".

[0239] The above-mentioned i-th leaf node #1 can be understood as the leaf node #1 with sequence number i, and the i-th hash bucket #1 can be understood as the hash bucket #1 with sequence number i, i=1,2,3,4.

[0240] It should be understood that Figure 7 This is only for illustration and does not constitute any limitation to this application. For example, Figure 7 The hash bucket #1 and each of the hash buckets #1 shown may include a different number of hash groups. For example, Figure 7 The hash bucket #1 and the hash value included in the hash bucket #1 may also be hash values ​​of a larger magnitude.

[0241] In the present application, the types of the first database and the second database involved in the above steps 110 to 130 are not specifically limited.

[0242] Optionally, the first database and the second database may be heterogeneous databases.

[0243] Optionally, the first database and the second database may be isomorphic databases.

[0244] Optionally, the first database may be a relational database or a non-relational database, and the second database may be a relational database or a non-relational database.

[0245] For example, the first database may be a relational database, and the second database may be a non-relational database. For example, the first database may be a relational database, and the second database may be a relational database.

[0246] Relational databases usually store data in tabular form, so data can be directly extracted from the relational database to construct the first data table. Non-relational databases usually store data in non-tabular form (for example, documents, key values, or graph structures, etc.), so before extracting data from the non-relational database to construct the first data table, the data to be verified in the non-relational database needs to be converted into a table storage form, where each row of the table can include a row identifier and one or more row data.

[0247] In most scenarios, the data in the production database (i.e., an example of the first database mentioned above) changes dynamically. During the data verification process, the verified data changes again and needs to be re-verified. If the update method is adopted on the original Merkle tree, the update of each leaf will cause the recalculation of the hash values ​​of the layers above it. When the number of layers of the Merkle tree is high, this calculation overhead will increase sharply. Another factor that needs to be considered is that when the incremental data increases rapidly, considering the overall verification performance, the original Merkle tree needs to be expanded and rebuilt, which will have an impact on the data that has been verified.

[0248] The data verification method provided in the present application can also verify the full data and the incremental data separately. Specifically, the full data is verified using a Merkle tree with a higher number of layers, and the incremental data is verified using a Merkle tree with a lower number of layers, thereby effectively reducing the computational overhead.

[0249] In the present application, the method for obtaining the incremental data to be verified from the data table is not specifically limited. For example, the existing method for obtaining the incremental data to be verified in the data table can be used. For example, other methods for obtaining the incremental data to be verified can also be used.

[0250] As an example, data table #1 (i.e., an example of the first data table) includes 10 rows of data, data table #2 (i.e., an example of the second data table) includes 10 rows of data, and data table #2 is obtained by copying data table #1. At time #1, the method of steps 110 to 130 can be used to generate Merkle tree #1 (i.e., an example of the first Merkle tree) according to data table #1, and generate Merkle tree #2 (i.e., an example of the second Merkle tree) according to data table #2, and determine whether data table #1 and data table #2 are consistent by comparing Merkle tree #1 with Merkle tree #2. At a moment after time #1, the data of rows 5 to 10 in data table #1 are updated. At this time, the updated data table #1 can be recorded as data table #3 (i.e., an example of the first data table), and data table #4 (i.e., an example of the second data table) is obtained by copying data table #3. In this case, only the incremental data can be checked for consistency using the method of steps 110 to 130. Specifically, a Merkle tree #3 (i.e., an example of the first Merkle tree) is generated based on data table #3, a Merkle tree #4 (i.e., an example of the second Merkle tree) is generated based on data table #4, and whether data table #3 and data table #4 are consistent is determined by comparing Merkle tree #3 with Merkle tree #4.

[0251] The data verification method provided by the present application can better meet the needs of accurate and fast verification of massive data in different scenarios (for example, online data or offline data). In the above technical scheme, the hash value of each first leaf node of the first Merkle tree corresponding to the first data table is determined according to the first hash bucket corresponding thereto, and the hash value of each second leaf node of the second Merkle tree corresponding to the second data table is determined according to the second hash bucket corresponding thereto. Since the hash rule of obtaining N second hash buckets by hash partitioning the second data table according to the row identifier is the same as the hash rule of obtaining N first hash buckets by hash partitioning the first data table according to the row identifier, it can be ensured that the data corresponding to the same row identifier in the second data table and the first data table are respectively mapped to the second hash bucket and the first hash bucket with the same sequence number, so that the generated first Merkle tree and the second Merkle tree have the same structure. Therefore, it is possible to directly determine whether the first data table and the second data table are consistent by comparing the hash values ​​of the root nodes of the above two Merkle trees. When generating the first Merkle tree and the second Merkle tree, since the hash values ​​of the leaf nodes of the two Merkle trees are obtained by performing an XOR operation on the data in the corresponding hash buckets, sorting of the row data can be avoided when performing consistency check on the first data table and the second data table.

[0252] When it is determined that the hash values ​​of the root nodes of the two Merkle trees are inconsistent, the data verification method provided by the present application can further determine the inconsistent row identifiers and corresponding row data by comparing the hash values ​​of the child nodes of the two Merkle trees. In order to meet the needs of data verification in different scenarios, the data verification method provided by the present application can also adaptively adjust the tree heights of the first Merkle tree and the second Merkle tree according to the size of the data set to be verified.

[0253] Next, combine Figure 8 The verification process of the data verification method 200 provided in this application is specifically introduced.

[0254] like Figure 8 As shown, the method 200 includes steps 210 to 290, which are described below. The execution body of steps 210 to 290 may be Figure 2 The data verification device 130 is shown.

[0255] Step 210, start.

[0256] The above step 210 indicates starting the data consistency check.

[0257] Step 220, inserting a check mark into the data to be copied in database #1 (ie, an example of the first database in the above method 100).

[0258] The method of inserting the verification mark into the data to be copied can be the same as the existing method, which will not be described in detail here.

[0259] Step 230 : Database # 2 (ie, an example of the second database in the method 100 ) copies the data to be copied.

[0260] Step 240: Database #1 detects the mark bit and obtains data table #1.

[0261] The method for obtaining data table #1 after detecting the mark bit from database #1 can be the same as the existing method, which will not be repeated here.

[0262] Step 250: Database #2 detects the mark bit and obtains data table #2.

[0263] Step 260 , generating a Merkle tree # 1 (ie, an example of the first Merkle tree in the above method 100 ) according to data table # 1 .

[0264] Step 270, generating Merkle tree #2 (i.e., an example of the second Merkle tree in the above method 100) according to data table #2

[0265] The method for determining the Merkle tree in the above step 260 and the above step 270 is the same as the method for determining the Merkle tree in method 100. For details, please refer to step 110 above, which will not be described in detail here.

[0266] Step 280, determine whether the hash value of the root node of Merkle tree #1 is the same as the hash value of the root node of Merkle tree #2.

[0267] The above determination of whether the hash value of the root node of Merkle tree #1 is the same as the hash value of the root node of Merkle tree #2 includes:

[0268] When it is determined that the hash value of the root node of Merkle tree #1 is the same as the hash value of the root node of Merkle tree #2, execute step 281;

[0269] When it is determined that the hash value of the root node of Merkle tree #1 is different from the hash value of the root node of Merkle tree #2, step 282 and step 283 are executed.

[0270] Step 281, determine whether data table #1 is consistent with data table #2.

[0271] Step 282, compare the hash value of the leaf node of Merkle tree #1 with the hash value of the leaf node of Merkle tree #2 to determine the inconsistent row identifiers.

[0272] Please refer to the method in step 130 above for details, which will not be described in detail here.

[0273] In the present application, the execution order of step 281 and step 282 is not specifically limited.

[0274] For example, after executing step 280, step 281 may be executed first and then step 282. For example, after executing step 280, step 282 may be executed first and then step 281.

[0275] Step 283: According to the inconsistent row identifiers determined above, inconsistent row data is determined from database #1 and database #2.

[0276] Please refer to the method in step 130 above for details, which will not be described in detail here.

[0277] Optionally, after step 283, the determined inconsistent row identifiers and corresponding row data may also be stored in a storage module of the data verification device, for example, Figure 2 Storage module 136 is shown.

[0278] Step 290, end.

[0279] The above step 290 indicates the end of the data consistency check.

[0280] It should be understood that the above Figure 8 This is only for illustration and does not limit the data verification process provided in this application. For example, in some implementations, after determining that data table #1 and data table #2 are inconsistent, step 282 and step 283 may not be performed.

[0281] Fig. 9 It is a schematic diagram of a Merkle tree determined according to the method provided in this application.

[0282] like Fig. 9 As shown, there are 4 Merkle trees, namely Merkle tree #3 (i.e., an example of the first Merkle tree mentioned above), Merkle tree #4 (i.e., another example of the first Merkle tree mentioned above), Merkle tree #5 (i.e., an example of the second Merkle tree mentioned above), and Merkle tree #6 (i.e., another example of the second Merkle tree mentioned above). Among them, the height of Merkle tree #3 and Merkle tree #5 is 3, and the height of Merkle tree #4 and Merkle tree #6 is 2. For a detailed description of Merkle tree #3, Merkle tree #4, Merkle tree #5, and Merkle tree #6, please refer to the above Figure 7 The content described will not be described in detail here.

[0283] Among them, Merkle tree #3 can be understood as a Merkle tree generated at time #1 based on the full data in data table #1 (i.e., an example of the first data table mentioned above). Merkle tree #4 can be understood as a Merkle tree generated at time #2 based on the incremental data in data table #1, where time #2 is a moment after time #1, and the incremental data in data table #1 at time #2 includes the row data corresponding to the following row identifiers in data table #1: “2”, “6”, “7” and “8”. Merkle tree #5 can be understood as a Merkle tree generated at time #1 based on the full data in data table #2 (i.e., an example of the second data table mentioned above). Merkle tree #6 can be understood as a Merkle tree generated at time #2 based on the incremental data in data table #2, and the incremental data in data table #1 at time #2 includes the row data corresponding to the following row identifiers in data table #2: “2”, “6”, “7” and “8”. Among them, data table #2 is the data table obtained after migrating or synchronizing data table #1.

[0284] The method of generating the above-mentioned Merkle tree #3, Merkle tree #4, Merkle tree #5, and Merkle tree #6 can refer to method 100, which will not be described in detail here.

[0285] When consistency check is required for data table #1 and data table #2, it includes:

[0286] By comparing Merkle tree #3 and Merkle tree #5, it can be determined whether the full data in data table #1 and data table #2 are consistent;

[0287] By comparing Merkle tree #4 and Merkle tree #6, it can be determined whether the incremental data in data table #1 and data table #2 are consistent.

[0288] Optionally, after determining that the full data in data table #1 and data table #2 are inconsistent, the inconsistent row identifiers can be determined by comparing the leaf nodes of Merkle tree #3 with the leaf nodes of Merkle tree #5, and the row data corresponding to the inconsistent row identifiers can be determined from data table #1 and data table #2 based on the inconsistent row identifiers.

[0289] Optionally, after determining that the incremental data in data table #1 and data table #2 are inconsistent, the inconsistent row identifiers can be determined by comparing the leaf nodes of Merkle tree #4 with the leaf nodes of Merkle tree #6, and the row data corresponding to the inconsistent row identifiers can be determined from data table #1 and data table #2 based on the inconsistent row identifiers.

[0290] In the embodiment of the present application, using an independent Merkle tree to perform data verification on incremental data can further save computing overhead and improve the efficiency of data consistency verification.

[0291] Combination of the above Figures 1 to 9 , describes in detail the data verification method provided by this application and the system architecture applicable to the method. Figures 10 to 12 The data verification device, data verification equipment and data verification system provided by the present application are described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0292] In the embodiment of the present application, the data verification device should include a processing unit and a determination unit. The data verification device may be the data verification device 130 mentioned above.

[0293] Optionally, in some implementations, the data verification device may further include a transceiver unit.

[0294] Next, combine Fig.10 , taking a data verification device including a processing unit and a determination unit as an example for introduction.

[0295] Fig.10 It is a schematic structural diagram of a data verification device 1000 provided in this application.

[0296] like Fig.10 As shown, the device 1000 includes: a processing unit 1001 and a determining unit 1002.

[0297] A processing unit 1001 is used to process a first data table in a first database to generate a first Merkle tree, where each row of the first data table includes a row identifier and row data, and the first Merkle tree includes N first leaf nodes, where the N first leaf nodes correspond one-to-one to N first hash buckets, and a hash value of each of the first leaf nodes is determined according to a corresponding first hash bucket, and the N first hash buckets are obtained by performing hash partitioning on the first data table according to the row identifier, and any two first hash buckets are different, and N is a positive integer greater than or equal to 2;

[0298] The processing unit 1001 is further used to process a second data table in a second database to generate a second Merkle tree, where the second data table is obtained by synchronizing or migrating the first data table to the second database, and the second Merkle tree includes N second leaf nodes, where the N second leaf nodes correspond one-to-one to N second hash buckets, and the hash value of each second leaf node is determined according to the corresponding second hash bucket, and the N second hash buckets are obtained by hash partitioning the second data table according to the row identifier, and the hash rule of the N second hash buckets obtained by hash partitioning the second data table according to the row identifier is the same as the hash rule of the N first hash buckets obtained by hash partitioning the first data table according to the row identifier, and any two second hash buckets are different;

[0299] The determination unit 1002 is configured to compare the first Merkle tree with the second Merkle tree to determine whether the first data table is consistent with the second data table.

[0300] Optionally, the first data table includes M rows, where M is a positive integer greater than or equal to 1.

[0301] The processing unit 1001 is also used for:

[0302] Perform hash processing on the M rows to obtain M first hash groups, where the M first hash groups correspond to the M rows one by one, each of the first hash groups includes a row identifier in the M rows and a hash value of row data corresponding to the row identifier, and each of the first hash groups includes a different row identifier;

[0303] Mapping the M first hash groups to the N first hash buckets;

[0304] The determining unit 1002 is further configured to:

[0305] Determine hash values ​​of the N first leaf nodes according to the N first hash buckets;

[0306] The processing unit 1001 is also used for:

[0307] The first Merkle tree is generated according to the hash values ​​of the N first leaf nodes.

[0308] Optionally, the hash value of each first leaf node is a hash value obtained by performing an XOR operation on the hash value included in the first hash group included in the corresponding first hash bucket.

[0309] Optionally, the determining unit 1002 is further configured to:

[0310] Determine a hash value of the first Merkle tree root node and a hash value of the second Merkle tree root node;

[0311] If the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, determining that the first data table is consistent with the second data table;

[0312] If the hash value of the first Merkle tree root node is different from the hash value of the second Merkle tree root node, it is determined that the first data table is inconsistent with the second data table.

[0313] Optionally, the determining unit 1002 is further configured to:

[0314] Determine that the hash value of the i-th first leaf node is different from the hash value of the i-th second leaf node, the hash value of the i-th first leaf node is determined according to the i-th first hash bucket, and the hash value of the i-th second leaf node is determined according to the i-th second hash bucket, where i is a positive integer and 1≤i≤N;

[0315] Compare the i-th first hash bucket with the i-th second hash bucket to determine the row identifiers of the first data table and the second data table that are inconsistent;

[0316] The processing unit 1001 is also used for:

[0317] According to the inconsistent row identifier, the row data corresponding to the inconsistent row identifier is queried from the first database and the second database respectively.

[0318] Optionally, the first data table includes all the data in at least one data table in the first database.

[0319] Optionally, the first data table includes incremental data in at least one data table in the first database.

[0320] Optionally, the height of the first Merkle tree is associated with the first data table.

[0321] Optionally, the first database and the second database are heterogeneous databases or homogeneous databases.

[0322] Optionally, the first database is a relational database or a non-relational database, and the second database is a relational database or a non-relational database.

[0323] Next, combine Fig.11 , taking the data verification device including the transceiver, processor and memory as an example for introduction.

[0324] Fig.11 1 is a schematic structural diagram of a data verification device 1000 provided by the present application. Fig.11 As shown, the device 1000 includes: a transceiver 1010, a processor 1020 and a memory 1030. The transceiver 1010, the processor 1020 and the memory 1030 communicate with each other through an internal connection path to transfer control and / or data signals. The memory 1030 is used to store a computer program, and the processor 1010 is used to call and run the computer program from the memory 1030 to control the transceiver 1020 to send and receive signals.

[0325] Specifically, the transceiver 1010 can be used to obtain the first data table and the second data table mentioned above, which will not be described in detail here.

[0326] Specifically, the function of the processor 1020 is Fig.10 The specific functions of the processing unit 1001 and the determining unit 1002 shown correspond to each other and are not described in detail here.

[0327] In the embodiment of the present application, the data verification device should include a processor, wherein the data verification device can be any one of the terminal devices described above.

[0328] Optionally, in some implementations, the data verification device may also include a transceiver.

[0329] Optionally, in some implementations, the data verification device may also include a memory.

[0330] Fig.12 1 is a schematic diagram of the structure of a system 1200 provided by the present application. Fig.12 As shown, the system 1200 includes: the above data verification device 1000 or the data verification device 1100. Optionally, the system 1200 may also include the above first database and the above second database.

[0331] An embodiment of the present application provides a computer program product. When the computer program product is run on a data verification device 1310, the data verification device 1310 can execute method 100 and / or method 200 in the above method embodiments.

[0332] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0333] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0334] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0335] The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0336] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0337] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0338] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0339] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (digitalvideo disc, DVD), or a semiconductor medium (such as a solid state drive), etc.

[0340] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0341] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0342] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship; the term "at least one" in this application can mean "one" and "two or more". For example, at least one of A, B and C can represent seven situations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, A and C exist at the same time, C and B exist at the same time, and A, B and C exist at the same time.

[0343] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data verification method, characterized in that: The method comprises: Processing a first data table in a first database to generate a first Merkle tree, wherein each row of the first data table includes a row identifier and row data, and the first Merkle tree includes N first leaf nodes, wherein the N first leaf nodes correspond one-to-one to N first hash buckets, and the hash value of each first leaf node is determined according to the corresponding first hash bucket, and the N first hash buckets are obtained by performing hash partitioning on the first data table according to the row identifier, and any two first hash buckets are different, and N is a positive integer greater than or equal to 2; Processing a second data table in a second database to generate a second Merkle tree, wherein the second data table is obtained by synchronizing or migrating the first data table to the second database, the second Merkle tree includes N second leaf nodes, the N second leaf nodes correspond one-to-one to N second hash buckets, the hash value of each second leaf node is determined according to the corresponding second hash bucket, the N second hash buckets are obtained by hash partitioning the second data table according to row identifiers, a hash rule for obtaining the N second hash buckets by hash partitioning the second data table according to row identifiers is the same as a hash rule for obtaining the N first hash buckets by hash partitioning the first data table according to row identifiers, and any two second hash buckets are different; The first Merkle tree is compared with the second Merkle tree to determine whether the first data table is consistent with the second data table.

2. The method according to claim 1, characterized in that The first data table includes M rows, where M is a positive integer greater than or equal to 1. The processing of the first data table in the first database to generate a first Merkle tree includes: Performing hash processing on the M rows to obtain M first hash groups, the M first hash groups corresponding to the M rows one by one, each of the first hash groups including a row identifier in the M rows and a hash value of row data corresponding to the row identifier, and each of the first hash groups includes a different row identifier; mapping the M first hash groups to the N first hash buckets; Determine hash values ​​of the N first leaf nodes according to the N first hash buckets; Generate the first Merkle tree according to the hash values ​​of the N first leaf nodes.

3. The method according to claim 2, characterized in that The hash value of each of the first leaf nodes is a hash value obtained by performing an XOR operation on the hash values ​​included in the first hash group included in the corresponding first hash bucket.

4. The method according to any one of claims 1 to 3, characterized in that The comparing the first Merkle tree with the second Merkle tree to determine whether the first data table is consistent with the second data table includes: Determine a hash value of the first Merkle tree root node and a hash value of the second Merkle tree root node; If the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, determining that the first data table is consistent with the second data table; If the hash value of the first Merkle tree root node is different from the hash value of the second Merkle tree root node, it is determined that the first data table is inconsistent with the second data table.

5. The method according to claim 4, characterized in that After determining that the first data table is inconsistent with the second data table, the method further includes: Determine that a hash value of an i-th first leaf node is different from a hash value of an i-th second leaf node, the hash value of the i-th first leaf node is determined according to the i-th first hash bucket, the hash value of the i-th second leaf node is determined according to the i-th second hash bucket, i is a positive integer, and 1≤i≤N; Compare the i-th first hash bucket with the i-th second hash bucket to determine the row identifiers of the first data table and the second data table that are inconsistent; According to the inconsistent row identifiers, the row data corresponding to the inconsistent row identifiers are queried from the first database and the second database respectively.

6. The method according to any one of claims 1 to 3, characterized in that The first data table includes all data in at least one data table in the first database.

7. The method according to any one of claims 1 to 3, characterized in that The first data table includes incremental data in at least one data table in the first database.

8. The method according to any one of claims 1 to 3, characterized in that The height of the first Merkle tree is associated with the first data table.

9. The method according to any one of claims 1 to 3, characterized in that: The first database and the second database are heterogeneous databases or homogeneous databases.

10. The method according to any one of claims 1 to 3, characterized in that: The first database is a relational database or a non-relational database, and the second database is a relational database or a non-relational database.

11. A data verification device, characterized in that: The device comprises: A processing unit, configured to process a first data table in a first database to generate a first Merkle tree, wherein each row of the first data table includes a row identifier and row data, and the first Merkle tree includes N first leaf nodes, wherein the N first leaf nodes correspond one-to-one to N first hash buckets, and the hash value of each first leaf node is determined according to the corresponding first hash bucket, and the N first hash buckets are obtained by performing hash partitioning on the first data table according to the row identifier, and any two first hash buckets are different, and N is a positive integer greater than or equal to 2; The processing unit is further used to process a second data table in a second database to generate a second Merkle tree, where the second data table is obtained by synchronizing or migrating the first data table to the second database, where the second Merkle tree includes N second leaf nodes, where the N second leaf nodes correspond one-to-one to N second hash buckets, where the hash value of each second leaf node is determined according to the corresponding second hash bucket, where the N second hash buckets are obtained by hash partitioning the second data table according to row identifiers, where a hash rule for obtaining the N second hash buckets by hash partitioning the second data table according to row identifiers is the same as a hash rule for obtaining the N first hash buckets by hash partitioning the first data table according to row identifiers, and any two second hash buckets are different; A determination unit is used to compare the first Merkle tree with the second Merkle tree to determine whether the first data table is consistent with the second data table.

12. The device according to claim 11, characterized in that The first data table includes M rows, where M is a positive integer greater than or equal to 1. The processing unit is also used for: Performing hash processing on the M rows to obtain M first hash groups, the M first hash groups corresponding to the M rows one by one, each of the first hash groups including a row identifier in the M rows and a hash value of row data corresponding to the row identifier, and each of the first hash groups includes a different row identifier; Mapping the M first hash groups to the N first hash buckets; The determining unit is further configured to: Determine hash values ​​of the N first leaf nodes according to the N first hash buckets; The processing unit is also used for: Generate the first Merkle tree according to the hash values ​​of the N first leaf nodes.

13. The device according to claim 12, characterized in that The hash value of each of the first leaf nodes is a hash value obtained by performing an XOR operation on the hash values ​​included in the first hash group included in the corresponding first hash bucket.

14. The device according to any one of claims 11 to 13, characterized in that The determining unit is further configured to: Determine a hash value of the first Merkle tree root node and a hash value of the second Merkle tree root node; If the hash value of the first Merkle tree root node is the same as the hash value of the second Merkle tree root node, determining that the first data table is consistent with the second data table; If the hash value of the first Merkle tree root node is different from the hash value of the second Merkle tree root node, it is determined that the first data table is inconsistent with the second data table.

15. The device according to claim 14, characterized in that The determining unit is further configured to: Determine that a hash value of an i-th first leaf node is different from a hash value of an i-th second leaf node, the hash value of the i-th first leaf node is determined according to the i-th first hash bucket, the hash value of the i-th second leaf node is determined according to the i-th second hash bucket, i is a positive integer, and 1≤i≤N; Compare the i-th first hash bucket with the i-th second hash bucket to determine the row identifiers of the first data table and the second data table that are inconsistent; The processing unit is also used for: According to the inconsistent row identifiers, the row data corresponding to the inconsistent row identifiers are queried from the first database and the second database respectively.

16. The device according to any one of claims 11 to 13, characterized in that The first data table includes all data in at least one data table in the first database.

17. The device according to any one of claims 11 to 13, characterized in that The first data table includes incremental data in at least one data table in the first database.

18. The device according to any one of claims 11 to 13, characterized in that The height of the first Merkle tree is associated with the first data table.

19. The device according to any one of claims 11 to 13, characterized in that The first database and the second database are heterogeneous databases or homogeneous databases.

20. The device according to any one of claims 11 to 13, characterized in that The first database is a relational database or a non-relational database, and the second database is a relational database or a non-relational database.

21. A data verification device, characterized in that: The method comprises at least one processor and a communication interface, wherein the at least one processor is configured to execute a computer program or instruction so that the data verification is performed according to the method according to any one of claims 1 to 10.

22. The data verification device according to claim 21, characterized in that: The apparatus further comprises at least one memory coupled to the at least one processor, the computer program or instructions being stored in the at least one memory.

23. A computer-readable storage medium, characterized in that: Used for storing computer instructions, when the computer instructions are executed, the method according to any one of claims 1 to 10 is implemented.

24. A data verification system, characterized in that: Includes a data verification device as described in claim 21 or 22.

25. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 10.

Images