Data checking method and electronic device
By using multi-version Merkle tree technology, the data consistency problem in incremental data migration is solved, ensuring data consistency without interrupting business processing and data migration, improving verification accuracy and efficiency, and saving storage space.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-23
- Publication Date
- 2026-03-20
AI Technical Summary
During incremental data migration, existing technologies require interrupting business processing and data migration of the source database to ensure data consistency, which cannot adapt to real-time update scenarios.
By employing multi-version Merkle tree technology, multiple versions of Merkle trees are obtained from the target data table and the source data table. The version numbers are compared to ensure data consistency and avoid interrupting business processing and data migration.
It enables incremental data migration without interrupting business processing and data migration in the source database, improving the accuracy and efficiency of data verification and saving storage space.
Smart Images

Figure CN115840753B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of database, and in particular to a data verification method and an electronic device. BACKGROUND
[0002] Data migration refers to migrating a user data table in a source database to a target database. The representation method and encoding format of data in the data table in the source database and the data table in the target database can be different, but the representation content of the user data table in the source database and the representation content of the user data table in the target database need to be consistent. One method for verifying whether the content of the user data table in the source database and the content of the user data table in the target database are consistent is to calculate a Merkle tree of the user data table in the source database before migration, and calculate a Merkle tree of the user data table in the target database after migration, so as to compare the Merkle tree of the user data table in the source database and the Merkle tree of the user data table in the target database. If the root nodes are consistent (the consistency of the root nodes also means that each child node is consistent), the representation content of the user data table in the source database and the representation content of the user data table in the target database are consistent. However, in incremental data migration, the data table in the source database is constantly changed, and the changes in the source database need to be migrated to the target database in near real time. If the traditional method is used to verify the consistency of the data content, the migration of the source database needs to be interrupted, and the target database needs to be migrated. SUMMARY
[0003] Embodiments of the present application provide a data verification method and an electronic device, which can avoid interrupting the business processing and data migration process of the source database during incremental data migration.
[0004] In a first aspect, an embodiment of the present application provides a data verification method, which comprises:
[0005] obtaining a first Merkle tree corresponding to a second version of a target data table, the first Merkle tree being generated after the target data table is updated according to update operations respectively indicated by at least one version whose version number is less than or equal to the second version;
[0006] obtaining a second Merkle tree corresponding to a third version of a source data table, the second Merkle tree being generated after the source data table is updated according to update operations respectively indicated by at least one version whose version number is less than or equal to the third version, the version number of the third version being greater than or equal to the second version;
[0007] verifying the update result of the target data table according to the first Merkle tree and the second Merkle tree.
[0008] The method further includes: obtaining a first Merkle tree corresponding to a second version of the target data table, and obtaining a second Merkle tree corresponding to a third version of the source data table, wherein the third version has a version number greater than or equal to a version number of the second version. In other words, when performing data verification, since the update operation of the target data table is lagging behind the operation on the source data table, it is only necessary to ensure that the obtained second Merkle tree corresponding to the third version of the source data table has a version number greater than or equal to the version number of the second version, so as to perform data verification. It is not necessary to force the version number of the target data table to be consistent with the version number of the source data table, and therefore, the business processing and data migration of the source data table can not be interrupted.
[0009] In a possible implementation, the obtaining the second Merkle tree corresponding to the third version of the source data table includes:
[0010] obtaining a multi-version Merkle tree of the source data table, the multi-version Merkle tree including a plurality of Merkle trees respectively corresponding to a plurality of versions of the source data table, and a maximum version number in the plurality of versions being greater than or equal to the version number of the second version;
[0011] obtaining the second Merkle tree corresponding to the third version from the multi-version Merkle tree.
[0012] The method can store the Merkle trees corresponding to the respective versions in the manner of the multi-version Merkle tree for the source data table, so as to save storage space.
[0013] In a possible implementation, the obtaining the second Merkle tree corresponding to the third version from the multi-version Merkle tree includes:
[0014] if the plurality of versions include a version having the same version number as the version number of the second version, obtaining a Merkle tree corresponding to the version having the same version number as the version number of the second version as the second Merkle tree corresponding to the third version;
[0015] if the plurality of versions do not include a version having the same version number as the version number of the second version, obtaining a version having a version number greater than the version number of the second version and having a minimum difference from the version number of the second version from the plurality of versions, and obtaining a Merkle tree corresponding to the version having the version number greater than the version number of the second version and having the minimum difference from the version number of the second version as the second Merkle tree corresponding to the third version.
[0016] The method has the advantages that if the multi-version Merkle tree has a version with the same version number as the second version, the version with the same version number is preferentially selected for comparison, thereby improving the accuracy of data verification. If the multi-version Merkle tree does not have a version with the same version number as the second version, a version with a version number greater than the second version and closest to the second version is selected for data verification. The version is relatively close, and the updated content is relatively small, facilitating subsequent verification and improving verification efficiency and accuracy.
[0017] In a possible implementation, the verifying the update result of the target data table according to the first Merkle tree and the second Merkle tree comprises:
[0018] If the first Merkle tree and the second Merkle tree are the same, it is determined that the update result of the target data table does not have an error.
[0019] If the first Merkle tree and the second Merkle tree are different, inconsistent content is determined according to the first Merkle tree and the second Merkle tree, the inconsistent content being used to represent the difference between the content after the second version indicated update operation is performed on the target data table and the content after the third version indicated update operation is performed on the source data table.
[0020] The actual update content between the second version and the third version is obtained.
[0021] If the set of inconsistent update content is a subset of the set of actual update content, it is determined that the update result of the target data table is questionable.
[0022] If the set of inconsistent update content is not a subset of the set of actual update content, it is determined that the update result of the target data table has an error.
[0023] The method respectively illustrates the data verification manner in the case that the first Merkle tree and the second Merkle tree are the same or different, thereby providing the possibility that data verification can be performed in the case that the second version of the target data table and the third version of the source data table are different, in other words, providing the possibility that data verification can be performed in the case that the second version of the target data table and the third version of the source data table are different, thereby avoiding interruption of data migration and source data table business.
[0024] In a possible implementation, the first Merkle tree corresponding to the second version is a Merkle tree corresponding to a current verification period, and the second version is a version corresponding to the most recent update operation performed on the target data table before the end time of the current verification period.
[0025] The method can perform periodic data verification on data migration, and further define the first Merkle tree of the second version in periodic verification and the definition of the second version in periodic verification. Through periodic verification, although the second version and the third version obtained in a certain verification period are different, the versions obtained in the same verification period can be the same after at least one period, the verification error can be found in time, the accuracy of verification is improved, and the business and the data migration process do not need to be interrupted.
[0026] In a possible implementation, the first Merkle tree corresponding to the second version of the target data table is obtained by:
[0027] obtaining the Merkle tree corresponding to the last verification period of the current verification period;
[0028] When there is an update operation on the target data table in the current verification period, the first Merkle tree is obtained according to the update operation on the target data table in the current verification period and the Merkle tree corresponding to the last verification period.
[0029] When there is no update operation on the target data table in the current verification period, the Merkle tree corresponding to the last verification period is determined as the first Merkle tree.
[0030] In the method, the first Merkle tree is calculated according to the update operation on the target data table in the current verification period and the Merkle tree corresponding to the last verification period, so that repeated calculation of the entire Merkle tree is avoided, and the calculation efficiency is improved.
[0031] In a possible implementation, the multi-version Merkle tree of the source data table is a multi-version Merkle tree corresponding to the current verification period, and the multi-version Merkle tree includes a plurality of Merkle trees corresponding to a plurality of versions respectively, wherein the versions are less than or equal to a first version, and the first version is a version corresponding to a latest update operation performed on the source data table before the end time of the current verification period.
[0032] The method further defines the multi-version Merkle tree of the source data table in periodic verification, that is, the latest multi-version Merkle tree of the source data table can be maintained at all times, so that storage space is saved.
[0033] In a possible implementation, the multi-version Merkle tree of the source data table is obtained by:
[0034] obtaining the multi-version Merkle tree corresponding to the last verification period of the current verification period;
[0035] when there is an update operation on the source data table in the current check period, obtaining the multi-version Merkle tree corresponding to the current check period according to the update operation on the source data table in the current check period and the multi-version Merkle tree corresponding to the last check period;
[0036] when there is no update operation on the source data table in the current check period, determining the multi-version Merkle tree corresponding to the last check period as the multi-version Merkle tree corresponding to the current check period.
[0037] The above method can obtain the multi-version Merkle tree of the source data table in the periodic check based on the multi-version Merkle tree of the last check period and the actual update operation in the current check period, thereby avoiding repeated calculation of the entire multi-version Merkle tree and improving the calculation efficiency.
[0038] In a second aspect, an electronic device is provided, including: a processor, a memory coupled to the processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions to perform:
[0039] obtaining a first Merkle tree corresponding to a second version of a target data table, the first Merkle tree being generated after the target data table is updated according to update operations respectively indicated by at least one version with a version number less than or equal to the second version;
[0040] obtaining a second Merkle tree corresponding to a third version of a source data table, the second Merkle tree being generated after the source data table is updated according to update operations respectively indicated by at least one version with a version number less than or equal to the third version, and the version number of the third version being greater than or equal to the second version;
[0041] verifying an update result of the target data table according to the first Merkle tree and the second Merkle tree.
[0042] In a third aspect, a computer storage medium is provided, including computer instructions, when the computer instructions are run on an electronic device, causing the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.
[0043] In a fourth aspect, a computer program product is provided, when the computer program product is run on an electronic device, causing the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.
[0044] It can be understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect and the computer program product provided in the fourth aspect can realize the method provided in the first aspect. Therefore, the beneficial effects achieved thereby can refer to the corresponding beneficial effects in the first aspect, which will not be described here again. BRIEF DESCRIPTION OF DRAWINGS
[0045] Figure 1 is an example diagram of a Merkel tree provided by an embodiment of the present application;
[0046] Figure 2 is an example diagram of a multi-version Merkel tree provided by an embodiment of the present application;
[0047] Figure 3 is an example diagram of a system architecture for data migration and data verification provided by an embodiment of the present application;
[0048] Figure 4 is a flowchart of a data verification method provided by an embodiment of the present application;
[0049] Figure 5 is an example of data verification provided by an embodiment of the present application;
[0050] Figure 6 is another example of data verification provided by an embodiment of the present application;
[0051] Figure 7 is a flowchart of another data verification method provided by an embodiment of the present application;
[0052] Figure 8 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0053] First, the terms related to the embodiments of the present application are described.
[0054] 1. Data migration
[0055] Data migration refers to migrating the content in a source data table in a source database to a target data table in a target database, and ensuring that the data in the source data table and the data in the target data table actually represent the same content, and the representation method and encoding format of the data in the source data table and the data in the target data table can be different. As a special case, the representation method and encoding format of the source data table and the target data table are generally different between heterogeneous databases (different databases developed using different design ideas and implementation methods).
[0056] 2. Incremental data migration
[0057] Incremental data migration is a data migration manner. In incremental data migration, since the business executed on the source database is continuously performed, various operations including insertion, deletion, update, etc. are continuously performed on the data in the source data table of the source database. The migration process is continuously performed, and these changes of the source data table also require the target data table in the target database to be migrated in near real time.
[0058] 3. Data verification
[0059] After the data is migrated from the source data table to the target data table, it is necessary to verify whether the data in the target data table is consistent with the actual content of the data in the source data table.
[0060] 4. Data verification of Merkle Tree
[0061] The Merkle Tree calculation process is shown in Figure 1 , assuming that at time t, the data table includes 5 rows of data, R1, R2, R3, R4, and R5. As shown in Figure 1 , first, the Hash values of the 5 rows of data are calculated, which are H1, H2, H3, H4, and H5, respectively. That is, the data of R1, R2, R3, R4, and R5 are input into the Hash function to obtain the Hash values H1, H2, H3, H4, and H5 corresponding to the 5 rows, respectively. In this application, H(Rx) represents Hash operation on the data of the xth row to obtain the Hash value corresponding to the xth row. For example, H(R1) represents Hash operation on row R1 to obtain the corresponding Hash value H1.
[0062] Then, the Hash values H1, H2, H3, H4, and H5 are grouped two by two, and the last H5 does not form a group with other nodes, so the last group only includes H5. Then, the Hash values of each group are input into the Hash function to calculate the Hash values of each group. As shown in Figure 1 , H1 and H2 are input into the Hash function to calculate the Hash value of H1 and H2 as H6. It should be noted that in the figure, H(H1*H2) can be understood as inputting H1 and H2 as two inputs of the Hash function to obtain the Hash value H6 of H1 and H2. Similarly, the Hash value H7 of H3 and H4 can be calculated, and the Hash value H(8) of H5 can be understood as including only one Hash value H5 in the last group of Hash values, so the other input of the Hash function can be set as a default value. The Hash values are recursively grouped and calculated in turn, and finally the Hash value of the root node, HRoot, is obtained.
[0063] The Merkle tree data verification method is an effective data verification method, which can locate the inconsistent position of the data row. For example, after some operations are performed on the source data table, the root node HRoot of the Merkle Tree obtained by calculating the source data table is HRoot{source}, and after the update content corresponding to the operation is migrated to the target data table, the root node HRoot of the Merkle Tree obtained by calculating the target data table is HRoot{target}.
[0064] If HRoot{source} = HRoot{target}, it indicates that the content represented by the source data table is consistent with the content represented by the target data table. If HRoot{source} and HRoot{target} are not equal, the Hash values of the corresponding nodes in the Merkle Tree of the source data table and the Merkle Tree of the target data table can be compared from top to bottom starting from the root node, and finally the inconsistent row or rows of data can be located, that is, the Hash values of these rows are not equal. For example, H9 in the Merkle Tree of the source data table and the Merkle Tree of the target data table is not equal, further, H6 and H7 in the Merkle Tree of the source data table and the Merkle Tree of the target data table are obtained, respectively, if H6 in the Merkle Tree of the source data table and the Merkle Tree of the target data table is equal, but H7 is not equal, further, H3 and H4 in the Merkle Tree of the source data table and the Merkle Tree of the target data table are obtained, H3 in the Merkle Tree of the source data table and the Merkle Tree of the target data table is equal, but H4 is not equal, then the content represented by R4 of the source data table and the target data table can be locked as inconsistent.
[0065] From the above data verification process, in order to realize the data verification between the source data table and the target data table, the source data table needs to pause the business processing after performing a certain operation, and wait until the content corresponding to the operation is completely updated to the target data table, then the Merkle Tree of the target data table and the source data table is obtained, and the content verification between the source data table and the target data table is further performed. Obviously, this method is not suitable for incremental data migration scenarios.
[0066] To adapt to the incremental data migration scene, the present application proposes a multi-version Merkle Tree. The multi-version Merkle Tree can be understood as that in the process of updating the data table, a new version of Merkle Tree can be continuously generated, and each Merkle Tree generated historically and currently can be stored. For example, the Merkle Tree can be generated at a certain check period, so as to store the Merkle Tree generated at each check period, thereby forming the multi-version Merkle Tree.
[0067] To save storage space, the multi-version Merkle Tree can continuously add new nodes on the multi-version Merkle Tree that has been generated, thereby forming the multi-version Merkle Tree. The following will be described in combination with Figure 2 The multi-version Merkle Tree is taken as an example. As shown in Figure 2 , the version of the Merkle tree node is represented by SCN. The historical version and the current version are saved in a chain table in the order of the version number, and the head of the chain table is the latest version (SCN maximum). For example, the row R4 is updated, and the updated row is R4', and the version before the update is SCN0, and the version after the update is SCN1. The nodes H4, H7, H9 and HRoot related to the row R4 are obtained, and the new nodes H4', H7', H9' and HRoot' of the nodes H4, H7, H9 and HRoot are added in the Merkle Tree according to the updated R4', and the version of the generated new node is defined as SCN1. As shown in Figure 2 , the new nodes of H4, H7, H9 and HRoot are sequentially added. As shown in Figure 2 It can be seen that through the multi-version Merkle Tree, Figure 2 two versions of Merkle Tree corresponding to the two versions can be obtained. It can be understood that for the Merkle Tree corresponding to the version SCN1, the nodes (such as nodes H1, H2, H3, H6) that have not changed from the version SCN0 to the version SCN1 adopt the value of the node of the last version (i.e. the version SCN0). In other words, Figure 2 the multi-version Merkle Tree includes the Merkle Tree corresponding to the version SCN0 and the Merkle Tree corresponding to the version SCN1.
[0068] The specific implementation of how to realize the data check in the incremental data migration process through the multi-version Merkle Tree can refer to the description of the subsequent embodiments, which will not be described here.
[0069] As shown in Figure 3 , a system architecture diagram for data check provided by the embodiment of the present application is shown. As shown inFigure 3 As shown in the figure, the system architecture includes a source database, a migration tool, a target database, and a data verification module. The present application migrates the update content of the source data table in the source database to the target data table in the target database. The migration tool can realize migration of the update content of the source data table to the target data table. The present application identifies the update operation implemented on the source data table in the source database by using a version number, for example, identifies the update operation on row R1 by using version SCN0, identifies the update operation on row R2 by using version SCN1, and so on. Specifically, the migration tool continuously extracts data from the source database and replays the update content of the source database to the target data table of the target database according to the version number sequence (SCN0, SCN1, …, SCNm, …, SCNn…) submitted by the source database. The replay can be understood as updating the target data table. It can be understood that for the same update content, the time of updating the target data table will lag behind the time of updating the source data table. For example, the update operation on row R2 identified by version SCN1 will lag behind the update operation on the row R2 of the source data table in the source database.
[0070] Optionally, after completing the update operation identified by a version, the source database can obtain a snapshot of the source data table as the snapshot corresponding to the version in the source data table. Similarly, after completing the update operation identified by a version, the target database can also obtain a snapshot of the target data table as the snapshot corresponding to the version in the target data table.
[0071] The data verification module is used to verify whether a migration error occurs after the content of the source data table is migrated to the target data table.
[0072] The following will be described in combination with the accompanying Figure 4 The process of data verification of the data verification module is exemplarily illustrated as follows: Figure 4 As shown in the figure, a flowchart of a data verification method provided by the embodiment of the present application is shown. The data verification method includes steps 101-107, and the execution order of each step can not be limited.
[0073] Please refer to steps 101-103 for an explanation of the update process of the multi-version Merkle Tree of the source data table in the source database:
[0074] 101, obtaining a snapshot of the source data table from the source database to construct an initial Merkle tree;
[0075] It can be understood that the snapshot can be a snapshot of the initial state of the source data table, or a snapshot obtained after a part of the update operation is performed on the source data table.
[0076] As shown in the figure, a flowchart of a data verification method provided by the embodiment of the present application is shown. The data verification method includes steps 101-107, and the execution order of each step can not be limited. Figure 4The source data table update part shown is a schematic diagram of the update of the Merkle tree of the source data table, with T as the verification period, for example, version SCN0 indicates an update to row R1; version SCN1 indicates an update to row R2; version SCN2 indicates an update to row R3; version SCN3 indicates an update to row R4; version SCN4 indicates an update to row R5; version SCN5 indicates an update to row R6; and version SCN6 indicates an update to row R7. It should be noted that versions SCN0-SCN6 can be understood as timestamps, which are identifiers of the updates to the rows in sequence over time T, but the actual data row updates can not necessarily be performed in the order of rows R1 to R7. For example, version SCN0 indicates an update to row R5, version SCN1 indicates an update to row R1, and version SCN2 indicates an update to row R6, in other words, the versions only indicate the various update operations in chronological order.
[0077] The present application can be to obtain a snapshot of the source data table after updating row R1 of the source data table, i.e., after performing the update operation indicated by version SCN0, to construct an initial Merkle tree, which is referred to as the Merkle tree corresponding to version SCN0 for ease of description.
[0078] 102. Obtain a set of modified row IDs of the source data table in the verification period from the migration tool;
[0079] 103. According to the set of modified row IDs of the source data table, obtain a corresponding partial snapshot, update the initial Merkle tree of the source data table, add a node copy, and obtain a multi-version Merkle tree.
[0080] For a verification period, which is referred to as an analysis verification period for ease of description, when the end time of the analysis verification period is reached, a version corresponding to the last completed update operation of the source data table is obtained as the version number of the analysis verification period, which is referred to as the first version for ease of description, and a snapshot corresponding to the first version is obtained, i.e., a snapshot corresponding to the first version. For example, Figure 5 As shown, in the analysis verification period [t, t+T], at the end time t+T of the analysis verification period, the source database has completed updates to rows R1, R2, and R3 of the source data table and is updating row R4, i.e., the source database has completed the update operation identified by version SCN2, so SCN2 is taken as the version number of the analysis verification period, and a snapshot corresponding to the version SCN2 is obtained, i.e., a snapshot after the update to R3 is completed in the source data table.
[0081] In an implementation, the Merkle tree corresponding to the first version can be calculated from scratch according to the snapshot corresponding to the first version.
[0082] In another implementation, in order to avoid repeatedly calculating the entire Merkle tree, the application can further obtain a modified row ID set of the source data table in the to-be-analyzed verification period, the modified row ID set including row identifiers modified in the to-be-analyzed verification period, and further obtain data of the modified row from the snapshot corresponding to the first version, so as to add corresponding nodes in the initial Merkle tree obtained in the last verification period, to obtain a multi-version Merkle tree, that is, to add on the basis of the initial Merkle tree in the last verification period, so as to avoid calculating the entire Merkle tree to improve efficiency. The adding manner can refer to the description of Figure 2 , and will not be described here. It can be understood that if the to-be-analyzed verification period is [t+T, t+2T], the multi-version Merkle tree in the last verification period [t, t+T] can be added to obtain the multi-version Merkle tree corresponding to the to-be-analyzed verification period [t+T, t+2T].
[0083] The following illustrates the updating process of obtaining the modified row ID set of the source data table in the to-be-analyzed verification period and the Merkle tree. Specifically, the version number of the last verification period of the to-be-analyzed verification period needs to be obtained. If the to-be-analyzed verification period is the first verification period, such as the verification period [t, t+T] in Figure 5 , the version number corresponding to the initial Merkle tree is SCN0. It can be understood that if the to-be-analyzed verification period is [t+T, t+2T], the last verification period is [t, t+T], and the version number of the last verification period is SCN2.
[0084] Since the migration tool can obtain the row update operation identified by each version, the source data table modified row ID set between the version number of the to-be-analyzed verification period and the version number of the last verification period of the to-be-analyzed verification period is further obtained from the migration tool.
[0085] As shown in Figure 5 , the to-be-analyzed verification period [t, t+T], the modified row ID between version SCN0 and version SCN2 of the source data table can be obtained, so as to constitute the modified row ID set. Since version SCN1 identifies updating row R2 and SCN2 identifies updating row R3, the row ID set includes row R2 and row R3, that is, row R2 and row R3 are modified in the verification period.
[0086] It can be understood that if the version number of the last check period is the same as the version number of the check period to be analyzed, it means that there is no update, and the multi-version Merkle tree does not need to be updated, and the multi-version Merkle tree of the last check period is directly used as the multi-version Merkle tree of the check period to be analyzed.
[0087] Further, in Figure 5 Continuing as an example, the updated data of rows R2 and R3 is obtained from the snapshot corresponding to version SCN2, so that the corresponding node is added in the initial Merkle tree corresponding to version SCN0, and the multi-version Merkle tree after adding the node is called the multi-version Merkle tree corresponding to version SCN2, which is also the multi-version Merkle tree of the check period [t, t+T]. The multi-version Merkle tree corresponding to version SCN2 includes the Merkle tree corresponding to SCN0 and the Merkle tree corresponding to SCN2. Similarly, for the check period [t+T, t+2T], the multi-version Merkle tree at time t+2T (i.e., the multi-version Merkle tree corresponding to the check period [t+T, t+2T]) is the multi-version Merkle tree corresponding to version SCN4, i.e., the multi-version Merkle tree is updated after performing the operation of updating row R5 identified by version SCN4. The multi-version Merkle tree corresponding to SCN4 includes the Merkle tree corresponding to SCN0, the Merkle tree corresponding to SCN2, and the Merkle tree corresponding to SCN4.
[0088] In the embodiment of the application, the data check module can update the multi-version Merkle tree of the source database once in each check period. It can be understood that if the versions of two adjacent periods are the same, the multi-version Merkle tree does not need to be updated.
[0089] It can be understood that in the above example, one check period to be analyzed is taken as an example for illustration. In the present application, the above execution steps can be performed in each check period, so as to obtain the multi-version Merkle tree of the source data table.
[0090] In the embodiment of the application, for the target data table of the target database, a multi-version Merkle tree can be used, or a multi-version Merkle tree can not be used, i.e., only the last generated Merkle tree is stored. The check period of the target data table can be the same as the check period of the source data table, i.e., the same check period is used for both. The following describes the update of the Merkle tree of the target data table in combination with steps 104-106 in Figure 4
[0091] 104, obtaining a snapshot of the target data table from the target database, and constructing an initial Merkle tree.
[0092] It is understandable that the snapshot can be a snapshot of the initial state of the target data table, or a snapshot obtained after performing some update operations on the target data table.
[0093] like Figure 5 The target data table update section shown is a schematic diagram of the Merkle tree update of the target data table, with T as the verification period. For example, version SCN0 indicates an update to row R1; version SCN1 indicates an update to row R2; version SCN2 indicates an update to row R3; version SCN3 indicates an update to row R4; version SCN4 indicates an update to row R5; version SCN5 indicates an update to row R6; and version SCN6 indicates an update to row R7.
[0094] This application can take a snapshot of the target data table after updating row R1 of the target data table, that is, after executing the update operation indicated by version SCN0, and construct an initial Merkle tree. For ease of description, this initial Merkle tree is referred to as the Merkle tree corresponding to version SCN0.
[0095] 105. Obtain the set of modified row IDs of the target data table within the verification period from the migration tool.
[0096] 106. Based on the set of modified row IDs of the target data table, obtain the corresponding snapshot data and update the Merkle tree of the target data table.
[0097] For a given verification period, we will refer to it as the verification period to be analyzed. When the end time of this verification period arrives, we retrieve the version corresponding to the most recent completed update operation on the target data table. This version is used as the version number of the verification period to be analyzed, and for ease of description, we will refer to it as the second version. It is understood that for the same verification period to be analyzed, the first version determined for the source data table and the second version determined for the target data table can be the same or different. The following will use... Figure 5 For example, for Figure 5 In the first verification cycle, for the source data table, the determined version for this verification cycle is SCN2, and for the target data table, the determined version is SCN1. In other words, at the end of verification cycle 1, the update operation identified by SCN2 has been completed for the source data table, but the update operation identified by SCN1 has only just been completed for the target data table. This is understandable. Figure 5 In the same inspection cycle, the update operation on the target data table lags behind the update operation on the source data table by one row (i.e., R3). This is just an example, and it can be any number of rows. This application does not limit it.
[0098] Further, a snapshot after the update operation identified by the second version is executed is acquired, i.e. the snapshot corresponding to the second version. As shown in FIG. 3, in the to-be-analyzed check period [t, t+T], at the end time t+T of the to-be-analyzed check period, the update of the rows R1 and R2 of the target data table has been completed, the row R3 is being updated, i.e. the target database has completed the update operation identified by the version SCN1, therefore SCN1 is taken as the version number of the to-be-analyzed check period, and the snapshot corresponding to the version SCN1 is acquired, i.e. the snapshot after the update of R2 in the target data table is completed. Figure 5
[0099] In an implementation manner, the Merkle tree of the target data table in the check period can be calculated from zero based on the snapshot corresponding to the second version.
[0100] In another implementation manner, in order to avoid repeatedly calculating the entire Merkle tree, the application can further acquire the modified row ID set of the target data table in the to-be-analyzed check period, the modified row ID set including the row identifiers modified in the to-be-analyzed check period, and further acquire the data of the modified row from the snapshot corresponding to the second version, so as to update the corresponding node in the initial Merkle tree obtained in the previous check period, thereby obtaining the Merkle tree corresponding to the second version, i.e. replacing the content of the corresponding node based on the initial Merkle tree in the previous check period, avoiding calculating the entire Merkle tree to improve the efficiency.
[0101] The update process of acquiring the modified row ID set of the target data table in the to-be-analyzed check period and the Merkle tree is exemplified below. Specifically, the version number of the previous check period of the to-be-analyzed check period needs to be acquired. If the to-be-analyzed check period is the first check period, such as the check period [t, t+T] in FIG. 3, the version number corresponding to the initial Merkle tree is SCN0. Figure 5
[0102] Since the migration tool can obtain the row operation identified by each version, the modified row ID set of the target data table between the version (i.e. the second version) of the to-be-analyzed check period and the version of the previous check period of the to-be-analyzed check period is further acquired from the migration tool.
[0103] As shown in FIG. 4, the modified row ID set of the target data table between the version SCN1 of the to-be-analyzed check period and the version SCN0 of the previous check period is acquired from the migration tool. Figure 5 As shown, the to-be-analyzed check period [t, t+T], the version SCN0 to the version SCN1 between the target data table modification row ID can be obtained, thereby constituting a modification row ID set. Since the version SCN1 identifies the update to the row R2, the row ID set includes the row R2, that is, the row R2 is modified in the check period.
[0104] It can be understood that if the version of the last check period is the same as the second version of the to-be-analyzed check period, it means that there is no update, and the Merkle tree does not need to be updated, and the Merkle tree of the last check period is directly taken as the Merkle tree of the to-be-analyzed check period. For example, Figure 5 In the check period 2 [t+T, t+2T], since the version of the check period 1 is SCN1, the version of the check period 2 is also SCN1, and the Merkle tree of the check period 1 is taken as the Merkle tree of the check period 2.
[0105] Further, as an example, Figure 5 Continuing as an example, the to-be-analyzed check period is [t, t+T], the updated data of the row R2 is obtained from the snapshot corresponding to the version SCN1, thereby updating the corresponding node of the initial Merkle tree corresponding to the version SCN0 to form a new Merkle tree, and the new Merkle tree is also the Merkle tree of the check period [t, t+T]. For the to-be-analyzed check period [t+T, t+2T], since the version of the check period is the same as the version of the last check period [t, t+T], the Merkle tree does not need to be updated, and the Merkle tree of the last check period [t, t+T] is taken as the Merkle tree of the check period [t+T, t+2T]. For the to-be-analyzed check period [t+2T, t+3T], since the version of the check period is SCN2, that is, the last update operation is performed on the operation identified by SCN2. Further, the row ID set updated between the version of the last check period [t+T, t+2T] and the version of the check period [t+2T, t+3T] is obtained, thereby updating the corresponding node on the basis of the Merkle tree of the last check period to obtain the Merkle tree of the check period [t+2T, t+3T].
[0106] Through continuous circulation, the existing Merkle tree is continuously updated, thereby replacing the existing Merkle tree, and obtaining the Merkle tree of the target data table.
[0107] The above steps 101 to 103 yield the multi-version Merkle tree for each verification period of the source data table, and steps 104 to 106 yield the Merkle tree for each verification period of the target data table. It is understood that this application can also use a multi-version Merkle tree on the target data table side. The specific generation method can refer to the generation method of the multi-version Merkle tree for each verification period of the source data table, and will not be repeated here. The data verification process is described below with an example using step 107.
[0108] 107. Perform data validation based on the Merkle tree of the target data table and the multi-version Merkle tree of the source data table.
[0109] Obtain the Merkle tree of the target data table, which can be obtained after performing the update operation identified by version two. For example... Figure 5 As shown, if data validation is performed after the end of validation period 1 and before the end of validation period 2, the Merkle tree of the target data table is updated after the update operation on row R2 identified by SCN1 is completed; the second version refers to SCN1. Similarly, if data validation is performed after the end of validation period 3 and before the end of validation period 4, the Merkle tree of the target data table is updated after the update operation on row R3 identified by SCN2 is completed; the second version refers to SCN2.
[0110] Retrieve the multi-version Merkle tree of the source data table. This multi-version Merkle tree can be obtained after performing the update operation identified by the first version. For example... Figure 5 As shown, if data validation is performed after the end of validation period 1 and before the end of validation period 2, the Merkle tree of the source data table is updated after the update operation on row R3 identified by SCN2 is completed, and the first version refers to SCN2. Similarly, if data validation is performed after the end of validation period 2 and before the end of validation period 3, the Merkle tree of the source data table is updated after the update operation on row R5 identified by SCN4 is completed, and the first version refers to SCN4.
[0111] It is understandable that for a single data verification, the Merkle tree of the target data table obtained at the time of the verification is the most recently updated Merkle tree, and the multi-version Merkle tree of the source data table obtained is also the most recently updated multi-version Merkle tree.
[0112] The multi-version Merkle tree can include a plurality of Merkle trees corresponding to a plurality of versions respectively. Further, a version whose version number is greater than or equal to the version number of the second version is determined from the plurality of versions as a third version, in other words, the version number of the third version is greater than or equal to the version number of the second version. The Merkle tree corresponding to the third version is obtained from the multi-version Merkle tree. It should be noted that if the plurality of versions includes the same version as the second version, the same version is taken as the third version, and if the plurality of versions does not include the same version as the second version, a version whose version number is greater than the version number of the second version and which is closest to the version number of the second version is taken as the third version.
[0113] The following is an example. Figure 5 For example, if the data check is performed after the end time of the check period 1 and before the end time of the check period 2, the version corresponding to the Merkle tree of the target data table is SCN1, and the multi-version Merkle tree of the source data table includes the Merkle tree corresponding to SCN0 and the Merkle tree corresponding to SCN2. A version greater than or equal to SCN1 is selected from SCN0 and SCN2 as a third version. Since there is no version identical to SCN1, SCN2 greater than SCN1 is taken as the third version, and the Merkle tree corresponding to SCN2 is obtained from the multi-version Merkle tree. It can be understood that if the data check is performed after the end time of the check period 3, the second version is SCN2, and the multi-version Merkle tree includes the Merkle trees corresponding to SCN0, SCN2, SCN4, and SCN6 respectively. SCN2 is selected as the third version from the multi-version, that is, the third version is identical to the second version.
[0114] The Merkle tree corresponding to the third version is compared with the Merkle tree of the target data table (i.e., the Merkle tree corresponding to the second version). If they are consistent, it means that the migration has no error. If they are inconsistent, it may be due to the version difference between the second version and the third version. Therefore, the inconsistent row IDs are determined according to the comparison result of the Merkle tree corresponding to the third version and the Merkle tree of the target data table, and the inconsistent row IDs form a first set.
[0115] Since it may be due to the version difference between the second version and the third version, the row IDs updated between the second version and the third version are obtained to form a second set. For example, the second version is SCN1 and the third version is SCN2. The version SCN1 to the version SCN2 updates the row R3, and therefore the row R3 forms the second set.
[0116] If the first set is a subset of the second set, the check result is suspicious, which may be caused by the inconsistency of the Merkle tree comparison result due to version difference, and the migration may not have an error. In one possible implementation, the check can be performed again after one or more check cycles. As shown in Figure 5 , if the check is performed at the end of check cycle 1, the comparison result of the Merkle tree corresponding to the third version and the Merkle tree of the target data table is inconsistent, and if the check is performed at the end of check cycle 3 after two check cycles, the comparison result may be consistent. In another possible implementation, the update processing of the source data table can be suspended, and the suspension duration needs to be long enough for the target data table to complete all data migration from the source data table to the target data table, that is, the target data table and the source data table reach the same progress. As shown in Figure 5 , if the check is performed at the end of check cycle 1 and the check result is suspicious, the update of the source data table is suspended, and the update of the target data table and the update of the source data table are waited to reach the same progress, the Merkle tree of the source data table and the Merkle tree of the target data table are updated and compared, if consistent, it is determined that the migration has no error, and if inconsistent, the migration has an error. After the Merkle tree is updated, the update operation of the source data table can be restarted, that is, the source database side business processing is restarted. In yet another possible implementation, if the check result is suspicious at a certain data check, the version of the Merkle tree of the source data table used in this check is SCNs. After the data migration tool completes all data migration of SCNs<=SCNs, the update is suspended. After the target data table is played back, the Merkle tree of the source data table and the target data table is updated to perform the check. If consistent, it is determined that the migration has no error, and if inconsistent, the migration has an error.
[0117] It should be noted that the above subset can be understood as a proper subset or a universal set. In other words, each row ID in the first set is in the second set. In Figure 6 , the first set is a universal set of the second set. In some embodiments, the first set can be a proper subset of the second set, which can be specifically referred to as the description of Figure 6 later, and is not described here in detail.
[0118] If the first set is not a subset of the second set, in other words, the first set includes a row ID that does not exist in the second set, it is determined that the data migration has an error.
[0119] Please refer to Figure 6 for a schematic diagram of data check provided by the present application, in Figure 6In the source data table update part, the multi-version Merkle tree of the source data table is updated every check period, for example, at the end of check period 1, the operations of adding rows R0-R3 and deleting row R2 have been completed, that is, the update operations indicated by versions SCN0-SCN4 have been performed, therefore, SCN4 is the version of check period 1, and the multi-version Merkle tree of the check period also includes the Merkle trees corresponding to SCN0 and SCN4.
[0120] As Figure 6 the source data table update part, the multi-version Merkle tree of the source data table is updated every check period, for example, at the end of check period 1, the operations of adding rows R0-R3 and deleting row R2 have been completed, that is, the update operations indicated by versions SCN0-SCN4 have been performed, therefore, SCN4 is the version of check period 1, and the multi-version Merkle tree of the check period also includes the Merkle trees corresponding to SCN0 and SCN4.
[0121] As Figure 7 the source data table update part, the multi-version Merkle tree of the source data table is updated every check period, for example, at the end of check period 1, the operations of adding rows R0-R3 and deleting row R2 have been completed, that is, the update operations indicated by versions SCN0-SCN4 have been performed, therefore, SCN4 is the version of check period 1, and the multi-version Merkle tree of the check period also includes the Merkle trees corresponding to SCN0 and SCN4.
[0122] If the check time is the end time of check period 2, the multi-version Merkle tree of the source data table includes the Merkle trees corresponding to SCN0, SCN4 and SCN8. Then, from SCN0, SCN4 and SCN8, a version SCN4 greater than or equal to SCN1 is selected. By comparing the Merkle tree corresponding to SCN1 and the Merkle tree corresponding to SCN4, it can be locked that the row R3 is different, but actually, the rows R2 and R3 are changed between the version SCN1 and the version SCN4, and only the last row R2 is deleted. In this scenario, the row ID set obtained by comparing the Merkle trees is a proper subset of the actual changed row ID set, that is, the first set mentioned above is a subset of the second set.
[0123] Please refer to Figure 5 , a flowchart of a data check method provided by the embodiment of the present application, in which the Merkle tree in the foregoing embodiment is replaced by a Merkle tree, as shown in the figure, the method can include but is not limited to the following steps:
[0124] 201, obtaining a first Merkle tree corresponding to a second version of a target data table, the first Merkle tree being generated by updating the target data table according to update contents indicated by at least one version whose version number is less than or equal to the second version.
[0125] The Merkle tree of the target data table can be updated continuously. For example, a check period can be set, and the Merkle tree of the target data table is updated in each check period. It can be understood that if the version of the current check period is the same as that of the last check period, the Merkle tree does not need to be updated.
[0126] In an implementation manner, the Merkle tree of the target data table can be updated on the basis of the existing Merkle tree of the target data table, and the updated Merkle tree replaces the existing Merkle tree of the target data table. In other words, the target data table can only have a latest updated Merkle tree. For details, refer to the foregoing description of the embodiments. Figure 5 For example, the first Merkle tree corresponding to the second version of the target data table can be the Merkle tree corresponding to SCN1, which is updated on the basis of the Merkle tree corresponding to SCN0, and the Merkle tree corresponding to SCN0 is updated to the Merkle tree corresponding to SCN1. The Merkle tree corresponding to SCN1 can be understood as being generated by updating the target data table according to the update content indicated by the version numbers SCN0-SCN1. In this implementation manner, the second version can be understood as the latest updated version of the target data table.
[0127] In another implementation manner, the multiple versions of the Merkle tree of the target data table can be stored respectively, or the Merkle tree corresponding to each version of the target data table can be stored respectively. In this implementation manner, the second version can be one of the multiple versions of the Merkle tree of the target data table. For example, Figure 8 As shown in FIG. 6, the Merkle tree corresponding to SCN0, the Merkle tree corresponding to SCN1, and the Merkle tree corresponding to SCN2 of the target data table can be stored respectively. The second version can be any one of SCN0, SCN1, and SCN2.
[0128] 202, obtaining multiple versions of the Merkle tree of the source data table, wherein the multiple versions of the Merkle tree include the Merkle tree corresponding to each version of the source data table;
[0129] The multiple versions of the Merkle tree of the source data table can be used to determine the Merkle tree corresponding to each version of the source data table. It can be understood that the Merkle tree corresponding to each version of the source data table can also be stored respectively, which is not limited in the present application. For details of generating the multiple versions of the Merkle tree of the source data table, refer to the foregoing description of the embodiments, which will not be repeated here.
[0130] In some embodiments, the multiple versions of the Merkle tree of the source data table can be the latest generated multiple versions of the Merkle tree.
[0131] 203, obtaining a second Merkle tree corresponding to the third version from the multi-version Merkle tree, the second Merkle tree being generated by updating the source data table according to the update content indicated by at least one version number less than or equal to the third version, the third version being a version in the multi-version whose version number is greater than the version number of the second version and has the smallest difference with the version number of the second version, or being a version whose version number is equal to the version number of the second version.
[0132] It is determined from the multi-version Merkle tree whether there is a version whose version number is equal to the version number of the second version, and if there is, it is taken as the third version. If there is no version whose version number is equal to the version number of the second version, it is further determined that the version in the multi-version whose version number is greater than the version number of the second version and has the smallest difference with the version number of the second version is taken as the third version.
[0133] The second Merkle tree corresponding to the third version is obtained from the multi-version Merkle tree.
[0134] 204, comparing the first Merkle tree with the second Merkle tree; if consistent, executing step 205, if inconsistent, executing step 206.
[0135] It is determined that the migration does not exist error.
[0136] 205, determining that the migration does not exist error.
[0137] 206, determining the inconsistent row ID according to the comparison result and constituting a first set; and obtaining the row ID actually changed between the second version and the third version and constituting a second set.
[0138] 207, if the first set is a subset of the second set, it is determined that whether the migration exists error is doubtful.
[0139] 208, if the first set is not a subset of the second set, it is determined that the migration exists error.
[0140] If the first Merkle tree and the second Merkle tree are inconsistent, the inconsistent row ID can be determined according to the comparison result, and the specific determination method can refer to the description of the foregoing embodiments, which will not be described here. The inconsistent row ID constitutes a first set. The row ID actually changed between the second version and the third version can be obtained by a migration tool, and the row ID actually changed constitutes a second set, which can refer to the description of the foregoing embodiments, which will not be described here. If the first set is a subset of the second set, it is determined that whether the migration exists error is doubtful, and the verification can be performed again after a certain verification period. If the first set is not a subset of the second set, it is determined that the migration exists error.
[0141] It can be understood that, if the second version and the third version are same in step 203, it is indicated that the target data table and the source data table are compared after being updated to the same version, and if the first Merkle tree and the second Merkle tree are inconsistent, it is indicated that the migration error is definitely occurred. Therefore, the application further determines whether the second version and the third version are same in the case that the first Merkle tree and the second Merkle tree are inconsistent, if yes, it is determined that the migration error is occurred, and if no, the actual changed row ID between the second version and the third version can be further acquired, the actual changed row ID constitutes the second set, and it is judged whether the first set is a subset of the second set, so as to determine whether the migration error is occurred.
[0142] It should be noted that each of the technical solutions (or referred to as each embodiment) of the application can be independently implemented or can be implemented based on certain internal relations. The application does not make any limitation. And the various terms and definitions between the embodiments can be mutually quoted. In each embodiment of the application, different implementations can be combined or independently implemented.
[0143] Referring to Figure 8 , Figure 8 is a structural schematic diagram of an electronic device provided by an embodiment of the application. The electronic device can be a server, a gateway device or a network device such as a base station or a router, or a terminal device such as a notebook computer, a desktop computer, a tablet computer or a mobile phone. The server can be a file server, a domain control server, a database server, a mail server, a Web server, a multimedia server, a communication server, a terminal server, an infrastructure server, a virtualization server, etc. The server can be tower type, rack type, blade type, etc. The electronic device can adopt an X86 architecture, a reduced instruction set computer (RISC) architecture, an advanced RISC machine (ARM) architecture, etc., but is not limited to this.
[0144] As Figure 8As shown, the electronic device can include, but is not limited to, a processor 810, a communication interface 820, and a memory 830, which are connected to each other through an internal bus 840. It should be understood that the electronic device 800 can be an electronic device in cloud computing, or an electronic device in an edge environment.
[0145] The processor 810 can be constituted by one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0146] The bus 840 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 840 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.
[0147] The memory 830 can include a volatile memory, such as a random access memory (RAM); the memory 830 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the memory 830 can also include a combination of the above types.
[0148] It should be noted that the memory 830 of the electronic device 800 stores Corresponding codes or computer instructions of each component of the data verification system, including the codes for the data verification module to perform data verification. The processor 810 executes these codes to realize the functions of data verification of the present application, that is, to execute the method of steps 101-107 and / or to execute the method of steps 201-208.
[0149] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, when the computer program is executed by a processor, can realize part or all steps of any one of the method embodiments described above.
[0150] The present application also provides a computer program, the computer program includes instructions, when the computer program is executed by a computer, so that the computer can execute part or all steps of any one of the methods for service scaling.
[0151] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.
[0152] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0153] The terms used in the above embodiments are only for the purpose of describing the particular embodiments, and are not intended to be a limitation of the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application means and includes any or all possible combinations of one or more listed items.
[0154] In the above embodiments, according to the context, the term "when" can be interpreted to mean "if" or "after" or "in response to determining" or "in response to detecting". Similarly, according to the context, the phrase "upon determining" or "if detecting (the stated condition or event)" can be interpreted to mean "if determining" or "in response to determining" or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)".
[0155] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like.
[0156] Those of ordinary skill in the art understand that all or part of the processes in the above embodiments can be implemented by a computer program to instruct the relevant hardware, which can be stored in a computer readable storage medium. The program can include the processes of the above method embodiments when executed. The aforementioned storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.
Claims
1. A data verification method, characterized in that, include: Obtain the first Merkle tree corresponding to the second version of the target data table. The first Merkle tree is generated after updating the target data table according to the update operations indicated by at least one version with a version number less than or equal to the second version. Obtain the second Merkle tree corresponding to the third version of the source data table. The second Merkle tree is generated after updating the source data table according to the update operations indicated by at least one version with a version number less than or equal to the third version. The version number of the third version is greater than or equal to the second version. The second Merkle tree corresponding to the third version is obtained from the multi-version Merkle tree of the source data table. The multi-version Merkle tree includes Merkle trees corresponding to multiple versions of the source data table. The largest version number among the multiple versions is greater than or equal to the version number of the second version. The update results of the target data table are verified based on the first Merkle tree and the second Merkle tree.
2. The method as described in claim 1, characterized in that, The step of obtaining the second Merkle tree corresponding to the third version of the source data table includes: Obtain the multi-version Merkle tree of the source data table; Obtain the second Merkle tree corresponding to the third version from the multi-version Merkle tree.
3. The method as described in claim 2, characterized in that, Obtaining the second Merkle tree corresponding to the third version from the multi-version Merkle tree includes: If the plurality of versions includes a version with the same version number as the second version, then the Merkle tree corresponding to the version with the same version number as the second version is obtained and used as the second Merkle tree corresponding to the third version; If there is no version with the same version number as the second version among the plurality of versions, then the version with a version number greater than the version number of the second version and the smallest difference from the version number of the second version is obtained from the plurality of versions, and the Merkle tree corresponding to the version with a version number greater than the version number of the second version and the smallest difference from the version number of the second version is obtained, and used as the second Merkle tree corresponding to the third version.
4. The method as described in claim 1, characterized in that, The step of verifying the update result of the target data table based on the first Merkle tree and the second Merkle tree includes: If the first Merkle tree and the second Merkle tree are the same, then it is determined that the update result of the target data table does not contain any errors. If the first Merkle tree and the second Merkle tree are different, then the inconsistent content is determined according to the first Merkle tree and the second Merkle tree. The inconsistent content is used to represent the difference between the content after performing the update operation indicated by the second version on the target data table and the content after performing the update operation indicated by the third version on the source data table. Obtain the actual update content between the second version and the third version; If the set of inconsistent content is a subset of the set of actual updated content, then the update result of the target data table is questionable. If the set of inconsistent content is not a subset of the set of actual updated content, then the update result of the target data table is determined to be erroneous.
5. The method according to any one of claims 1-4, characterized in that, The first Merkle tree corresponding to the second version is the Merkle tree corresponding to the current verification period, and the second version is the version corresponding to the most recent update operation performed on the target data table before the end time of the current verification period.
6. The method as described in claim 5, characterized in that, The step of obtaining the first Merkle tree corresponding to the second version of the target data table includes: Obtain the Merkle tree corresponding to the previous verification period in the current verification period; When there is an update operation on the target data table during the current verification period, the first Merkle tree is obtained based on the update operation on the target data table during the current verification period and the Merkle tree corresponding to the previous verification period. When there is no update operation on the target data table within the current verification period, the Merkle tree corresponding to the previous verification period is determined as the first Merkle tree.
7. The method according to any one of claims 2-4, characterized in that, The multi-version Merkle tree of the source data table is the multi-version Merkle tree corresponding to the current verification period. The multi-version Merkle tree includes Merkle trees corresponding to multiple versions with version numbers less than or equal to the first version. The first version is the version corresponding to the most recent update operation performed on the source data table before the end time of the current verification period.
8. The method as described in claim 7, characterized in that, The process of obtaining the multi-version Merkle tree of the source data table includes: Obtain the multi-version Merkle tree corresponding to the previous verification period in the current verification period; When there is an update operation on the source data table during the current verification period, the multi-version Merkle tree corresponding to the current verification period is obtained based on the update operation on the source data table during the current verification period and the multi-version Merkle tree corresponding to the previous verification period. When there is no update operation on the source data table within the current verification period, the multi-version Merkle tree corresponding to the previous verification period is determined as the multi-version Merkle tree corresponding to the current verification period.
9. An electronic device, characterized in that, It includes a processor and a memory, the memory being coupled to the processor; the memory is used to store computer instructions; the processor is used to execute the computer instructions. Obtain the first Merkle tree corresponding to the second version of the target data table. The first Merkle tree is generated after updating the target data table according to the update operations indicated by at least one version with a version number less than or equal to the second version. Obtain the second Merkle tree corresponding to the third version of the source data table. The second Merkle tree is generated after updating the source data table according to the update operations indicated by at least one version with a version number less than or equal to the third version. The version number of the third version is greater than or equal to the second version. The second Merkle tree corresponding to the third version is obtained from the multi-version Merkle tree of the source data table. The multi-version Merkle tree includes Merkle trees corresponding to multiple versions of the source data table. The largest version number among the multiple versions is greater than or equal to the version number of the second version. The update results of the target data table are verified based on the first Merkle tree and the second Merkle tree.
10. The electronic device as claimed in claim 9, characterized in that, The processor is specifically used to execute: Obtain the multi-version Merkle tree of the source data table; Obtain the second Merkle tree corresponding to the third version from the multi-version Merkle tree.
11. The electronic device as claimed in claim 10, characterized in that, The processor is specifically used to execute: If the plurality of versions includes a version with the same version number as the second version, then the Merkle tree corresponding to the version with the same version number as the second version is obtained and used as the second Merkle tree corresponding to the third version; If there is no version with the same version number as the second version among the plurality of versions, then the version with a version number greater than the version number of the second version and the smallest difference from the version number of the second version is obtained from the plurality of versions, and the Merkle tree corresponding to the version with a version number greater than the version number of the second version and the smallest difference from the version number of the second version is obtained, and used as the second Merkle tree corresponding to the third version.
12. The electronic device as claimed in claim 9, characterized in that, The processor is specifically used to execute: If the first Merkle tree and the second Merkle tree are the same, then it is determined that the update result of the target data table does not contain any errors. If the first Merkle tree and the second Merkle tree are different, then the inconsistent content is determined according to the first Merkle tree and the second Merkle tree. The inconsistent content is used to represent the difference between the content after performing the update operation indicated by the second version on the target data table and the content after performing the update operation indicated by the third version on the source data table. Obtain the actual update content between the second version and the third version; If the set of inconsistent content is a subset of the set of actual updated content, then the update result of the target data table is questionable. If the set of inconsistent content is not a subset of the set of actual updated content, then the update result of the target data table is determined to be erroneous.
13. The electronic device according to any one of claims 9-12, characterized in that, The first Merkle tree corresponding to the second version is the Merkle tree corresponding to the current verification period, and the second version is the version corresponding to the most recent update operation performed on the target data table before the end time of the current verification period.
14. The electronic device as claimed in claim 13, characterized in that, The processor is specifically used to execute: Obtain the Merkle tree corresponding to the previous verification period in the current verification period; When there is an update operation on the target data table during the current verification period, the first Merkle tree is obtained based on the update operation on the target data table during the current verification period and the Merkle tree corresponding to the previous verification period. When there is no update operation on the target data table within the current verification period, the Merkle tree corresponding to the previous verification period is determined as the first Merkle tree.
15. The electronic device according to any one of claims 10-12, characterized in that, The multi-version Merkle tree of the source data table is the multi-version Merkle tree corresponding to the current verification period. The multi-version Merkle tree includes Merkle trees corresponding to multiple versions with version numbers less than or equal to the first version. The first version is the version corresponding to the most recent update operation performed on the source data table before the end time of the current verification period.
16. The electronic device as claimed in claim 15, characterized in that, The processor is specifically used to execute: Obtain the multi-version Merkle tree corresponding to the previous verification period in the current verification period; When there is an update operation on the source data table during the current verification period, the multi-version Merkle tree corresponding to the current verification period is obtained based on the update operation on the source data table during the current verification period and the multi-version Merkle tree corresponding to the previous verification period. When there is no update operation on the source data table within the current verification period, the multi-version Merkle tree corresponding to the previous verification period is determined as the multi-version Merkle tree corresponding to the current verification period.
Citation Information
Patent Citations
Data synchronization method and device, storage medium and computer equipment
CN111917834A
Data verification method and device, computing equipment and storage medium
CN113626416A