Incremental-data identification, positioning and updating method

Abstract

The present invention belongs to the technical field of incremental-data synchronization. Provided is an incremental-data identification, positioning and updating method. The incremental-data identification, positioning and updating method based on an Ethereum MPT in the present invention uses a storage structure of an Ethereum MPT in the blockchain field and performs adaptive fine-tuning on the storage structure, and applies the hash pointer and prefix sharing characteristics of the Ethereum MPT to incremental-data identification, positioning and updating in the field of data synchronization.

Description

Incremental data identification, positioning and updating method TECHNICAL FIELD

[0001] The present application relates to the technical field of incremental data synchronization, in particular to an incremental data identification, positioning and updating method. BACKGROUND

[0002] The current mainstream incremental data synchronization methods are as follows:

[0003] Incremental data synchronization based on timestamp, this method needs to determine a fixed timestamp field, which has certain invasiveness to the data structure, and the data provider in the actual business often does not allow it. The source database running system and the client database running system may not be synchronized, which will cause errors in determining "which data is incremental" according to the timestamp, resulting in data omission or repeated synchronization.

[0004] Incremental data synchronization based on flag, this method needs to determine a fixed flag field, which has certain invasiveness to the data structure, and the data provider in the actual business often does not allow it, and the client updates the flag after synchronizing the data, which cannot meet the scene of one source to multiple ends.

[0005] Incremental data synchronization based on CDC (Change Data Capture), this method directly reads the system log of the source database, which has invasiveness to the source database, and has great difficulty in compatibility and adaptation, not all databases support CDC natively.

[0006] Full acquisition and comparison of incremental data, this method is a most traditional brute force solution, which is reliable but often unrealistic, it consumes great resources, has high time cost, poor real-time performance and low efficiency.

[0007] SUMMARY

[0008] In order to solve the above technical problems, the present application provides an incremental data identification, positioning and updating method.

[0009] The technical scheme of the present application is:

[0010] An incremental data identification, positioning and updating method, which applies the data structure of Ethereum MPT tree in the field of blockchain to the field of data synchronization after adaptive fine-tuning, realizes precise data synchronization in multiple scenes such as ensuring data consistency, one source to multiple ends, not allowing to modify the data structure of the data provider and compatible with multiple database types.

[0011] The data structure of Ethereum MPT tree in the field of blockchain is adapted and fine-tuned to the field of data synchronization, so as to realize precise data synchronization in multiple scenarios such as ensuring data consistency, one-source-to-multiple-ends, not allowing modification of data provider data structure, and compatibility with multiple database types.

[0012] The hash pointer and prefix sharing characteristics of the MPT tree are fully utilized to realize data synchronization, the hash pointer characteristics are utilized to realize incremental data identification, positioning and updating, and the prefix sharing characteristics are utilized to compress the storage space of the tree and improve the search efficiency, so as to realize second-level incremental data synchronization at an acceptable storage cost.

[0013] The hash state of the MPT tree reflects the data state of the data table, and the incremental data is identified by comparing the hash pointers of the root nodes of the MPT trees of the client and the source; the incremental data is positioned by searching and comparing the MPT trees of the client and the source; and the incremental data of the client is updated by the read-write lock mechanism and the re-verification of the hash pointer before transaction submission.

[0014] The data structure of the MPT tree is adapted and fine-tuned, a MPT primary key mapping table is designed between the MPT tree and the data table, and a customized MPT primary key generation rule is designed, which fully utilizes the prefix compression characteristics of the MPT tree, improves the search efficiency and saves the storage space, reduces the branch node to 10 elements, and increases a data primary key element in the leaf node to realize bidirectional search of the MPT primary key and the data primary key.

[0015] Through a database adaptation layer, the source data structure and the database are zero-intruded, and the query, insertion and update operations of multiple databases are realized in the adaptation layer.

[0016] Further,

[0017] When the client performs full synchronization for the first time, it acquires the source read lock first. If there is a write lock at this time, it cannot be acquired, and it will be retried after a period of time until it is acquired. The read locks between several nodes are not mutually exclusive, and several nodes can perform full synchronization at the same time.

[0018] The MPT primary key mapping table of the source is synchronized to the client.

[0019] The data table data is full-synchronized to the client.

[0020] The same rule as the source is used to build the MPT tree of the client. After the construction, the root node hash pointer of the MPT tree of the client is compared with the root node hash pointer of the MPT tree of the source. If they are consistent, the read lock is released. If they are not consistent, it means that there is a problem in the data storage during the full synchronization process, and the full synchronization needs to be performed again.

[0021] Further,

[0022] Get write lock before insert or update data, if there is a client doing full or incremental synchronization at this time, retry after a period of time until get, in order to ensure data consistency, do not allow data synchronization during writing data.

[0023] After getting the write lock, generate database insert or update transaction but do not commit, if it is data update, get the MPT primary key generated before, if it is data insert, generate new MPT primary key, then calculate the full field hash value of the data, update <MPT primary key, data full field hash value> to MPT tree, and calculate the hash pointer of the affected parent node in reverse until the root node hash pointer is updated, after updating the MPT tree, commit the insert and update transaction and release the write lock.

[0024] Further,

[0025] Client incremental data identification, positioning and updating

[0026] Incremental data identification: compare the root node hash pointer of the client MPT tree and the root node hash pointer of the source end MPT tree, if they are the same, it means that the client and the source end data are consistent, and there is no need to update, if they are different, it means that the source end has incremental data, which needs to be updated;

[0027] Incremental data positioning: when the hash pointer is different from the source end, try to get the source end read lock, if there is no insert or update operation at this time, the lock is successful, if the source end is updating data, it needs to wait, and try to get it again after waiting for a period of time;

[0028] The read locks between several nodes are not mutually exclusive, and several nodes can do incremental synchronization at the same time; after getting the read lock, the client starts to locate the incremental data, starting from the root node of the MPT tree, comparing the hash pointer of each node of the client and the source end, if they are the same, skip and do not search its child node, if they are different, continue to search its child node, until the child nodes of different nodes are leaf nodes, compare the contents of the leaf nodes of the client and the source end one by one to locate the leaf node that needs to be updated, if the client does not have the leaf node, identify it as data to be inserted, otherwise identify it as data to be updated, find the leaf node that needs to be updated, take the database primary key in the third element of the leaf node, and send a request to the source end to get the original data;

[0029] Incremental data update: first generate a database insertion update transaction, if the current data needs to be inserted, get the MPT primary key from the source end through the data primary key, otherwise directly get the MPT primary key corresponding to the data from the client, and calculate the data full field hash value, update the MPT primary key and data full field hash value to the client MPT tree in turn, and calculate the hash pointer of the affected parent node in reverse, until the root node hash pointer is updated, finally check whether it is the same as the MPT tree root node hash pointer of the source end, if the same, release the read lock, submit the client database insertion update transaction, if different, it means that the data synchronization is wrong, retry the configured number of times, if still wrong, release the read lock, mark the error state and issue a warning.

[0030] The beneficial effects of the present application are

[0031] The application applies the hash pointer and prefix sharing characteristics of the Ethereum MPT tree to incremental data synchronization to achieve the following purposes:

[0032] 1. Meet the data synchronization consistency of the source end and the client, solve the missing number problem.

[0033] 2. Support incremental data synchronization of one source to multiple ends.

[0034] 3. Adapt to the scene where the data provider in the actual business does not allow to modify the source end data structure and cannot intrude into the database.

[0035] 4. Compatible with multiple database types.

[0036] 5. Achieve second-level incremental data synchronization at an acceptable storage cost. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of the use of the three nodes of the MPT tree after fine-tuning;

[0038] Figure 2 is a schematic diagram of the sample data storage structure;

[0039] Figure 3 is a flowchart of the source end data update;

[0040] Figure 4 is a flowchart of the client identification, positioning and updating. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] To solve the above problems, a database adaptation layer is developed to avoid data structure intrusion and be compatible with multiple database types. An Ethereum MPT tree is introduced to achieve consistency of source and client data, solve the problem of missing numbers, and meet the synchronization scenario of one source to multiple clients.

[0043] MPT tree, or Merkle Patricia Tree, is a data structure that combines the features of Merkle Tree and Patricia Trie (also known as prefix compression tree). It supports the storage of <key, value> form data. In blockchain systems such as Ethereum, MPT tree is widely used to store and verify key information within blocks, especially account status and transaction data.

[0044] MPT tree has the feature of Merkle tree hash pointer connection. Except for leaf nodes, each node contains the hash value of its child node content, which means that the integrity and consistency of the entire tree structure and its content can be quickly verified through the hash of the root node. The hash state of MPT tree reflects the data state of the data table, which can quickly identify data updates during data synchronization and ensure the verifiability and credibility of the synchronization result.

[0045] The connection feature of Merkle tree hash pointer is not enough. Storing a Merkle tree will occupy a large storage space, and finding the value node through key is less efficient. MPT tree also has the feature of prefix sharing of prefix compression tree. Finding the leaf node of value through key is efficient, with time complexity of O(m), where m is the length of key. And using compression algorithm saves storage space, providing guarantee for second-level data synchronization on the basis of acceptable storage cost.

[0046] The core process of data synchronization is as follows:

[0047] (1) Source database constructs MPT tree

[0048] The source database needs to construct an MPT tree for the data table to be synchronized. The MPT tree stores data in the format of <primary key, full field hash value>. Since the primary key in the source database may be a self-incrementing primary key or a 32-bit uuid or a joint primary key, the situation is complex and it is difficult to fully utilize the prefix feature of MPT tree, which is not conducive to storage space compression and lookup performance. Therefore, a primary key mapping table is constructed to map the primary key in the source data table to the primary key in the MPT tree. The MPT primary key consists of pure digits from 0 to 9, with a fixed length of 10 bits, which can accommodate up to 10 billion data and meet the actual business requirements. When generating MPT primary keys, the same prefix principle is adopted to fully utilize the prefix sharing feature of MPT tree. When the same prefix is used up, a new prefix is used.

[0049] The MPT tree is constructed using the MTP primary key of the data as the key of the MPT tree and the data full field hash as the value, in the format of <MTP primary key, data full field hash>, so the data full field hash can be quickly found through the MTP primary key.

[0050] As shown in FIG. 1, FIG. 2 and Table 1, the original MPT tree has three kinds of nodes, branch nodes: one 17-element node [1, 2...e, f, value], leaf nodes: one two-element node <key encoding path, value>, and extension nodes: one two-element node <key encoding path, child node hash pointer>. In order to realize bidirectional lookup of the primary key mapping, that is, quickly finding the data primary key through the MTP primary key, the leaf node of the MPT tree is modified to add an element for storing the data primary key, so that the data full field hash and the data primary key can be quickly found through the MTP primary key. In order to reduce the number of branches of the MPT tree and reduce the storage space, the branch node is modified. The MPT tree branch node can store 17 elements, the first 16 elements of which are indexed as [1, 2...e, f] 16 values. The branch is too much, the MPT primary key generated by the application is a 10-bit fixed-length pure number, and the branch node is reduced to 10 elements only indexed to 10. Due to the fixed-length characteristics of the MPT primary key, the value can only exist in the leaf node, and the last element for storing the value in the branch node is deleted. The MPT tree is constructed according to the above rules. Table 1 is an example of the MPT primary key mapping table data.

[0051] Table 1:

[0052] (2) Client first full synchronization

[0053] When the client first full synchronization, the source end read lock is first acquired. If there is a write lock at this time, the source end will not be able to be acquired, and after a period of time, the retry is performed until the acquisition is completed. The read locks among multiple nodes are not mutually exclusive, and multiple nodes can simultaneously perform full synchronization.

[0054] The source end MPT primary key mapping table is synchronized to the client.

[0055] The data table data is full synchronized to the client.

[0056] Using the same rules as the source end, the client MPT tree is constructed. After the construction, the root node hash pointer of the client MPT tree and the root node hash pointer of the source end MPT tree are compared. If they are consistent, the read lock is released. If they are not consistent, it means that there is a problem in the data storage during the full synchronization process, and the full synchronization needs to be re-performed.

[0057] (3) Source end updates data

[0058] As shown in FIG. 3, a write lock is acquired before data insertion or update, if there is a client performing full or incremental synchronization at this time, the retry is performed after a period of time until acquisition, in order to ensure data consistency, data synchronization is not allowed during data writing.

[0059] After the write lock is acquired, a database insertion or update transaction is generated but not committed, if it is data update, the MPT primary key generated before is directly acquired, if it is data insertion, a new MPT primary key is generated, then the full field hash value of the data is calculated, <MPT primary key, full field hash value of the data> is updated to the MPT tree, and the hash pointer of the affected parent node is reversely calculated until the root node hash pointer is updated, after the MPT tree is updated, the insertion and update transactions are committed and the write lock is released.

[0060] (4) Client incremental data identification, positioning and updating

[0061] As shown in FIG. 4, incremental data identification: comparing the hash pointer of the root node of the client MPT tree and the hash pointer of the root node of the source MPT tree, if they are the same, it means that the data of the client and the source is consistent, and there is no need to update, if they are different, it means that there is incremental data in the source, and it needs to be updated.

[0062] Incremental data positioning: when the hash pointer is different from the source, try to acquire the source read lock, if there is no insertion or update operation in the source at this time, the lock is successful, if the source is updating data, it needs to wait, after waiting for a configured time, try to acquire again. The read locks among multiple nodes are not mutually exclusive, and multiple nodes can perform incremental synchronization at the same time. After the read lock is acquired, the client starts to position the incremental data, starting from the root node of the MPT tree, comparing the hash pointer of each node of the client and the source, if they are the same, skip and do not search the child nodes, if they are different, continue to search the child nodes, until the child nodes of the different nodes are leaf nodes, compare the contents of the leaf nodes of the client and the source one by one to locate the leaf node that needs to be updated, if the client does not have the leaf node, identify it as data to be inserted, otherwise identify it as data to be updated, after finding the leaf node that needs to be updated, take the database primary key in the third element of the leaf node, and send a request to the source to get the original data. Using this method to position incremental data, the SQL polling and full comparison method are replaced by tree comparison, the time complexity is O(logN), which can quickly locate the insertion and update of a small amount of data in a large amount of data, and reduce the query pressure on the source database.

[0063] Incremental data update: first generate a database insertion update transaction, if the current data needs to be inserted, get the MPT primary key from the source end through the data primary key, otherwise directly get the MPT primary key corresponding to the data from the client, calculate the data full field hash value, update to the client MPT tree in turn, and calculate the hash pointer of the affected parent node in reverse, until the root node hash pointer is updated, finally check whether it is the same as the MPT tree root node hash pointer of the source end, if the same, release the read lock, commit the client database insertion update transaction, if different, it means that the data synchronization is wrong, retry the configured number of times, if still wrong, release the read lock, mark the error state and issue a warning.

[0064] The application has good technical effects in scenarios that need to ensure data consistency, uses the data structure in the field of blockchains, improves data verifiability and credibility, and determines whether there is data update through quick comparison of a value of the MPT tree root node hash pointer, can greatly reduce the query pressure on the source end database, and easily proves the consistency with the source end data when the data is used, checks whether the synchronization is successful before each synchronization is completed, and has great performance improvement compared with the full data comparison mode, especially in the case of a large amount of data and irregular small updates, the performance is superior.

[0065] The application has good technical effects in one-source-to-multiple-end data synchronization, and through maintenance of the MPT tree, no matter how different the data of the client and the source end is, the same MPT tree of the source end can be compared to identify and locate the data that needs to be updated by the client respectively, until the data of the client is consistent with the data of the source end.

[0066] The application has good technical effects in scenarios that do not allow modification of the data provider data structure and heterogeneous database synchronization, and through a database adaptation layer, zero-invasion of the source system is achieved, and query, insertion and update operations of multiple databases are realized in the adaptation layer.

[0067] The above description is only the preferred embodiment of the application, which is only used to illustrate the technical scheme of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application is included in the protection scope of the application.

Claims

1. A method for identifying, locating, and updating incremental data, characterized in that: The hash pointer and prefix sharing features of the MPT tree are used to achieve data synchronization, the hash pointer feature is used to realize incremental data identification, positioning, and update, and the prefix sharing feature is used to compress the storage space of the tree and improve search efficiency.

2. The method according to claim 1, characterized in that The hash state of the MPT tree is used to reflect the data state of the data table. Incremental data identification is achieved by comparing the hash pointer of the client MPT tree root node with the hash pointer of the source MPT tree root node. Incremental data location is achieved by searching and comparing the client MPT tree and the source MPT tree; Client incremental data updates are achieved through a read-write lock mechanism and hash pointer rechecking before committing transactions.

3. The method according to claim 2, characterized in that An MPT primary key mapping table is designed between the MPT tree and the data table, and the MPT primary key generation rules are customized. The branch nodes are reduced to 10 elements, and a data primary key element is added to the leaf node to realize bidirectional search between the MPT primary key and the data primary key.

4. The method according to claim 2, characterized in that During the first full synchronization, the client first acquires a read lock on the source. If the source has a write lock, the lock cannot be acquired. The client will retry after a set interval until the lock is acquired. The read locks between multiple nodes are not mutually exclusive, and multiple nodes can perform full synchronization at the same time. Synchronize the source-side MPT primary key mapping table to the client; Synchronize all data in the data table to the client.

5. The method according to claim 4, characterized in that Use the same rules as the source end to build the client MPT tree. After building, compare the root node hash pointer of the client MPT tree with the root node hash pointer of the source MPT tree. If they are consistent, release the read lock, such as If there is inconsistency, it means that there was a problem with data storage during the full synchronization process, and full synchronization needs to be repeated.

6. The method according to claim 2, characterized in that Obtain a write lock before inserting or updating data. If a client is performing full or incremental synchronization at this time, retry after a period of time until the lock is obtained. To ensure data consistency, data synchronization is not allowed during data writing.

7. The method according to claim 6, characterized in that After acquiring the write lock, a database insert or update transaction is generated but not committed. If it is a data update, the previously generated MPT primary key is directly obtained. If it is a data insert, a new MPT primary key is generated. Then the full-field hash value of the data is calculated, the MPT primary key and the full-field hash value of the data are updated to the MPT tree, and the hash pointer of the affected parent node is reversely calculated until the root node hash pointer is updated. After updating the MPT tree, the insert and update transaction is committed and the write lock is released.

8. The method according to claim 2, characterized in that Client incremental data identification, positioning, and update: Incremental data identification: Compare the hash pointer of the client's MPT tree root node with the hash pointer of the source's MPT tree root node. If they are the same, it indicates that the client and source data are consistent and do not need to be updated. If they are different, it indicates that the source has incremental data and needs to be updated. Incremental data positioning: When the hash pointer differs from the source, the system attempts to acquire a read lock on the source. If the source is not performing any insert or update operations, the lock succeeds. If the source is updating data, the system needs to wait for a configured time before retrying to acquire the lock. The read locks between several nodes are not mutually exclusive, and several nodes can perform incremental synchronization at the same time. After obtaining the read lock, the client starts to locate the incremental data, starting from the root node of the MPT tree, and compares the hash pointers of each node on the client and the source. If they are the same, the search for their child nodes is skipped. If they are different, the search for their child nodes is continued until the child nodes of the different nodes are leaf nodes. The client and source leaf nodes are compared one by one. The content of the point is located at the leaf node that needs to be updated. If the client does not have this leaf node, it is identified as data to be inserted. Otherwise, it is identified as data to be updated. After finding the leaf node to be updated, the database primary key in the third element of the leaf node is obtained, and a request is sent to the source end to obtain the original data. Incremental data update: First, generate a database insert update transaction. If the current data needs to be inserted, obtain the MPT primary key from the source through the data primary key. Otherwise, obtain the MPT primary key corresponding to the data directly from the client, and calculate the hash value of the data's full fields. Update the MPT primary key and the hash value of the data's full fields to the client's MPT tree in sequence, and reversely calculate the hash pointer of the affected parent node until the root node hash pointer is updated. Finally, check whether it is the same as the MPT tree root node hash pointer on the source side. If they are the same, release the read lock and submit the client database insert update transaction. If they are different, it means that the data synchronization is wrong. If the error still occurs after the configured number of retries, release the read lock, mark the error status, and issue an early warning.

Images