Efficient data storage management method and structure based on sector division and Merkle tree

Abstract

The invention discloses an efficient data storage management method and structure based on sector division and a Merkle tree, and the method comprises the following steps: after receiving a data storage request of a user, dividing original data submitted by the user into a plurality of sectors with fixed sizes; performing Hash calculation on the sector data to construct a sub-Merkle tree, and submitting a root node and a tree height of the sub-Merkle tree as metadata to a storage contract; calculating the length of a sector array according to the height of the sub-Merkle tree, comparing the number of sectors with the length of the sector array, and ensuring that the number of the sectors is consistent with the length of the sector array in a zero filling manner; and the storage contract allocates an initial position of metadata storage according to the submitted metadata, and updates the whole Merkle tree maintained by the storage contract. By optimizing the data storage process and the structural design, the system architecture and the storage logic are simplified, and the efficiency of decentralized data storage management is improved.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to an efficient data storage management method and structure based on sector division and Merkle tree. Background Art

[0002] At present, blockchain storage solutions mainly include: fully on-chain storage solutions, hybrid solutions combining IPFS (InterPlanetary FileSystem) and blockchain indexing, and dedicated storage chain solutions.

[0003] The fully on-chain storage solution works by storing all data submitted by users directly as transaction content on the blockchain. Each full node needs to synchronize and store the complete data content, and smart contracts process and verify by directly accessing the on-chain data.

[0004] The hybrid solution of IPFS combined with blockchain index adopts the method of separating data and index, using distributed storage systems such as IPFS to store data, and blockchain only saves indexes. Users first upload data to the IPFS network, and IPFS generates CID (Content Identifier) ​​through a multi-hash algorithm, and then records CID on the blockchain.

[0005] The working process of dedicated storage chain solutions, such as Filecoin, mainly revolves around the storage proof mechanism. Storage nodes need to seal the data and use PoRep (Proof of Replication) to prove that the data is indeed stored, and regularly provide PoSt (Proof of Spacetime) to prove that the data is continuously stored.

[0006] In the process of implementing the technical method of the embodiment of the present invention, the inventor of this patent found at least the following technical problems in the prior art: In a fully on-chain storage solution, smart contracts need to process and verify complete data, resulting in high contract execution costs. As the amount of data grows, blockchain data expands rapidly, affecting the synchronization efficiency of nodes. Each full node needs to store all data, resulting in a huge waste of storage resources. The fully on-chain storage solution needs to process a large amount of data synchronization and verification logic, and the system is highly complex, resulting in high system maintenance costs and difficult upgrades.

[0007] In the hybrid solution of IPFS combined with blockchain index, since all nodes provide storage space voluntarily, there is no incentive layer and data persistence is poor. The hybrid solution of IPFS combined with blockchain index requires the maintenance of a complex distributed network, which makes the system architecture complex, resulting in high system maintenance costs and difficult upgrades.

[0008] In the dedicated storage chain solution, the entire process involves a complex zero-knowledge proof mechanism, requires specialized hardware support, has numerous system components, a complex system architecture, high maintenance costs, and difficulty in upgrading.

[0009] Furthermore, due to the high system complexity and high hardware requirements of existing blockchain storage solutions, the entry threshold for storage nodes is relatively high.

[0010] In summary, the existing blockchain storage solutions have complex system architecture, low storage efficiency, and high entry barriers for storage nodes, which cannot meet actual usage needs. Summary of the invention

[0011] The embodiments of the present invention provide an efficient data storage management method and structure based on sector division and Merkle tree, which solves the problem that the existing blockchain storage solution cannot meet the actual usage requirements.

[0012] An embodiment of the present invention provides, on one hand, an efficient data storage management method based on sector division and Merkle tree, including: After receiving the data storage request from the user, the original data submitted by the user is divided into multiple sectors of fixed size; Hash the sector data to construct a sub-Merkle tree, and submit the root node and tree height of the sub-Merkle tree as metadata to the storage contract; Calculate the sector array length based on the height of the child Merkle tree, compare the number of sectors with the sector array length, and ensure that the number of sectors is consistent with the sector array length by zero padding; The storage contract allocates the starting location of metadata storage based on the submitted metadata and updates the entire Merkle tree maintained by the storage contract.

[0013] Optionally, the original data submitted by the user is divided into multiple sectors of fixed size, including: Obtain the data size of the original data submitted by the user; The original data is divided into multiple data segments of fixed size, each data segment corresponds to a sector.

[0014] Optionally, the sector array length is calculated according to the tree height of the sub-Merkle tree, specifically: Tree height based on child Merkle trees a , through the formula l =2^( a -1), calculate the length of the sector array l .

[0015] Optionally, the entire Merkle tree maintained by the update storage contract is specifically: Update the entire Merkle tree based on the root node of the sub-Merkle tree provided by the user; or Calculate the Keccak-256 hash value of each sector and update the entire Merkle tree.

[0016] Optionally, after updating the entire Merkle tree maintained by the storage contract, also include: Receive the original data submitted by the user to the storage node of the network and store the original data.

[0017] Optionally, compare the number of sectors with the length of the sector array and ensure that the number of sectors is consistent with the length of the sector array by zero padding, including: When the number of sectors is less than the length of the sector array, additional sectors are added and the sectors are filled with data by zero filling to ensure that the number of sectors is consistent with the length of the sector array; When the number of sectors is equal to the length of the sector array, execute the subsequent steps.

[0018] Optionally, after the entire Merkle tree maintained by the updating storage contract is updated, the following steps are further included: Perform hash calculation of the sub-Merkle tree on the sector data and compare the hash calculation result with the root node in the metadata to verify the data; When the hash calculation result is consistent with the root node in the metadata, the verification is successful; When the hash calculation result is inconsistent with the root node in the metadata, it indicates that the verification has failed.

[0019] On the other hand, an embodiment of the present invention further provides an efficient data storage management structure based on sector division and Merkle tree, the structure comprising: Sector array, including multiple sectors of fixed size; A Merkle tree has a leaf node that corresponds to a sector in the sector array, and the node of the Merkle tree stores the hash value of the sector.

[0020] Optionally, the number of sectors is a power of 2.

[0021] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: The present invention divides the original data submitted by the user into sectors of fixed size, so that the storage does not need to pay attention to application details such as data type, storage size, submission time period, storage location, etc., and only needs to divide different original data into different numbers of sectors, and only use a simplest storage logic model and interface. Through the unified sector organization method, there is no need to consider complex factors, which simplifies the system architecture, helps to reduce maintenance costs, and facilitates system upgrades; the present invention adopts a strategy of separating metadata from actual data, that is, only metadata is submitted to the storage contract, and the actual data is not submitted to the storage contract, which can achieve efficient storage management; the present invention uses zero filling The method ensures that the number of sectors is consistent with the length of the sector array to form a complete sub-Merkle tree, which is not only convenient for later verification and maintenance, but also can improve the efficiency and stability of data storage management; the storage contract of the present invention maintains the root of the Merkle tree, and the metadata records the root and tree height of the Merkle tree, which can ensure accurate sector allocation and facilitate efficient verification of data status in the later stage; the system architecture of the present invention is simple, the hardware requirements are not high, and the entry threshold of storage nodes can be lowered; because the storage contract of the present invention only receives a small amount of metadata, it can allocate storage for users, which can reduce the burden on the smart contract platform and is particularly suitable for deployment and operation on the smart contract platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of an efficient data storage management method based on sector division and Merkle tree in one embodiment of the present invention; Figure 2 It is a general schematic diagram of an efficient data storage management structure based on sector division and Merkle tree in one embodiment of the present invention; Figure 3 The present invention is a verification diagram of an efficient data storage management structure based on sector division and Merkle tree in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present invention provide an efficient data storage management method and structure based on sector division and Merkle tree, which solves the problem that the existing blockchain storage solution cannot meet the actual usage requirements.

[0024] In order to better understand the above-mentioned efficient data storage management method and structure based on sector division and Merkle tree, the following will be described in detail in conjunction with the drawings and specific implementation methods of the specification. Obviously, the embodiments described in the present invention are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] An efficient data storage management method based on sector division and Merkle tree in an embodiment of the present invention includes steps S1 to S4. Figure 1 , and describe the method steps in detail.

[0026] Step S1: After receiving a data storage request from a user, the original data submitted by the user is divided into multiple sectors of fixed size.

[0027] In step S1, the original data submitted by the user is divided into multiple sectors of fixed size, which specifically includes: obtaining the data amount of the original data submitted by the user; dividing the original data into multiple data segments of fixed size, each data segment corresponds to a sector.

[0028] In the specific implementation process, for example, when a user needs to store data, a data storage request is first sent. When the storage contract receives the data storage request from the user, the original data submitted by the user is first divided into sectors of a fixed size. Among them, the size of the sector can be set to 256 bytes, which is used to store a fixed size of bytes. Of course, in the actual application process, the size of the sector can be set to other fixed sizes, such as 512 bytes, 1024 bytes, etc., which is not limited by the present invention.

[0029] like Figure 2 As shown, assuming that the size of the original data submitted by the user is 1280 bytes and the size of each sector is 256 bytes, after step S1, the original data is divided into 5 sectors of 256 bytes, and the number of sectors is 5.

[0030] Step S2: Hash the sector data to construct a sub-Merkle tree, and submit the root node and tree height of the sub-Merkle tree as metadata to the storage contract.

[0031] Continuing with the previous embodiment, in the specific implementation process, for example: for each sector of data, calculate its hash value. Hash functions (such as SHA-256) will map data of any length to a hash value of fixed length. This hash value can be regarded as a unique identifier for the sector data. Use the calculated sector hash value to build a sub-Merkle tree. The construction of the sub-Merkle tree is completed by recursively merging two hash values ​​(usually by connecting the two hash values ​​and then calculating the hash again) until only one root hash value remains. Submit the root node and tree height of the sub-Merkle tree to the storage contract for storage. Figure 2 As shown, the root node of the sub-Merkle tree corresponds to Root, and the height of the sub-Merkle tree is 4.

[0032] It should be noted that the sub-Merkle tree refers to the Merkle tree formed by hashing the sector data after the original data submitted by the user is divided into multiple sectors. The sub-Merkle tree is a temporary Merkle tree. After it is constructed, the root and height of this temporary Merkle tree are submitted to the storage contract as metadata.

[0033] Step S3: Calculate the sector array length according to the height of the child Merkle tree, compare the number of sectors with the sector array length, and ensure that the number of sectors is consistent with the sector array length by zero padding.

[0034] In step S3, the length of the sector array is calculated based on the height of the sub-Merkle tree. Specifically, a , through the formula l =2^( a -1), calculate the length of the sector array l .

[0035] Continuing with the previous embodiment, in the specific implementation process, for example: Figure 2 As shown, substituting the height of the sub-Merkle tree, 4, into the formula l=2^(a-1), the length of the sector array can be calculated l is 8.

[0036] In step S3, the number of sectors is compared with the length of the sector array, and zero-filling is used to ensure that the number of sectors is consistent with the length of the sector array, specifically including: when the number of sectors is less than the length of the sector array, an additional number of sectors are added, and the sectors are filled with data by zero-filling to ensure that the number of sectors is consistent with the length of the sector array; when the number of sectors is equal to the length of the sector array, subsequent steps are executed.

[0037] Continue to use the previous embodiment, for example: Figure 2 As shown, since the number of sectors is 5 and the length of the sector array is 8, the number of sectors 5 is less than the length of the sector array 8, 3 sectors are added, and the 3 sectors are filled with data by zero filling, so that the number of sectors and the length of the sector array are both 8.

[0038] Step S4: The storage contract allocates the starting location of metadata storage based on the submitted metadata and updates the entire Merkle tree maintained by the storage contract.

[0039] In step S4, the entire Merkle tree maintained by the storage contract is updated, specifically: according to the root node of the sub-Merkle tree provided by the user, the entire Merkle tree is updated; or the Keccak-256 hash value of each sector is calculated to update the entire Merkle tree.

[0040] After steps S1 to S4, the present invention forms an efficient data storage management structure based on sector division and Merkle tree, the structure comprising: Sector array, including multiple sectors of fixed size; A Merkle tree has a leaf node that corresponds to a sector in the sector array, and the node of the Merkle tree stores the hash value of the sector.

[0041] The number of sectors is a power of 2, forming a complete sub-Merkle tree, which facilitates later verification and maintenance and improves storage efficiency.

[0042] After updating the entire Merkle tree maintained by the storage contract in step S4, it also includes: receiving the original data submitted by the user to the storage node of the network, and storing the original data. The present invention can submit the original data to the storage contract, and the actual data is stored in the node, realizing the strategy of separating the metadata on-chain from the actual data.

[0043] After updating the entire Merkle tree maintained by the storage contract in step S4, it also includes: performing hash calculation of the sub-Merkle tree on the sector data, and comparing the hash calculation result with the root node in the metadata to verify the data; when the hash calculation result is consistent with the root node in the metadata, it indicates that the verification is successful; when the hash calculation result is inconsistent with the root node in the metadata, it indicates that the verification failed. It should be noted that the hash calculation result can only be compared with the root of the subtree corresponding to the user in the entire Merkle tree (that is, the root node in the submitted metadata).

[0044] In the blockchain storage solution, the verification process of the fully on-chain storage solution needs to process the complete data; the hybrid solution of IPFS combined with blockchain indexing requires multiple hash verifications, and the data verification mechanism is complex, requiring the implementation of a special cryptographic proof protocol. It is difficult for smart contracts to efficiently verify data integrity, and the verification process requires access to complete data, which increases the complexity of the system; the dedicated storage chain solution uses a complex storage proof mechanism, such as Filecoin's PoRep and PoSt proofs, which has cumbersome verification logic and requires special hardware acceleration. An overly complex verification mechanism not only increases the complexity of system implementation, but also makes the verification process computationally intensive, requiring special hardware support, and smart contracts need to process heavy verification logic, which is severely limited by the computing and storage resource limitations of the contract platform.

[0045] In response to this technical problem, in the data verification phase, in order to efficiently verify whether the data in the sector is complete and has not been tampered with, considering that the Merkle tree is a tree data structure, each leaf node represents the hash value of a data block, and each non-leaf node is the hash of the hash values ​​of its two child nodes. Ultimately, the root node of the tree represents the Merkle root of the entire data set, which is a unique hash value. Only the Merkle root needs to be maintained to ensure data integrity, which significantly reduces the verification cost, because the original data submitted by the user will only affect the root of its child Merkle tree. As long as the root of its child Merkle tree is well maintained, the correctness of the entire storage Merkle tree can be maintained. When the data changes, the root of its child Merkle tree will change accordingly, which is inconsistent with the Merle root of the submitted metadata, and can be quickly verified, which is simple and convenient without traversing the entire tree. By recalculating the child Merkle tree from the data block in the sector, a new Merkle root is obtained. Then compare the recalculated new Merkle root with the Merkle root stored in the storage contract. If the two are consistent, it indicates that the verification is successful, that is, the data in the sector is complete and has not been tampered with; if the two are inconsistent, it indicates that the verification has failed, that is, the data in the sector has been damaged or tampered with.

[0046] Continuing with the previous embodiment, in the specific implementation process, for example: Figure 3 As shown, when the submitted metadata is the value of Hash2, when performing data verification, it is only necessary to perform hash calculation of the sub-Merkle tree for Data5 to Data8, and compare the hash calculation result with the metadata to determine whether the data stored by the user is the original data submitted.

[0047] Existing blockchain storage solutions generally lack flexibility in data management. The location of data stored on the full chain is fixed by the order of blocks; the content addressing mechanism of the hybrid solution of IPFS combined with blockchain index binds the data location to the content; and the data allocation of the dedicated storage chain solution is subject to a complex proof mechanism. This fixed data location method limits the scalability of the system and is difficult to adapt to different application scenarios.

[0048] In order to improve the flexibility of data management and quickly locate the location where data has changed, the present invention provides two methods for quickly locating the location where data has changed.

[0049] Method 1: Compare the hash value before the change with the hash value after the change. If the hash value has not changed, the data corresponding to the sector will not change either. Based on the changed hash value, the location where the data has changed can be quickly located.

[0050] Method 2: Calculate the hash calculation result of the sector data, compare the hash calculation result with the root value of the leaf node of the sub-Merkle tree stored in the metadata, and quickly locate the location where the data has changed based on the leaf node whose hash calculation result is inconsistent with the root value of the leaf node of the sub-Merkle tree stored in the metadata.

[0051] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An efficient data storage management method based on sector division and Merkle tree, characterized in that: include: After receiving the data storage request from the user, the original data submitted by the user is divided into multiple sectors of fixed size; Hash the sector data to construct a sub-Merkle tree, and submit the root node and tree height of the sub-Merkle tree as metadata to the storage contract; Calculate the sector array length based on the height of the child Merkle tree, compare the number of sectors with the sector array length, and ensure that the number of sectors is consistent with the sector array length by zero padding; The storage contract allocates the starting location of metadata storage based on the submitted metadata and updates the entire Merkle tree maintained by the storage contract.

2. The method according to claim 1, characterized in that Divide the original data submitted by the user into multiple sectors of fixed size, including: Obtain the data size of the original data submitted by the user; The original data is divided into multiple data segments of fixed size, each data segment corresponds to a sector.

3. The method according to claim 1, characterized in that The sector array length is calculated according to the tree height of the sub-Merkle tree, specifically: Tree height based on child Merkle trees a , through the formula l =2^( a -1), calculate the length of the sector array l .

4. The method according to claim 1, wherein: The entire Merkle tree maintained by the update storage contract is specifically: Update the entire Merkle tree based on the root node of the sub-Merkle tree provided by the user; or Calculate the Keccak-256 hash value of each sector and update the entire Merkle tree.

5. The method according to claim 1, characterized in that After updating the entire Merkle tree maintained by the storage contract, it also includes: Receive the original data submitted by the user to the storage node of the network and store the original data.

6. The method according to claim 1, characterized in that Compare the number of sectors with the length of the sector array and ensure that the number of sectors is consistent with the length of the sector array by zero padding, including: When the number of sectors is less than the length of the sector array, additional sectors are added and the sectors are filled with data by zero filling to ensure that the number of sectors is consistent with the length of the sector array; When the number of sectors is equal to the length of the sector array, execute the subsequent steps.

7. The method according to claim 1, characterized in that After the entire Merkle tree maintained by the storage contract is updated, it also includes: Perform hash calculation of the sub-Merkle tree on the sector data and compare the hash calculation result with the root node in the metadata to verify the data; When the hash calculation result is consistent with the root node in the metadata, the verification is successful; When the hash calculation result is inconsistent with the root node in the metadata, it indicates that the verification has failed.

8. An efficient data storage management structure based on sector division and Merkle tree, characterized in that: The structure comprises: Sector array, including multiple sectors of fixed size; A Merkle tree has a leaf node that corresponds to a sector in the sector array, and the node of the Merkle tree stores the hash value of the sector.

9. The structure according to claim 1, characterized in that The number of sectors is a power of 2.

Images