Distributed data storage method and device

Abstract

According to the distributed data storage method and device, DAG topology and a hybrid consensus mechanism are fused, strong consistency verification of data integrity is achieved through combination of TrustBFT deterministic consensus and Merkle tree certification, cross-fragment asynchronous calling and a two-stage confirmation process are designed through a dynamic interaction mechanism, transaction atomicity is ensured, and the transaction efficiency is improved. Meanwhile, verification delay is controlled at a millisecond level through a self-adaptive block sorting and state check point technology, and the method is an elastic framework, supports fragment dynamic increase and decrease, committee rotation and ten-thousand-level node expansion, and remarkably improves the throughput and the attack resistance of a system, thereby solving the contradiction among efficiency, safety and expansibility of distributed storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a method and device for distributed storage of data. BACKGROUND

[0002] Distributed storage is a data storage technology that uses disk space on each machine in an enterprise through a network and forms a virtual storage device from these scattered storage resources. Data is stored in various corners of the enterprise. However, existing distributed storage generally relies on centralized nodes for coordination and management, which has a single point of failure risk, and the storage and integrity verification logic is highly coupled, making it difficult to dynamically adjust the strategy according to application requirements.

[0003] In the field of blockchain, the throughput bottleneck of single-chain architecture is significant (such as Bitcoin only 7TPS), which cannot meet the needs of high-concurrency scenarios; the DAG-based solution (such as IOTA) introduces parallel topology, but lacks an efficient integrity verification mechanism, and relies on probabilistic consensus when interacting across shards, resulting in high confirmation delay (such as OHIE confirmation time exceeding 10 minutes), and the data consistency between shards is not guaranteed. In addition, existing solutions have defects in node dynamic management and large-scale shard expansion, and cannot flexibly respond to node joining / leaving or load fluctuations, resulting in insufficient system flexibility. SUMMARY

[0004] The present application provides a method and device for distributed storage of data to solve the problem that existing data cannot be effectively verified for integrity after distributed storage.

[0005] In a first aspect, the present application provides a method for distributed storage of data, the method comprising: determining the number of shards and determining the shards corresponding to each node in the node set N, wherein each node corresponds to a shard uniquely, and the shards are distributed DAG topology, and mapping the shards of the DAG topology to a tree structure; storing the original data to the nodes of the corresponding shards according to a predetermined storage principle, and storing the data storage structure after the shards, with a Merkle tree verification path; According to the original data and its attached Merkle tree verification path, the stored data is subjected to intra-shard and inter-shard consensus to verify the integrity of the distributed storage of the data, and after the verification is passed, the storage of the data is completed.

[0006] Optionally, determining the number of shards includes: determining the number of shards based on the data storage efficiency optimization principle under the current hardware configuration condition and determining the number of nodes in each shard.

[0007] Optionally, the determining of the shard corresponding to each node in the node set N comprises: determining the shard corresponding to each node in the node set N according to the data transmission distance, the communication hop count and the data attribute; or, the remainder of the node ID / shard number. determining the shard corresponding to each node.

[0008] Optionally, the method further comprises: generating a unique node ID by a DHT algorithm , and binding the node to the shard corresponding thereto by the node ID ; and the nodes in the shard form a parallel chain, and each node in the parallel chain is assigned a storage responsibility according to the data type; the storing of the original data on the node corresponding to the shard according to the preset storage principle comprises: packing the original data into a block , the block including a transaction list , a parent block Merkle root , and a transaction Merkle root , and storing the data on the node corresponding to the shard according to the data type.

[0009] Optionally, after the determining of the shard corresponding to each node in the node set N, the method further comprises: establishing a contract relationship between the node and the shard corresponding thereto, and updating the state of the contract to after the node stores data. Each shard generates a state checkpoint periodically, writes into a consensus queue by an ElasticDAG partitioning and sorting algorithm, and stores and processes subsequent data according to the consensus queue.

[0010] Optionally, the stored data is subjected to intra-shard and inter-shard consensus, comprising: verifying the Merkle consistency with the transaction list according to a hybrid consensus protocol, updating the shard state , and broadcasting the state checkpoint.

[0011] Optionally, the method further comprises: cross-shard calling of the stored data. The cross-shard calling of the stored data comprises: proving the source data according to the called data., the target shard returns the source data after verifying the legality of the call.

[0012] Optionally, after receiving the returned data, the shard submits or rolls back the transaction through a two-phase confirmation mechanism, and the source shard updates the state after the target shard succeeds, otherwise, an exception handling logic is triggered.

[0013] In a second aspect, the present application provides a device for distributed storage of data, comprising: a processing unit configured to determine the number of shards and determine the shards corresponding to each node in the node set N, wherein each node corresponds to one shard uniquely, and the shards form a distributed DAG topology, and map the shards of the DAG topology to a tree structure; a storage unit configured to store the original data to the nodes corresponding to the shards according to a preset storage principle, and store the data structure after storing the shards, with a Merkle tree verification path; a verification unit configured to perform intra-shard and inter-shard consensus on the stored data according to the original data and the Merkle tree verification path attached thereto, to verify the integrity of the data stored in a distributed manner, and complete the storage of the data after the verification is passed.

[0014] In a third aspect, the present application provides a computer readable storage medium, wherein the storage medium stores a computer program, and the program is executed by a processor to implement any of the methods described above.

[0015] The present application has the following advantages: The present application combines DAG topology and hybrid consensus mechanism, determines consensus through TrustBFT and combines Merkle tree proof, realizes strong consistency verification of data integrity, and designs cross-shard asynchronous call and two-phase confirmation process through dynamic interaction mechanism, ensures transaction atomicity, controls verification delay to milliseconds through adaptive block sorting and state checkpointing technology, and the present application is a flexible architecture supporting dynamic increase and decrease of shards, committee rotation and ten-thousand-node expansion, significantly improves system throughput and attack resistance, and solves the contradiction between efficiency, security and scalability of distributed storage.

[0016] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments, and are not intended to limit the scope of the application. Furthermore, the drawings are not necessarily drawn to scale. In the drawings: Figure 1 is a flowchart of a method for distributed storage of data according to an embodiment of the present application; Figure 2 is a flowchart of another method for distributed storage of data according to an embodiment of the present application; Figure 3 is a structural diagram of an apparatus for distributed storage of data according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The present application will be further described by the following detailed description in conjunction with the appended drawings and examples. It should be understood that the specific embodiments described herein are intended to be merely exemplary of the application and not limiting thereof.

[0019] Existing distributed storage generally relies on centralized nodes for coordination and management, which has a single point of failure risk, and the storage and integrity verification logic are highly coupled, so it is difficult to dynamically adjust the strategy according to the application requirements.

[0020] In the field of blockchain, the throughput bottleneck of single-chain architecture is significant (such as Bitcoin only 7TPS), which cannot meet the demand of high concurrency scenarios; the DAG-based scheme (such as IOTA) introduces parallel topology, but lacks an efficient integrity verification mechanism, and relies on probabilistic consensus when interacting across shards, resulting in high confirmation delay (such as OHIE confirmation time exceeding 10 minutes), and the data consistency between shards is not guaranteed. In addition, existing solutions have defects in node dynamic management and large-scale shard expansion, and cannot flexibly respond to node joining / leaving or load fluctuations, resulting in insufficient system flexibility.

[0021] To solve the above problems, an embodiment of the present application provides a method for distributed storage of data, as shown in Figure 1 , the method comprises: S101, determining the number of shards , and determining the shards corresponding to each node in the node set N; In an embodiment of the present application, each node corresponds to a shard uniquely, and the shard is a distributed DAG topology structure, and the shard of the DAG topology structure is mapped to a tree structure; Specifically, in the present embodiment, the number of shards is determined based on the principle of optimal data storage efficiency under the current hardware configuration condition , and the number of nodes in each shard is determined.

[0022] Specifically, the shards corresponding to each node in the node set N can be determined based on data transmission distance, number of communication hops, and data attributes; or, node ID / number of shards. The remainder is used to determine the partition corresponding to each node.

[0023] In specific implementation, embodiments of the present invention can input the node set N and the number of shards. The consensus protocol parameters (such as committee size s) are then used, and all nodes join the network using a DHT algorithm (such as the Chord protocol), generating a unique node ID. Maintain the virtual node table .

[0024] initialization Parallel chain DAG instance Storage responsibilities are allocated according to data type: User data fragmentation: To determine the shard to which a node belongs, use node ID%N. Smart contract sharding: Each contract corresponds to a unique shard. Store contract code With state.

[0025] S102. According to the preset storage principle, store the original data to the corresponding sharded nodes, and attach the Merkle tree verification path to the data storage structure after sharding. The specific embodiments of the present invention are based on Split the data into the corresponding instance: Raw data packaged into blocks ,Include (Transaction list) (Parent block Merkle root) (Trade Merkle root); The state update record of the smart contract is as follows It is stored in the corresponding shard of the contract; Each shard periodically generates status checkpoints. The data is written to the consensus queue using the ElasticDAG partitioning and sorting algorithm.

[0026] In other words, the embodiments of the present invention generate unique node IDs using the DHT algorithm. and through node ID Bind nodes to their corresponding shards; and the nodes within a shard constitute... There are several parallel chains, and each node within a parallel chain is assigned storage responsibilities according to data type; then the original data is packaged into blocks. This block Including transaction list , parent block Merkle root , transaction Merkle root , and stored on the nodes of the corresponding shards according to the data type.

[0027] In specific implementation, the method according to the embodiment of the application further includes establishing a contract relationship between the node and the corresponding shard, and updating the state of the contract as after the node stores the data. Each shard generates a state checkpoint periodically, writes the state checkpoint into a consensus queue through an ElasticDAG partitioning and sorting algorithm, and stores subsequent data according to the state of the consensus queue.

[0028] S103, according to the original data and the verification path of the accompanying Merkle tree, performing intra-shard and inter-shard consensus on the stored data to verify the integrity of the distributed storage of the data, and completing the storage of the data after the verification is passed.

[0029] In the embodiment of the application, the intra-shard and inter-shard consensus on the stored data is performed according to a hybrid consensus protocol to verify the Merkle consistency of the transaction list, update the shard state , and broadcast the state checkpoint.

[0030] In specific implementation, the method according to the embodiment of the application further includes cross-shard calling of the stored data, that is, according to the proof of the called source data , routing to the target shard through an asynchronous calling mechanism, and returning the source data after the target shard verifies the legality of the calling.

[0031] It should be noted that after the shard receives the returned data, the embodiment of the application further submits or rolls back the transaction through a two-phase confirmation mechanism, and updates the state from the source shard after the target shard is successful, and if the target shard is not successful, triggers an abnormal processing logic.

[0032] In the embodiment of the application, the two-phase confirmation is to roll back within the shard and between the shards, that is, if a transaction confirmation is not successful, it will not stop at the middle stage, but return to the state before the transaction, and initiate the next state.

[0033] Overall, the embodiments of the present application achieve a technical breakthrough by decoupling the distributed storage layer and the integrity verification layer: in terms of storage, the present application relies on smart contract sharding technology, uses DHT routing to achieve fast data positioning, and allows a single node to participate in multiple shards through node virtualization, thereby improving storage resource utilization; in terms of verification, the present application combines DAG topology and hybrid consensus mechanism, combines TrustBFT deterministic consensus and Merkle tree proof to achieve strong consistency verification of data integrity; from the perspective of dynamic interaction mechanism design, the present application realizes cross-shard asynchronous calling and two-stage confirmation process (i.e. intra-shard and inter-shard consensus on stored data) to ensure transaction atomicity, while the present application controls the verification delay to the millisecond level through adaptive block sorting and state checkpointing technology; in addition, the present application adopts a flexible architecture, which enables the present application to support dynamic increase and decrease of shards, committee rotation, and ten-thousand-node expansion, significantly improving system throughput and attack resistance, and solving the contradiction between efficiency, security and scalability of existing solutions.

[0034] The embodiments of the present application will be explained and described in detail below with reference to the accompanying drawings. Figure 2 The method described in the embodiments of the present application will be explained and described in detail through a specific example: First, the present application first initializes the nodes and shards, which can specifically input a node set N, a number of shards , consensus protocol parameters (such as committee size s), and finally output a distributed storage network architecture, node and shard mapping relationship.

[0035] The full node of the embodiments of the present application is added to the network through the DHT algorithm (such as the Chord protocol) to generate a unique node ID , and maintain a virtual node table .

[0036] Initialization Parallel chain DAG instance , assign storage responsibilities according to data types: User data shards: Determine the belonging shard, node ID%N determines the belonging shard; Smart contract shard: each contract corresponds to a unique shard , stores contract code and state; The data sharding and storage strategy in the embodiments of the present application specifically includes: Input raw data D, data type label ; output the data storage structure after sharding, with a Merkle tree verification path.

[0037] Specifically, the embodiments of the present application are according to shard the data to the corresponding instance: Raw data packaged into blocks ,Include (Transaction list) (Parent block Merkle root) (Trade Merkle root).

[0038] The state update record of the smart contract is as follows It is stored in the corresponding shard of the contract; Each shard periodically generates status checkpoints. The data is written to the consensus queue using the ElasticDAG partitioning and sorting algorithm. The data writing and integrity verification in this embodiment of the invention specifically includes: Transaction initiation and routing are the processes by which users input their transactions. (including signature) ), target identifier (such as contract) Output: The original data after routing, awaiting consensus within the shard.

[0039] Light Node Broadcast To the full node, the full node resolves the target shard using the DHT routing algorithm: If it is a contract call, through Locate to fragment .

[0040] Cross-shard transactions are marked as It is temporarily stored in the buffer pool.

[0041] In this embodiment of the invention, intra-shard consensus and verification are based on the input transaction set. Current sharding status Output the confirmed block data, along with a TrustBFT signature or PoW proof of work.

[0042] Nodes within a shard run a hybrid consensus protocol: Consortium blockchain scenario: TrustBFT protocol, the committee is elected from N, and after reaching a consensus, a confirmed block is generated; In public blockchain scenarios: PoW is used to select candidate nodes, and then TrustBFT is used to confirm blocks.

[0043] Verifying block integrity is a check Consistent with the Merkle of the transaction list. Then update the shard status. And broadcast the status checkpoint.

[0044] The cross-shard interaction and dynamic management embodiments of the present invention specifically include: Cross-shard transaction processing is the input of cross-shard transactions. , source shard , target shard ; output cross-shard interaction result, ensure atomicity.

[0045] The specific application is to execute local logic by the source shard, generate cross-shard call request , with source data Merkle proof . Through the asynchronous call mechanism, it is routed to the target shard, and the target shard verifies the legality and executes the logic to return the result . The source shard receives , and submits or rolls back the transaction through a two-phase confirmation mechanism: if the target shard is successful, the source shard updates the state; otherwise, trigger the exception handling logic.

[0046] In specific implementation, the input of the node dynamic management of the embodiment of the application is the new node joining request , and the leaving node ; the output is the node list and the shard load balancing state after dynamic adjustment.

[0047] Specifically, the new node in the embodiment of the application is authenticated by PoW+TEE, synchronizes the target shard historical block and checkpoint data (block synchronization protocol based on ElasticDAG).

[0048] When the node leaves, data migration is triggered: the stored shard data is migrated to other nodes, and the DHT routing table is updated.

[0049] The method described in the embodiment of the application will be explained and described in detail below taking a decentralized e-commerce platform as an example: Suppose there is a decentralized e-commerce platform, on which users can purchase goods, and suppliers can manage goods information.

[0050] The system initialization and shard construction in the embodiment of the application specifically include: First, node and shard initialization is performed: input: node set N: suppose there are 4 initial nodes: `N1`, `N2`, `N3`, `N4`.

[0051] Shard number N chain : decide to create 3 parallel chain shards, that is, `N chain = 3`. Consensus protocol parameters: committee size `s = 3` (for TrustBFT).

[0052] Output: distributed storage network architecture: a network composed of 4 nodes and 3 shards.

[0053] Node and shard mapping relationship: 1. Full nodes join the network through DHT algorithms, generating unique node IDs: `N1`, `N2`, `N3`, `N4` join the network.

[0054] Maintain a virtual node table (similar to the Chord protocol), for example: `VirtualTable[N1] = {shard0, shard1}` `VirtualTable[N2] = {shard1, shard2}` `VirtualTable[N3] = {shard0, shard2}` `VirtualTable[N4] = {shard0, shard1, shard2}` (as a full node) 2. Initialize N chain Parallel chain DAG instances: `Instance0` (named: `Shard_UserAccounts`): mainly handles user account balances, identity information, etc.

[0055] `Instance1` (named: `Shard_Products`): mainly handles product information, inventory, etc.

[0056] `Instance2` (named: `Shard_Orders`): mainly handles order creation, payment status, etc.

[0057] Assign storage responsibilities by data type: User data shards: User ID `UserID_Alice`, if `Hash(UserID_Alice) % 3 = 0`, Alice's user data (such as account balance) is stored in `Shard_UserAccounts`.

[0058] User ID `UserID_Bob`, if `Hash(UserID_Bob) % 3 = 1`, Bob's user data is stored in `Shard_Products` (in this case Bob may be a supplier).

[0059] Smart contract shards: `Contract_Pay` (handles payment logic) -> assigned to `Shard_UserAccounts` (`shard_pay`).

[0060] `Contract_Item` (manages product details, price, inventory) -> assigned to `Shard_Products` (`shard_item`).

[0061] `Contract_Order` (creates and manages orders) -> assigned to `Shard_Orders` (`shard_order`).

[0062] Data Sharding and Storage Strategy in the Invention Embodiment: Input: Suppose a vendor Bob wants to list a new product "Smart Watch" with a price of $150 and a stock of 100.

[0063] Raw Data D: `NewProduct_SmartWatch = {ID: SW001, Name: "Smart Watch", Price: 150, Stock: 100}`.

[0064] Data Type Label Type(D): `Type(D) = "Product_Creation"`.

[0065] Output: Sharded data storage structure with Merkle tree verification path.

[0066] Detailed Steps: 1. Shard the data to the corresponding Instance according to Type(D): `Type(D)` is `"Product_Creation"`, which usually involves calling `Contract_Item`. According to the assignment in 3.1.1, `Contract_Item` is stored in `Shard_Products` (`Instance1`).

[0067] The state update record of the smart contract is StateUpdatek: Bob calls `Contract_Item.createProduct(NewProduct_SmartWatch)`.

[0068] Shard_Products generates a state update, StateUpdate_SW001, recording the creation of the new product and its initial state.

[0069] This state update is packaged into a block in Shard_Products.

[0070] The transaction data is packaged into a block Block_i: The nodes on Shard_Products package StateUpdate_SW001 and other pending product-related transactions into Block_P_1 (Product Shard Block 1).

[0071] Block_P_1 contains: TxList_P_1: contains StateUpdate_SW001.

[0072] RefRoot_P_1: references the Merkle root of the previous block (in the DAG of Shard_Products).

[0073] TxRoot_P_1: the Merkle root of TxList_P_1.

[0074] Each shard periodically generates a state checkpoint Checkpoint_i: Shard_UserAccounts, Shard_Products, Shard_Orders periodically generate Checkpoint containing a snapshot of the current state according to their internal DAG structure, and write it into their respective consensus queue, ready to be verified and recorded.

[0075] For example, Checkpoint_P_X contains the current state root of all products on Shard_Products.

[0076] The data writing and integrity verification in the embodiments of the present application specifically includes: ​Transaction initiation and routing: Input: Suppose user Alice wants to buy a "smart watch" (ID: SW001) at $150. User transaction Tx: `Tx_Alice_Buy_SW001 = {Sender: UserID_Alice, Receiver:Contract_Order.buy(SW001, $150), Sig:...}`. Target contracts: involves `Contract_Pay` (for payment), `Contract_Item` (for updating inventory), `Contract_Order` (for creating an order). Output: Routed transaction data, waiting for consensus within the shard.

[0077] 1. Light node broadcasts Tx to full nodes, full nodes resolve target shards by DHT routing algorithm: Alice's wallet (light node) broadcasts `Tx_Alice_Buy_SW001`.

[0078] Full nodes in the network (e.g. `N4`) receive this transaction and resolve that this transaction involves: Payment: Target contract `Contract_Pay`, located in `Shard_UserAccounts` (`shard0`).

[0079] Inventory update: Target contract `Contract_Item`, located in `Shard_Products` (`shard1`).

[0080] Order creation: Target contract `Contract_Order`, located in `Shard_Orders` (`shard2`).

[0081] Since it involves multiple shards, this is a cross-shard transaction.

[0082] If it's a contract call: Locate by `Hash(CID)%N_chain`.

[0083] `Hash(Contract_Pay.ID)%3 = 0` -> `Shard_UserAccounts`.

[0084] `Hash(Contract_Item.ID)%3 = 1` -> `Shard_Products`.

[0085] `Hash(Contract_Order.ID) % 3 = 2` -> `Shard_Orders`.

[0086] CrossTx is temporarily stored in BufferPool in this embodiment: `Tx_Alice_Buy_SW001` is marked as `CrossTx_Alice_SW001` and put into global `BufferPool`, waiting for cross-shard coordination processing.

[0087] In-shard consensus and verification: input: Suppose `Shard_UserAccounts` needs to process the Alice account deduction part (`Tx_Alice_Pay`), which is part of `CrossTx_Alice_SW001`.

[0088] Transaction set TxSet: `TxSet_UA = {Tx_Alice_Pay, other_transactions_UA}`.

[0089] Current shard state State_k: `State_UA_Current` (contains Alice's current balance).

[0090] Output: confirmed block data with TrustBFT signature or PoW proof of work.

[0091] 1. In-shard nodes run hybrid consensus protocol: Consortium chain scenario: Suppose `Shard_UserAccounts` adopts a consortium chain mode.

[0092] Nodes `N1`, `N3`, `N4` are selected as the committee of `Shard_UserAccounts` (`s=3`).

[0093] `N1` proposes a block `Block_UA_X` containing `TxSet_UA`.

[0094] `N3` and `N4` verify and sign, reaching TrustBFT consensus.

[0095] Generate confirmed block `ConfirmedBlock_UA_X` (contains `Tx_Alice_Pay`).

[0096] Public chain scenario: If Shard_UserAccounts is a public chain, PoW filters candidate nodes, and then TrustBFT confirms blocks. Here, consortium chain is used for simplification.

[0097] The embodiment of the application verifies the block integrity: it checks the Merkle consistency of TxRoot_i and the transaction list: the comparison between TxRoot in ConfirmedBlock_UA_X and the Merkle root of TxSet_UA ensures that all transaction data is completely included and has not been tampered with.

[0098] Update the shard state State_k: State_UA_Current <- State_UA_Current + TxSet_UA (Alice's balance is deducted).

[0099] Broadcast the new State_UA_Current as a state checkpoint Correspondingly, the embodiment of the application also provides a device for distributed storage of data, which is shown in Figure 3 The device comprises: a processing unit configured to determine the number of shards and determine the shards corresponding to each node in the node set N, wherein each node uniquely corresponds to a shard, and the shard is a distributed DAG topology, and map the shards of the DAG topology to a tree structure; a storage unit configured to store the original data to the nodes corresponding to the shards according to a preset storage principle, and store the data storage structure after the storage of the shards, with a Merkle tree verification path; a verification unit configured to perform intra-shard and inter-shard consensus on the stored data according to the original data and the Merkle tree verification path attached thereto, to verify the integrity of the distributed storage of the data, and complete the storage of the data after the verification is passed.

[0100] ​In summary, in terms of storage architecture, compared with the centralized or single-chain architecture of the prior art, the storage efficiency and scalability are limited, and since the application adopts a combination of distributed sharding and DAG topology, supports parallel storage of data and effective utilization of fork blocks, and can greatly improve the throughput and flexibility of the storage system. In terms of integrity verification, the traditional scheme usually relies on a single Merkle tree verification or probabilistic consensus mechanism, which is insufficient in terms of verification certainty and efficiency, and the application introduces a hybrid consensus mechanism (TrustBFT+PoW) and state checkpoint technology, through a deterministic consensus and layered verification process, to ensure strong consistency and efficiency of data integrity verification. For cross-shard interaction, the existing scheme often relies on a relay chain or global coordination node, resulting in high interaction delay and strong coupling, and the application proposes an asynchronous call and two-stage confirmation mechanism to decouple cross-shard logic and intra-shard logic, significantly reducing the complexity and delay of cross-shard interaction, while ensuring the atomicity of transactions. In terms of elastic expansion capability, the traditional scheme is mostly static sharding or limited to node size, and is difficult to adapt to large-scale node dynamic changes, and the application supports elastic expansion of a million-node scale through dynamic sharding adjustment, node virtualization and other technologies, effectively handles load fluctuations, and improves the scalability and resource utilization of the system.

[0101] In addition, the embodiment of the application also provides a computer readable storage medium, and the storage medium stores a computer program.

[0102] The related content of the device embodiment and the storage medium embodiment of the application can be understood by referring to the method embodiment of the application, and will not be discussed in detail here.

[0103] Although the preferred embodiments of the application have been disclosed for the purpose of example, those skilled in the art will realize that various modifications, additions and substitutions are possible, therefore, the scope of the application should not be limited to the above-mentioned embodiments.

Claims

1. A method for distributed data storage, characterized in that, The method includes: Determine the number of fragments And determine the shards corresponding to each node in the node set N, wherein each node uniquely corresponds to a shard, and the shard is a distributed DAG topology, and map the shards of the DAG topology to a tree structure; The original data is stored on the corresponding sharded nodes according to the preset storage principles, and the data storage structure after sharding is accompanied by a Merkle tree verification path. Based on the original data and its accompanying Merkle tree verification path, consensus is performed on the stored data within and between shards to verify the integrity of the distributed storage. After successful verification, the data storage is completed.

2. The method according to claim 1, characterized in that, Determine the number of fragments ,include: Under the current hardware configuration, the number of shards should be determined based on the principle of optimal data storage efficiency. And determine the number of nodes in each shard.

3. The method according to claim 1, characterized in that, The determination of the partitions corresponding to each node in the node set N includes: The fragments corresponding to each node in the node set N are determined based on the data transmission distance, the number of communication hops, and the data attributes. or, Node ID / Number of Shards The remainder is used to determine the partition corresponding to each node.

4. The method according to claim 1, characterized in that, The method further includes: generating a unique node ID using the DHT algorithm. and through node ID Bind a node to its corresponding shard; the nodes within a shard consist of... There are several parallel chains, and each node in the parallel chain is assigned storage responsibilities according to data type; The step of storing the original data to the corresponding shard nodes according to the preset storage principles includes: Pack the raw data into blocks This block Including transaction list parent block Merkle root Merkle Root Trading And store them on the corresponding shard nodes according to their data types.

5. The method according to claim 4, characterized in that, After determining the partitions corresponding to each node in the node set N, the method further includes: Establish a contract relationship between the node and its corresponding shard, and after the node stores data, update and record the state of the contract. ; Each shard periodically generates status checkpoints. The data is written to the consensus queue using the ElasticDAG partitioning and sorting algorithm, and subsequent data storage and processing are performed based on the status of the consensus queue.

6. The method according to claim 4, characterized in that, Consensus is performed on the stored data within and between shards, including: Verification based on the hybrid consensus protocol To ensure consistency with the Merkle data in the transaction list, update the shard status. And broadcast the status checkpoint.

7. The method according to claim 6, characterized in that, The method further includes: performing cross-shard calls on the stored data; The method of cross-shard access to stored data includes: Proof based on the source data invoked The call is routed to the target shard via an asynchronous call mechanism. After verifying the validity of the call, the target shard returns the source data.

8. The method according to claim 7, characterized in that, After receiving data, the shard submits or rolls back the transaction through the consensus confirmation mechanism within and between shards. The source shard updates the status after the target shard succeeds; otherwise, the exception handling logic is triggered.

9. A distributed data storage device, characterized in that, The device includes: Processing unit, used to determine the number of fragments. And determine the shards corresponding to each node in the node set N, where each node uniquely corresponds to a shard, and the shards are distributed DAG topology structures, and map the shards of the DAG topology structure to a tree structure; The storage unit is used to store the original data to the corresponding sharded nodes according to the preset storage principles, and to store the data structure after sharding, along with the Merkle tree verification path. The verification unit is used to perform intra-shard and inter-shard consensus on the stored data based on the original data and its accompanying Merkle tree verification path, so as to verify the integrity of the distributed storage of the data, and complete the storage of the data after the verification is passed.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for distributed storage of data according to any one of claims 1-8.

Images