Erasure code-based storage optimization method for decentralized storage system

Abstract

The invention discloses a decentralized storage system-oriented storage optimization method based on erasure codes, which comprises the following steps of: partitioning a file stream and organizing the file stream into a basic Merkel directed acyclic graph, carrying out erasure code coding on the partitioned file stream, and constructing an EC-Merkle DAG; leaf blocks of the EC-Merkle DAG and the basic Merkel directed acyclic graph are stored through consistent hash, and non-leaf blocks of the EC-Merkel DAG and the basic Merkel directed acyclic graph are copied and stored to all nodes; metadata consensus is carried out through a metadata consensus mechanism based on Merk-CRDT; and obtaining an identifier set of the leaf data block based on the basic Merkel directed acyclic graph, further completing file obtaining operation, and if the storage node of the leaf data block is down or the data is damaged, repairing the file. According to the method, the storage efficiency of the data in the decentralized storage scene can be improved, and the integrity and security of the data are ensured.

Description

Technical Field

[0001] The present invention belongs to the field of decentralized distributed storage optimization, and in particular relates to a storage optimization method based on erasure codes for a decentralized storage system. Background Art

[0002] The design concept of decentralized storage complies with the principles of content addressing and peer-to-peer networks, and can realize decentralized storage and transmission of data, aiming to create a global, decentralized network for storage and file sharing. In a decentralized storage network, files are divided into data blocks, supporting different data encoding formats. The default size of a data block is 256KB, and each data block has a unique CID. The final constructed file also has a corresponding CID unique identifier. Thanks to this design, each decentralized storage node will automatically deduplicate data blocks to minimize storage space usage. However, decentralized storage currently does not support erasure code fault tolerance. When storing large files, traditional multi-copy methods need to be used for fault tolerance, which will cause a lot of storage overhead. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a storage optimization method based on erasure coding for a decentralized storage system to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above objectives, the present invention provides a storage optimization method based on erasure coding for a decentralized storage system, comprising:

[0005] Obtain a file stream for block processing and organize it into a basic Merkle directed acyclic graph; construct an EC-Merkle DAG by performing erasure coding on the block-processed file stream; store the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to specific nodes based on a consistent hash ring, and store the non-leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to all nodes;

[0006] Metadata consensus is achieved through a Merkle-CRDT-based metadata consensus mechanism;

[0007] If file acquisition is performed, based on the identifier of the basic Merkle directed acyclic graph, the BitSwap algorithm is used to search whether the corresponding key-value pair exists locally. If not, the nodes that establish point-to-point communication and other nodes in the decentralized network are searched to obtain non-leaf blocks, and the identifier set of leaf data blocks is obtained based on the non-leaf blocks; the file acquisition operation is completed based on the identifier set. If the storage node of the leaf data block is down or the data is damaged, the file is repaired. During the repair process, the file acquisition operation is synchronized according to the identifier set.

[0008] Optionally, the process of obtaining the basic Merkle directed acyclic graph includes:

[0009] Obtain a file stream, cut the file stream into a number of leaf data blocks and organize them into a basic Merkle directed acyclic graph structure; wherein the leaf data blocks have the same size.

[0010] Optionally, the process of storing the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to a specific node based on the consistent hashing ring includes:

[0011] The leaf data blocks in the basic Merkle directed acyclic graph are encoded based on the erasure code to obtain the check blocks corresponding to each leaf data block; an EC-Merkle DAG is constructed based on the check blocks and the leaf data blocks; the EC-Merkle DAG and the leaf data blocks in the basic Merkle directed acyclic graph are deduplicated according to the hash values ​​of the leaf data blocks and the check blocks and the hash ring composed of the nodes in the decentralized cluster, and the deduplicated leaf data blocks and the check blocks are subjected to consistent hashing and stored in specific nodes.

[0012] Optionally, the process of constructing EC-Merkle DAG includes:

[0013] A check block generation rule is preset, and a check block is obtained according to the erasure code and the check block generation rule; a stripe is obtained based on the data block and the check block generated by one encoding, and an EC-Merkle DAG is constructed based on the stripe.

[0014] Optionally, the metadata consensus mechanism based on Merkle-CRDT includes two core components: DAG-Syncer and Broadcaster, wherein DAG-Syncer is used to synchronize the Merkle-DAG structure between nodes, and Broadcaster is used to broadcast CRDT operations.

[0015] Optionally, the process of performing file repair includes:

[0016] If the storage node of a leaf data block crashes or the data is damaged, the identifier of the corresponding EC-Merkle DAG is obtained based on the identifier of the basic Merkel directed acyclic graph, and the non-leaf node is obtained based on the identifier of the EC-Merkle DAG; the corresponding leaf data block is obtained based on the non-leaf node and the erasure code decoding operation is performed, and the leaf data block and the check block are obtained according to the erasure code decoding operation to repair the file.

[0017] Optionally, the method further includes:

[0018] Get the pinset of the decentralized storage cluster through the consensus module, traverse the pinset to analyze whether the Pin type is ClusterDAGPin and whether EC-Merkle DAG exists, determine the use of erasure coding based on the analysis results, and check the file integrity based on the use of erasure coding; if erasure coding is used, reconstruct the complete file based on the erasure coding, and re-pin the lost data blocks and check blocks to the cluster.

[0019] The present invention also discloses a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned storage optimization method based on erasure codes for decentralized storage systems.

[0020] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the storage optimization method based on erasure codes for a decentralized storage system are implemented.

[0021] The present invention also discloses a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned storage optimization method based on erasure code for a decentralized storage system are implemented.

[0022] Compared with the prior art, the present invention has the following advantages and technical effects:

[0023] The present invention truncates the file stream added to the decentralized storage in equal proportions and encodes it using erasure codes, and constructs EC-Merkle DAG based on Merkle DAG and the check blocks obtained by erasure code encoding. In the process of constructing EC-Merkle DAG, firstly, a basic Merkle DAG is constructed from the original file, and then the RootCID is consensused in the cluster of decentralized nodes through Merkle-CRDT, so that each node can restore the entire file through Merkle DAG. Then it is necessary to further construct EC-Merkle DAG, take multiple data blocks as a stripe of erasure code encoding and decoding, encode the erasure code according to a given code rate, and then insert the check blocks obtained by one encoding into the end of the encoding stripe corresponding to EC-Merkle DAG. It is then necessary to store Merkle DAG and EC-Merkle DAG. Because there are a large number of decentralized nodes, in order to balance the IO throughput of each node and give full play to the data recovery effect of the erasure code, each block of the stripe (all leaf data blocks or check blocks) needs to be stored in all nodes, and the leaf blocks are mapped to different storage nodes on the hash ring composed of cluster nodes through consistent hashing according to the number of Replicas. The storage method of non-leaf blocks is: copy the non-leaf blocks to all storage nodes. Through Merkle-CRDT, the hash ring of each node and the information stored by the node, these metadata, are reached in the cluster. By setting up a recovery node to continuously check the active nodes in the cluster and restore the data and redistribute the data when the node is down. The example of the present invention can improve the storage efficiency of data in decentralized storage scenarios and ensure the integrity and security of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A schematic diagram of the interaction between a decentralized storage client and a decentralized storage cluster service according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a file storage process according to an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of block allocation based on consistent hashing according to an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of Pin types maintained by a decentralized storage cluster according to an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of an acquisition block according to an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of file acquisition according to an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of file recovery according to an embodiment of the present invention;

[0032] Figure 8 A schematic diagram of file storage according to an embodiment of the present invention;

[0033] Fig. 9 Schematic diagram of the Merkle-CRDT consensus algorithm of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Embodiment 1

[0037] like Figure 1-5 As shown, this embodiment provides a storage optimization method based on erasure coding for a decentralized storage system, including:

[0038] Step S1: decentralized storage cluster client part;

[0039] Step S2: file storage part;

[0040] Step S3: metadata consensus part;

[0041] Step S4: file acquisition and recovery part;

[0042] Step S5: Batch restore file parts.

[0043] In step S1, the decentralized storage cluster client consists of a command line and a Client package. When the client is a command line, after the user enters the command, the command parsing unit in the command line will parse the command, identify its type (such as adding files, getting files, batch recovery files, etc.), the file path involved, and the erasure code parameters (if any), and the file transfer unit will quickly transfer the user-specified file to the Client package through the HTTP protocol in a stable and efficient manner. After receiving the command and file information, the request processing unit in the Client package will properly process and reasonably assign tasks according to the command type. For example, if it is a command to add a file using an erasure code, the erasure code and EC-Merkle DAG construction module will be called. This module first divides the file into data blocks according to 256KB, uses the erasure code algorithm (such as RS code) to generate check blocks according to the set code rate (K data blocks generate M check blocks), calculates the hash value of each block, constructs Merkle DAG and constructs EC-Merkle DAG, and then passes the relevant information to the subsequent processing unit. If it is a file acquisition task, the request processing unit will trigger the Merkle DAG and retrieval and EC-Merkle DAG decoding and retrieval modules. The decoding and retrieval module first obtains MetaPin (data structure of file metadata participating in consensus) according to the incoming CID (file unique identifier), and then obtains ClusterDAGPin (data structure containing erasure code information participating in consensus) through MetaPin's Reference, reads the CID and other metadata of EC-Merkle DAG, and then retrieves data blocks from cluster nodes accordingly. During retrieval, dual-process concurrent acquisition is adopted. One process obtains all blocks from the decentralized storage peer, and the other process obtains blocks from the first process. Once enough decoded blocks (such as K blocks in a stripe) are obtained, the decoding and recovery of the remaining blocks in the stripe begins. The two methods work together. After any one successfully obtains all blocks, the other is interrupted to save resources. After obtaining the complete file stream, it is packaged with tar and streamed to the Client. The Client processes the data stream to obtain the complete file and decompresses it to the user-specified path. If it is a batch recovery file task, the batch recovery module of the Client package is started. It first communicates with the consensus layer through the consensus proxy module to obtain the pinset (a collection of data structures participating in the consensus), then traverses each file in the pinset, checks whether the Pin type is ClusterDAGPin and whether there is erasure code metadata, so as to determine whether the file is encoded with erasure code. If it is encoded and damaged, let the decentralized storage node service obtain EC-Merkle DAG and detect it. If it is damaged, it will be restored using the erasure code algorithm, and the restored data block or check block will be re-transmitted to the surviving node through consistent hash mapping according to the original parameters and type to ensure the integrity of the file.

[0044] In step S2, the whole process of file storage is as follows: Figure 2 As shown, use the command line to add folders or files locally or on the network. First, read the file stream into the system through the HTTP protocol and enter the Chunking module, which cuts the file into data blocks of 256KB each. Next, the Layout module organizes these data blocks into the form of Merkle DAG, and then sends the data blocks to the DAGService. On the one hand, the DAGService sends the data blocks to a specific decentralized storage peer via HTTP, and on the other hand, it sends the data blocks to the RS module. After receiving the data blocks, the RS module first integrates them into blocks. When K data blocks are accumulated, encoding operations are performed. Before encoding, align the length of each data block (truncate the long block and fill it with zeros to the previous block, and fill the short block with zeros itself), then encode the new Go coroutine to obtain the check block, add the check block to the stripe, and then send the stripe to the Adder module. As shown Figure 3 As shown, the Adder module distributes multiple data blocks and check blocks to different nodes of the decentralized storage cluster through a consistent hashing algorithm. The specific details of consistent hash allocation are as follows: first create a hash ring ranging from 0 to 2^256-1 for block allocation. Assume that there are two actual nodes A and B, and each node generates three virtual nodes (A-0, A-1, A-2 and B-0, B-1, B-2); then create multiple virtual nodes on each physical storage node. The number depends on the system situation and is generally a multiple of the number of physical nodes to ensure load balancing and fault tolerance; then use the SHA-256 algorithm to calculate the hash value for each data block and check block to ensure that it is evenly distributed between 0 and 2^256-1; then map the calculated hash value to the hash ring, which is divided into several intervals, each corresponding to a virtual node; then map the virtual nodes A-0, A-1, A-2 to the physical node A, and map B-0, B-1, B-2 to the physical node B; finally, determine the corresponding virtual node based on the position of the hash value of the data block and check block on the hash ring, and then allocate them to the corresponding physical node for storage. At the same time, since RS encoding requires input blocks of equal length and unequal length blocks are padded with zeros for alignment, the original file size must be saved for correct interception when the last strip is decoded, and the file size and the order of each block are recorded during encoding and stored as metadata in ClusterDAGPin. Since RS encoding requires the input blocks to be equal in length to work, zero padding is required for unequal length blocks for alignment, so the original length of each block needs to be saved so that the data block can be correctly intercepted during RS decoding. In addition, the order of each block in the slice needs to remain unchanged during RS decoding. Therefore, the CID and order of each block need to be recorded during encoding and stored as metadata in ClusterDAGPin.

[0045] In step S3, Merkle-CRDT is responsible for persisting key metadata such as the CID of the Merkle DAG file, the CID of the ClusterPin file, the CID of the EC-Merkle DAG, and the order of data blocks and check blocks in the stripe to ClusterDAGPin, and finally saves them to each decentralized storage cluster node. Figure 4 As shown in the figure, Pin is an additional data structure maintained by the cluster, which is used to store file information that needs to participate in the consensus. There are different types, which can indicate the pinning status of the file in the cluster. The consensus layer will further maintain the file and pinset status to ensure the consistency and accuracy of metadata in the cluster. By using the characteristics of Merkle DAG and the convergence properties of CRDT, the system can achieve data integrity and eventual consistency in a decentralized environment. At the same time, it uses distributed technologies (such as DHT and PubSub) to achieve high scalability of metadata dissemination, use content addressing to ensure security, and use KV storage to ensure that the blocks stored in each node are deduplicated.

[0046] In step S4, use the decentralized storage client ecget CID command to download the file (especially when the file may be damaged). First, get the MetaPin of the file according to the incoming Merkle DAG CID, then get the ClusterDAGPin through the Reference pointer of MetaPin, read the ClusterDAGPin content to get the CID and other metadata of EC-Merkle DAG. Figure 5 As shown in the figure, when concurrently acquiring blocks, the acquisition time varies due to different storage nodes, especially when the node is down or the data block is damaged. In order to avoid wasting time on decoding after the acquisition of the file timeout, a dual-coroutine strategy is adopted. The default coroutine returns success after acquiring all blocks from the decentralized storage peer; the other coroutine acquires blocks from the default coroutine, and once it acquires enough blocks for decoding (such as K data blocks of RS code), it starts RS decoding to recover the remaining blocks in the stripe. The two methods are executed concurrently, and any one method that successfully acquires all blocks will interrupt the other to save resources. After acquiring the recovered data block or check block, it is fixed to the surviving decentralized storage peer according to the original parameters and type. There is no need to repeat the complete file addition operation, only need to re-perform consistent hash mapping and transfer the data block or check block. Finally, the file is packaged with tar and streamed to the Client. The Client processes the data stream to obtain the complete file and decompresses it to the user-specified path.

[0047] The main content of step S5 is similar to step S4, except that the pinset of the consensus module needs to be obtained at the beginning. Then, files encoded with erasure codes are selected from the pinset, and then these files are checked for block damage. If there is damage, the complete file is reconstructed using an erasure code algorithm (such as RS code), and the lost data blocks and check blocks are redistributed to the surviving nodes.

[0048] Embodiment 2

[0049] like Figure 6-9 As shown, this embodiment provides a storage optimization method based on erasure coding for a decentralized storage system, including:

[0050] Obtain a file stream for block processing and organize it into a basic Merkle directed acyclic graph; construct an EC-Merkle DAG by performing erasure coding on the block-processed file stream, store the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to specific nodes based on a consistent hash ring, and store the non-leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to all nodes;

[0051] In some specific implementations, the process of obtaining the basic Merkle directed acyclic graph includes:

[0052] Obtain a file stream, cut the file stream into a number of leaf data blocks and organize them into a basic Merkle directed acyclic graph structure; wherein the leaf data blocks have the same size.

[0053] In some specific embodiments, the process of storing the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to a specific node based on the consistent hashing ring includes:

[0054] The leaf data blocks in the basic Merkle directed acyclic graph are encoded based on the erasure code to obtain the check blocks corresponding to each leaf data block; an EC-Merkle DAG is constructed based on the check blocks and the leaf data blocks; the EC-Merkle DAG and the leaf data blocks in the basic Merkle directed acyclic graph are deduplicated according to the hash values ​​of the leaf data blocks and the check blocks and the hash ring composed of the nodes in the decentralized cluster, and the deduplicated leaf data blocks and the check blocks are subjected to consistent hashing and stored in specific nodes.

[0055] In some specific implementations, the process of constructing an EC-Merkle DAG includes:

[0056] The check block generation rule is preset, and the check block is obtained according to the erasure code and check block generation rule; the stripe is obtained based on the data block and check block generated by one encoding, and the EC-Merkle DAG is constructed based on the stripe.

[0057] Specifically, when a file needs to be stored in the cluster in the form of erasure coding, the file content will first be read into a node of the decentralized storage cluster through the HTTP protocol. Subsequently, the file will be segmented by the Chunking module according to the size of each block of 256KB, and the data blocks will be organized into a Merkle DAG structure using the Layout module. Next, the Adder module will send the leaf data block to the DAGService for further processing by the decentralized storage cluster. The DAGService will perform consistent hashing based on the hash value of the data block and the hash ring composed of the nodes in the decentralized cluster, and store the data block in the virtual node. Finally, the data block will be mapped to a certain node, and the leaf data block will be sent to a specific decentralized storage peer through the HTTP protocol, and then the non-leaf data block will be stored in all peers. At the same time, the EC-Chunking module encodes the leaf data blocks into several leaf check blocks at a fixed erasure code rate, combines the blocks involved in one encoding into a stripe, and then sends the different blocks of the entire stripe to the Adder module. This module is responsible for sending the leaf data blocks and check blocks to the DAGService for further processing by the decentralized storage cluster. The DAGService performs consistent hashing based on the hash values ​​of the data blocks and check blocks and the hash ring composed of the nodes in the decentralized cluster, and stores the data blocks and check blocks in virtual nodes. Finally, the data blocks and check blocks will be mapped to a certain node.

[0058] Metadata consensus is achieved through a Merkle-CRDT-based metadata consensus mechanism;

[0059] In some specific implementations, the metadata consensus mechanism based on Merkle-CRDT includes two core components: DAG-Syncer and Broadcaster, where DAG-Syncer is responsible for synchronizing the Merkle-DAG structure between nodes, and Broadcaster is responsible for broadcasting CRDT operations.

[0060] Specifically, the consensus method of metadata is mainly based on Merkle-CRDT implementation. Specifically, Merkle DAG is used as the transmission and persistence layer to implement CRDT (Conflict-Free Replicated Data Types), so that the system has decentralization and eventual consistency. Merkle-DAG, as a logical clock, can simplify the design and implementation of convergent data types under weak message delivery guarantees and a large number of copies. The metadata synchronization mechanism of Merkle-CRDT includes two core components: DAG-Syncer and Broadcaster. DAG-Syncer is responsible for synchronizing the Merkle-DAG structure between nodes, and Broadcaster is responsible for broadcasting CRDT operations. These two components jointly ensure the causal consistency of each object. After the erasure code is introduced, this embodiment uses DAG-Syncer consensus to each node for the non-leaf data blocks of Merkle DAG and EC-Merkle DAG, and the leaf data blocks and check blocks are mapped to the actual decentralized nodes after calculating the consistent hash through the hash ring. In terms of consistency assurance, the immutability and self-verification of Merkle-DAG ensure data integrity, and the convergence property of CRDT ensures that all nodes eventually reach a consistent state, allowing the system to operate in a loose network environment without relying on a specific messaging protocol. At the same time, the system achieves highly scalable metadata dissemination through distributed technologies such as DHT and PubSub, and utilizes the security and deduplication characteristics of content addressing. This metadata consensus mechanism based on Merkle-CRDT not only ensures data consistency, but also has good scalability and fault tolerance.

[0061] If file acquisition is performed, based on the identifier of the basic Merkle directed acyclic graph, the BitSwap algorithm is used to search whether the corresponding key-value pair exists locally. If not, the nodes that establish point-to-point communication and other nodes in the decentralized network are searched to obtain non-leaf blocks, and the identifier set of leaf data blocks is obtained based on the non-leaf blocks; the file acquisition operation is completed based on the identifier set. If the storage node of the leaf data block is down or the data is damaged, the file is repaired. During the repair process, the file acquisition operation is synchronized according to the identifier set.

[0062] In some specific implementations, the process of repairing a file includes:

[0063] If the storage node of a leaf data block crashes or the data is damaged, the identifier of the corresponding EC-Merkle DAG is obtained based on the identifier of the basic Merkel directed acyclic graph, and the non-leaf node is obtained based on the identifier of the EC-Merkle DAG; the corresponding leaf data block is obtained based on the non-leaf node and the erasure code decoding operation is performed, and the leaf data block and the check block are obtained according to the erasure code decoding operation to repair the file.

[0064] Specifically, there are two main methods for obtaining files: the first is to directly obtain the original file content through Merkle DAG, but this method can only be used when each data block of the file is well preserved; the second method is suitable for use when it is uncertain whether decentralized nodes can all serve normally. After obtaining non-leaf nodes through EC-Merkle DAG on the decentralized storage cluster, further map the hash table and the hash of the data blocks and check blocks in the leaf nodes to the physical nodes storing the corresponding data, and restore the file content through limited stripe data blocks and check blocks. When obtaining a file, you first need to pass in the rootCID of the file, then obtain MetaPin through rootCID, and then obtain ClusterDAGPin according to the Reference of MetaPin, and deserialize the metadata in CBOR format to obtain the rootCID of the EC-Merkle DAG corresponding to the file, which is called ECRootCID in this embodiment. Because Merkle DAG can be regarded as a separate file, the rootCID content can be directly obtained through DAGReader. When obtaining data blocks concurrently, because the storage nodes may involve many decentralized nodes, each node needs to establish multiple stable p2p connections with other decentralized nodes in the cluster to obtain the data of the data blocks, so the nodes that obtain the data can pass. If you wait until the acquisition of the file times out before performing erasure code decoding, a lot of time will be wasted in the case of block loss. Therefore, two methods can be used to obtain all blocks at the same time: Method 1 directly obtains all data from the decentralized storage peer through Merkle DAG, and returns success only after the entire file is obtained; Method 2 is to obtain non-leaf data blocks and check blocks through EC-Merkle DAG, and decode while obtaining. After EC-Merkle DAG is fully restored, it means that each data block and check block in the leaf block are cached locally, and then the content of the entire file can be obtained through Merkle DAG. In order to speed up the acquisition of data, you can pass a parameter to obtain the file content on the client. The two methods of obtaining files are executed concurrently. After the first method successfully obtains the file content, the context of method 2 can be interrupted to save resources; if the first method is blocked, the first method can obtain the file after method 2 recovers EC-Merkle DAG. If the file can only be obtained after it is restored through EC-Merkle DAG, the restored data blocks and check blocks can be fixed to the currently surviving decentralized peers according to the original parameters and types. Because there is no need to repeatedly fix the same data blocks and check blocks, there is no need to re-execute the complete file adding process.For different blocks, data blocks and check blocks can be mapped to different decentralized nodes on the hash ring according to consistent hashing, and the data blocks and check blocks are transmitted to the corresponding decentralized peers through persistent p2p BlockStream (point-to-point file blocks). The file is then compressed by tar and sent to the Client using streaming. After the Client processes the data stream, it obtains the complete file and decompresses it to the path specified by the user.

[0065] In some embodiments, the method further comprises:

[0066] Get the pinset of the decentralized storage cluster through the consensus module, traverse the pinset to analyze whether the Pin type is ClusterDAGPin and whether EC-Merkle DAG exists, determine the use of erasure coding based on the analysis results, and check the file integrity based on the use of erasure coding; if erasure coding is used, reconstruct the complete file based on the erasure coding, and re-pin the lost data blocks and check blocks to the cluster.

[0067] Specifically, the steps to recover all files are similar to those to recover some files, except that there is no incoming CID. Instead, the consensus module is used to obtain the pinset of the decentralized storage cluster. Then, by traversing and checking whether the Pin type is ClusterDAGPin and the existence of erasure code metadata, it is determined whether the file is encoded with erasure code, and then the integrity of the file is checked. If a file is damaged, the erasure code is used to reconstruct the complete file, and the lost data blocks and check blocks are re-pinned to the cluster.

[0068] The present invention also discloses a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the above-mentioned storage optimization method based on erasure codes for decentralized storage systems.

[0069] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the storage optimization method based on erasure codes for a decentralized storage system are implemented.

[0070] The present invention also discloses a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned storage optimization method based on erasure code for a decentralized storage system are implemented.

[0071] In the field of storage, erasure codes are another method of protecting data availability and fault tolerance in addition to multiple copies. The erasure code used in this embodiment is the Reed-Solomon code (RS code), because the RS code can adapt to any number of data disks and redundant disks, and the peers of decentralized storage can also be abstracted as disks when they are simple storage nodes, so the decentralized nature of decentralized storage is very suitable for RS code optimization. It is feasible to use erasure codes for storage in a decentralized network. Its principle is mainly to use the check block obtained by multiplying the data block by the generator matrix, and distribute data in the decentralized storage network. And when an error occurs during data transmission or storage and part of the data is lost, the original block can be restored by multiplying the inverse matrix of the generator matrix corresponding to the error-free block with the error-free block, which can adapt well to unreliable storage and transmission environments.

[0072] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A storage optimization method based on erasure coding for a decentralized storage system, characterized in that: The following steps are involved: Obtain the file stream for block processing and organize it into a basic Merkle directed acyclic graph; construct the EC-Merkle DAG by erasure coding the block-processed file stream; Based on the consistent hashing ring, the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph are stored in specific nodes, and the non-leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph are stored in all nodes; Metadata consensus is achieved through a Merkle-CRDT-based metadata consensus mechanism; If a file is to be retrieved, based on the identifier of the basic Merkle directed acyclic graph, the BitSwap algorithm is used to search whether the corresponding key-value pair exists locally. If not, the node that establishes point-to-point communication and other nodes in the decentralized network are searched to obtain non-leaf blocks, and the identifier set of the leaf data block is obtained based on the non-leaf blocks; The file acquisition operation is completed based on the identifier set. If the storage node of the leaf data block is down or the data is damaged, the file is repaired. During the repair process, the file acquisition operation is synchronously performed according to the identifier set.

2. The method according to claim 1, characterized in that The process of obtaining the basic Merkle directed acyclic graph includes: Obtain a file stream, cut the file stream into a number of leaf data blocks and organize them into a basic Merkle directed acyclic graph structure; wherein the leaf data blocks have the same size.

3. The method according to claim 2, characterized in that The process of storing the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to a specific node based on the consistent hashing ring includes: The leaf data blocks in the basic Merkle directed acyclic graph are encoded based on the erasure code to obtain the check blocks corresponding to each leaf data block; an EC-Merkle DAG is constructed based on the check blocks and the leaf data blocks; the EC-Merkle DAG and the leaf data blocks in the basic Merkle directed acyclic graph are deduplicated according to the hash values ​​of the leaf data blocks and the check blocks and the hash ring composed of the nodes in the decentralized cluster, and the deduplicated leaf data blocks and the check blocks are subjected to consistent hashing and stored in specific nodes.

4. The method according to claim 3, characterized in that The process of constructing EC-Merkle DAG includes: A check block generation rule is preset, and a check block is obtained according to the erasure code and the check block generation rule; a stripe is obtained based on the data block and the check block generated by one encoding, and an EC-Merkle DAG is constructed based on the stripe.

5. The method according to claim 1, characterized in that The metadata consensus mechanism based on Merkle-CRDT includes two core components: DAG-Syncer and Broadcaster. DAG-Syncer is used to synchronize the Merkle-DAG structure between nodes, and Broadcaster is used to broadcast CRDT operations.

6. The method according to claim 1, characterized in that The process of file repair includes: If the storage node of a leaf data block crashes or the data is damaged, the identifier of the corresponding EC-Merkle DAG is obtained based on the identifier of the basic Merkel directed acyclic graph, and the non-leaf node is obtained based on the identifier of the EC-Merkle DAG; the corresponding leaf data block is obtained based on the non-leaf node and the erasure code decoding operation is performed, and the leaf data block and the check block are obtained according to the erasure code decoding operation to repair the file.

7. The method according to claim 1, characterized in that The method further comprises: Get the pinset of the decentralized storage cluster through the consensus module, traverse the pinset to analyze whether the Pin type is ClusterDAGPin and whether EC-Merkle DAG exists, determine the use of erasure coding based on the analysis results, and check the file integrity based on the use of erasure coding; if erasure coding is used, reconstruct the complete file based on the erasure coding, and re-pin the lost data blocks and check blocks to the cluster.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the erasure code-based storage optimization method for a decentralized storage system as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the storage optimization method based on erasure coding for a decentralized storage system described in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the storage optimization method based on erasure coding for a decentralized storage system described in any one of claims 1 to 7 are implemented.

Images