A method for erasure code-based storage optimization for a decentralized storage system

Abstract

The application discloses a kind of based on erasure code's storage optimization method for decentralized storage system, comprising: file stream is divided into blocks and is organized as basic Merkle directed acyclic graph, and the file stream of block is encoded with erasure code, and EC-Merkle DAG is constructed;The leaf block of EC-Merkle DAG and basic Merkle directed acyclic graph is stored by consistent hashing, and the non-leaf block of both is stored to all nodes by replication;Metadata consensus is carried out by the metadata consensus mechanism based on Merkle-CRDT;The identifier set of leaf data block is obtained based on basic Merkle directed acyclic graph, and further file acquisition operation is completed, if there is the storage node of leaf data block downtime or data damage, then file repair is carried out.The application can improve the storage efficiency of data in decentralized storage scene, and ensure the integrity and security of data.

Description

Technical Field

[0001] The present invention belongs to the field of decentralized distributed storage optimization, and in particular relates to a storage optimization method based on erasure codes for decentralized storage systems. Background Art

[0002] The design concept of decentralized storage aligns with the principles of content addressing and peer-to-peer networking, enabling decentralized data storage and transmission. It aims to create a global, decentralized network for storing and sharing files. In a decentralized storage network, files are segmented into data blocks, supporting various data encoding formats. The default block size is 256KB, and each block has a unique CID. The resulting file is also uniquely identified by a corresponding CID. Thanks to this design, each decentralized storage node automatically deduplicates data blocks, minimizing storage space usage. However, decentralized storage currently does not support erasure coding fault tolerance. When storing large files, traditional multiple copies are required for fault tolerance, which results in significant storage overhead. Summary of the Invention

[0003] To solve the above technical problems, the present invention proposes a storage optimization method based on erasure codes for decentralized storage systems to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above objectives, the present invention provides a storage optimization method based on erasure coding for a decentralized storage system, comprising:

[0005] Obtain a file stream, segment it into blocks, and organize it into a basic Merkle directed acyclic graph (DAG). Construct an EC-Merkle DAG by erasure coding the segmented file stream. Store the leaf blocks of the EC-Merkle DAG and the basic Merkle DAG at specific nodes based on a consistent hashing ring, while storing the non-leaf blocks of the EC-Merkle DAG and the basic Merkle DAG at all nodes.

[0006] Metadata consensus is achieved through a Merkle-CRDT-based metadata consensus mechanism;

[0007] If file acquisition is performed, based on the identifier of the basic Merkle directed acyclic graph, the BitSwap algorithm is used to search whether the corresponding key-value pair exists locally. If not, the nodes that establish point-to-point communication and other nodes in the decentralized network are searched to obtain non-leaf blocks, and the identifier set of the leaf data block is obtained based on the non-leaf block; the file acquisition operation is completed based on the identifier set. If the storage node of the leaf data block is down or the data is damaged, the file is repaired. During the repair process, the file acquisition operation is synchronized according to the identifier set.

[0008] Optionally, the process of obtaining the basic Merkle directed acyclic graph includes:

[0009] Obtain a file stream, truncate the file stream into several leaf data blocks, and organize them into a basic Merkle directed acyclic graph structure; wherein the leaf data blocks have the same size.

[0010] Optionally, the process of storing leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to specific nodes based on the consistent hashing ring includes:

[0011] The leaf data blocks in the basic Merkle directed acyclic graph are encoded based on the erasure code to obtain the check blocks corresponding to each leaf data block; an EC-Merkle DAG is constructed based on the check blocks and the leaf data blocks; the leaf data blocks in the EC-Merkle DAG and the basic Merkle directed acyclic graph are deduplicated according to the hash values ​​of the leaf data blocks and the check blocks and the hash ring composed of the nodes in the decentralized cluster; the deduplicated leaf data blocks and the check blocks are subjected to consistent hashing and stored in specific nodes.

[0012] Optionally, the process of constructing an EC-Merkle DAG includes:

[0013] A check block generation rule is preset, and a check block is obtained according to the erasure code and the check block generation rule; a stripe is obtained based on the data block and the check block generated by one encoding, and an EC-Merkle DAG is constructed based on the stripe.

[0014] Optionally, the Merkle-CRDT-based metadata consensus mechanism includes two core components: DAG-Syncer and Broadcaster, where DAG-Syncer is used to synchronize the Merkle-DAG structure between nodes, and Broadcaster is used to broadcast CRDT operations.

[0015] Optionally, the file repair process includes:

[0016] If the storage node of a leaf data block crashes or the data is damaged, the identifier of the corresponding EC-Merkle DAG is obtained based on the identifier of the underlying Merkel directed acyclic graph, and the non-leaf node is obtained based on the identifier of the EC-Merkle DAG; based on the non-leaf node, the corresponding leaf data block is obtained and erasure code decoding is performed, and the leaf data block and check block are obtained based on the erasure code decoding operation to repair the file.

[0017] Optionally, the method further includes:

[0018] Obtain the pinset of the decentralized storage cluster through the consensus module. By traversing the pinset, analyze whether the Pin type is ClusterDAGPin and whether the EC-Merkle DAG exists. Based on the analysis results, determine the use of erasure coding and check file integrity based on the use of erasure coding. If erasure coding is used, reconstruct the complete file based on the erasure coding and re-pin the lost data blocks and check blocks to the cluster.

[0019] The present invention also discloses a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the above-mentioned storage optimization method based on erasure codes for decentralized storage systems.

[0020] The present invention also discloses a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned storage optimization method based on erasure code for decentralized storage systems are implemented.

[0021] The present invention also discloses a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the above-mentioned storage optimization method based on erasure code for decentralized storage systems.

[0022] Compared with the prior art, the present invention has the following advantages and technical effects:

[0023] The present invention proportionally truncates the file stream added to decentralized storage and encodes it using erasure codes. An EC-Merkle DAG is constructed based on the Merkle DAG and the check blocks obtained by erasure coding. In constructing the EC-Merkle DAG, a basic Merkle DAG is first constructed from the original file. Then, a Merkle-CRDT is used to reach consensus on the RootCID in a cluster of decentralized nodes, allowing each node to restore the entire file through the Merkle DAG. The EC-Merkle DAG is then further constructed, treating multiple data blocks as a stripe for erasure coding and decoding. After encoding the erasure code at a given bit rate, the check blocks obtained from the encoding are inserted into the end of the corresponding encoded stripe of the EC-Merkle DAG. Furthermore, it is necessary to store Merkle DAG and EC-Merkle DAG. Because there are a large number of decentralized nodes, in order to balance the IO throughput of each node and give full play to the data recovery effect of the erasure code, each block of the stripe (both leaf data blocks and check blocks) needs to be stored on all nodes, and the leaf blocks are mapped to different storage nodes on the hash ring composed of cluster nodes through consistent hashing according to the number of Replicas. The storage method of non-leaf blocks is: copy the non-leaf blocks to all storage nodes. Through Merkle-CRDT, the hash ring of each node and the information stored by the node, such as metadata, are reached in the cluster to reach consensus. By setting up a recovery node to continuously check the active nodes in the cluster and restore and redistribute the data when the node goes down. The example of the present invention can improve the storage efficiency of data in decentralized storage scenarios and ensure the integrity and security of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0025] Figure 1 A schematic diagram of the interaction between a decentralized storage client and a decentralized storage cluster service according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a file storage process according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of block allocation based on consistent hashing according to an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of the Pin types maintained by the decentralized storage cluster according to an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of an acquisition block according to an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of file acquisition according to an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of file recovery according to an embodiment of the present invention;

[0032] Figure 8 A schematic diagram of file storage according to an embodiment of the present invention;

[0033] Figure 9 Schematic diagram of the Merkle-CRDT consensus algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Example 1

[0037] like Figure 1-5 As shown, this embodiment provides a storage optimization method based on erasure coding for a decentralized storage system, including:

[0038] Step S1: decentralized storage cluster client part;

[0039] Step S2: file storage part;

[0040] Step S3: metadata consensus part;

[0041] Step S4: file acquisition and recovery part;

[0042] Step S5: Batch restore file parts.

[0043] In step S1, the decentralized storage cluster client consists of a command line and a Client package. When the client is a command line, after the user enters a command, the command parsing unit within the command line parses the command, identifying its type (e.g., adding a file, retrieving a file, batch restoring files, etc.), the file paths involved, and erasure coding parameters (if any). Simultaneously, the file transfer unit quickly and stably and efficiently transfers the user-specified file to the Client package via the HTTP protocol. Upon receiving the command and file information, the request processing unit within the Client package appropriately processes and assigns tasks based on the command type. For example, if the command is to add a file using erasure coding, the erasure coding and EC-Merkle DAG construction module is invoked. This module first splits the file into data blocks of 256KB each, applies an erasure coding algorithm (e.g., RS code) to generate parity blocks at a set bit rate (K data blocks generate M parity blocks), calculates the hash value for each block, constructs a Merkle DAG, and constructs an EC-Merkle DAG, then passes the relevant information to subsequent processing units. For file retrieval tasks, the request processing unit triggers the Merkle DAG and retrieval and EC-Merkle DAG decoding and retrieval modules. The decoding and retrieval module first retrieves the MetaPin (a data structure containing file metadata participating in consensus) based on the passed CID (unique file identifier). It then uses the MetaPin's reference to retrieve the ClusterDAGPin (a data structure containing erasure coding information participating in consensus), reads the EC-Merkle DAG's CID and other metadata, and then retrieves data blocks from cluster nodes. Retrieval utilizes two concurrent coroutines: one coroutine retrieves all blocks from a decentralized storage peer, while the other retrieves blocks from the first. Once sufficient decoded blocks have been retrieved (e.g., K blocks in a stripe), decoding and recovering the remaining blocks in the stripe begins. These two methods work in tandem; once either successfully retrieves all blocks, the other is interrupted to conserve resources. After the complete file stream is retrieved, it is packaged as a tarball and streamed to the client. The client processes the data stream to obtain the complete file and decompresses it to a user-specified path. For batch file recovery tasks, the client's batch recovery module is initiated. It first communicates with the consensus layer through the consensus proxy module to obtain the pinset (a collection of data structures participating in the consensus), then iterates over each file in the pinset, checking whether the Pin type is ClusterDAGPin and whether there is erasure code metadata to determine whether the file is encoded with an erasure code. If it is encoded and damaged, the decentralized storage node service obtains the EC-Merkle DAG and detects it. If damaged, it is restored using the erasure code algorithm. The recovered data block or check block is re-transmitted to the surviving node through consistent hash mapping according to the original parameters and type, ensuring file integrity.

[0044] In step S2, the whole process of file storage is as follows Figure 2 As shown, use the command line to add folders or files locally or on the network. First, read the file stream into the system through the HTTP protocol and enter the Chunking module, which cuts the file into data blocks of 256KB each. Next, the Layout module organizes these data blocks into Merkle DAG form and then sends the data blocks to the DAGService. On the one hand, the DAGService sends the data blocks to a specific decentralized storage peer via HTTP, and on the other hand, it sends the data blocks to the RS module. After receiving the data blocks, the RS module first integrates them into blocks, and when K data blocks are accumulated, the encoding operation is performed. Before encoding, the length of each data block is aligned (long blocks are truncated and padded with zeros to the previous block, and short blocks are padded with zeros themselves), and then the new Go coroutine is used to encode the check block, the check block is added to the stripe, and then the stripe is sent to the Adder module. As shown Figure 3 As shown in the figure, the Adder module distributes multiple data blocks and check blocks to different nodes of the decentralized storage cluster through a consistent hashing algorithm. The specific consistency hash allocation is as follows: first, create a hash ring ranging from 0 to 2^256-1 for block allocation. Assume that there are two actual nodes A and B, and each node generates three virtual nodes (A-0, A-1, A-2 and B-0, B-1, B-2); then create multiple virtual nodes on each physical storage node. The number depends on the system situation and is generally a multiple of the number of physical nodes to ensure load balancing and fault tolerance; then use the SHA-256 algorithm to calculate the hash value for each data block and check block to ensure that it is evenly distributed between 0 and 2^256-1; then map the calculated hash value to the hash ring, which is divided into several intervals, each interval corresponding to a virtual node; then map the virtual nodes A-0, A-1, A-2 to physical node A, and map B-0, B-1, B-2 to physical node B; finally, determine the corresponding virtual node based on the position of the hash value of the data block and check block on the hash ring, and then allocate them to the corresponding physical node for storage. At the same time, because RS encoding requires input blocks of equal length and unequal length blocks are padded with zeros for alignment, the original file size must be saved for correct truncation when decoding the last stripe. During encoding, the file size and the order of each block are recorded and stored as metadata in ClusterDAGPin. Since RS encoding requires equal-length input blocks to work, and zeros are padded for alignment of unequal length blocks, the original length of each block must be saved so that the data block can be correctly truncate during RS decoding. RS decoding also requires that the order of each block in the slice remain unchanged. Therefore, the CID and order of each block must be recorded during encoding and stored as metadata in ClusterDAGPin.

[0045] In step S3, Merkle-CRDT is responsible for persisting key metadata such as the CID of the Merkle DAG file, the CID of the ClusterPin file, the CID of the EC-Merkle DAG, and the order of the data blocks and check blocks in the stripe to ClusterDAGPin, and finally saving them to each decentralized storage cluster node. Figure 4 As shown in the figure, Pin is an additional data structure maintained by the cluster to store file information required for consensus. It comes in different types, indicating the pinning status of files within the cluster. The consensus layer further maintains file and pinset status to ensure metadata consistency and accuracy within the cluster. Leveraging the characteristics of Merkle DAG and the convergence properties of CRDT, the system achieves data integrity and eventual consistency in a decentralized environment. Distributed technologies such as DHT and PubSub enable highly scalable metadata dissemination, content addressing ensures security, and key-value storage ensures that every node stores deduplicated blocks.

[0046] In step S4, use the decentralized storage client ecget CID command to download the file (especially when the file may be damaged). First, get the MetaPin of the file according to the incoming Merkle DAG CID, then get the ClusterDAGPin through the Reference pointer of MetaPin, read the ClusterDAGPin content to get the CID and other metadata of EC-Merkle DAG. Figure 5 As shown, when concurrently retrieving blocks, retrieval times can vary depending on the storage node, especially if a node crashes or a data block is corrupted. To avoid wasted decoding time after a file retrieval timeout, a dual-coroutine strategy is adopted. The default coroutine returns success after retrieving all blocks from the decentralized storage peer. Another coroutine retrieves blocks from the default coroutine and, once it has enough blocks to decode (e.g., K data blocks for an RS code), begins RS decoding to recover the remaining blocks in the stripe. These two methods execute concurrently; if either method successfully retrieves all blocks, the other is interrupted, saving resources. After retrieving the recovered data block or checksum, it is pinned to a surviving decentralized storage peer using its original parameters and type. There's no need to repeat the complete file append operation; instead, the consistent hash mapping is re-performed and the data block or checksum is transferred. Finally, the file is packaged using tarballs and streamed to the client. The client processes the data stream to obtain the complete file and decompresses it to the user-specified path.

[0047] Step S5 is similar to step S4, except that the consensus module's pinset is initially obtained. Erasure-coded files are then selected from the pinset and checked for block corruption. If so, the complete file is reconstructed using an erasure coding algorithm (such as RS) and the lost data and parity blocks are redistributed to surviving nodes.

[0048] Example 2

[0049] like Figure 6-9 As shown, this embodiment provides a storage optimization method based on erasure coding for a decentralized storage system, including:

[0050] Obtain a file stream, segment it into blocks, and organize it into a basic Merkle directed acyclic graph (DAG). Erasure code the segmented file stream to construct an EC-Merkle DAG. Based on a consistent hashing ring, the leaf blocks of the EC-Merkle DAG and the basic Merkle DAG are stored on specific nodes. Meanwhile, the non-leaf blocks of the EC-Merkle DAG and the basic Merkle DAG are stored on all nodes.

[0051] In some specific implementations, the process of obtaining the basic Merkle directed acyclic graph includes:

[0052] Obtain a file stream, truncate the file stream into several leaf data blocks, and organize them into a basic Merkle directed acyclic graph structure; wherein the leaf data blocks have the same size.

[0053] In some specific embodiments, the process of storing leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to a specific node based on the consistent hashing ring includes:

[0054] The leaf data blocks in the basic Merkle directed acyclic graph are encoded based on the erasure code to obtain the check blocks corresponding to each leaf data block; an EC-Merkle DAG is constructed based on the check blocks and the leaf data blocks; the leaf data blocks in the EC-Merkle DAG and the basic Merkle directed acyclic graph are deduplicated according to the hash values ​​of the leaf data blocks and the check blocks and the hash ring composed of the nodes in the decentralized cluster; the deduplicated leaf data blocks and the check blocks are subjected to consistent hashing and stored in specific nodes.

[0055] In some specific implementations, the process of constructing an EC-Merkle DAG includes:

[0056] Preset check block generation rules, obtain check blocks according to erasure codes and check block generation rules; obtain stripes based on the data blocks and check blocks generated by one encoding, and construct EC-Merkle DAG based on the stripes.

[0057] Specifically, when a file needs to be stored in the cluster in an erasure-coded manner, the file content will first be read into a node of the decentralized storage cluster via the HTTP protocol. Subsequently, the file will be split into 256KB blocks by the Chunking module, and the data blocks will be organized into a Merkle DAG structure using the Layout module. Next, the Adder module will send the leaf data block to the DAGService for further processing by the decentralized storage cluster. The DAGService will perform consistent hashing based on the hash value of the data block and the hash ring composed of the nodes in the decentralized cluster, and store the data block in the virtual node. Finally, the data block will be mapped to a node, and the leaf data block will be sent to a specific decentralized storage peer via the HTTP protocol. The non-leaf data block will then be stored in all peers. At the same time, the EC-Chunking module encodes the leaf data blocks into several leaf check blocks at a fixed erasure code rate, combines the blocks involved in one encoding into a stripe, and then sends the different blocks of the entire stripe to the Adder module. This module is responsible for sending the leaf data blocks and check blocks to the DAGService for further processing by the decentralized storage cluster. The DAGService will perform consistent hashing based on the hash values ​​of the data blocks and check blocks and the hash ring composed of the nodes in the decentralized cluster, and store the data blocks and check blocks in virtual nodes. Finally, the data blocks and check blocks will be mapped to a certain node.

[0058] Metadata consensus is achieved through a Merkle-CRDT-based metadata consensus mechanism;

[0059] In some specific implementations, the Merkle-CRDT-based metadata consensus mechanism includes two core components: DAG-Syncer and Broadcaster. DAG-Syncer is responsible for synchronizing the Merkle-DAG structure between nodes, and Broadcaster is responsible for broadcasting CRDT operations.

[0060] Specifically, the metadata consensus method is mainly based on Merkle-CRDT implementation. Specifically, Merkle DAG is used as the transmission and persistence layer to implement CRDT (Conflict-Free Replicated Data Types), so that the system has decentralization and eventual consistency. Merkle-DAG, as a logical clock, can simplify the design and implementation of convergent data types under weak message delivery guarantees and a large number of replicas. The metadata synchronization mechanism of Merkle-CRDT includes two core components: DAG-Syncer and Broadcaster. DAG-Syncer is responsible for synchronizing the Merkle-DAG structure between nodes, and Broadcaster is responsible for broadcasting CRDT operations. These two components together ensure the causal consistency of each object. After the introduction of erasure codes, this embodiment uses DAG-Syncer to consensus the non-leaf data blocks of Merkle DAG and EC-Merkle DAG to each node, and the leaf data blocks and check blocks are mapped to the actual decentralized nodes after calculating the consistent hash through the hash ring. In terms of consistency, the immutability and self-verification of Merkle-DAG ensure data integrity, while the convergence properties of CRDT ensure that all nodes eventually reach a consensus, enabling the system to operate in a loose network environment without relying on a specific messaging protocol. Furthermore, the system utilizes distributed technologies such as DHT and PubSub to achieve highly scalable metadata dissemination, leveraging the security and deduplication properties of content addressing. This Merkle-CRDT-based metadata consensus mechanism ensures data consistency while also offering excellent scalability and fault tolerance.

[0061] If file acquisition is performed, based on the identifier of the basic Merkle directed acyclic graph, the BitSwap algorithm is used to search whether the corresponding key-value pair exists locally. If not, the nodes that establish point-to-point communication and other nodes in the decentralized network are searched to obtain non-leaf blocks, and the identifier set of the leaf data block is obtained based on the non-leaf block; the file acquisition operation is completed based on the identifier set. If the storage node of the leaf data block is down or the data is damaged, the file is repaired. During the repair process, the file acquisition operation is synchronized according to the identifier set.

[0062] In some specific implementations, the process of repairing a file includes:

[0063] If the storage node of a leaf data block crashes or the data is damaged, the identifier of the corresponding EC-Merkle DAG is obtained based on the identifier of the underlying Merkel directed acyclic graph, and the non-leaf node is obtained based on the identifier of the EC-Merkle DAG; based on the non-leaf node, the corresponding leaf data block is obtained and erasure code decoding is performed, and the leaf data block and check block are obtained based on the erasure code decoding operation to repair the file.

[0064] Specifically, there are two main methods for retrieving files: the first is to directly obtain the original file content through the Merkle DAG, but this method can only be used when every data block of the file is intact. The second method is suitable for situations where it is uncertain whether decentralized nodes are all functioning properly. After obtaining non-leaf nodes through the EC-Merkle DAG on the decentralized storage cluster, the hash table and the hash of the data blocks and check blocks in the leaf nodes are further mapped to the physical nodes storing the corresponding data. The file content is then restored using a limited number of striped data blocks and check blocks. When retrieving a file, the rootCID of the file must be entered. The MetaPin is then obtained from the rootCID. The ClusterDAGPin is then obtained based on the MetaPin's reference. The CBOR-formatted metadata is then deserialized to obtain the rootCID of the EC-Merkle DAG corresponding to the file, which is referred to as the ECRootCID in this embodiment. Because the Merkle DAG can be treated as a single file, the rootCID content can be directly retrieved through the DAGReader. When retrieving data blocks concurrently, because storage nodes may involve many decentralized nodes, each node needs to establish multiple stable peer-to-peer connections with other decentralized nodes in the cluster to retrieve the data blocks. This ensures that the node retrieving the data can proceed. Waiting until the file retrieval timeout to perform erasure code decoding would waste significant time in the event of a lost block. Therefore, two methods can be used to retrieve all blocks simultaneously: Method 1 directly retrieves all data from decentralized storage peers using the Merkle DAG, returning a success response only after the entire file has been retrieved. Method 2 retrieves non-leaf data blocks and parity blocks using the EC-Merkle DAG, then decodes them as they are retrieved. Once the EC-Merkle DAG is fully restored, indicating that every leaf data block and parity block has been locally cached, the entire file content can be retrieved using the Merkle DAG. To speed up data retrieval, the client can pass a parameter to retrieve the file content. The two methods can be executed concurrently. Once the first method successfully retrieves the file content, the second method's context is interrupted, saving resources. If the first method is blocked, the second method can resume retrieving the file after the EC-Merkle DAG is restored. If the file is only accessible after recovery via EC-Merkle DAG, the recovered data and check blocks can be pinned to a surviving decentralized peer using the original parameters and types. This eliminates the need to re-run the entire file-adding process because the same data and check blocks do not need to be pinned repeatedly.For different blocks, consistent hashing is used to map data and checksums to different decentralized nodes on the hash ring. These blocks are then transmitted to the corresponding decentralized peers via a persistent peer-to-peer BlockStream (peer-to-peer file blocks). The files are then compressed using tarballs and streamed to the client. The client processes the data stream, extracts the complete file, and decompresses it to a user-specified path.

[0065] In some embodiments, the method further comprises:

[0066] Obtain the pinset of the decentralized storage cluster through the consensus module. By traversing the pinset, analyze whether the Pin type is ClusterDAGPin and whether the EC-Merkle DAG exists. Based on the analysis results, determine the use of erasure coding and check file integrity based on the use of erasure coding. If erasure coding is used, reconstruct the complete file based on the erasure coding and re-pin the lost data blocks and check blocks to the cluster.

[0067] Specifically, the steps for restoring all files are similar to restoring some files. The difference is that there is no incoming CID. Instead, the consensus module uses the pinset of the decentralized storage cluster. Then, by traversing and checking whether the Pin type is ClusterDAGPin and the existence of erasure code metadata, it is determined whether the file is encoded with erasure code. Then, the file is checked for integrity. If a file is damaged, the erasure code is used to reconstruct the complete file, and the lost data blocks and check blocks are re-pinned to the cluster.

[0068] The present invention also discloses a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the above-mentioned storage optimization method based on erasure codes for decentralized storage systems.

[0069] The present invention also discloses a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned storage optimization method based on erasure code for decentralized storage systems are implemented.

[0070] The present invention also discloses a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the above-mentioned storage optimization method based on erasure code for decentralized storage systems.

[0071] In the field of storage, erasure codes are another method of protecting data availability and fault tolerance in addition to multiple copies. The erasure code used in this embodiment is the Reed-Solomon code (RS code). Because the RS code can adapt to any number of data disks and redundant disks, and the peers of decentralized storage can also be abstracted as disks when acting as simple storage nodes, the decentralized nature of decentralized storage is very suitable for optimizing RS codes. It is feasible to use erasure codes for storage in a decentralized network. Its principle is mainly to use the check blocks obtained by multiplying the data blocks according to the generator matrix and distribute data in the decentralized storage network. In addition, when errors occur during data transmission or storage and some data is lost, the original blocks can be restored by multiplying the inverse matrix of the generator matrix corresponding to the error-free blocks with the error-free blocks, which can well adapt to unreliable storage and transmission environments.

[0072] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A storage optimization method based on erasure coding for a decentralized storage system, characterized in that: The following steps are involved: Obtain the file stream, divide it into blocks, and organize it into a basic Merkle directed acyclic graph; construct the EC-Merkle DAG by erasure coding the block-processed file stream; Based on the consistent hashing ring, the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph are stored in the preset nodes, and the non-leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph are stored in all nodes; The process of storing the leaf blocks of the EC-Merkle DAG and the basic Merkle directed acyclic graph to the preset nodes based on the consistent hashing ring includes: Encode leaf data blocks in a basic Merkle directed acyclic graph based on erasure codes to obtain check blocks corresponding to each leaf data block; construct an EC-Merkle DAG based on the check blocks and the leaf data blocks; deduplicate the leaf data blocks in the EC-Merkle DAG and the basic Merkle directed acyclic graph based on the hash values ​​of the leaf data blocks and the check blocks and the hash ring formed by the nodes in the decentralized cluster; perform consistent hashing on the deduplicated leaf data blocks and the check blocks, and store them in a preset node; The process of constructing EC-Merkle DAG includes: A check block generation rule is preset, and a check block is obtained according to the erasure code and the check block generation rule; a stripe is obtained based on the data block and the check block generated by the one-time encoding, and an EC-Merkle DAG is constructed based on the stripe; Metadata consensus is achieved through a Merkle-CRDT-based metadata consensus mechanism; If a file is to be retrieved, the BitSwap algorithm is used to search for the corresponding key-value pair locally based on the identifier of the basic Merkle directed acyclic graph. If not, the node that establishes point-to-point communication and other nodes in the decentralized network are searched to obtain non-leaf blocks, and the identifier set of the leaf data block is obtained based on the non-leaf block. The file retrieval operation is completed based on the identifier set. If the storage node of the leaf data block is down or the data is damaged, the file is repaired. During the repair process, the file retrieval operation is synchronized according to the identifier set. The file repair process includes: If the storage node of a leaf data block crashes or the data is corrupted, the identifier of the corresponding EC-Merkle DAG is obtained based on the identifier of the underlying Merkel directed acyclic graph, and the non-leaf node is obtained based on the identifier of the EC-Merkle DAG; based on the non-leaf node, the corresponding leaf data block is obtained, and the erasure code decoding operation is performed to obtain the complete leaf data block and check block, completing the file repair.

2. The method according to claim 1, characterized in that The process of obtaining the basic Merkle directed acyclic graph includes: Obtain a file stream, truncate the file stream into several leaf data blocks, and organize them into a basic Merkle directed acyclic graph structure; wherein the leaf data blocks have the same size.

3. The method according to claim 1, characterized in that The Merkle-CRDT-based metadata consensus mechanism includes two core components: DAG-Syncer and Broadcaster. DAG-Syncer is used to synchronize the Merkle-DAG structure between nodes, and Broadcaster is used to broadcast CRDT operations.

4. The method according to claim 1, wherein The method further comprises: Obtain the pinset of the decentralized storage cluster through the consensus module. By traversing the pinset, analyze whether the Pin type is ClusterDAGPin and whether the EC-Merkle DAG exists. Based on the analysis results, determine the use of erasure coding and check file integrity based on the use of erasure coding. If erasure coding is used, reconstruct the complete file based on the erasure coding and re-pin the lost data blocks and check blocks to the cluster.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the erasure code-based storage optimization method for a decentralized storage system as recited in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the erasure code-based storage optimization method for a decentralized storage system described in any one of claims 1 to 4 are implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the erasure code-based storage optimization method for a decentralized storage system described in any one of claims 1 to 4 are implemented.

Images