A high-efficiency DAG construction, traversal and export method and device for distributed storage

Abstract

The application relates to an efficient DAG construction, traversal and export method and device for distributed storage, which adopts a streaming and layered parallel processing architecture. For multiple sub-trees of the same level in the DAG, the internal construction processes of the sub-trees are independent of each other. The DAG construction does not need to wait for all leaf nodes to be generated before the upper layer construction, but can adopt a mode of 'generating and processing in parallel'. Once the nodes at the bottom accumulate to a number sufficient to constitute a parent node, a task is started to construct the parent node, and the parent node is pushed to a processing queue at a higher layer. The method breaks the inherent serial dependence to adapt to the application requirements of high throughput and low delay, thereby significantly improving the experience of users when uploading and downloading files.

Description

Technical Field

[0001] This invention belongs to the field of interplanetary file system technology, and relates to an efficient method and apparatus for constructing, traversing and exporting DAGs for distributed storage. Background Technology

[0002] The InterPlanetary File System (IPFS) is a decentralized distributed storage protocol designed to build an efficient and reliable data storage and access system through distributed file storage and peer-to-peer transmission technologies.

[0003] Compared to traditional centralized storage models, IPFS significantly improves data storage reliability and access speed by using content addressing technology, version management, and a distributed storage network.

[0004] IPFS is a content-addressable data model where all content is organized as a directed acyclic graph (Merkle DAG). In IPFS, adding content, retrieving content, and exporting the DAG are three common operations, with the specific processes as follows:

[0005] (1) Add content:

[0006] The client submits file content to be added to the shared network via CoreAPI or HTTP API;

[0007] Chunking: The file is divided into chunks according to a chunking strategy;

[0008] DAG Construction: Construct a Merkle DAG to represent the file / directory structure;

[0009] Provider announcement: Nodes announce their ownership of this content through a distributed hash table (DHT).

[0010] (2) Search content:

[0011] Path resolution: Converts paths (such as / ipfs / Qm…) into CIDs;

[0012] Local search: Check if the content exists locally;

[0013] Provider discovery: If not located locally, use DHT to find the provider;

[0014] Content exchange: Requesting blocks from the provider using Bitswap;

[0015] DAG traversal: Follow the links in the DAG to retrieve all the required blocks;

[0016] Content assembly: Reconstructing the original content from the blocks.

[0017] (3) DAG Export: Compared with directly downloading the file content of the search content, the DAG export function does not care whether the specific data inside the DAG is a directory or a file. Instead, it packages all the original databases that make up the DAG into a single file. The packaged file format is called Content Addressable Archive (.car).

[0018] Currently, there are research and optimization solutions in the industry dedicated to improving the efficiency of content routing. For example, some solutions optimize the "provider announcement" process (such as "Optimistic Provide"), aiming to more quickly publish provider records to the appropriate location in the DHT network when a user uploads a file, thereby shortening the time it takes for other nodes to discover the content.

[0019] However, in modern network environments, especially in enterprise intranets, data centers, or regional high-speed networks, network transmission rates are extremely high and inter-node latency is extremely low. In such scenarios, the time required to find the provider record via DHT accounts for a small percentage of the total data operation time and is no longer the main performance bottleneck. The real performance bottleneck has shifted from "network addressing" to the processing efficiency of the IPFS core data structure DAG itself on the local node. Regardless of network speed, data upload and download must involve the construction and traversal of this critical DAG data structure, and the inefficiency of these operations in existing technologies has become a new and more significant constraint. Specifically, this manifests in:

[0020] (1) The inherent computational bottleneck of DAG construction limits the upload speed: The core of the data addition operation is to divide the file into blocks, hash it and assemble it into a complete DAG on the local node. This process is a computationally intensive and I / O intensive task. Especially when processing large files or a large number of small files, its inherent serial computation dependence makes it impossible to fully utilize CPU and storage resources in parallelization.

[0021] (2) The inherent serial dependency of DAG traversal limits download speed: Data retrieval is essentially a traversal of the DAG. Existing technologies require starting from the root CID, parsing and obtaining the CID list of the next layer of nodes layer by layer, and then initiating a request. This serial dependency pattern of "request-parse-request" makes the total download time directly affected by the depth and breadth of the DAG. Even if the speed of each network provider lookup is fast, the accumulated latency of hundreds or thousands of serialized data block requests constitutes the main time overhead of the download operation.

[0022] (3) The inefficiency of full traversal in DAG export limits data management capabilities: DAG export operation is also constrained by the inefficient serial traversal mechanism. It requires a complete traversal of all nodes of the target DAG to serialize it into a CAR file. For DAGs with complex structures and a large number of nodes, this process is extremely time-consuming, seriously affecting the efficiency of backup, migration and archiving of large-scale data. Summary of the Invention

[0023] To address the problems existing in the traditional methods mentioned above, this invention proposes an efficient DAG construction, traversal, and export method and apparatus for distributed storage. This method focuses on optimizing DAG construction, traversal, and export, breaking its inherent serial dependency to adapt to the application requirements of high throughput and low latency, thereby significantly improving the user experience when uploading and downloading files.

[0024] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0025] An efficient DAG construction method for distributed storage includes the following steps:

[0026] The leaf node producer module reads the data source and streams the read data source into the leaf nodes of the DAG. The leaf nodes are then distributed as a data stream through an output channel.

[0027] The hierarchical processor receives node streams from the lower-level channel, caches and counts the node streams, and when the number of cached nodes reaches a preset threshold, the cached nodes are merged into a new parent node and pushed to the upper-level channel.

[0028] By using process control and cascading modules, leaf node producer modules and multiple level processors are cascaded together through data channels. The output channel of the upper level module is used as the input channel of the lower level module, thereby realizing recursive or iterative merging from leaf nodes to root nodes until a unique root node is generated.

[0029] An efficient DAG construction device for distributed storage includes:

[0030] The leaf node producer module is used to read the data source and stream the read data source into the leaf nodes of the DAG, and distribute the leaf nodes as a data stream through an output channel.

[0031] The hierarchical processor is used to receive node streams from the lower-level channel, cache and count the node streams, and when the number of cached nodes reaches a preset threshold, the cached nodes are merged into a new parent node and pushed to the upper-level channel.

[0032] The process control and cascading module is used to cascade the leaf node producer module and multiple level processors through data channels, and use the output channel of the upper level module as the input channel of the lower level module, thereby realizing recursive or iterative merging from the leaf node to the root node until a unique root node is generated.

[0033] An efficient DAG traversal method for distributed storage is provided. This method is used to traverse a DAG constructed using the aforementioned efficient DAG construction method for distributed storage. The method includes the following steps:

[0034] Step 1: During the initialization of the concurrent prefetcher, a concurrent worker pool consisting of a fixed number of pre-defined background worker coroutines is started; the prefetcher is the SlidingPrefetcher concurrent prefetcher.

[0035] Step 2: Starting from the root node of the DAG, after parsing the CID of each data block, immediately submit it to the internal task queue of the prefetcher, thereby triggering it to download from the IPFS network;

[0036] Step 3: Each worker coroutine in the data node processing pool obtains a CID from the queue channel in Step 2 and calls the prefetcher's synchronization mechanism to wait. Once the prefetcher confirms that the data corresponding to the CID has been downloaded locally, the coroutine immediately parses and extracts the original file data and puts it into a result channel.

[0037] Step 4: Consume data blocks sequentially from the result channel and use a buffered writer to efficiently write the aggregated data to disk;

[0038] Steps 1 to 4 enable CID lookup, download, parsing, and writing of data blocks. Each stage can handle data blocks corresponding to different CIDs, allowing the node parsing logic and network I / O prefetching to be completely separated in structure, giving the DAG traversal structure pipelined operation performance.

[0039] In one embodiment, an efficient DAG traversal method for distributed storage is applied to the DAG download process to obtain the CID node pool during concurrent DAG download traversal based on sliding prefetching and a multi-stage pipeline; wherein the specific process of concurrent DAG download based on sliding prefetching and a multi-stage pipeline includes:

[0040] An efficient DAG traversal method for distributed storage is used to obtain the CID node pool.

[0041] Each coroutine in the concurrent worker pool obtains a CID from the task queue of the traversal phase, and then calls the synchronization method of the concurrent prefetcher to wait until the concurrent prefetcher confirms that the data node corresponding to the CID has been downloaded and cached. When the node is ready, the coroutine immediately parses and extracts the original file data from it, and then puts the original file data into a result channel.

[0042] The data module is consumed from the result channel, and a buffered writer is used to aggregate the data blocks in memory and write them to the disk in sequence.

[0043] An efficient DAG traversal device for distributed storage, the device being used to traverse a DAG constructed using the aforementioned efficient DAG construction method for distributed storage; comprising:

[0044] The CID lookup module is used to start a concurrent worker pool consisting of a preset fixed number of background worker coroutines when the concurrent prefetcher is initialized; the prefetcher is the SlidingPrefetcher concurrent prefetcher.

[0045] The download module, starting from the root node of the DAG, immediately submits the CID of each data block to the internal task queue of the prefetcher after parsing it, thereby triggering its download from the IPFS network;

[0046] The parsing module is used by each worker coroutine in the data node processing pool to obtain a CID from the queue channel in step 2 and call the prefetcher's synchronization mechanism to wait. Once the prefetcher confirms that the data corresponding to the CID has been downloaded to the local machine, the coroutine immediately parses and extracts the original file data and puts it into a result channel.

[0047] The write module is used to consume data blocks sequentially from the result channel and efficiently write the aggregated data to disk using a buffered writer.

[0048] The CID lookup, download, parsing, and writing modules are implemented to achieve CID lookup, download, parsing, and writing of data blocks. Each stage can handle data blocks corresponding to different CIDs, which completely separates the node parsing logic and network I / O prefetching in structure, giving the DAG traversal structure pipelined operation performance.

[0049] A first efficient DAG derivation method for distributed storage is provided, which is used to derive the DAG constructed using the aforementioned efficient DAG construction method for distributed storage. The first efficient DAG derivation method for distributed storage includes:

[0050] Leveraging the modularity of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. Without changing the high-level DAG traversal algorithm, when a metadata-only export instruction is received, a custom metadata extraction proxy is used to replace the standard block storage read component in the data access chain.

[0051] When the metadata extraction agent receives a node read request for a specific CID, it selectively reads the information that constitutes the metadata, dynamically synthesizes a new, lightweight placeholder node in memory based on the read information, and returns the placeholder node to the higher-level DAG traversal algorithm.

[0052] A first efficient DAG derivation device for distributed storage, the first derivation device being used to derive a DAG constructed using the aforementioned efficient DAG construction method for distributed storage; the first derivation device includes:

[0053] The metadata proxy definition module is used to leverage the modularity of the IPFS data access architecture to decouple the high-level DAG traversal logic from the low-level block storage access logic. Without changing the high-level DAG traversal algorithm, when a metadata-only export instruction is received, a custom metadata extraction proxy is used to replace the standard block storage read component in the data access chain.

[0054] The DAG export module is used when the metadata extraction agent receives a node read request for a specific CID. It selectively reads the information that constitutes the metadata, dynamically synthesizes a new, lightweight placeholder node in memory based on the read information, and returns the placeholder node to the higher-level DAG traversal algorithm.

[0055] A second efficient DAG derivation method for distributed storage is provided, which is used to derive a DAG constructed using the aforementioned efficient DAG construction method for distributed storage. The second efficient DAG derivation method for distributed storage includes:

[0056] Leveraging the modular nature of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. An improved high-level DAG traversal algorithm replaces the original high-level traversal algorithm without altering the low-level block storage reading mechanism. In the improved high-level DAG traversal algorithm, when the traverser processes a node, it calls the GetNodeMetadata(CID) or GetLinks(CID) interface to parse and return only the node's metadata from the data block, ignoring the file's content itself. The node's metadata includes at least the links and the data block size.

[0057] A third efficient DAG derivation method for distributed storage is provided, which is used to derive the DAG constructed using the aforementioned efficient DAG construction method for distributed storage. The third efficient DAG derivation method for distributed storage includes:

[0058] Leveraging the modular nature of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. Without changing the high-level DAG traversal logic and the low-level block storage access logic, an independent metadata index is pre-built for all data blocks when writing to IPFS nodes. When performing metadata-priority DGA export, the traverser is configured to directly query the metadata index. The metadata index includes at least the data block size and the linked list.

[0059] One of the above technical solutions has the following advantages and beneficial effects:

[0060] The aforementioned efficient DAG construction, traversal, and derivation method and apparatus for distributed storage employs a streaming, hierarchical parallel processing architecture. For multiple subtrees at the same level in the DAG, the internal construction process of each subtree is independent. DAG construction does not require waiting for all leaf nodes to be generated before proceeding to the next level; instead, it can adopt a "generate-while-process" mode: once the number of nodes at the lower level accumulates to a sufficient number to form a parent node, a task is immediately initiated to construct this parent node and push it to a higher-level processing queue. This method breaks its inherent serial dependency to adapt to the application requirements of high throughput and low latency, thereby significantly improving the user experience when uploading and downloading files. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of the existing mechanism for serial DAG construction in one embodiment;

[0063] Figure 2 This is a flowchart illustrating an efficient DAG construction method for distributed storage in one embodiment.

[0064] Figure 3 This is a schematic diagram of an efficient DAG construction process for distributed storage in one embodiment;

[0065] Figure 4 This is a schematic diagram of the DAG sequential traversal in one embodiment;

[0066] Figure 5 This is a flowchart of a DAG concurrent download based on sliding prefetching and a multi-stage pipeline in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0069] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0070] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0071] Existing DAG serial construction mechanisms, such as Figure 1 As shown, in the current Kubo implementation of IPFS (i.e., the Go language implementation of IPFS), when adding a large file, IPFS uses a layout strategy called "Balanced" by default to construct the DAG. Although this strategy can generate a relatively balanced tree structure, its construction process is completely serial, resulting in significant efficiency issues. Figure 1 Taking a file containing 6 data blocks (leaf nodes) as an example, and setting the maximum number of links (Maxlinks) for each node to 3, the detailed steps are as follows:

[0072] (1) The system sequentially reads the first 3 data blocks (L1, L2, L3) of the file and links them to a parent node to generate the first subtree (Root1).

[0073] (2) When the number of child nodes of Root1 reaches the Maxlinks limit (3), the system will create a new root node (Root2) and use the completed Root1 as its first child node (at this time it is called SubRoot1).

[0074] (3) The system continues to read the remaining 3 data blocks (L4, L5, L6) of the file sequentially, constructs a new subtree (SubRoot2), and links it to Root2 as the second child node. Finally, Root2 is the root node of the MerkelDAG tree corresponding to the file.

[0075] As can be seen from the above process, the construction of subRoot1 and subRoot2 do not affect each other and can be carried out simultaneously. Therefore, this invention takes this as a starting point to design a high-performance DAG construction method in PFS based on a streaming pipeline.

[0076] In one embodiment, such as Figure 2 , Figure 3 As shown, an efficient DAG construction method for distributed storage is provided, which may include the following processing steps 100 to 102:

[0077] Step 100: Read the data source through the leaf node producer module, and stream the read data source into the leaf nodes of the DAG. Distribute the leaf nodes as a data stream through an output channel.

[0078] Specifically, the leaf node producer module starts an independent coroutine that continuously reads data from a file, generates leaf nodes, and sends them out as a data stream through a channel.

[0079] The function of the leaf node producer module is to read the data source and stream it into leaf nodes of the DAG, which are then distributed through an output pipeline.

[0080] Step 101: Receive node streams from the lower-level channel through the hierarchical processor, cache and count the node streams. When the number of cached nodes reaches a preset threshold, merge the cached nodes into a new parent node and push the new parent node to the upper-level channel.

[0081] Specifically, one or more hierarchical processor modules, each configured to: (a) receive lower-level nodes from an input channel; (b) internally buffer and count the received nodes; (c) when the node count reaches a preset threshold, construct a parent node from the buffered nodes; and (d) send the newly generated parent node to an output channel. A flow control and cascading module is responsible for cascading leaf node producers and multiple hierarchical processors through data channels, using the output channel of the upper-level module as the input channel of the lower-level module, thereby realizing recursive or iterative merging from leaf nodes to the root node until a unique root node is generated.

[0082] The internal construction processes of multiple subtrees at the same level in a Directed Acyclic Graph (DAG) are independent of each other. DAG construction does not need to wait for all leaf nodes to be generated before proceeding to higher levels; instead, it can adopt a "generate-as-you-go" model: once the number of nodes at the lower level (whether leaf or intermediate) accumulates enough to form a parent node, a task is immediately initiated to construct this parent node and pushed to a higher-level processing queue.

[0083] The hierarchical processor is the core of the pipeline. It receives node streams from lower-level channels. Once the number of received nodes accumulates to the maximum number of connections (maxLinks), it immediately calls buildParentNode to merge these nodes into a new parent node and pushes this newly generated parent node into the upper-level channel.

[0084] Step 102: The leaf node producer module and multiple level processors are cascaded through data channels by the process control and cascading module. The output channel of the upper level module is used as the input channel of the lower level module, thereby realizing the recursive or iterative merging from the leaf node to the root node until a unique root node is generated.

[0085] Specifically, regarding recursion and merging: the `processLevel` function is called recursively. After processing all nodes in the current level, it checks the number of parent nodes in the output. If the number is greater than 1, it means that merging needs to continue upwards. Therefore, it uses the output of its current level as input and calls itself again to enter the merging process at higher levels. This process repeats continuously until only a root node remains.

[0086] The aforementioned efficient DAG construction method for distributed storage employs a streaming, hierarchical parallel processing architecture. For multiple subtrees at the same level in the DAG, the internal construction process of each subtree is independent. DAG construction does not require waiting for all leaf nodes to be generated before proceeding to the next level; instead, it can adopt a "generate-while-processing" model: once the number of nodes at the lower level is sufficient to form a parent node, a task is immediately initiated to construct this parent node and push it to a higher-level processing queue. This method breaks its inherent serial dependency to adapt to the application requirements of high throughput and low latency, thereby significantly improving the user experience when uploading and downloading files.

[0087] It should be understood that, although the above Figure 2 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 2 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0088] In one embodiment, an efficient DAG construction apparatus for distributed storage is also provided, comprising:

[0089] The leaf node producer module is used to read the data source and stream the read data source into the leaf nodes of the DAG, and distribute the leaf nodes as a data stream through an output channel.

[0090] The hierarchical processor is used to receive node streams from the lower-level channel, cache and count the node streams, and when the number of cached nodes reaches a preset threshold, the cached nodes are merged into a new parent node and pushed to the upper-level channel.

[0091] The process control and cascading module is used to cascade the leaf node producer module and multiple level processors through data channels, and use the output channel of the upper level module as the input channel of the lower level module, thereby realizing recursive or iterative merging from the leaf node to the root node until a unique root node is generated.

[0092] It is understood that for a detailed explanation of the efficient DAG construction device for distributed storage, please refer to the corresponding explanations of the embodiments of the efficient DAG construction method for distributed storage above, and will not be repeated here. Each module in the above-described efficient DAG construction device for distributed storage can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or it can be stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0093] In one embodiment, an efficient DAG traversal method for distributed storage is also provided. This method is used to traverse a DAG constructed using the aforementioned efficient DAG construction method for distributed storage; it includes the following steps:

[0094] Step 1: During the initialization of the concurrent prefetcher, a concurrent worker pool consisting of a fixed number of pre-defined background worker coroutines is started; the prefetcher is the SlidingPrefetcher concurrent prefetcher.

[0095] Step 2: Starting from the root node of the DAG, after parsing the CID of each data block, immediately submit it to the internal task queue of the prefetcher, thereby triggering it to download from the IPFS network;

[0096] Step 3: Each worker coroutine in the data node processing pool obtains a CID from the queue channel in Step 2 and calls the prefetcher's synchronization mechanism to wait. Once the prefetcher confirms that the data corresponding to the CID has been downloaded locally, the coroutine immediately parses and extracts the original file data and puts it into a result channel.

[0097] Step 4: Consume data blocks sequentially from the result channel and use a buffered writer to efficiently write the aggregated data to disk;

[0098] Steps 1 to 4 enable CID lookup, download, parsing, and writing of data blocks. Each stage can handle data blocks corresponding to different CIDs, allowing the node parsing logic and network I / O prefetching to be completely separated in structure, giving the DAG traversal structure pipelined operation performance.

[0099] In one embodiment, an efficient DAG traversal method for distributed storage is applied to the DAG download process to obtain a CID node pool during concurrent DAG download traversal based on sliding prefetching and a multi-stage pipeline. The specific process of concurrent DAG download based on sliding prefetching and a multi-stage pipeline includes: obtaining the CID node pool using an efficient DAG traversal method for distributed storage; each coroutine in the concurrent worker pool obtains a CID from the task queue of the traversal phase, then calls the synchronization method of the concurrent prefetcher to wait until the concurrent prefetcher confirms that the data node corresponding to the CID has been downloaded and cached; when a node is ready, the coroutine immediately parses and extracts the original file data, then puts the original file data into a result channel; the data module consumes data from the result channel, and uses a buffered writer to aggregate data blocks in memory and write them sequentially to disk.

[0100] Specifically, Figure 4 This demonstrates the process of traversing a DAG in an existing IPFS implementation. Figure 4 In UnixFS (type=file), the data type is file, tsize is the number of elements in data block CID1, and Cq, cVt, and KqW are the amounts of metadata in data blocks 1, 2, and 3, respectively. The existing IPFS implementation traverses the DAG as follows: obtain the CID of chunk1, and only after finding the provider, downloading node data, and writing to disk will chunk2 be processed, and so on until chunk5 is written to disk. However, during the process of writing chunk1 to disk, the system can obtain the CID of chunk2 and perform operations such as searching for chunk2 on the network. Therefore, this invention takes this as its starting point, decoupling DAG traversal from node acquisition and network download from disk writing, to achieve concurrent downloading based on sliding prefetching and a multi-stage pipeline. To achieve decoupling and parallelization of network data download and disk writing, this application designs a DAG concurrent download method based on sliding prefetching and a multi-stage pipeline, which is an efficient DAG traversal method for distributed storage. This method employs a high-performance concurrent processing pipeline composed of multiple independent stages. Its core is a prefetcher called SlidingPrefetcher, which works in conjunction with three asynchronously executed coroutine stages (traversal, processing, and writing) to transform the serial download process into an efficient parallel job. The specific process is as follows:

[0101] (1) SlidingPrefetcher Concurrent Prefetch: The concurrent prefetcher serves as the download engine for the entire pipeline. During initialization, it starts a concurrent worker pool consisting of a fixed number (e.g., 128) of background worker coroutines. The Content Identifier (CID) of all data blocks to be downloaded is submitted to an internal task queue. The coroutines in the worker pool act as consumers, continuously and concurrently retrieving CIDs from the queue and independently calling the underlying data service to perform network lookups and data block downloads. Downloaded data nodes are stored in an internal cache and their status is synchronized through a notification channel associated with the CID. When a node downloads successfully, its corresponding notification channel is closed, and subsequent processes can confirm that the data is ready by waiting for the channel to close, thus achieving efficient asynchronous waiting.

[0102] (2) DAG Traversal and Prefetch Triggering: A separate coroutine is responsible for traversing the DAG topology. When it identifies a data block node, it immediately submits the node's CID to the concurrent prefetcher, triggering its background download in a non-blocking manner. Subsequently, the same CID is placed in a queue channel, transferring ownership to the next processing stage of the pipeline. To reduce scheduling overhead, this stage can also employ a batch distribution strategy. The concurrent prefetch module, including a task queue and a concurrent work unit, is responsible for receiving CIDs and retrieving data blocks from the network in parallel, while also providing a synchronous notification mechanism for data readiness.

[0103] (3) Data Node Processing: Consists of a concurrent coroutine pool. Each worker coroutine in the pool obtains a CID from the queue channel of the previous stage, and then calls the prefetcher's synchronization method to wait until the prefetcher confirms that the data node corresponding to the CID has been downloaded and cached. Once the node is ready, the coroutine immediately parses and extracts the original file data ([] bytes) from it, and then puts these binary data blocks into a result channel. A parallel processing module is used to wait for the data ready notification from the prefetcher module and parse the content of the downloaded data blocks;

[0104] (4) Serialization write: This stage consists of a single write coroutine that consumes data blocks from the result channel and uses a buffered writer (bufio.NewWriter) to aggregate these data blocks in memory and write them to disk in sequence. This single write point design avoids disk I / O contention and maximizes write throughput.

[0105] The DAG concurrent download process based on sliding prefetching and multi-stage pipeline is as follows: Figure 5As shown, the method for accelerating DAG data structure retrieval decouples the DAG topology traversal process, the network acquisition process of data blocks, and the data writing process, and processes them through an asynchronous concurrent pipeline. The pipeline includes a traversal and prefetch triggering phase, a parallel processing phase, and a serialization and writing phase.

[0106] In one embodiment, an efficient DAG traversal apparatus for distributed storage is also provided. The apparatus is used to traverse a DAG constructed using the aforementioned efficient DAG construction method for distributed storage; it includes:

[0107] The CID lookup module is used to start a concurrent worker pool consisting of a preset fixed number of background worker coroutines when the concurrent prefetcher is initialized; the prefetcher is the SlidingPrefetcher concurrent prefetcher.

[0108] The download module, starting from the root node of the DAG, immediately submits the CID of each data block to the internal task queue of the prefetcher after parsing it, thereby triggering its download from the IPFS network;

[0109] The parsing module is used by each worker coroutine in the data node processing pool to obtain a CID from the queue channel in step 2 and call the prefetcher's synchronization mechanism to wait. Once the prefetcher confirms that the data corresponding to the CID has been downloaded to the local machine, the coroutine immediately parses and extracts the original file data and puts it into a result channel.

[0110] The write module is used to consume data blocks sequentially from the result channel and efficiently write the aggregated data to disk using a buffered writer.

[0111] The CID lookup, download, parsing, and writing modules are implemented to achieve CID lookup, download, parsing, and writing of data blocks. Each stage can handle data blocks corresponding to different CIDs, which completely separates the node parsing logic and network I / O prefetching in structure, giving the DAG traversal structure pipelined operation performance.

[0112] In one embodiment, a first efficient DAG derivation method for distributed storage is also provided. This first efficient DAG derivation method for distributed storage is used to derive a DAG constructed using the aforementioned efficient DAG construction method for distributed storage. The first efficient DAG derivation method for distributed storage introduces a proxy mechanism at the data access layer to intercept data block read requests, specifically including the following steps:

[0113] Step 200: Utilize the modularity of the IPFS data access architecture to decouple the high-level DAG traversal logic from the low-level block storage access logic; without changing the high-level DAG traversal algorithm, when a metadata-only export instruction is received, replace the standard block storage read component with a custom metadata extraction proxy in the data access link.

[0114] Specifically, to address the I / O intensiveness and inefficiency caused by traditional DAG export operations that involve reading the entire data block, this invention proposes a metadata-priority DAG export method based on a storage access proxy. The core idea is to introduce a dedicated metadata extraction proxy in the underlying data access layer, without altering the upper-layer DAG traversal algorithm, to intercept and transform node read requests, thereby exporting only the structural metadata of the DAG rather than the complete data.

[0115] Step 201: When the metadata extraction agent receives a node read request for a specific CID, it selectively reads the information that constitutes the metadata, dynamically synthesizes a new, lightweight placeholder node in memory based on the read information, and returns the placeholder node to the high-level DAG traversal algorithm.

[0116] Specifically, the existing IPFS DAG export... <dircid>The API for exporting a DAG structure in IPFS exports the entire DAG graph associated with a specified CID, including all intermediate and leaf nodes. This means the system tracks and packages all data blocks of all files in the directory. This mechanism is reasonable in scenarios requiring a complete backup of the entire directory. However, its drawbacks are significant in the more common "on-demand access" scenario. For example, a user might only need file1, but is forced to download the contents of files file1 through file200. Therefore, this invention addresses this issue by designing a DAG export mode that only exports metadata. The user then finds the corresponding file's CID based on the metadata and calls `ipfs get` (the API for downloading files in IPFS) to download the required content.

[0117] This method leverages the modularity of the IPFS data access architecture, which decouples the high-level DAG traversal logic from the low-level block storage access logic. When the system receives a metadata-only export instruction, a custom metadata extraction proxy replaces the standard block storage read component in the data access chain. This seamlessly redirects all read requests for DAG nodes originating from the upper-level traversal algorithm to this proxy, enabling it to perform in-flight processing before the data is returned to the traverser.

[0118] When the metadata extraction agent receives a node read request for a specific Content Identifier (CID), it does not read the complete data block corresponding to that CID from physical storage. Instead, it selectively reads only information sufficient to constitute metadata, such as the original size of the data block and its internal list of links. The agent then dynamically synthesizes a new, lightweight "placeholder" node in memory. This node contains the actual size and link information, but its data payload is empty. Finally, the agent returns this synthesized "placeholder" node to the upper-level traversal algorithm. Because the placeholder node structurally preserves the necessary link information, the traversal algorithm can continue its traversal process normally along the links, completely unaware that the underlying data has been replaced. In this way, the entire export process operates on lightweight metadata nodes, fundamentally avoiding large-scale disk I / O operations, greatly shortening export time and reducing resource consumption.

[0119] The core components involved in this method include: a standard DAG traversal module; a configurable data access layer in which the standard block storage reader can be replaced; and a metadata extraction proxy module, which, when configured to replace the standard reader, is responsible for performing the aforementioned functions of interception, selective reading, dynamic synthesis of "placeholder" nodes, and returning them to the traversal module.

[0120] In one embodiment, a first efficient DAG derivation apparatus for distributed storage is also provided. This first derivation apparatus is used to derive a DAG constructed using the aforementioned efficient DAG construction method for distributed storage. The first derivation apparatus includes:

[0121] The metadata proxy definition module is used to leverage the modularity of the IPFS data access architecture to decouple the high-level DAG traversal logic from the low-level block storage access logic. Without changing the high-level DAG traversal algorithm, when a metadata-only export instruction is received, a custom metadata extraction proxy is used to replace the standard block storage read component in the data access chain.

[0122] The DAG export module is used when the metadata extraction agent receives a node read request for a specific CID. It selectively reads the information that constitutes the metadata, dynamically synthesizes a new, lightweight placeholder node in memory based on the read information, and returns the placeholder node to the higher-level DAG traversal algorithm.

[0123] In one embodiment, a second efficient DAG derivation method for distributed storage is also provided. This second efficient DAG derivation method for distributed storage is used to derive a DAG constructed using the above-described efficient DAG construction method for distributed storage. The second efficient DAG derivation method for distributed storage includes:

[0124] Leveraging the modular nature of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. An improved high-level DAG traversal algorithm replaces the original high-level traversal algorithm without altering the low-level block storage reading mechanism. In the improved high-level DAG traversal algorithm, when the traverser processes a node, it calls the GetNodeMetadata(CID) or GetLinks(CID) interface to parse and return only the node's metadata from the data block, ignoring the file's content itself. The node's metadata includes at least the links and the data block size.

[0125] Specifically, the core idea of ​​the second efficient DAG export method for distributed storage lies in separating metadata extraction from complete data block reading during DAG traversal. This method modifies the high-level traversal algorithm: it doesn't change the underlying block storage and reading mechanism, but directly modifies the upper-level DAG traversal algorithm itself. A new traversal mode specifically designed for metadata export can be developed. In this mode, when processing a node, the iterator does not call the general GetNode(CID) method (which returns complete node data), but instead calls a specialized GetNodeMetadata(CID) or GetLinks(CID) interface. Specifically, when the iterator needs node metadata, it calls the GetNodeMetadata(CID) interface, which is specifically used to parse and return all the source information required by the node from the data block, such as the data block size and creation timestamp. If the traversal only needs to know the topology, it calls the GetLinks(CID) interface, which only parses and returns the list of child node links contained in the node from the data block. The difference between this solution and the present invention is that the modification of this solution lies in the high-level business logic (traversal algorithm), while the modification of the present invention lies in the low-level data access logic.

[0126] In one embodiment, a third efficient DAG derivation method for distributed storage is also provided. This third efficient DAG derivation method for distributed storage is used to derive a DAG constructed using the above-described efficient DAG construction method for distributed storage. The third efficient DAG derivation method for distributed storage includes:

[0127] Leveraging the modular nature of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. Without changing the high-level DAG traversal logic and the low-level block storage access logic, an independent metadata index is pre-built for all data blocks when writing to IPFS nodes. When performing metadata-priority DGA export, the traverser is configured to directly query the metadata index. The metadata index includes at least the data block size and the linked list.

[0128] Specifically, based on a pre-built metadata index: This approach trades space for time by pre-creating an independent metadata index (e.g., using a key-value database) for all data blocks when data is written to IPFS nodes. This index uses the CID as the primary key, storing the metadata (size, link list, etc.) of the data block corresponding to that CID. When performing a metadata-first DAG export, the iterator is configured to completely bypass the main block store and directly query this lightweight metadata index. Since the index only contains metadata, the query process is very fast, with no large-scale disk I / O. This approach shifts the cost of optimization from dynamic computation "during export" to preprocessing and additional storage overhead "during import."

[0129] This method derives the topology of a DAG without reading the entire data block.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.< / dircid>

Claims

1. An efficient DAG construction method for distributed storage, characterized in that, Including the following steps: The leaf node producer module reads the data source and streams the read data source into the leaf nodes of the DAG. The leaf nodes are then distributed as a data stream through an output channel. The hierarchical processor receives node streams from the lower-level channel, caches and counts the node streams, and when the number of cached nodes reaches a preset threshold, the cached nodes are merged into a new parent node and pushed to the upper-level channel. By using process control and cascading modules, leaf node producer modules and multiple level processors are cascaded together through data channels. The output channel of the upper level module is used as the input channel of the lower level module, thereby realizing recursive or iterative merging from leaf nodes to root nodes until a unique root node is generated.

2. A high-efficiency DAG construction device for distributed storage, characterized in that, include: The leaf node producer module is used to read the data source and stream the read data source into the leaf nodes of the DAG, and distribute the leaf nodes as a data stream through an output channel; The hierarchical processor is used to receive node streams from the lower-level channel, cache and count the node streams, and when the number of cached nodes reaches a preset threshold, the cached nodes are merged into a new parent node and the new parent node is pushed to the upper-level channel. The process control and cascading module is used to cascade the leaf node producer module and multiple hierarchical processors through a data channel, and use the output channel of the upper-level module as the input channel of the lower-level module, thereby realizing recursive or iterative merging from the leaf node to the root node until a unique root node is generated.

3. An efficient DAG traversal method for distributed storage, characterized in that, The method is used to traverse a DAG constructed using the efficient DAG construction method for distributed storage as described in claim 1; it includes the following steps: Step 1: During the initialization of the concurrent prefetcher, a concurrent worker pool consisting of a preset fixed number of background worker coroutines is started; the prefetcher is the SlidingPrefetcher concurrent prefetcher; Step 2: Starting from the root node of the DAG, after parsing the CID of each data block, immediately submit it to the internal task queue of the prefetcher, thereby triggering it to download from the IPFS network; Step 3: Each worker coroutine in the data node processing pool obtains a CID from the queue channel in Step 2 and calls the prefetcher's synchronization mechanism to wait. Once the prefetcher confirms that the data corresponding to the CID has been downloaded locally, the coroutine immediately parses and extracts the original file data and puts it into a result channel. Step 4: Consume data blocks sequentially from the result channel and use a buffered writer to efficiently write the aggregated data to disk; Steps 1 to 4 enable CID lookup, download, parsing, and writing of data blocks. Each stage can handle data blocks corresponding to different CIDs, allowing the node parsing logic and network I / O prefetching to be completely separated in structure, giving the DAG traversal structure pipelined operation performance.

4. The efficient DAG traversal method for distributed storage according to claim 3, characterized in that, The efficient DAG traversal method for distributed storage is applied to the DAG download process to obtain the CID node pool during concurrent DAG download traversal based on sliding prefetching and multi-stage pipeline; the specific process of concurrent DAG download based on sliding prefetching and multi-stage pipeline includes: The CID node pool is obtained using the efficient DAG traversal method for distributed storage. In the concurrent worker pool, each coroutine obtains a CID from the task queue of the traversal phase, and then calls the synchronization method of the concurrent prefetcher to wait until the concurrent prefetcher confirms that the data node corresponding to the CID has been downloaded and cached. When the node is ready, the coroutine immediately parses and extracts the original file data from it, and then puts the original file data into a result channel. The data module is consumed from the result channel, and a buffered writer is used to aggregate the data blocks in memory and write them to the disk in sequence.

5. A high-efficiency DAG traversal device for distributed storage, characterized in that, The apparatus is used to traverse a DAG constructed using the efficient DAG construction method for distributed storage as described in claim 1; including: The CID lookup module is used to start a concurrent worker pool consisting of a preset fixed number of background worker coroutines when the concurrent prefetcher is initialized; the prefetcher is the SlidingPrefetcher concurrent prefetcher. The download module, starting from the root node of the DAG, immediately submits the CID of each data block to the internal task queue of the prefetcher after parsing it, thereby triggering its download from the IPFS network; The parsing module is used by each worker coroutine in the data node processing pool to obtain a CID from the queue channel in step 2 and call the prefetcher's synchronization mechanism to wait. Once the prefetcher confirms that the data corresponding to the CID has been downloaded to the local machine, the coroutine immediately parses and extracts the original file data and puts it into a result channel. The write module is used to consume data blocks sequentially from the result channel and efficiently write the aggregated data to disk using a buffered writer. The CID lookup, download, parsing, and writing modules are implemented to achieve CID lookup, download, parsing, and writing of data blocks. Each stage can handle data blocks corresponding to different CIDs, which completely separates the node parsing logic and network I / O prefetching in structure, giving the DAG traversal structure pipelined operation performance.

6. A first method for efficiently deriving a DAG for distributed storage, characterized in that, The method is used to derive a DAG constructed using the efficient DAG construction method for distributed storage as described in claim 1; the method includes: Leveraging the modularity of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. Without changing the high-level DAG traversal algorithm, when a metadata-only export instruction is received, a custom metadata extraction proxy is used to replace the standard block storage read component in the data access chain. When the metadata extraction agent receives a node read request for a specific CID, it selectively reads the information that constitutes the metadata, dynamically synthesizes a new, lightweight placeholder node in memory based on the read information, and returns the placeholder node to the high-level DAG traversal algorithm.

7. A first-order efficient DAG derivation device for distributed storage, characterized in that, The apparatus is used to derive a DAG constructed using the efficient DAG construction method for distributed storage as described in claim 1; The first type of export device includes: The metadata proxy definition module is used to leverage the modularity of the IPFS data access architecture to decouple the high-level DAG traversal logic from the low-level block storage access logic. Without changing the high-level DAG traversal algorithm, when a metadata-only export instruction is received, a custom metadata extraction proxy is used to replace the standard block storage read component in the data access chain. The DAG export module is used to selectively read the information that constitutes the metadata when the metadata extraction agent receives a node read request for a specific CID. Based on the read information, it dynamically synthesizes a new, lightweight placeholder node in memory and returns the placeholder node to the higher-level DAG traversal algorithm.

8. A second efficient DAG derivation method for distributed storage, characterized in that, The method is used to derive a DAG constructed using the efficient DAG construction method for distributed storage as described in claim 1; the method includes: Leveraging the modular nature of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. An improved high-level DAG traversal algorithm replaces the original high-level traversal algorithm without altering the low-level block storage reading mechanism. In the improved high-level DAG traversal algorithm, when the traverser processes a node, it calls the GetNodeMetadata(CID) or GetLinks(CID) interface to parse and return only the node's metadata from the data block, ignoring the file's content itself. The node's metadata includes at least the links and the data block size.

9. A third efficient method for deriving a DAG for distributed storage, characterized in that, The method is used to derive a DAG constructed using the efficient DAG construction method for distributed storage as described in claim 1; the method includes: Leveraging the modularity of the IPFS data access architecture, the high-level DAG traversal logic is decoupled from the low-level block storage access logic. Without changing the high-level DAG traversal logic and the low-level block storage access logic, an independent metadata index is pre-established for all data blocks when writing to IPFS nodes. When performing metadata-priority DGA export, the traverser is configured to directly query the metadata index. The metadata index includes at least the data block size and the linked list.

Images