Method and equipment for segmenting and restoring data set

Abstract

The invention aims to provide a data set segmentation and restoration method and equipment. The method comprises the following steps: storing information of each file in a traversed file directory for storing a big data set into a file list; acquiring information of a current to-be-processed file from the file list, determining a file path according to a set comparison condition and the information of the current to-be-processed file, and adding the file path to a target file in the subgraph; and converting a target file into an IPLD node, constructing a Merkel tree relative to the file directory by using the IPLD node according to a file path of the target file on each node, and generating root identification information of the slice sub-root graph. Downloading root identification information of all slice sub-root graphs; and searching the remaining file fragments under the same file according to the file paths of the found file fragments, and restoring the remaining file fragments into one file according to the sequence. Therefore, the cut fragments are basically consistent in size, the original directory layer structure is reserved, and the complete file in the downloaded fragments can be directly used.

Description

technical field [0001] The present application relates to the computer field, and in particular to a method and device for splitting and restoring data sets. Background technique [0002] With the development of society, the data generated by people is becoming larger and larger, and there is an urgent need for a method of segmenting, storing and restoring large data sets. IPFS (InterPlanetary File System, Interplanetary File System) builds a peer-to-peer distributed file system, realizing decentralized storage in the true sense. [0003] If you import the entire large data set (TB, PB level) into IPFS to generate a root identification information (CID), although the steps are simple, you will encounter many problems, such as the import time is too long, and it may fail. If you need to extract a certain part of the file, you have to download the entire data set. Then, at this time, it is necessary to segment the large data set. The existing segmentation schemes are as foll...

AI Technical Summary

Problems solved by technology

[0011] One purpose of this application is to provide a method and device for splitting and restoring data sets, which can solve the problem of large gaps in the size of fragments after splitting large data sets in the prior art, difficulty in extracting some files, and inability to retain the original directory hierarchy structure and other issues

Images