Small file optimization storage method based on HDFS

Abstract

The invention discloses a small file optimization storage method based on a HDFS, wherein the efficiency for the HDFS to read small files is improved, and the overall performance of the system is improved. The method includes the steps that first, the small files are combined and undergo storage preprocessing, wherein the storage preprocessing on the small files is achieved through filtering of the filers, combination of the small files, generation of metadata and generation of object IDs; second, after the files are stored into the HDFS in a combined mode, the mapping relations between the small files and combined files in the HDFS are stored into the metadata of the small files in a mode of file metadata, a directory structure of the files is stored in a file name, and the metadata are stored in a mode of distributed clusters on the basis of a Chord protocol; third, the directory structure of the files is optimized, and generated key values of the metadata are decomposed into Directory IDs and Small File IDs. The Directory IDs serve as key values for the metadata to skip into nodes in a metadata cluster, and therefore the files under the same directory are stored into the same node. The Small File IDs are generated in metadata nodes, and therefore each of the metadata corresponds to unique ID identification in the whole system.

Description

technical field [0001] The invention relates to an HDFS-based small file optimized storage method, in particular to a method for realizing optimized storage of distributed and massive small file data, and belongs to the field of distributed data optimized storage. Background technique [0002] With the rapid development of Internet technology, the amount of data is increasing exponentially. Now such an Internet era of information explosion has brought a profound impact on the lives of Internet users. However, for the Internet application field, how to solve the storage of these massive data is a huge challenge. Under these challenges, traditional data storage and processing technologies, such as relational databases, can no longer meet the needs of technological development, and emerging mass data storage technologies continue to emerge. Such as the parallel file system GFS used by Google; the MapReduce programming model for massive data; Amazon's Simple Storage Service (S3...

AI Technical Summary

Problems solved by technology

[0009] Purpose of the invention: Aiming at the problem of low utilization of massive small file data storage resources and low file access efficiency in the HDFS distributed file system, the present invention provides an optimized storage method for small files based on HDFS to improve the reading efficiency of HDFS for small files , to improve the overall performance of the system

Images