Optimization method based on HDFS small file storage

Abstract

The invention discloses an optimization method based on HDFS small file storage. The method includes the following steps that S1, a metadata server used for storing metadata information of user space is established; S2, a user file used for storing all small files is established for each user; S3, a metadata structure is defined and used for recording detailed metadata information of all the small files of each user, and the metadata structure records the deviant of the small files in the user file and the sizes of the small files; S4, the small files of the users are managed according to the corresponding relation between the metadata structure and the small files of the users. The user space metadata server used for storing the metadata information of the user space is introduced, the small files are operated by means of the corresponding relation between the metadata structure and the small files of the users, and the performance of an HDFS for storing and processing the small files can be improved.

Description

technical field [0001] The invention belongs to the field of performance optimization of a distributed file system, in particular to an optimization method based on HDFS small file storage. Background technique [0002] With the large-scale growth of information volume, enterprises invest more and more in data storage. There is an urgent need for new storage solutions to change the status quo, save storage costs, and reduce storage investment. Cloud storage has emerged as the times require. The prototype of cloud computing was designed by Google to turn waste into treasure. Therefore, the cloud storage architecture has its unique advantages, which cannot be replaced by traditional centralized storage. In the current form of large-scale data growth, the advantages of cloud storage over traditional storage models include cost reduction, on-demand allocation, strong scalability, strong flexibility, strong fault tolerance, and convenient data migration. [0003] Hadoop is an op...

AI Technical Summary

Problems solved by technology

[0008] The purpose of the present invention is to overcome the problem in the prior art that storing a large number of small files in HDFS will cause the NameNode node to use too much memory and cause performance degradation, and provide a correspondence between the metadata structure and the user's small files To operate on small files, an optimization method based on HDFS small file storage that can improve the performance of HDFS system storage and processing small files

Images