High-performance file storage and management system based on HDFS

Abstract

The invention provides a high-performance file storage and management system based on an HDFS. A file preprocessing module and a file merging module are provided for overcoming the defect that an HDFS(Hadoop Distributed File System) supports small file storage, a pre-fetching cache module is provided for improving the file reading efficiency, a fragmentation module is provided for improving the overall space utilization rate, and a data encryption module is provided for protecting private files of a user. The invention provides a universal small file merging strategy and an HDFS client optimization scheme, and combines the advantages of an HAR file archiving method and a MapFile method so as to be suitable for any type of files. Specifically, the system optimizes a small file merging strategy and provides a defragmentation mechanism which is easy to update so as to improve the space utilization rate of an HDFS cluster. Meanwhile, a hotspot file strategy based on frequency statistics improves the reading speed of data, and improves the safety of a user account and a private file based on MD5 login encryption and AES file encryption.

Description

technical field [0001] The present invention relates to the field of computer science, and more specifically, relates to an HDFS-based high-performance file storage and management system. Background technique [0002] With the popularization and application of the mobile Internet, global data information is growing explosively, and emerging industries such as artificial intelligence, machine learning, and cloud computing are developing rapidly. The development of traditional computer storage systems can no longer meet the needs of large amounts of data generation and application. At present, the commonly used distributed file systems include Hadoop, Lustre, MogileFS, GoogleFS, etc. Hadoop has a series of advantages such as high reliability, strong scalability, fast storage speed, and high fault tolerance, and HDFS is exactly Distributed file system for Hadoop. [0003] However, the calculation and storage of big data is not the only problem faced by the huge information flo...

AI Technical Summary

Problems solved by technology

There are two problems with HAR file archiving: reading a file requires traversing two layers of indexes, which may be less efficient than direct reading. Second, once a file is created, it cannot be modified. If you want to add or delete a merged file, you only need Can create new files again

Some of these methods are suitable for specific scenarios and data, and some implementations are too complex to achieve wide application

[0012] To address the security issues of HDFS, Hadoop introduces encryption space, but it has the disadvantages of a single encryption algorithm, no support for iterative directory encryption, high usage rights, and no application-level encryption

Images