Mass small file processing method based on HDFS

Abstract

The invention provides a mass small file processing method based on an HDFS (Hadoop Distributed File System). The method comprises the following steps of generating a file ID (Identifier) through small file filtration and metadata reading to complete preprocessing on files; opening up a memory cache region; building a file uploading queue; performing time-delay storage on the files; merging the small files into a Mapfile file of a <key, value> structure in the cache region to be stored; storing file metadata into a distributive database Hbase, wherein the Hbase realizes the persistence in the HDFS; expressing the file status by a Status flag bit to complete operations such as fast reading of the small files of the cache region and Mapfile fragment merging, so that the real-time insert update and delete operation of the HDFS on the small files can be supported. The method has the advantages that the reading efficiency of the HDFS on the small files is improved, so that the system supports the real-time update operation on the small files, and the integral performance of the system is improved.

Description

technical field [0001] The invention relates to a method for processing massive small files based on HDFS, belonging to the field of distributed data optimized storage. Background technique [0002] With the rise of Internet web2.0, the amount of network data is increasing exponentially. In the era of big data, traditional data storage technology can no longer meet the needs of technological development. Hadoop Distributed File System, HDFS for short, is a distributed file system. At present, in the field of distributed file storage technology represented by HDFS, HDFS is widely used to efficiently process various large files. [0003] HDFS is optimized for high data throughput application scenarios. In other words, it is developed for accessing large files. If you access a large number of small files, you need to continuously jump from a Datanode (a data node of HDFS, which provides storage blocks for HDFS) to Another Datanode, severely impacting performance. Finally, pr...

AI Technical Summary

Problems solved by technology

Hadoop itself provides a solution: HadoopArchive, which is the Hadoop archive format (hereinafter referred to as HAR). Reading a file through HAR is actually less efficient than reading a file directly from HDFS; Suzhou Liangjiang Technology Co., Ltd. in its In the patent, SequenceFile (a key-value file format of Hadoop) is used to package files. Since SequenceFile has no direct index, it is inefficient to retrieve the entire file every time it is read; Mapfile (a key-value file format of Hadoop) type format) is an indexed SequenceFile, but requires additional memory to save the index file Metadata (metadata)

[0005] In addition, several merging file solutions provided by Hadoop must be packaged and uploaded at one time. In this way, the file cannot be deleted, modified, or appended after the file is uploaded.

In the patent of Beijing University of Aeronautics and Astronautics, the interface of HDFS to read files is improved, and the MapReduce (Hadoop programming model) model is used for processing, but this method is not suitable for the environment of online real-time storage modification, and cannot achieve high concurrency Image upload request processing

This prevents the performance of HDFS from being fully utilized in many application areas.

Images