Method for accessing archived file based on B + tree index

Abstract

The invention discloses a B + tree index-based archived file access method, which comprises the following steps of: file division: calling a file filter, and dividing an archived file into a large file, a small file and an ultra-small file according to the size of the archived file; file merging processing: sending the small files and the ultra-small files to a file merging module, and merging and classifying the small files and the ultra-small files into index files and data files by using a file merging strategy; constructing an index system: constructing the index system for storing the index file in an HDFS (Hadoop Distributed File System); file uploading: directly uploading a large file to the HDFS file system; meanwhile, the index file and the data file which are merged and classified are uploaded to the HDFS file system; and file index query: querying the small file information according to the file size based on the index system. According to the technical scheme, the problem that in an original scheme, the small file access performance is poor can be solved, and the small file retrieval speed is increased while NameNode memory consumption is reduced.

Description

technical field [0001] The present application relates to the technical field of file indexing, and more specifically, relates to an access method for archiving files based on a B+ tree index. Background technique [0002] When the original HDFS stores a large number of small files, because each small file will generate metadata in HDFS and save it in the memory of the NameNode node, the following three problems will occur, mainly as follows: [0003] (1) Impact on NameNode performance: Since the metadata information of small files is stored in the NameNode master node server, the metadata of each small file requires about 250 bytes of memory. Each small file exists in the form of data blocks in HDFS. For each block with default three copies, its metadata requires about 368 bytes of memory. As a result, when too many small files are stored in HDFS, a large amount of memory space of the NameNode will be occupied, which seriously degrades the performance of the NameNode. [...

AI Technical Summary

Problems solved by technology

This will cause too many small files stored in HDFS, which will occupy a large amount of memory space of NameNode and seriously degrade the performance of NameNode.

[0004] (2) Poor processing performance: when processing a large number of small files, each request can only process a single file, and small files need to be read one by one, requiring MapReduce (HDFS data processing framework) to perform multiple processing on different nodes in the cluster Read and write operations, which take far longer than processing a single file of the same size as this batch of small files

[0005] (3) Long storage time: Since each small file uploaded by the client will send a request to the NameNode node, the NameNode node needs to check its own metadata information after receiving the request to confirm whether the target file exists, whether the parent directory exists, etc. Feedback on whether the client can upload the target file, it will take a lot of time

However, after processing according to the above method, when we need to access the small files in the large file, because the metadata information before the small file is stored in the memory of the NameNode to ensure fast access, it is now dumped into these three file structures , the file access mechanism has changed, resulting in poor access performance

[0015] Therefore, how to provide an access method based on the B+ tree index archive file, which can solve the problem of poor access performance of small files in the original solution, and realize the reduction of NameNode memory consumption while improving the speed of retrieval of small files has become an important issue. Technical problems to be solved urgently by those skilled in the art

Images