Method based on Hadoop small file optimization and reverse index establishment

Abstract

The invention discloses a method based on Hadoop small file optimization and reverse index establishment. A large number of small files can be uploaded to an HDFS, and reverse indexes can be established for the files in the HDFS. The method comprises small file optimization and reverse index establishment and mainly comprises the steps that (1) a user uploads a large number of small files corresponding to HDFS blocks in size to small file queues in Hadoop; (2) the size of the small files in the file queues is calculated regularly, (3) the files, meeting requirements, in the small file queues are combined through the Sequence file method and then are uploaded to the HDFS; (4) the reverse indexes are established for the files in the HDFS. According to the method based on Hadoop small file optimization and reverse index establishment, the defect that the Hadoop small file process is inconvenient is overcome, the processing performance of the small files can be optimized, the memory is released, and the retrieval speed and efficiency are improved.

Description

technical field [0001] The invention relates to the field of system processing, in particular to a method based on small file optimization and inverted index in Hadoop. Background technique [0002] When the size of a file is smaller than the block size on HDFS (Distributed File System), such a file is called a small file in Hadoop. A large number of small files will seriously affect the scalability and performance of Hadoop. [0003] (1) In the HDFS distributed file system, any file information and file block information are stored in the form of an object in the memory of the NameNode (master node). Each object occupies about 150 bytes. A large number of small files make the NameNode Memory usage severely limits cluster scaling. [0004] (2) The speed of HDFS accessing a large number of small files is much slower than accessing large files of the same size. It is primarily designed for streaming access to large files. Reading small files usually results in a large numb...

AI Technical Summary

Problems solved by technology

At present, there are two retrieval methods to retrieve unstructured data. The first is the sequential scanning method, which scans the unstructured document from the head to the end of the document. If the required information is found in the document, the document is scanned Record it, and then select the next document to continue scanning from beginning to end until all documents are scanned and return to the recorded document, which is less efficient

Images