A file retrieval system based on hdfs

Abstract

HDFS-based file retrieval system, including system configuration module, file management module, index management module, retrieval portal module, MongoDB database, HDFS cluster, Spark cluster, and ElasticSearch cluster; the file management module stores files in the HDFS cluster; the index management module passes The Spark cluster creates an index and stores it in the ElasticSerch cluster; the retrieval portal module sends the retrieval conditions to the ElasticSearch cluster for index matching to obtain the retrieval results, and the MongoDB database is used to store the records produced in the file retrieval process; the HDFS cluster and the Spark cluster of the present invention Both ElasticSearch and ElasticSearch clusters are distributed to reduce query load and improve query efficiency; the master-slave architecture has horizontal scalability and stability, which is convenient for improving the overall processing capacity of the cluster, and the system is in a stable working state; the replica redundancy strategy is adopted to ensure indexing reliability and integrity.

Description

technical field [0001] The invention belongs to the field of distributed search engines, and in particular relates to an HDFS-based file retrieval system. Background technique [0002] The traditional full-text retrieval system is implemented based on Lucene. Using Lucene can realize the establishment, optimization and query of file index. But when the number of index files increases to a certain amount, due to the internal mechanism of Lucene, a large amount of data is loaded into the memory, and will be discarded after the query is completed. After a large amount of data occupies the memory, the Java Virtual Machine (JVM) will be frequently shortened and recycled, resulting in a serious bottleneck in query performance. And the traditional file retrieval systems are stand-alone systems. With the advent of the big data era, stand-alone file retrieval systems cannot handle massive amounts of data, and its index creation takes a long time and query efficiency is low. [00...

AI Technical Summary

Problems solved by technology

But when the number of index files increases to a certain amount, due to the internal mechanism of Lucene, a large amount of data is loaded into the memory, and will be discarded after the query is completed.

After a large amount of data occupies the memory, the Java Virtual Machine (JVM) will be frequently shortened and recycled, resulting in a serious bottleneck in query performance

Moreover, traditional file retrieval systems are stand-alone systems

With the advent of the era of big data, the stand-alone file retrieval system cannot handle massive amounts of data, and its index creation takes a long time and the query efficiency is low

Images