Spark SQL-based distributed full text retrieval system and method

Abstract

The invention relates to a Spark SQL-based distributed full text retrieval system and method. The system comprises an SQL translation layer, a data source management layer, a parallel calculation layer and a distributed storage layer; an SQL-based full text retrieval method and translation processes, among modules of the SQL translation layer, of full text retrieval SQL statements are proposed; a full text retrieval process parallelization method is designed in a data source management module; and in a retrieval optimization module, two index storage models and corresponding primitive table data reduction strategies during query are designed, wherein a partition align connection algorithm which is used for reducing primitive table data during query and has a complexity of O (n) is designed for an index appointed column-based storage model. Under the two storage models, the index construction time is shortened to 0.6% / 0.5% of the traditional database, the query time is shortened to the 1% / 10% of the traditional database, and the index storage amount is decreased to 55.0% of the traditional database. According to the method, the Spark SQL data analysis function is strengthened, and the requirements for traditional business migration and full text retrieval carried out on mass data in the existing businesses can be satisfied.

Description

technical field [0001] The present invention relates to data analysis and information retrieval technology under massive data, and more specifically relates to a distributed full-text retrieval system and method based on Spark SQL. It belongs to the field of software technology. Background technique [0002] With the development of technologies such as cloud computing and the Internet of Things, as well as the emergence of blogs and social networks, application models based on location services (LBS) have emerged (see literature: Meng Xiaofeng, Cixiang. Big Data Management: Concepts, Technologies and Challenges [J]. Computer Research and Development, 2013, (01): 146-169.), the types and scale of data are growing at an unprecedented rate, and the valuable value contained in big data has become the driving force for people to store and process big data (see literature: Cheng Xueqi , Jin Xiaolong, Wang Yuanzhuo, Guo Jiafeng, Zhang Tieying, Li Guojie. A review of big data syste...

AI Technical Summary

Problems solved by technology

[0006] The technology of the present invention solves the problem: overcomes the problem that existing data analysis tools do not support full-text retrieval under massive data, provides a distributed full-text retrieval system and method based on Spark SQL, enhances the data analysis function of Spark SQL, and can effectively meet traditional business requirements Migration and existing business needs for full-text retrieval of massive data

Images