An automatic identification and optimization method based on massive data-like SQL retrieval scenarios

Abstract

The invention provides an automatic recognition optimization method based on mass data type SQL retrieval scene, and belongs to the statistical and analysis field of the mass data. The method comprises five aspects to perform the optimization: introducing lucene as an optional storage medium; adding a bloomfilter index (bf index) for a retrieval field of each data file; dividing different retrieval scenes and selecting an optimal storage medium; converting a SQL-like statement into a lucene statement when performing the lucene retrieval scene; and an adding an valid session-level setting for the lucene storage medium and the bf index. The session-level validity of the bf index is judged while retrieving the mass data, a to-be-retrieved data file list is reduced through the bf index, and then the session-level validity of the lucene storage medium is re-judged. Through the adoption of the method provided by the invention, the resource consumption of the cluster while retrieving the mass data is effectively lowered, and the retrieval performance of the mass data is greatly improved.

Description

technical field [0001] The invention belongs to the technical field of statistical analysis of massive data, and relates to a technical scheme for automatically identifying massive data retrieval scenarios based on SQL mode, and selecting corresponding means to speed up retrieval. Background technique [0002] With the rapid development of information science and technology, various forms of massive data such as webpage files, text data, multimedia data, etc. are continuously generated, resulting in a rapid expansion of data scale, and the application fields of various types of data are also expanding. The applications are as follows Features: First, the data scale is large and continues to grow, and these data need to be saved for statistical analysis; second, the requirements for complex query operations and online transaction processing capabilities are high, and the requirements for response time are relatively strict. Moreover, this is carried out under the condition of...

AI Technical Summary

Problems solved by technology

However, the retrieval performance requirements of various retrieval scenarios cannot be met only through the massive data storage of a single storage medium. Therefore, on this basis, the present invention provides an automatic identification and optimization technology for SQL retrieval scenarios of massive data to meet the requirements of different retrieval scenarios. High Performance Retrieval Requirements for Scenes

Images