A method for storage and near-real time query of time-sensitive data based on open source big data

Abstract

The invention provides a method for storage and near-real time query of time-sensitive data based on open source big data. The method comprises the steps of establishing a near-real time query processing platform having an internal storage space and an external storage space; defining a file storage strategy and performing data processing and calculating on source data files in the internal storage space so that the source data files are stored in the external storage space after being arranged according to the time-sensitive characteristics thereof; performing reverse index with the time-sensitive characteristics of the data files as the filter conditions, establishing point index and range index to generate index information and storing the index information into the external storage space and caching the information into the internal storage space; inquiring the index information and searching the point index or range index to obtain relevant file path lists, and reading source data files corresponding to query requests according to the file path lists. Fully based on the time-sensitive characteristics, the data filter strategy is designed to reduce data scanning quantity, and thus the near-real time query feedback of big data is realized.

Description

technical field [0001] The invention belongs to the field of database technology and information processing, and in particular relates to a storage and near real-time query method for time-sensitive data based on open source big data. Background technique [0002] With the development of wireless technology and the advancement of terminal equipment, the trend of data massification has been demonstrated in various industries. In the field of scientific research, such as astronomical observation data, meteorological data, ocean monitoring data, etc., with the maturity of the sensor network, the collection of these data becomes easy, resulting in the explosive growth of log information; in addition, in the field of decision-making, such as stock trading Daily transaction data, corporate reports, and Weibo data in the market are also booming. In addition to the more significant characteristics of these data in terms of quantity, the potential correlation between data also has r...

AI Technical Summary

Problems solved by technology

The data operations supported by HBase are very limited, and because it only supports efficient queries based on primary keys, the definition of primary keys is very important; the performance support for range queries is good, but the performance of large-scale scanning is extremely poor

In addition, frequent insertion or update of HBase will greatly affect system performance, and because it does not support secondary indexes, it currently only supports indexes on primary keys, so the query performance for non-primary keys is poor

[0022] In order to overcome the shortcomings in the existing technology that big data queries cannot provide second-level feedback speed, open source databases do not support secondary indexes, high database storage expansion rate, slow data insertion and update speed, and poor performance of analysis services based on data scanning, etc., The present invention proposes a storage and near-real-time query method based on open source big data to solve time-sensitive data. By fully combining time-sensitive characteristics, a data filtering strategy is designed to reduce the amount of data scanning

Images