Real-time data storage and query method

Abstract

The invention provides a real-time data storage and query method. The method comprises the steps that when storage is conducted, partition is conducted on data to be stored by adopting consistency Hash sharding, primary index data carried with index information of the consistency Hash sharding are obtained from at least one distributed node of a distributed storage system, merger processing is conducted on at least two primary index data of one same distributed node to form a distributed file block corresponding to the distributed node, and a clustered index is established for the primary index data in the distributed file block; secondary index data carried with clustered index information are obtained from the distributed file block corresponding to the distributed node, and the secondary index data in the distributed file block are written in a disk in which the corresponding distributed node is located; when query is conducted, by means of the index information of the consistency Hash sharding established when the data to be inquired are stored and the clustered index information established in the distributed file block, a needed result can be located accurately and rapidly.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a real-time data storage and query method. Background technique [0002] In recent years, with the rapid development and popularization of computer and information technology, the scale of industrial application systems has expanded rapidly, and the data generated by industrial applications has grown explosively. Since the value of data is positively correlated with real-time performance, that is, the newer the data, the more valuable it is. Therefore, how to effectively store, analyze, and process increasing large-scale real-time data is a hot research topic now. [0003] At present, traditional methods mainly use databases and data warehouses or distributed file systems based on Hadoop / Spark to complete the storage and reading of large-scale real-time data, and then realize the analysis and processing of real-time data. Specifically, the method of storing and reading real-ti...

AI Technical Summary

Problems solved by technology

Specifically, the method of storing and reading real-time data using databases and data warehouses is mainly completed by building B+ trees or bitmap indexes in traditional databases, but this method occupies a relatively large amount of database memory space when storing real-time data. And it takes a long time to query real-time data, and it is difficult to meet the performance requirements of large-scale real-time data storage and query

The method of data storage based on Hadoop/Spark distributed file system i

Images