A time series retrieval index system and method for super large-scale data

Abstract

The invention discloses an index system and an index method for retrieving time sequences of ultra-large-scale data. The index system comprises a vertical hierarchical index module and a corresponding time axis index module. The vertical hierarchical index module comprises a plurality of hierarchies of indexes, each hierarchy of indexes comprises a hash function and a plurality of data sets, and data in the data sets are mapped into a corresponding next hierarchy of data set by the aid of the hash function of the corresponding next hierarchy of indexes; the time axis index module is used for creating event lists and time lists. The index system and the index method have the advantages that original big data sets can be ultimately distributed into a plurality of small data sets by means of hierarchy -by- hierarchy hash mapping by the aid of the hierarchical index module, operation such as query processing, data loading and storage optimization can be independently executed on each small data set, accordingly, the vertical hierarchical index module can be combined with the time axis index module, risks of full-table scanning operation in time sequence retrieval operation procedures can be prevented, and the time sequence retrieval speeds can be greatly increased; the index system and the index method for retrieving the time sequences of the ultra-large-scale data can be widely applied to the field of data processing.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a time-series indexing system and method for ultra-large-scale data. Background technique [0002] The field of time-series data management of time-series database also involves the time-series retrieval operation on the data in the database. Internally, the time-series database also implements various time-series data management functions efficiently by establishing a data index for the time-series data. In general, these data indexes in the time series database are mainly divided into two categories, one is the index based on the B+ tree structure, and the other is the index based on the R tree structure. For example, there are several specific index structures such as Time Index, Snapshot Index, CheckpointIndex, Archivable Time Index, Overlapping B+ tree, etc. [0003] Timeline Index is an index structure proposed by Martin Kaufman et al. in 2013. It mainly serves the time-ser...

AI Technical Summary

Problems solved by technology

[0007] As mentioned above, the time-series indexes of various time-series databases can effectively solve various time-series data management problems in time-series databases. However, they have a common feature that they are all designed for traditional relational time-series databases. They are designed to deal with conventional-scale data volumes, usually millions of data, and they cannot cope with the technical challenges brought about by the ultra-large-scale data of more than 1 billion in the current big data era

When the total amount of data in the data set to be processed becomes larger and larger, the above-mentioned time series index of the time series database will have serious performance and efficiency problems, making it impossible to return effective time series retrieval results within an acceptable time

Images