Method and system for locality non-cluster index based on streaming data

Abstract

The invention discloses a method and a system for locality non-cluster index based on streaming data. The method comprises: a real-time update step: updating index records generated aimed at each received piece of streaming data in a Hash index table in real time, the index record recording an index key in the streaming data, a major key corresponding to the index key when the index key appears for the first time, and the number of data covered from first appearance to last appearance at present; a writing-in step: when a trigger condition is achieved, writing the index records in the Hash index table in an index table, and continuing to execute the real-time update step. The method greatly reduces space of the index table and bandwidth cost generated by establishing the index table. The index method combines random access and sequential scanning, effectively uses time locality characteristic of the streaming data, and is more aligned with an access model of a storage medium, and improves inquiry efficiency of index data.

Description

technical field [0001] The invention relates to the field of large-scale data processing, in particular to a localized non-clustered index method and system based on streaming data. Background technique [0002] The current streaming index technology mainly establishes a corresponding index for each piece of data in real time through the indexing method of the traditional database. [0003] Such as figure 1 Shown is a schematic diagram of an index establishment method in the prior art. [0004] figure 1 The left table in the middle is the streaming data continuously received by the streaming data processing system, which is displayed in the form of a data table. The primary key in the table is used to identify each piece of data in a time-increasing manner, and the column to be indexed is the The data items that can be indexed, each piece of data can also have other data items such as data columns. [0005] In the existing technology, in order to facilitate retrieval and...

AI Technical Summary

Problems solved by technology

[0008] 1. It takes a lot of space to store the index table, and it is very expensive to build the index table in real time for streaming data

[0009] 2. Sequential scanning of an index table will lead to random access to the data table, loss of spatial locality, and degradation of read performance

[0010] 3. Does not take advantage of the temporal locality of streaming data

Images