A hbase secondary index query and storage system and its query method

Abstract

The present invention relates to the technical field of data processing, in particular to an HBase secondary index query and storage system and query method. The HBase secondary index query and storage system comprises a client, a query processing module, a query execution engine module and an HBase storage module. According to the HBase secondary index query and storage system and query method, filter columns originally stored in each index table are combined into a main filter column table to enable filter column data requiring redundancy to be greatly reduced, thereby reducing storage redundancy of data; and meanwhile, the queries to all index tables are finally integrated into the query to the main filter column table through a mixer filter, the main filter column table is stored in a rapid storage medium at the same time, and the query efficiency is relatively high; moreover, the query efficiency is greatly improved by using the rapid random access performance of an SSD hard disc.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to an HBase secondary index query and storage system and a query method thereof. Background technique [0002] With the development and application of big data technology, HBase has gradually become a NoSQL distributed storage system widely used in the industry. It is highly reliable, column-oriented, and open source, and has been successfully used in production systems by companies such as Facebook and Alibaba. How to efficiently store and query secondary indexes on HBase is a research hotspot in the industry. At present, the widely used scheme architecture is as follows: image 3 As shown, the solution mainly includes three modules: HBase storage module, query processing module, and query execution engine. The HBase storage module is responsible for the storage of original data and index data. It has the characteristics of distributed, large capacity, and fast response. Si...

AI Technical Summary

Problems solved by technology

[0003] Existing secondary index storage and query schemes require redundant filter columns in each index table of the data table, resulting in large data redundancy; when querying, if the index table does not contain all the filter conditions in the query conditions, the The index table cannot be searched. If all the index tables are not searchable, it is necessary to use the query condition to construct a filter and perform a full table scan on the data table, which is extremely inefficient

In addition, each index table stores a filter list, and the data storage redundancy is large. In the existing scheme, when querying different index tables, it is necessary to filter the filter columns of different index tables, and the query efficiency is low.

Images