Implementation method of HBase secondary index

Abstract

The invention particularly relates to an implementation method of HBase secondary index. According to the implementation method of the HBase secondary index, an Elasticsearch search engine, a NiFi data flow platform and an HBase distributed column storage database are integrated; wherein the NiFi data flow platform is responsible for extracting source data and writing the source data into the Elasticsearch engine and the HBase distributed column storage database, the Elasticsearch engine is responsible for storing indexed data and a rowkey primary key of the HBase distributed column storage database, and the HBase distributed column storage database is responsible for storing full detailed data; then an Elasticsearch engine, according to the query condition, searches a rowkey primary key of the HBase distributed column storage database, and the detailed data stored in the HBase distributed column storage database is queryed by using the rowkey primary key as the query condition, so that an efficient query retrieval function is provided for the HBase. According to the implementation method of the HBase secondary index, the memory cache space and the disk storage space of the server can be saved, efficient query and retrieval functions are provided for the HBase, and the query and retrieval efficiency is greatly improved.

Description

technical field [0001] The invention relates to the technical field of data retrieval, in particular to a method for realizing an HBase secondary index. Background technique [0002] With the rapid development of computer technology and network technology, massive data is stored in the HBase database. In the HBase database, only the Rowkey primary key is used as the first-level index. If you want to perform data retrieval and query on the non-primary key fields of HBase, you often need to scan the entire table through a distributed computing framework such as MapReduce / Spark, and the hardware resource consumption and time delay will be relatively high. . HBase cannot satisfy the fast and complex query function of data. The advantages and disadvantages of HBase data storage are as follows: [0003] Apache HBase is a Hadoop database, a distributed, scalable, big data storage database. The HBase distributed column store database can host very large tables on commodity hardw...

AI Technical Summary

Problems solved by technology

HBase is based on the rowkey primary key query speed in milliseconds, but HBase is not suitable for complex logic queries, which often require full table scanning, which consumes a lot of resources

Images