Method and system for statistically querying HBase based on analysis performed by Hive on HFile

Abstract

The present invention relates to a method and system for statistically querying a HBase based on analysis performed by a Hive on an HFile. The method comprises the following steps: step S1, creating a detailed-list storage and query table; step S2, adding an original detailed list to the HBase based on the detailed-list storage and query table, and storing the original detailed list as an HFILE file in an HDFS of the HBase; and step S3, the Hive performing relational and statistical query based on the HFILE file. The method also comprises: directly accessing the HBase to search for a detailed list. According to the method and system provided by the present invention, a concurrent data reading operation can be completed efficiently; distributed computing is implemented by using MapReduce so as to execute a computing task in a distributed environment and output a result, wherein computing is independent from each other and can be dynamically expanded; response efficiency of a high-concurrent reading operation on big data is greatly improved; and data is not backed up, meeting the requirement of data consistency.

Description

technical field [0001] The invention relates to the field, in particular to a method and system for statistically querying HBase based on Hive analysis of HFile. Background technique [0002] There are three schemes for querying HBase in the prior art. [0003] Solution 1: Direct access to HBASE to read data. The storage application initiates a read operation command, and the HBase client accepts the data read request, and determines which RegionServer the data should be handed over to for processing based on the read data and the currently cached region. If the Region has no cache, then request Zookeeper to get the RegionServer. The client sends the read request to a specific RegionServer, and the RegionServer first queries in the data cache, and returns directly if it hits. Otherwise, the data is read from the distributed file system, cached and returned to the client. [0004] Solution 2: Based on distributed computing, use HIVE to associate HBASE with statistical que...

AI Technical Summary

Problems solved by technology

[0006] The above three solutions have certain defects in the tight coupling of system architecture, high concurrency, and data consistency. Difficult to separate and cut

High concurrency: Hive is currently used for statistics, the Hbase service node has a large load, and the waiting time for operations is long

Images