A hbase column aggregation method

Abstract

The invention discloses a column aggregation method of HBASE. Based on the HBase server query mechanism, the system architecture of the method is provided with an aggregation scanner module, and the description information of the aggregation table is expanded, and column aggregation attributes are added therein; in the query When merging, the multi-version data of the same cell is aggregated and calculated according to the column aggregation attribute to realize the aggregation on the cell; when the data is merged, the HBase server will have multiple versions according to the column aggregation attribute information in the aggregation table description. The data is aggregated and the result is retained, and the non-result data is cleared after the merge operation is completed. Compared with the prior art, the method disclosed in the present invention has better improvements in terms of real-time performance, performance and resource usage of aggregation table query.

Description

technical field [0001] The invention discloses an HBase column aggregation method and relates to the technical field of computer information storage. Background technique [0002] HBase is a high-reliability, high-performance, column-oriented, and scalable open source non-relational database implemented with reference to Google's BigTable, using HDFS as the underlying storage. With the development and application of big data technology, HBase has gradually become a NoSQL distributed storage system widely used in the industry. It is highly reliable, column-oriented, and open source, and has been successfully used in production systems by companies such as Facebook and Alibaba. [0003] The HBase data model is shown in the following table: [0004] [0005] The most basic unit of HBase is the column (Column, Qualifier); one column or multiple columns form a column cluster (Family, Store), and one column cluster or multiple column clusters form a row (Row), and the unique r...

AI Technical Summary

Problems solved by technology

[0016] 1) MapReduce aggregation scheme: actually adopt the Read-Write method, that is, first read out the existing values ​​in the library, rewrite them after calculation, and increase the IO request; MapReduce needs to re-scan all the data in the table every time and repeat the calculation Relatively large, waste of computing resources; MapReduce batch calculation is relatively poor in real-time; cannot support convective data storage, only fixed files can be used as input

[0017] 2) Scan&Endpoint aggregation scheme: each aggregation query needs to query all relevant data for aggregation operation, which requires more resources, and is prone to timeout exceptions, which affects query performance; the historical data that needs to be aggregated in the data table requires a long-term Save, increase data redundancy, consume storage, aggregate query performance is getting slower and slower, and resource waste is increasing; Scan&Endpoint server can only achieve aggregation on the same region, and the client needs a second aggregation operation

Images