Big data multidimensional data analysis method and system based on Hadoop and HBase

Abstract

The invention provides a big data multidimensional data analysis method based on Hadoop and HBase, and the method comprises the steps: defining a multidimensional data model, the dimension of the multidimensional data model being selected from fields of a table of source data according to the needs of a user, and configuring a needed analysis measurement; reading corresponding data from a data source according to the dimension n of the defined multi-dimensional analysis data model, calculating the data by adopting Hadoop to obtain data cubes of 2n combinations, and storing the data cubes intoan HBase database; inputting a database query statement, and forwarding the database query statement to a query engine through a Rest service layer, converting the database query statement into a corresponding API statement conforming to HBase query by the query engine, and obtaining data or a data set conforming to query conditions in the HBase through routing selection according to the API statement and in combination with a query mechanism of the HBase. The invention further provides a big data multi-dimensional data analysis system based on Hadoop and HBase. Real-time query, grouping and aggregation under a super-large data set are met, and the data processing efficiency is improved.

Description

technical field [0001] The invention relates to the field of databases, in particular to a Hadoop and HBase-based big data multidimensional data analysis method and system. Background technique [0002] Hadoop is an open source framework of Apache for processing large data sets. The Hadoop framework application engineering provides a distributed storage and computing environment across computer clusters. Hadoop is designed to expand from a single server to thousands of machines, and each machine can provide local computing and storage. At its core, Hadoop mainly has two levels, namely: processing / computing layer (MapReduce), and storage layer ( Hadoop Distributed File System). [0003] HBase is a high-reliability, high-performance, column-oriented, and scalable distributed NoSQL database. Its design goal is to solve the theoretical and implementation limitations of relational databases when processing massive data. The number of records (rows) in each table in HBsae can re...

AI Technical Summary

Problems solved by technology

[0004] Encountered a bottleneck in the use of traditional databases and big data platforms, that is, the traditional architecture only supports vertical expansion, and the data processing capability is improved by adding hardware resources such as memory and CPU on a machine, but the relative data grows exponentially, and single-machine expansion soon to reach the limit

Although the Hadoop big data platform can store and calculate large-scale data, it cannot provide efficient interactive query, and the query efficiency is low

Images