Big data-based online analytical processing system and method

Abstract

The invention discloses a big data-based online analytical processing system and method. The system can be used for carrying out quick multi-dimensional query and analysis on data sets with different scales and levels under a Hadoop environment. A query plan selected through query, planning and estimation comprises MDX query supporting Hive and Hbase precomputation cache mechanism-based multi-dimensional query. According to the system and method, optimization of the MDX query supporting Hive data warehouses on extensible cluster nodes and of the Hbase precomputation cache mechanism-based multi-dimensional query are realized, the low-delay multi-dimensional query requirements of the data sets with different scales and levels are satisfied, and the OLAP multi-dimensional query of different OLAP data organization models under a single data source background is solved. Aiming at the performance optimization problem of Hive multi-dimensional query on large-scale data sets, an Hbase cache-based segmented layered dimensionality-reduction aggregation algorithm is proposed, and the algorithm brings MOLAP for solving the multi-dimensional query calculation of large-scale data into a big data OLAP system, so that the extendibility and effectiveness of the multi-dimensional query of data with different scales and levels under a big data background are greatly enhanced.

Description

technical field [0001] The present invention relates to an analysis and processing method under the environment of big data Hadoop (a software platform for developing and running big data), in particular to an online analysis and processing system and method based on big data. Background technique [0002] In recent years, with the continuous development of OLAP (On-Line Analytical Processing) technology, OLAP system products emerge in an endless stream, but most of them are ROLAP (Relational Database On-Line Analytical Processing) systems based on relational databases or a single MOLAP (Multidimensional Database On-Line Analytical Processing) system. )system. Although, the continuous enhancement of single-node memory scalability and column-oriented in-memory database technology has improved the query performance of ROLAP systems. However, the scale of terabytes to petabytes of data generated by enterprise-level applications has exceeded the maximum query limit that traditi...

AI Technical Summary

Problems solved by technology

[0003] 1. The traditional ROLAP system's MDX (a query language that supports multidimensional object and data definition and operation) query is limited by specific databases, and cannot be completed on scalable cluster nodes. Support Hive (a data warehouse tool based on Hadoop) data warehouse MDX query, the traditional ROLAP system architecture has become a bottleneck factor restricting the performance of multi-dimensional query analysis in terms of scalability

[0004] 2. In MDX query, the traditional ROLAP system lacks certain flexibility

Therefore, it only involves low-latitude small-scale data multi-dimensional query, and the performance of aggregation calculation in memory will be restricted by large-scale, high-latitude data volume query requirements

At the same time, when faced with large-scale data processing, the excessive connection operations of ROLAP restrict the query processing performance

Images