A system and method for online analysis and processing based on big data

An online analytical processing and big data technology, applied in the field of analytical processing, can solve problems such as query processing performance constraints, constraints, and lack of flexibility in ROLAP systems

CN106372114BActive Publication Date: 2019-09-10UNIV OF ELECTRONICS SCI & TECH OF CHINA
7 Cites 2 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2019-09-10

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses an online analysis and processing system and method based on big data. The system can perform fast multi-dimensional query and analysis on data sets of different scales and levels under the Hadoop environment. Query plans selected through query plan evaluation include Hive-supported MDX queries and multi-dimensional queries based on Hbase's precomputing caching mechanism. Realize support for MDX query of Hive data warehouse on scalable cluster nodes, multi-dimensional query optimization based on Hbase pre-computing cache mechanism, meet the low-latency multi-dimensional query requirements of data sets of different scale levels, and solve different OLAP data organization models under the background of a single data source OLAP multidimensional query. Aiming at the performance optimization problem of Hive multi-dimensional query on large-scale data sets, a layer-by-layer dimensionality reduction aggregation algorithm based on Hbase cache is proposed. This algorithm brings MOLAP, which solves the multidimensional query calculation of large-scale data, into the big data OLAP system, which greatly enhances the scalability and efficiency of multidimensional query of data at different scale levels under the background of big data.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to an analysis and processing method under the environment of Hadoop (a software platform for developing and operating large data processing) of big data, and in particular to an online analysis and processing system and method based on big data. Background technique

[0002] In recent years, with the continuous development of OLAP (Online Analytical Processing) technology, OLAP system products emerge in endlessly, but most of them are ROLAP (Relational Database Online Analytical Processing) systems based on relational databases or a single MOLAP (Multidimensional Database Online Analytical Processing) system. )system. Although, the continuous enhancement of single-node memory scalability and column-oriented memory database technology has improved the query performance of the ROLAP system. However, the terabyte to petabyte data scale generated by enterprise-level applications has exceeded the maximum query limit that trad...

Examples

Embodiment Construction

[0081] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

[0082] Such as figure 1 As shown, an online analysis and processing system based on big data, the system includes user interface module, query planning module, MDX query interpretation module, HQL query interpretation module, metadata management module, MDX aggregation cache module, Cube construction based on Hbase Cache module and data storage module;

[0083] The user interface module accepts MDX and HQL query requests from users on data sets of different scales for calling by the query planning module;

[0084] The MDX query interpretation module is responsible for explaining and executing query processing to MDX, completing the entire interpretation and query calculation, and realizing the online analysis and processing of reading dime...