OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform

A processing method and query processing engine technology, applied in the field of parallel OLAP query processing and OLAP query processing, can solve problems affecting overall performance and occupation, and achieve low network transmission delay and synchronization cost, high fault tolerance performance, and high query processing performance. Effect

Active Publication Date: 2012-09-12
RENMIN UNIVERSITY OF CHINA
View PDF3 Cites 66 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

During the processing of MapReduce's star connection, a large amount of materialized data and data distribution will occupy a large amount of disk I / O and network bandwidth, seriously affecting the overall performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform
  • OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform
  • OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] Parallel OLAP processing includes two stages: local OLAP processing and global OLAP processing. The performance depends on the performance of the local OLAP query processing engine and network transmission performance. The key issues to be solved for OLAP processing on massive data warehouses are data distribution model, parallel OLAP query processing model and network communication model.

[0040] Therefore, the invention discloses an OLAP query processing method on a database and Hadoop hybrid platform. The method includes reverse star storage model, distributed cache technology, DDTA-JOIN technology, hybrid replica management technology, hybrid OLAP query processing engine technology, and merge technology based on database hash aggregation algorithm. Among them, the reverse star storage model is used to realize the distributed storage of the data warehouse star storage model. The distributed cache technology manages the memory of the working nodes as a virtual share...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an OLAP (On Line Analytical Processing) inquiry processing method facing a database and a Hadoop mixing platform. In the method, on the basis of a multicopy fault-tolerance mechanism of a Hadoop, when the OLAP inquiry processing is carried out, firstly, the OLAP inquiry processing is executed on a main working copy and an inquiry processing result is recorded into an aggregation result table of a local database; and when a working node goes wrong, node information of a fault-tolerance copy corresponding to the main working copy is searched by a namenode and a MapReduce task is called to complete the OLAP inquiry processing task on the fault-tolerance copy. According to the invention, a database technology and a Hadoop technology are combined and the storage performance of the database and the high extendibility and high availability of the Hadoop are combined by a mode of double storage engines and double inquiry processing engines; and the database inquiry processing and the MapReduce inquiry processing are integrated in a loosely coupling mode by utilizing a master-slave copy management mechanism so as to not only ensure the high inquiry processing performance, but also ensure the high fault-tolerance performance.

Description

technical field [0001] The invention relates to an OLAP (Online Analytical Processing) query processing method, in particular to a parallel OLAP query processing method oriented to a database and Hadoop hybrid platform, and belongs to the technical field of database management. Background technique [0002] OLAP (On-Line Analytical Processing, Online Analytical Processing) is designed to meet specific query and report requirements for decision support or multidimensional environments. Data warehouses generally use a multidimensional model to store subject-oriented analytical data sets, mainly using a star storage model of multiple dimension tables and a single fact table. The core of OLAP query is star-join, that is, grouping and aggregation calculation is performed on the joining results on the basis of joining the fact table with multiple dimension tables. The connection operation between the fact table and the dimension table mainly adopts the hash connection technology....

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/283
Inventor 张延松王珊
Owner RENMIN UNIVERSITY OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap