Query processing method and system for materialized view of big data platform

A technology of big data platform and materialized view, which is applied in the direction of network data query, special data processing application, network data index, etc. It can solve the problems that it is impossible to provide materialized view storage space, materialized view occupies a large storage space, etc., and shorten the processing time. Time, easy to expand, reduce the effect of network data transmission

Active Publication Date: 2019-07-12
ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER COMPANY +1
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Materialized views take up a lot of storage space. Although the big data platform has a large storage space, it is impossible to provide unlimited storage space for materialized views.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Query processing method and system for materialized view of big data platform
  • Query processing method and system for materialized view of big data platform
  • Query processing method and system for materialized view of big data platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] The present invention is described in detail below in conjunction with accompanying drawing:

[0049] Such as figure 2 As shown, the specific implementation of the query processing method of the big data platform materialized view of the patent of the present invention is mainly divided into two parts: the materialized view selection method and the materialized view placement method.

[0050] 1. Materialized view selection method:

[0051] 1) Because the big data query changes in each cycle, materialized view selection is performed periodically.

[0052] 2) In the cloud computing nodes, randomly select a computing node as the master node, which collects the query set of the previous cycle.

[0053] 3) Generate an MVPP structure diagram from the query set. All query processing plans (Multi-View Processing Plan, abbreviated as MVPP) express an overall query processing strategy for a query set in the form of a directed acyclic graph. figure 1 The leaf nodes in the dat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a query processing method and system for big data platform materialized views. The steps for selecting the materialized views are as follows: generating an MVPP (Multi-View Processing Plan) structure diagram for a given query collection, obtaining a collection of all nonleaf nodes according to the structure diagram, calculating a value of each nonleaf node in the collection, and obtaining a materialized view collection by employing a revenue maximization oriented materialized view greedy selection algorithm; and the steps for placing the materialized views are as follows: establishing a materialized view associated weight matrix for the materialized view collection obtained at the steps for selecting the materialized views, calculating the value of each element in the matrix, arranging all computational nodes in a descending order according to magnitudes of materialized storage spaces, taking the node with the largest materialized storage space in all computational nodes, and obtaining the materialized views placed at the node. According to the materialized view placing method disclosed by the invention, the network data transmission among the nodes can be reduced, and the processing time is shortened.

Description

technical field [0001] The invention relates to the field of big data query processing, in particular to a query processing method and system for a materialized view of a big data platform. Background technique [0002] With the continuous deepening and advancement of smart grid construction, the amount of data generated by grid operation and equipment monitoring has grown exponentially. The data that grid companies can store and process has reached an unprecedented level, and it has increased rapidly at a speed exceeding Moore's Law. The core value of big data lies in the storage and analysis of massive data. In a commercial environment, data processing service providers package big data processing into services and sell them to users. With the development of commercial application of big data, its data processing has the following characteristics: [0003] 1) The number of big data platform users is huge. Big data service providers hope to serve more customers in order ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/953
CPCG06F16/951
Inventor 马艳辜超姚金霞邹立达陈素红孔刚刘兴华张世栋朱文兵冯兰新
Owner ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER COMPANY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap