Structured query language (SQL) based MapReduce operation generating method and system

A job and job flow technology, applied in the field of data processing, can solve problems such as limited optimization and does not support rapid analysis of large-scale data, and achieve the effect of ensuring execution efficiency

Active Publication Date: 2014-04-30
CHINA AGRI UNIV
View PDF3 Cites 37 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, due to the fixed execution process of the MapReduce model, the query plan generated by compil

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Structured query language (SQL) based MapReduce operation generating method and system
  • Structured query language (SQL) based MapReduce operation generating method and system
  • Structured query language (SQL) based MapReduce operation generating method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053]The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0054] The present invention proposes a method and system for generating MapReduce jobs based on SQL. On the basis of the existing grammatical analyzer Antlr tool generating an abstract syntax tree AST for SQL statements, a query plan tree structure is further generated, and finally in the query plan Based on the tree, a MapReduce job is generated with an operation task optimization strategy to ensure that the generated MapReduce job program has hig...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a structured query language (SQL) based MapReduce operation generating method and system. The method includes analyzing received SQL sentences to generate an abstract syntax tree, converting the abstract syntax tree into a query plan tree, optimizing the query plan tree, and generating a corresponding MapReduce operation according to the optimized query plan tree. According to the system, an another tool for language recognition (Antlr) is based on to generate the abstract syntax tree (AST) from the SQL sentences, further, the AST is based on to generate a query plan tree, and finally, the MapReduce operation is generated on the basis of the query plan tree; meanwhile, corresponding optimization policies are further provided, so that the generated MapReduce operation has higher execution efficiency.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a SQL-based MapReduce job generation method and system. Background technique [0002] With the rapid development of modern information technology, especially the Internet, the amount of data in various fields and application platforms is also growing explosively. The rapidly expanding amount of data has greatly increased the intensity and complexity of computing tasks for various applications on this basis. At the same time, the increasing number and wide application of low-cost, high-performance commercial computers and storage devices enable large-scale, scalable distributed computing systems to be quickly established to support these calculations and applications. In recent years, the distributed computing framework MapReduce (MR) has provided a practical and effective solution to the challenges faced by large-scale data analysis, relying on its excellent automatic ta...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/44G06F17/30
Inventor 李林韩洪林曹津赵明明叶思菁朱德海张晓东姚晓闯
Owner CHINA AGRI UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap