Method for generating Spark and Hadoop program code by midcore based on model transformation

A program code and model conversion technology, applied in the field of code generation, can solve problems such as unfriendliness and domain complexity, and achieve the effect of reducing complexity and improving development efficiency

Active Publication Date: 2018-06-12
NANJING UNIV
View PDF6 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the framework uses the methods in the Mapper and Reducer classes to represent data operations, which has certain domain complexity and is not friendly to the expression of daily data operations.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for generating Spark and Hadoop program code by midcore based on model transformation
  • Method for generating Spark and Hadoop program code by midcore based on model transformation
  • Method for generating Spark and Hadoop program code by midcore based on model transformation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0056] The method will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

[0057] figure 1 Generate spark and hadoop code method flow and architecture diagram for the midcore of the embodiment of the present invention. Spark and hadoop are the most representative big data platforms today, both of which are based on the realization of the mapreduce programming model. This method realizes the syntax analysis of the midcore language, and automatically generates the program codes of the two platforms, so as to realize the QVT-R as the It represents the bridge between the existing model transformation language and the specific big data platform, and the method has good scalability. This method is based on the idea of ​​MDE, shields the details of specific big data platforms, and reduces the complexity of development of multiple platforms. like figure 1 Therefore, the method for generating Spark and Hadoop program codes b...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method for generating Spark and Hadoop program code by using a model transformation intermediate language midcore. The method comprises the steps that step1, two xtext itemsare built, the midcore language is respectively abstracted, a syntax tree is extracted, a model is built, and xtext grammar parser codes are written, step 2, xtend is used for writing a transformationrule mapping from midcore to Spark code in one of the items, Step 3, xtend is used for writing a transformation rule mapping from midcore to Hadoop code in another one of the items. The method for generating Spark and Hadoop program code by using a model transformation intermediate language midcore can map the codes of the Spark and Hadoop platforms by the intermediate language converted by an existing model conversion language, realize the bridging between the model transformation language and the Mapreduce program, and support the midcore transformed into Mapreduce code through a direct description module, thereby reducing the complexity of Mapreduce program development.

Description

technical field [0001] The present invention relates to a code generation method, which mainly utilizes model conversion technology to realize the task of automatically generating midCore scripts by QVT-R language, and belongs to computer technology, model conversion technology, domain-specific language and code generation technology big data distributed data processing cross-technical application areas. Background technique [0002] Data transformation is a common processing method for heterogeneous data. With the close cooperation and mergers between enterprises and organizations, heterogeneous data and information resources exist widely, seriously affecting the information exchange between departments. The goal of data conversion is to convert heterogeneous data information, eliminate various errors in the data, optimize the data model, and obtain an ideal data model. [0003] There are a large number of Domain Specific Languages ​​(DSL for short) in the computer field,...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F8/35G06F8/30
CPCG06F8/30G06F8/35
Inventor张天刘拨杰潘敏学李宣东
OwnerNANJING UNIV