Data processing method, system and device based on tagSQL

A technology of data processing and Hadoop cluster, applied in the field of big data, can solve problems such as lack of mastery of the interface language, programmers unable to quickly carry out big data development work, etc.

Active Publication Date: 2019-09-17
中国邮政集团有限公司广东省分公司
View PDF8 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Spark SQL provides several calling methods: command line, java api, python api, scala api, these calling methods are suitable for different users, however, because the command line of SQL programming in the prior art can only satisfy single-line statements call, and most SQL programmers do not master the other three interface languages, so programmers cannot quickly carry out big data development work

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, system and device based on tagSQL
  • Data processing method, system and device based on tagSQL
  • Data processing method, system and device based on tagSQL

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, it is only set for the convenience of illustration and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art sexual adjustment.

[0045] refer to figure 1 , a data processing method based on tagSQL, comprising the following steps:

[0046] S101. Receive tagSQL xml file and process request;

[0047] Specifically, the tagSQL is a tagged SQL language. The tagSQL xml is a file formed by tag SQL language. Send tagSQL xml files and process requests through kettle (an open source ETL scheduling tool) or tagSQLDevTool (debugging tool). mainServer (server) receives tagSQL xml files and process requests sent by kettle or tagSQLDevTool. Among them, k...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method, system and device based on a tagSQL. The method comprises the following steps: receiving a tagSQL xml file and a process request; acquiring a tagSQL application process according to the process request; enabling the tagSQL application process to analyze the tagSQL xml file to obtain a first process; forwarding the process request to a tagSQL application process, enabling the tagSQL application process to obtain a second process from the first process, and executing the second process, wherein the tagSQL xml file comprises information that the client carries out service logic processing according to a tagSQL xml label requirement. According to the method, the Spark SQL is changed into the programming language similar to the SQL storage process, so that the Spark SQL is compatible with multiple interface languages. The method can be widely applied to the technical field of big data.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data processing method, system and device based on tagSQL. Background technique [0002] Apache Spark (an open-source cluster computing system based on memory computing) is a fast and general-purpose computing engine designed for large-scale data processing. It has the advantages of Hadoop MapReduce, but it is different from MapReduce (programming model) in that: running The output results can be stored in memory, so there is no need to read and write HDFS (Hadoop Distributed File System, the core sub-project of the Hadoop project), so Spark (open source cluster computing environment) can be better applied to data mining and machine learning that require iteration The MapReduce algorithm. Spark SQL is a module for Spark to process data. Different from the API (Application Programming Interface) of the basic Spark RDD (Resilient Distributed Dataset), the interface prov...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/28G06F16/22
CPCG06F16/27G06F16/284G06F16/2246
Inventor 曾洁林
Owner 中国邮政集团有限公司广东省分公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap