Predo-based method and system for supporting multi-data-source hybrid execution of SQL and python scripts

A hybrid execution and multi-data source technology, applied in the field of data analysis, can solve problems such as cumbersome process, affecting data analysis efficiency, complex operation, etc., and achieve the effect of improving efficiency

Pending Publication Date: 2020-04-14
象辑科技股份有限公司
View PDF5 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In the whole process, only step 4 is the specific data analysis process, and the rest are repeated preparations for data analysis. The process is cumbersome and the operation is complicated, which greatly affects the efficiency of data analysis.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Predo-based method and system for supporting multi-data-source hybrid execution of SQL and python scripts

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in combination with the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0065] see figure 1 , the present invention provides a technical solution: a Presto-based method and system for mixed execution of SQL and python scripts supporting multiple data sources, the method and system are specifically divided into data source management, grammar editor, grammar parser and script Executor; first build a multi-data source query platform based on Presto, realize dynamic addition and deletion of data sources, and use the same set of SQL syntax to query data from differe...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Predo-based method and system for supporting multi-data-source hybrid execution of SQL and python scripts. The method and the system are specifically divided into data sourcemanagement, a grammar editor, a grammar parser and a script executor. The method comprises the following steps: constructing a multi-data-source query platform based on Predo to realize dynamic addition and deletion of data sources and query data of different data sources by using the same SQL grammar; defining a set of grammar rules capable of executing SQL and python codes in a mixed manner, wherein grammatical rule is as follows: the whole script is composed of a command line code block and a python code block, the command line code block comprises an SQL statement, the command line statement takes ':' as an end mark, a python code block needs to be wrapped with '%python', that is, the python code block starts with the '%python' and ends with the '%python'. According to the method provided by the invention, the multi-data-source query process is simplified, the cross-language barrier of SQL and python codes is eliminated, data analysts do not need to process the connection problemof python and SQL queries any more, and only pay attention to the data analysis part, so that the data analysis efficiency is greatly improved.

Description

technical field [0001] The invention relates to the technical field of data analysis, in particular to a Presto-based method and system for mixed execution of SQL and python scripts supporting multiple data sources. Background technique [0002] Currently, in the field of data analysis, analysts need to perform the following steps when using python to query and analyze the data of multiple SQL-like systems: [0003] Step 1, write query SQL for different SQL-like systems; Step 2, write link execution codes for different types of SQL systems and execute the written SQL for data query; Step 3, write data conversion code to convert query results into DataFrame; Step 4 , write the data analysis code to analyze the data; Step 5, execute the python code to get the analysis result. [0004] In the whole process, only step four is the specific data analysis process, and the rest are repeated preparations for data analysis. The process is cumbersome and the operation is complicated, ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/41G06F8/30G06F16/242
CPCG06F8/427G06F8/37G06F8/315G06F16/2433
Inventor 邱珩陈俊锋胡宗宝
Owner 象辑科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap