Data flow modeling and processing system to interact with data processing tool

A processing system and data processing technology, applied in the field of data processing, can solve the problems of project operation interruption, loss of data, inflexibility, etc., to achieve the effect of reducing difficulty, increasing the number of suitable people, and increasing processing efficiency

Active Publication Date: 2018-05-15
BEIJING ADVANCED DIGITAL TECH
View PDF6 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] 2. There is no common and standard API for data exchange between different kinds of data processing systems, engines and tools
[0006] 3. Without a GUI interface, the data processing process cannot be easily managed
Users cannot easily manipulate or make changes
[0008] 4. There is no common data exchange format. Each module in the data flow is different and has its own exchange format. Combining them in a data flow processing solution is not easy for them to exchange data smoothly
Once the design of the data flow processing scheme is completed, it will be very difficult for users to make adjustments and changes
[0009] 5. Th...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data flow modeling and processing system to interact with data processing tool
  • Data flow modeling and processing system to interact with data processing tool
  • Data flow modeling and processing system to interact with data processing tool

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0047] Such as figure 1 A data flow modeling and processing system interacting with data processing tools according to the present invention includes:

[0048] Graphical User Interface (GUI for short, also known as Graphical User Interface) refers to a computer-operated user interface displayed graphically. Users can create data flow diagrams through simple operations in the Graphical User Interface, giving users a simple Practical editing platform;

[0049] The interface server is connected to the graphical user interface, receives the data flow diagram of the graphical user interface, and parses it into a command function. The data flow diagram includes the overall processing process of data, and there will be a step in which a corresponding data processing tool is required. processing, the interface server first parses out the data processing tools needed in the data flow diagram, and then parses out what data processing these data processing tools need to do at the corres...

Embodiment 2

[0076] On the basis of the data flow modeling and processing system interacting with data processing tools described in Embodiment 1, this embodiment provides a specific implementation example, image 3 Describes a data processing process, including SQLquery defined by the user, querying the data stored in the HIVE table, sending the query results to the calculation engine Spark, Spark performs further calculations and conversions, and then outputs the converted data to HIVE. This data is again stored in another table in HIVE. In order to realize the above-mentioned data processing goal, in the system of the present invention, the user first needs to edit the data flow diagram on the graphical user interface, and drag and drop the components corresponding to SQLquery, Spark and HIVE to the editing area for editing, according to image 3 Edit the corresponding data flow diagram. After the editing is completed, run the data flow diagram, the GUI will send the operations of the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data flow modeling and processing system to interact with a data processing tool. The data flow modeling and processing system comprises: a graphics user interface for creating a data flow diagram; an interfacial server connected to the graphics user interface for receiving the data flow diagram of the graphics user interface and analyzing into a command function; an executive end comprising the data processing tool, wherein the executive end is connected with the interfacial interface to receive the command function sent by the interfacial server for the purpose ofoperating and to transmit final results to the graphics user interface through the interfacial server. The data flow modeling and processing system has the advantages that the graphics user interfaceand the interfacial server allow a user to design and execute a data flow diagram through simple graphic design, processing efficiency is improved, designing and execution difficulty is relieved, thesystem is more widely applicable, visual operations can be performed, execution results can be fed back, a problem with execution of the data processing tool can be discovered and solved in time, andit is avoided that the prior art takes massive checking to locate the problem.

Description

technical field [0001] The invention relates to a data flow modeling and processing system interacting with data processing tools, and belongs to the technical field of data processing. Background technique [0002] The rapid development of emerging information technologies and application models such as cloud computing, the Internet of Things, mobile interconnection, and social media has prompted a sharp increase in the amount of global data and pushed human society into the era of big data. There are many open source and closed source processing systems, engines, and tools for big data processing in the market today. For example, not all tools in the Hadoop ecosystem are compatible with each other. These tools are written in different languages ​​by different teams, have different APIs and different functions. There is no common way to easily integrate them in a computing environment. This results in a lot of software programming and integration work for every data anal...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/451G06F17/30
CPCG06F16/2428G06F16/252
Inventor 耐尔屈朝晖
Owner BEIJING ADVANCED DIGITAL TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap