Data flow modeling and processing system to interact with data processing tool

Abstract

The invention relates to a data flow modeling and processing system to interact with a data processing tool. The data flow modeling and processing system comprises: a graphics user interface for creating a data flow diagram; an interfacial server connected to the graphics user interface for receiving the data flow diagram of the graphics user interface and analyzing into a command function; an executive end comprising the data processing tool, wherein the executive end is connected with the interfacial interface to receive the command function sent by the interfacial server for the purpose ofoperating and to transmit final results to the graphics user interface through the interfacial server. The data flow modeling and processing system has the advantages that the graphics user interfaceand the interfacial server allow a user to design and execute a data flow diagram through simple graphic design, processing efficiency is improved, designing and execution difficulty is relieved, thesystem is more widely applicable, visual operations can be performed, execution results can be fed back, a problem with execution of the data processing tool can be discovered and solved in time, andit is avoided that the prior art takes massive checking to locate the problem.

Description

technical field [0001] The invention relates to a data flow modeling and processing system interacting with data processing tools, and belongs to the technical field of data processing. Background technique [0002] The rapid development of emerging information technologies and application models such as cloud computing, the Internet of Things, mobile interconnection, and social media has prompted a sharp increase in the amount of global data and pushed human society into the era of big data. There are many open source and closed source processing systems, engines, and tools for big data processing in the market today. For example, not all tools in the Hadoop ecosystem are compatible with each other. These tools are written in different languages ​​by different teams, have different APIs and different functions. There is no common way to easily integrate them in a computing environment. This results in a lot of software programming and integration work for every data anal...

AI Technical Summary

Problems solved by technology

[0005] 2. There is no common and standard API for data exchange between different kinds of data processing systems, engines and tools

[0006] 3. Without a GUI interface, the data processing process cannot be easily managed

Users cannot easily manipulate or make changes

[0008] 4. There is no common data exchange format. Each module in the data flow is different and has its own exchange format. Combining them in a data flow processing solution is not easy for them to exchange data smoothly

Once the design of the data flow processing scheme is completed, it will be very difficult for users to make adjustments and changes

[0009] 5. The existing technology requires fixed software codes, and the implementation process is also very strict and inflexible

This makes the entire development cycle very long, during which data cannot be used and data may be lost

[0011] 7. When the system is running, people cannot monitor the running status of each processing link in the data flow processing solution. If there is congestion in one of the links, it will slow down the speed of the entire system, making it difficult to meet the needs of big data companies.

Such an easy way of operating will cause the operation of the entire project to be interrupted, or the operation will fail

Images