Visual ETL flow management and scheduling monitoring method

Abstract

The invention discloses a visual ETL flow management and scheduling monitoring method comprising the following steps: a workflow is created graphically through flow chart drawing using an interface drag assembly; a workflow chart is drawn according to a data processing unit needing to be dragged, flow dependency and related branch, and a combined flow assembly; for storage of the flow in a drawing board, all objects in the drawing board, the attributes of the objects, the X-axis and Y-axis positions of the objects and the connecting lines between the objects are stored in the form of JSON objects and sent to the background, and the background converts the JSON objects into a JBPM flow and converts flow attributes into JBPM attributes; and ETL workflow scheduling monitoring is implemented in an automatic on-time scheduling mode. An interactive, graphical and draggable design is adopted to make an ETL workflow easy to create and maintain. Based on the characteristic of 'what you see is what you get', cumbersome manual intervention is reduced, the use threshold of an ETL system is lowered, the labor cost of enterprises is reduced, and the work efficiency is improved.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a visualized ETL process management and scheduling monitoring method. Background technique [0002] Data is the basis for enterprises, governments and even countries to use scientific management and decision-making analysis. In actual production applications, due to the different sources and storage formats of data in different original databases, specific system implementation and data integration are prone to problems. ETL (Extract-Transform-Load, data extraction, transformation and loading) The system is a powerful tool to solve this problem. However, as a professional tool, its complexity, error-proneness, and cumbersome configuration make it have a high threshold for use. How to design it to make it easy to operate, easy to maintain, and easy to expand is the key An important issue in the ETL development process. In today's big data era, if any data migration a...

AI Technical Summary

Problems solved by technology

In actual production applications, due to the different sources and storage formats of data in different original databases, specific system implementation and data integration are prone to problems. ETL (Extract-Transform-Load, data extraction, transformation and loading) The system is a powerful tool to solve this problem. However, as a professional tool, its complexity, error-proneness, and cumbersome configuration make it have a high threshold for use. How to design it to make it easy to operate, easy to maintain, and easy to expand is the key An important issue in the ETL development process. In today's big data era, if any data migration and loading requires professional data developers to use ETL tools, it will greatly reduce the work efficiency of the enterprise and increase labor costs.

Images