Visual ETL flow management and scheduling monitoring method

A process management and process technology, applied in the field of big data, can solve the problems of reducing enterprise work efficiency, different storage formats, increasing labor costs, etc., and achieve the effects of reducing manual intervention, easy creation, and reducing labor costs

Inactive Publication Date: 2016-09-28
CHINA ELECTRONICS TECH CYBER SECURITY CO LTD
View PDF2 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In actual production applications, due to the different sources and storage formats of data in different original databases, specific system implementation and data integration are prone to problems. ETL (Extract-Transform-Load, data extraction, transformation and loading) The system is a powerful tool to solve this problem. However, as a professional tool, its complexity, error-proneness, and cumber

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Visual ETL flow management and scheduling monitoring method
  • Visual ETL flow management and scheduling monitoring method
  • Visual ETL flow management and scheduling monitoring method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] A visual ETL process management and scheduling monitoring method, such as figure 1 shown, including the following:

[0017] 1. Visual management of ETL workflow:

[0018] 1. Graphical creation of ETL workflow:

[0019] In the present invention, the creation of the ETL workflow is completed by dragging and dropping components on the interface to draw a flow chart. Users can drag and drop data processing units, process dependencies and related branches, and merged process components as needed to draw an ETL workflow diagram. When the user chooses to save the process in the sketchpad, all objects, object properties, object X-axis and Y-axis of the sketchpad will be displayed. The location and connection between objects are saved as JSON objects and sent to the background. The background converts the JSON objects into JBPM processes, and converts process attributes into JBPM attributes. In order not to affect the query efficiency of the database, the present invention su...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a visual ETL flow management and scheduling monitoring method comprising the following steps: a workflow is created graphically through flow chart drawing using an interface drag assembly; a workflow chart is drawn according to a data processing unit needing to be dragged, flow dependency and related branch, and a combined flow assembly; for storage of the flow in a drawing board, all objects in the drawing board, the attributes of the objects, the X-axis and Y-axis positions of the objects and the connecting lines between the objects are stored in the form of JSON objects and sent to the background, and the background converts the JSON objects into a JBPM flow and converts flow attributes into JBPM attributes; and ETL workflow scheduling monitoring is implemented in an automatic on-time scheduling mode. An interactive, graphical and draggable design is adopted to make an ETL workflow easy to create and maintain. Based on the characteristic of 'what you see is what you get', cumbersome manual intervention is reduced, the use threshold of an ETL system is lowered, the labor cost of enterprises is reduced, and the work efficiency is improved.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a visualized ETL process management and scheduling monitoring method. Background technique [0002] Data is the basis for enterprises, governments and even countries to use scientific management and decision-making analysis. In actual production applications, due to the different sources and storage formats of data in different original databases, specific system implementation and data integration are prone to problems. ETL (Extract-Transform-Load, data extraction, transformation and loading) The system is a powerful tool to solve this problem. However, as a professional tool, its complexity, error-proneness, and cumbersome configuration make it have a high threshold for use. How to design it to make it easy to operate, easy to maintain, and easy to expand is the key An important issue in the ETL development process. In today's big data era, if any data migration a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06Q10/10
CPCG06Q10/103
Inventor 常梦楠唐明任红雷
Owner CHINA ELECTRONICS TECH CYBER SECURITY CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap