Real-time ETL dataflow conversion processing technical method and system

Abstract

The invention relates to the technical field of data processing, and in particular relates to a real-time ETL dataflow conversion processing technical method and a real-time ETL dataflow conversion processing technical system. The processing method comprises steps, provides topological graph configuration, and converts the same into analogue-SQL by using an internal converter, so that configuration is flexible; furthermore, work can still be performed under the condition that a network is interrupted and a machine goes down, so that stability of data conversion is ensured; meanwhile an internal processing module provides a dynamic configuration converting mode, so that the problems that in the field of big data conversion, at present, the data cannot be effectively standardized and converted in real time, resources are wasted due to a single point problem, and meanwhile, a JStorm task is hard codes and cannot be changed in real time and thus the resources cannot be dynamically adjustedare solved.

Description

technical field [0001] The invention relates to the technical field of big data real-time stream processing, in particular to a real-time ETL data stream conversion processing technology method and system. Background technique [0002] ETL is the process of describing the source after extraction and converting it to the target end, which is used to construct the data warehouse. JStorm is a distributed real-time computing engine, a system similar to Hadoop MapReduce. The user implements a task according to the specified programming specification, and then submits the task to The JStorm system runs 7*24 hours. Once a worker in the middle fails unexpectedly, the scheduler immediately assigns a new worker to replace the failed worker. Therefore, from an application point of view, a JStorm application is a distributed application that complies with a certain programming specification. From a system perspective, JStorm has a scheduling system similar to MapReduce. From the persp...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is: in order to solve the problem of not being able to effectively standardize and convert data in real time in the field of big data conversion, and there is a single point problem of wasting resource occupation; at the same time, since JStorm tasks are all hard-coded, they cannot be changed in real time and cannot be dynamically adjusted Resource problem, the present invention provides a kind of real-time ETL data stream conversion processing technical method, includes each step of processing method, provides a kind of topological map configuration, internal converter turns into SQL-like, configuration is flexible; And in network interruption, machine downtime It can still work in the case of a computer, ensuring the stability of data conversion; at the same time, the internal processing module provides a dynamic configuration conversion method, which effectively solves the problem of not being able to standardize and convert data in real time and effectively in the field of big data conversion. There is a single point problem that wastes resource occupation; at the same time, because JStorm tasks are hard-coded, they cannot be changed in real time and cannot dynamically adjust resource issues

Images