Real-time ETL dataflow conversion processing technical method and system

A technology for conversion processing and data flow, which is applied in electrical digital data processing, special data processing applications, instruments, etc. It can solve problems such as inability to dynamically adjust resources, standardization, and inability to real-time effective data, and achieve flexible configuration and guarantee stability. Effect

Inactive Publication Date: 2018-04-13
上海中畅数据技术有限公司
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is: in order to solve the problem of not being able to effectively standardize and convert data in real time in the field of big data conversion, and there is a single point problem of wasting resource occupation; at the same time, since JStorm tasks are all hard-coded, they cannot be changed in real time and cannot be dynamically adjusted Resource problem, the present invention provides a kind of real-time ETL data stream conversion processing technical method, includes each step of processing method, provides a kind of topological map configuration, internal convert

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time ETL dataflow conversion processing technical method and system
  • Real-time ETL dataflow conversion processing technical method and system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0022] The present invention will now be described in further detail with reference to the drawings. These drawings are all simplified schematic diagrams, which merely illustrate the basic structure of the present invention in a schematic manner, so they only show the structures related to the present invention.

[0023] A real-time ETL data stream conversion processing technology method, including the following specific steps:

[0024] 1) First carry out data standardization, standardize various log data into log templates, and standardize system performance indicators and business indicators into indicator templates. The template is a form of time series presentation, which mainly includes timestamps, dimensions, measurement values, and added values Etc., the diversification of data adopts standardized data to provide a basis for transformation;

[0025] 2) Put the above standard data Avro serialization into the kafka message queue;

[0026] 3) The kafka Spout module of Jstorm dese...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of data processing, and in particular relates to a real-time ETL dataflow conversion processing technical method and a real-time ETL dataflow conversion processing technical system. The processing method comprises steps, provides topological graph configuration, and converts the same into analogue-SQL by using an internal converter, so that configuration is flexible; furthermore, work can still be performed under the condition that a network is interrupted and a machine goes down, so that stability of data conversion is ensured; meanwhile an internal processing module provides a dynamic configuration converting mode, so that the problems that in the field of big data conversion, at present, the data cannot be effectively standardized and converted in real time, resources are wasted due to a single point problem, and meanwhile, a JStorm task is hard codes and cannot be changed in real time and thus the resources cannot be dynamically adjustedare solved.

Description

technical field [0001] The invention relates to the technical field of big data real-time stream processing, in particular to a real-time ETL data stream conversion processing technology method and system. Background technique [0002] ETL is the process of describing the source after extraction and converting it to the target end, which is used to construct the data warehouse. JStorm is a distributed real-time computing engine, a system similar to Hadoop MapReduce. The user implements a task according to the specified programming specification, and then submits the task to The JStorm system runs 7*24 hours. Once a worker in the middle fails unexpectedly, the scheduler immediately assigns a new worker to replace the failed worker. Therefore, from an application point of view, a JStorm application is a distributed application that complies with a certain programming specification. From a system perspective, JStorm has a scheduling system similar to MapReduce. From the persp...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 朱志刚朱明磊
Owner 上海中畅数据技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap