Data governance method based on datax

Abstract

The invention discloses a data governance method based on data x, which comprises the following steps that: step 1, the data x completes a single data synchronization operation called Job, after the data x receives a Job, a process is started to complete the whole operation synchronization process, and a data xJob module is a central management node of a single operation and undertakes the functions of data cleaning, subtask segmentation, TaskGroup management and the like; and step 2, after the dataxJob is started, the Job is segmented into a plurality of small Tasks according to different source end segmentation strategies so as to facilitate concurrent execution, and each Task is responsible for synchronization work of a part of data. The method has the beneficial effects that the problem of individual type distortion of data transmission is perfectly solved, monitoring of traffic and data volume of a whole link of operation during operation is provided, and dirty data detection, rich data conversion functions, accurate speed control and a robust fault-tolerant mechanism are provided.

Description

technical field [0001] The invention relates to the technical field of data governance, in particular to a datax-based data governance method. Background technique [0002] In the era of big data, with the development of enterprises, each business line, product line, and department will undertake various information systems to facilitate their own business. With the continuous deepening of informatization construction, the phenomenon of “data islands” caused by the independence and independence of business systems is particularly common. Businesses are not integrated, processes are not interoperable, and data is not shared. This allows enterprises to analyze and utilize data and develop reports. , analysis and mining, etc. have brought great difficulties. In this case, in order to realize the systematic operation and management of enterprise global data (information islands, data statistics, data analysis, data mining), dSS (decision support system), BI (business Intelligen...

AI Technical Summary

Problems solved by technology

[0014] 1. The cost of learning and configuring existing ETL tools is high, and some tools do not support real-time data processing. For some industries with high real-time requirements, real-time data processing and management decision report presentation cannot be done;

[0015] 2. Since there are many versions of business databases on the market, most ETLs only support the collection and loading of some data sources. This is also a particularly common problem in data governance. The root cause of the problem is that most data companies continue to launch their own For database products, there is a trend of a hundred flowers blooming in the database market; this makes version adaptation difficult;

[0016] 3. The existing ETL data governance tools do not adopt a distributed and scalable technical framework and do not support concurrent processing of massive tasks, resulting in untimely data processing

Images