Big data quality management task scheduling method based on pipeline model and task combination

Abstract

The invention discloses a big data quality management task scheduling method based on pipeline model and task combination. The big data quality management task scheduling method comprises the following steps: 1, reading dirty data from various underlying heterogeneous big data sources; 2, defining a series of data quality detection and repair tasks, and sending the tasks to a task scheduler; 3, enabling the task scheduler to classify the received data quality management tasks; 4, performing merging processing on the combinable classified tasks; 5, sequentially executing various tasks through aparallel data processing function; and 6, uniformly outputting and feeding back an execution result of the data quality detection and repair tasks. The problem that an existing data quality management system is insufficient in performance in a big data scene can be solved, the data quality management task execution efficiency is improved, and meanwhile data quality detection and restoration are considered.

Description

technical field [0001] The present invention relates to the field of big data quality management, in particular to a big data quality management task scheduling method based on pipeline model and task merging, especially relates to multiple heterogeneous big data sources at the bottom layer, including various data quality problems, and provides a Unified data quality management task scheduling method. Background technique [0002] In today's big data era, data quality issues are getting more and more attention. It is not only a basic data processing job, but also helps to clean data with quality problems, integrate clean data, and provide high-quality data services. It is a necessary prerequisite for users to develop upper-level applications, mine data value, and make correct decisions, and directly affects the social value and economic benefits that big data can bring. In addition, in practical applications, data quality management runs through the entire life cycle of dat...

AI Technical Summary

Problems solved by technology

[0005] Purpose of the invention: For the problems and deficiencies in the above-mentioned prior art, the purpose of the invention is to provide a large data quality management task scheduling based on pipeline model and task merging method to solve the problem of low performance of the existing system when processing multiple data management tasks in a big data scenario, while taking into account the detection and repair of various data quality problems

Images