Data quality management system applied to big data system

Abstract

The invention discloses a data quality management system applied to a big data system. The data quality management system comprises an application domain configuration module, a data source configuration module, a management index configuration module and a task execution module, wherein the application domain configuration module is used for configuring an application domain which the data to bedetected belongs to, and generating application domain configuration information storage; the data source configuration module is used for configuring a data source which the data to be detected belongs to, and generating data source configuration information storage; the management index configuration module is used to create the management index storage for the data information in the data source; the task execution module can be used to invoke the management index to according to an execution command to perform data check on the data information accessed by the application domain to the data source. Also included is a method of operating the data quality management system applied to the big data system. For the integration of large environmental data centers in a big data environment monitoring platform and the collection of numerous service data of many systems of the Provincial Environmental Protection Agency, the data quality management system can perform data inspection on the data of the environmental big data center, screen out dirty or abnormal data, and improve the data quality of the environmental big data center.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a data quality management system applied to a big data system and a method for using the system to manage the data quality of a big data system. Background technique [0002] Since the current big data center generally integrates and collects the business data of many systems, there are various types of data integrated and collected, and the data quality is uneven, including a large amount of dirty data or abnormal data, and the data quality is low, reducing Data utilization efficiency. Contents of the invention [0003] The purpose of the present invention is to provide a data quality management system applied to big data systems. By establishing a layer of data supervision platform between the data sources of many application systems and the built big data center, it solves the problem of existing big data problems. Due to the wide range of data sources and vario...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to provide a data quality management system applied to big data systems. By establishing a layer of data supervision platform between the data sources of many application systems and the built big data center, it solves the problem of existing big data problems. Due to the wide range of data sources and various types of data in the center, the quality of data is uneven and the efficiency of data utilization is reduced.

Images