Method and system for data quality monitoring

Abstract

The invention provides a data quality monitoring method comprising the steps of (1) configuring monitoring rules: data quality monitoring rules are realized by means of a dispatching front end; (2) transmitting the monitoring rules: the monitoring rules are transmitted to a computing framework in a manner of configuration information by a dispatching system and the computing framework further transmits the rule information in a context manner; (3) identifying the monitoring rules; the computing framework acquires the monitoring rules through the context information and tacitly approves that the configuration information starting with a ruler character string is used as the configuration of the data quality monitoring rules, and the computing framework applies the monitoring rules to each line of to-be-produced data, judges whether each line of data meets the monitoring rules and performs statistical counting on lines of data meeting the monitoring rules; (4) outputting monitoring data: when data production is over, the data quality monitoring data is output in a counter manner. The invention also provides a data quality monitoring system.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a data quality monitoring method and a data quality monitoring system. Background technique [0002] Existing data quality monitoring solutions can only target existing data, develop MapReduce jobs independently according to business needs, implement business monitoring rules in the MapReduce job code, and run the MapReduce job to scan the entire data, and finally output data quality Situation information. For example, Chinese patent application: system and method for data quality monitoring, application number: CN201210225743.X. [0003] The existing data quality monitoring technology can only target the already produced data, and then run the MapReduce job alone to scan the entire data, and finally get the data quality of this data, that is to say, it is necessary to scan the data repeatedly, and cannot When the data is output, the quality of the data can be obtai...

AI Technical Summary

Problems solved by technology

[0003] The existing data quality monitoring technology can only target the already produced data, and then run the MapReduce job alone to scan the entire data, and finally get the data quality of this data, that is to say, it is necessary to scan the data repeatedly, and cannot When the data is output, the quality of the data can be obtained directly. If there is too much data to monitor the data quality and the cluster computing pressure is relatively high, this solution will occupy a large amount of computing resources of the cluster and seriously affect other production data. output

At the same time, different MapReduce monitoring jobs need to be developed for different data. If there are a lot of data to be monitored, many MapReduce jobs need to be developed for related monitoring, which increases the development workload.

Images