Data quality monitoring method and system and computer equipment

Abstract

The invention provides a data quality monitoring method and system and computer equipment, which are used for monitoring the data quality of an offline data warehouse, and the method comprises the following steps: creating a data quality monitoring task, and storing monitoring task information in a document database, the monitoring task information comprising monitoring configuration information and alarm configuration information; submitting the monitoring task to a data warehouse based on a cluster so as to monitor data in the data warehouse in an offline state; processing a monitoring result obtained by executing the monitoring task and corresponding monitoring task information, and then storing the monitoring result and the corresponding monitoring task information into a relational database; the relational database is detected, an alarm is given according to a detection result, and the detection result comprises the monitoring result and the alarm configuration information. By flexibly configuring the data quality monitoring task, more effective and timely data monitoring can be realized, the timeliness of alarm information (or notification) can be ensured, and the data quality monitoring can be effectively ensured.

Description

technical field [0001] The present invention relates to the Internet field, in particular to a data quality monitoring method, system and computer equipment. Background technique [0002] With the advent of the big data era, more and more applications and services are built based on data, and the importance of data is self-evident. The guarantee of data quality is the basis of all data analysis and data mining. The quality of data is directly related to the accuracy of information, and also affects the survival and competitiveness of enterprises. Therefore, ensuring data availability and security is an important link that cannot be ignored. [0003] The existing open source data quality monitoring component Apache Griffin is a model-driven solution. Based on the target data set, users can choose from different dimensions (such as checking whether the data quantity and data accuracy of the source end and the target end are consistent after the execution of the offline task, ...

AI Technical Summary

Problems solved by technology

[0006] In order to solve at least one of the following technical problems: the data quality cannot be verified in a timely, effective and accurate manner, the problem data or wrong data cannot be notified to the corresponding business personnel in time, and the follow-up process data processing is invalid due to wrong data or problem data Sexuality, or even waste of cluster resources (or computing resources, etc.), and poor monitoring results due to data quality (problem data or wrong data)

Images