Multi-source heterogeneous data quality verification method and system based on unified SQL

Abstract

The invention discloses a multi-source heterogeneous data quality verification method and system based on a unified SQL. According to one aspect of the invention, the method comprises the following steps: S1, creating a quality verification rule; S2, creating a quality verification task; S3, creating a quality inspection job; S4, creating a quality verification scheduling plan; S5, submitting a scheduling plan to execute verification logic; and S6, generating a quality report; according to another aspect of the invention, the system comprises a task job configuration module, a job scheduling module, a verification execution module and a data quality report generation and data quality analysis module. The method has the beneficial effects that visual, flexible and unified quality management can be carried out on data in different storage forms, and meanwhile, complex business requirements can be met.

Description

technical field [0001] The invention relates to the field of data quality management, in particular to a multi-source heterogeneous data quality verification method and system based on unified SQL. Background technique [0002] In recent years, with the vigorous development of big data technology, as the support of big data, the value of various network data is also increasing. A large amount of data is generated on the network every day. These data may cause data quality problems in each stage of the complete life cycle of planning, acquisition, storage, sharing, maintenance, application, and extinction. Once these problems occur, subsequent business processes may be affected. There are catastrophic consequences, so it is very necessary to identify, measure, monitor and warn these data that may cause quality problems. In addition, in the entire data life cycle, there will be different storage forms, which may be relational databases such as mysql, postgres, oracle, or non-...

AI Technical Summary

Problems solved by technology

In addition, in the entire data life cycle, there will be different storage forms, which may be relational databases such as mysql, postgres, oracle, or non-relational databases such as elasticserch, mongoDB, etc. Difficulty in data quality management

Images