Method and device for monitoring data quality

Abstract

The invention discloses a method and device for monitoring data quality, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: taking source data and / or output data of a data calculation task as to-be-verified data; obtaining a verification rule of the to-be-verified data from a preset data verification rule set; executing a data verification task on the to-be-verified data based on the verification rule, wherein the data verification task and the data calculation task are carried out asynchronously. According to the embodiment of the invention, the data quality can be ensured, and the possibility of data calculation task errors is greatly reduced; the workload and complexity of data calculation and verification are reduced; data verification rules can be modified conveniently, and personalized customization of a data verification strategy is supported.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for monitoring data quality. Background technique [0002] Different data marts need to share data, and different data calculation tasks (including data model calculation, data analysis calculation, etc.) generate new data by processing old data. Data errors may be caused by database node loss (unavailable), node crash, upstream data error, etc. If you can't detect data errors in advance and continue to let the erroneous data flow downstream, it will have an impact on downstream related businesses, and may even have a fatal impact on some key businesses. If the data is found to be wrong after the data has been affected, it may cause huge losses, and it will be more difficult to remedy the wrong data. [0003] In order to avoid the occurrence of the above situation, in the prior art, the data is verified when the data calculation task is performed, that is,...

AI Technical Summary

Problems solved by technology

[0005] 1) Since the data calculation task includes data verification, the source code of the task is complex and the amount of data development is large;

[0006] 2) It is inconvenient to modify the data verification strategy: After the data verification program is written, it is very inconvenient to modify the data verification strategy; since the source code of the data calculation task needs to be directly modified, the risk of source code errors will also increase;

[0007] 3) The feedback of data error information is not timely: the source data can only be verified when the data calculation task is executed, and the data error cannot be found before the data calculation task is executed;

[0008] 4) There is repeated verification: every time a data calculation task is performed, the data needs to be verified, which wastes database resources and time

Images