A data quality detection method and system based on multi-dimensional tags

Abstract

The present application discloses a data quality detection method and system based on multi-dimensional tags. Based on known types of data items and detection rule base, use multi-dimensional label analysis algorithm to mark corresponding dimension labels for known types of data items, and use dimension labels to dynamically adjust the quality inspection process of known types of data items; based on unknown types of data items and Combined with the detection rule base, use the rule similarity evaluation algorithm to recommend a quality detection engine for unknown types of data sources, and verify the results of the quality detection engine to obtain an effective quality detection rule set; save the quality detection process of known types of data items and an effective quality detection rule set and update the multidimensional labeling rule base. This solution solves the problems of poor accuracy and weak timeliness caused by fixed detection rule templates through two algorithms: multi-dimensional labeling algorithm and rule similarity evaluation, realizes fast and accurate detection of data quality and timely feedback of detection results, and improves the quality of data sources.

Description

A method and system for data quality detection based on multi-dimensional labels technical field The application relates to data processing technology field, be specifically related to a kind of data quality detection method based on multidimensional label and system. Background technique [0002] "Big data" requires new processing modes to have stronger decision-making power, insight discovery power and process optimization ability. It is only because of this power that big data becomes a massive, high-growth and diversified information asset. As the big data systems in various places continue to Intermittent access to different industries, generating raw data from a variety of data sources, and reprocessing to form the final information assets. The quality of each data source is the basis for whether the big data system can be effective. Whether there are quality problems in various data sources, timely warning and improving the quality of data sources, reducing ...

AI Technical Summary

Problems solved by technology

[0004] The purpose of this application is to propose a data quality detection method and system based on multi-dimensional tags to solve the problems of poor accuracy and weak timeliness caused by fixed detection rule templates

Images