Method and device for dirty data detection

Abstract

The present invention provides a dirty data detection method and device, which includes normalizing the attribute type of the original data and then analyzing the attribute characteristics, so as to distinguish between the original data items of a clear type and unspecified types According to the distinguished results, match the original data with an appropriate dirty data detection scheme. In addition, the original data is classified based on different classification methods, and the dirty data ratio of each classification is counted after detection using the matched dirty data detection scheme, and the dirty data detection scheme used is adjusted according to the dirty data ratio obtained, and the The proportion of dirty data in each category is counted, and finally the dirty data detection scheme used when the proportion of dirty data is the highest is selected as the priority dirty data detection scheme for the same data item. The invention can quickly and accurately identify dirty data in massive original data, greatly improves the analysis and utilization value of big data, and reduces the construction cost of big data system.

Description

technical field [0001] The invention relates to the technical field of computers, in particular to a method and device for dirty data detection. Background technique [0002] The big data system needs to access a large amount of raw data of various types and weak causal relationship every day, including network logs, pictures, geographical locations, etc. The raw data is generated very quickly and contains a large amount of dirty data. The processing speed of the big data system There are very strict requirements. The traditional dirty data detection rules can only be manually pre-set corresponding detection rules for known types of raw data. However, due to the characteristics of poor correlation and unclear attribute types of raw data, many raw data When parsing and warehousing, the corresponding detection rules are not matched, resulting in a lot of dirty data entering the big data system, which seriously affects the quality of big data service business development. Ther...

AI Technical Summary

Problems solved by technology

[0005] 1) The method of dirty data detection is single: it can only be matched by a single template or regular expression, and the meaning corresponding to the original data item attribute cannot be automatically analyzed, and the corresponding detection rules can be configured

[0006] 2) The scope of dirty data detection is small: only the corresponding detection rules can be pre-set for the original data items of the specified type, and the original data items of the unspecified type cannot be detected, resulting in a lot of dirty data entering the big data system. Affect the quality of big data service business development;

Images