Data consistency evaluation method based on data pattern clustering

Abstract

The invention discloses a data consistency evaluation method based on data mode clustering. The method is applied to the big data analysis processing field, and solve the problem that in the prior art, consistency evaluation of multi-mode coexistence fields cannot be achieved. The method comprises the following steps: firstly, carrying out pattern clustering on to-be-evaluated fields read from a database according to a determined pattern clustering algorithm, then, determining a standard pattern in clustered patterns, and finally, carrying out pattern matching on values of the to-be-evaluatedfields by adopting the standard pattern to obtain dirty data. The method is especially suitable for application scenarios in which data engineers do not know about services and which modes exist reasonably are difficult to determine.

Description

technical field [0001] The invention belongs to the field of big data analysis and processing, and in particular relates to a consistency evaluation technology for structured data. Background technique [0002] Structured data, simply put, is a database. It is easier to understand when combined with typical scenarios, such as enterprise ERP and financial systems; medical HIS database; education card; government administrative approval; other core databases, etc. [0003] It basically includes high-speed storage application requirements, data backup requirements, data sharing requirements, and data disaster recovery requirements. [0004] Structured data, also known as row data, is logically expressed and realized by a two-dimensional table structure, strictly follows the data format and length specifications, and is mainly stored and managed by a relational database. The opposite of structured data is unstructured data that is not suitable for representation by two-dimensi...

AI Technical Summary

Problems solved by technology

[0005] In today's big data era, there are many types of businesses and complex application systems. In the process of system construction, the importance of data quality is often ignored, and insufficient measures are not taken, resulting in data quality problems with the gradual deepening of the application of the system and data. A little bit of exposure, such as data validity, accuracy, consistency, etc.

In fact, this evaluation method is very inefficient and cannot solve the following problems:

[0006] If the field allows the existence of multiple modes, the data engineer does not understand the business, and does not know which modes exist reasonably. At this time, it is impossible to specify a standard mode for judging whether the data is reasonable

Images