Method for quickly classifying massive database tables

Abstract

The invention relates to a method for quickly classifying massive database tables. The method comprises the steps of: calculating a mutual information entropy to obtain a key attribute of each table;constructing the feature vector of the selected attribute according to the metadata information of the attribute field type and the data content abstract; and clustering the key attributes by using aclustering algorithm of machine learning, labeling a clustering center, forming a training set, training a classification algorithm, applying the trained classification algorithm to residual attributeclassification, performing sampling judgment on a classification result, reversely optimizing the classification algorithm, and outputting categories of all database table attribute fields. Accordingto the method, field feature vectors are constructed in combination with database field metadata information and field content, the to-be-analyzed database key fields are clustered, the data field (tagging) is set, the training set is constructed, the classification algorithm of industry features is trained, and the workload of manual processing is reduced.

Description

technical field [0001] The invention relates to database technology, in particular to a method for quickly classifying massive database tables. Background technique [0002] In the process of data warehouse construction, data cataloging and cleaning consume a lot of manpower and material resources, and one of the most important tasks is to classify database tables. By classifying and labeling the database tables, identifying the data field to which it belongs (for example, the field represents customers, products, quantities, amounts, etc.), and establishing a data catalog, it is helpful to fill in missing metadata information and assist in the formulation of data quality rule discovery For data quality issues, etc., follow-up data governance and improvement will be carried out in a targeted manner. [0003] Existing data classification methods need to be carried out by personnel based on database design documents, database table structure notes, etc., which largely rely on...

AI Technical Summary

Problems solved by technology

[0003] Existing data classification methods need to be implemented by personnel based on database design documents, database table structure notes, etc., which largely rely on human experience, and each piece of metadata information needs to be confirmed one by one, which is time-consuming and laborious

When faced with massive data types and data scale, the labor cost is very huge

Images