Text type data identification method and system and storage medium

Abstract

The invention relates to a text type data identification method and system and a storage medium. The method comprises the steps: firstly obtaining each field in a data table, connecting all field information to obtain a field set, then constructing a training set, performing byte-based 1-gram feature extraction on the field set in the training set, inputting the extracted features into a bayes classifier for training, and finally performing feature extraction on the to-be-identified data table; and inputting the extracted features into the classifier for identification, and outputting a classification identification result by the classifier. According to the method, single Chinese characters or words serve as semantic units to serve as classification features, the sparsity problem of the features in a training sample is avoided, all field contents of the category are connected into a large file to be processed, Bayesian probability calculation is used for the classification problem. The prior probability of the field and the 1-gram category conditional probability are used as main variables for calculating the probability, and the addition of scores is realized by using logarithm operation in classification, so that the recognition accuracy is improved.

Description

technical field [0001] The invention belongs to the technical field of machine learning, and more specifically relates to a method, system and storage medium for text type data recognition. Background technique [0002] With the advent of the digital economy era, data, as a new factor of production, accelerates the integration and development of traditional industrial productivity, and promotes changes in the quality, efficiency, and power of economic development. In this context, my country attaches great importance to data security protection and escorts the development of the national data economy. my country actively learns from the advanced experience of foreign data security management, promulgates laws and regulations such as the "Network Security Law", clearly puts forward security requirements such as data classification, encryption, etc., and regards data asset sorting, data classification and classification management as basic measures and prerequisites for data s...

AI Technical Summary

Problems solved by technology

[0005] This disclosure uses a single Chinese character or a word as a semantic unit as a classification feature, avoiding the sparsity of the feature in the training sample; concatenating all the field contents of the category into a large file for processing; using Bayesian probability calculation for the classification problem , especially the prior probability of the field and the 1-gram category conditional probability are used as the main variables to calculate the probability; the prior probability and conditional probability are generated during the training process; the logarithmic operation is used in the classification to realize the addition of scores, which solves the problem of traditional The problem that the regular expression data asset sorting method cannot be effectively applied to most text type data

Images