Text classification method based on block partition and position weight

Abstract

The invention discloses a text classification method based on block partition and position weight. The method comprises the following steps of: after performing basic pre-processing on an input training or test text, extracting section information from the text; performing statistical analysis on block information by taking one section as a basic text block; performing block partition on the text content again, according to the block size distribution or pre-defined block ratio, including operation, such as combining text blocks; extracting characteristic words and quantized weight, acquiring the posterior probability of the characteristic words on class, analyzing the distribution of the characteristic words having the biggest posterior probability class and according with a text class label, and generating a text vector; and finishing classification model training or text classification with a classifier. The method can be applied to a text representation stage of a text classification system. The text classification effect is improved through the rich and traditional expression of the text content when the text vector is created by using the characteristic words.

Description

Technical field [0001] The invention relates to a text classification method based on block division and position weight, which belongs to the field of electronic text organization and classification. Background technique [0002] Text classification usually evaluates the pros and cons of text classification methods from two perspectives: one is classification performance, such as accuracy, recall, and F1 value, and the other is implementation performance, such as time efficiency and storage efficiency. The process of realizing text classification includes the steps of text preprocessing, text representation, classifier training, and classification application. The text representation includes details such as feature selection, feature weight requantization, and feature dimensionality reduction. [0003] Conventional text classification methods are mostly focused on the selection and improvement of classifiers, mainly based on support vector machines, K nearest neighbors, Bayesian ...

AI Technical Summary

Problems solved by technology

Due to the single expression mode of the former, the text information cannot be retained to the greatest extent; although the latter carries richer information, it usually has different text sizes and large differences in sentence length, resulting in poor adaptability of the sentence segmentation analysis mode and large consumption of storage resources. And it is not easy to find an effective feature word weight adjustment mode and other issues

Images