A text classification method and text classification device

Abstract

The invention relates to a text classification method and a text classification device. The method comprises the following steps: an NLP preprocessing step, which analyzes the user dialogue text with a natural language processing method, and obtains a word set and semantic annotation results about the user dialogue text; a multidimensional feature selection step, for the word set and Semantic annotation results are combined according to various rules to obtain a vectorized representation of the semantic information contained in the user dialogue text; and a classification step is to calculate a probability estimation value for the user dialogue classification obtained in the multi-dimensional feature selection step. According to the text classification method and text classification system of the present invention, the advantages of statistics and deep learning methods can be integrated, and a text classification solution oriented to customer needs can be realized through multi-dimensional feature selection.

Description

technical field [0001] The invention relates to text processing technology, in particular to a text classification method and a text classification device. Background technique [0002] At present, the implementation schemes of text classification technology are mainly divided into statistical learning methods and deep learning methods. The former is mainly based on the feature selection method. The word and sentence-level features of the text are selected through indicators such as TF-IDF, PMI, and chi-square value, and the feature vector representing the text is obtained, and the feature vector is obtained by machine learning. The probability of each label is used as the final classification standard; the latter is mainly based on model construction, using the discrete information of the text as input, through the serial and parallel structure of the multi-layer neural network, supplemented by the back propagation algorithm to update the network weight , directly get the ...

AI Technical Summary

Problems solved by technology

[0003] However, both statistical learning methods and deep learning methods have their flaws

The former's over-reliance on feature selection leads to limited improvement space for subsequent algorithms, and discretized features often lead to problems such as data sparseness and semantic gaps; the latter has a black-box structure, and end-to-end learning methods are difficult to generalize and are difficult to Difficult to achieve domain transfer, extremely dependent on the size of training data

Images