Text classification method based on TF-IDF matrix and capsule network

Abstract

The invention discloses a text classification method based on a TF-IDF matrix and a capsule network. The method comprises the steps of adopting the TF-IDF matrix for analysis of text data obtained after word segmentation processing to obtain a global threshold value meeting a preset condition alpha, and performing personalized analysis on each piece of text data to obtain a threshold value tj corresponding to each piece of text data and a set Salpha formed by the threshold values tj; comparing the global threshold value alpha with a threshold value alphai obtained by personality analysis of the text; obtaining a final threshold betai corresponding to each piece of text data and a set Sbeta formed by the final threshold betai, processing the text data set according to the finally obtained set Sbeta, carrying out word frequency analysis, and labeling the text data set according to the appearance sequence of vocabularies in the dictionary so as to realize text vector embedding; and through a doc2vec algorithm, expressing the embedded text vector by using a text matrix, and training a capsule network text classification model by taking the embedded text vector as an input of capsule network-based text classification. According to the method, some vocabularies which have low influence on text classification in the text data can be effectively removed, and text features are reduced.

Description

technical field [0001] The invention relates to the fields of natural language processing and information retrieval, in particular to a text classification method based on TF-IDF (term frequency-inverse text frequency index) matrix and capsule network. Background technique [0002] The text classification process first needs to perform text preprocessing and text feature processing on the data to obtain the feature vector of the text, which lays the foundation for the subsequent operations of the classification process. In the process of feature processing, traditional machine learning methods need to manually specify the specific form of features to represent the original data. [0003] Traditional text classification methods have high text feature dimensions, sparse data, and weak representation capabilities. Existing text classification methods such as the KNN (K-Nearest Neighbor) algorithm mainly rely on the limited surrounding samples, but the interpretability of the o...

AI Technical Summary

Problems solved by technology

Existing text classification methods such as the KNN (K-Nearest Neighbor) algorithm mainly rely on the limited surrounding samples, but the interpretability of the output is not strong, and the amount of calculation is large. When the samples are unbalanced, it may cause when the input When there is a new sample, the samples of the high-capacity class among the K neighbors of the sample account for the majority

Images