Short text classification method based on multiple weak supervision integration

Abstract

The invention discloses a short text classification method based on multiple weak supervision integration, and the method comprises the steps: obtaining an original data set and a knowledge base, andcarrying out the data preprocessing; carrying out knowledge extraction on the preprocessed data; representing the extracted knowledge as an annotation function, and using the annotation function for data annotation; carrying out label integration through a conditional independent model; training a classification model based on a full-connection neural network; evaluating and optimizing the classification model to obtain an optimal model; and performing short text classification by utilizing the optimal model. According to the short text classification method based on multiple weak supervisionintegration, explicit knowledge and implicit knowledge are completely expressed in a mode of combining keyword matching, regular expression and remote supervision clustering; by means of probability labels generated by a label integration mechanism, automatic labeling of label-free data is achieved, the problem of data sparsity of short texts is relieved, and the problem of unbalanced classification of the short texts is effectively solved.

Description

technical field [0001] The invention belongs to the field of natural language processing, and in particular relates to a short text classification method based on multiple weak supervision integration. Background technique [0002] Under the background of mobile Internet, the development of instant messaging not only promotes the surge of short text, but also makes the research and application of short text classification more and more important. [0003] Supervised machine learning mainly relies on manually labeled data and good feature representation. Good feature expression can be learned automatically with deep learning. However, due to the thousands of parameters that need to be learned, supervised deep learning is still inseparable from a large amount of labeled data. In fact, the training data for supervised learning is still dominated by manual annotation. Manual labeling is very expensive and time-consuming. Furthermore, as real-world applications continue to ch...

AI Technical Summary

Problems solved by technology

[0009] In view of this, the present invention is committed to providing a short text classification method based on multiple weak supervision integration, which can solve the problems of label bottleneck, data sparseness and unbalanced classification in short text classification as a whole.

The method of the present invention not only innovatively introduces three sources of weak supervision: keyword matching, regular expressions, and far-supervised clustering into short text annotations for the particularity of short texts; it also proposes a multiple Weakly supervised integration method, which integrates the discrete labels directly output by multiple weak supervisions into probability labels, in order to solve the imbalanced classification problem

Images