A Word Segmentation Method for Network Text Based on Domain Adaptability

Abstract

The invention discloses a domain adaptation-based word division method of social network text. Through building an integrated neural network and using a self-training learning method, cross-domain news corpus and labeled data and unlabeled data in a social network are utilized to train an integrated neural network model. The method specifically comprises: dividing the social network text into labeled and unlabeled datasets, and using the datasets as input; using the news domain corpus as source corpus, and pre-training source classifiers on the news source corpus; integrating the source classifiers through a manner of assigning weights to the source classifiers; using the social network corpus to train the integrated neural network model; and utilizing the well-trained integrated neural network model to carry out prediction, and thus improving an effect of word division of the social network. The method can be used to solve the problem of a poor effect caused by very insufficient data in the social network, and can effectively improve the effect of word division of the social network text.

Description

technical field [0001] The invention belongs to the field of natural language processing, relates to word segmentation of social network texts, and in particular to a method for word segmentation of social network texts based on domain adaptability. Background technique [0002] For word segmentation tasks in the traditional news field, statistical methods have initially achieved good results, mainly including conditional random fields and perceptron models. However, these models need to extract a large number of features, so the generalization ability is limited. [0003] In recent years, more and more neural network-based methods have been used for automatic feature extraction, among which there are more word segmentation models, mainly including convolutional neural network (Convolutional Neural Network, CNN), long short-term memory neural network ( Long Short Term Memory Network, LSTM) etc. Although these neural network-based methods are very effective, training these ...

AI Technical Summary

Problems solved by technology

[0004] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a word segmentation method based on domain-adaptive social network texts. By establishing an integrated neural network model and adopting a self-training learning method, using news domain corpus, a small amount of Annotated data and a large amount of unlabeled data are used to train the integrated neural network model, thereby improving the effect of word segmentation in social networks and solving the problem of poor results caused by too little data in social networks

Images