Self-adaptive Chinese word segmentation method based on embedded representation

Abstract

The embodiment of the invention discloses a self-adaptive Chinese word segmentation method based on embedded representation and belongs to the field of information processing. The method is characterized in that an embedded representation layer of a character is shared by a word segmentation network and a character language model. As for embedded representation of the character, on the one hand, hidden multi-granularity local features of a to-be-segmented text is obtained by means of the word segmentation network based on convolutional neural network; then label probability of the character is obtained through a forward network layer; finally, label inference is used to obtain the optimum segmentation result in the sentence level; on the other hand, an unlabelled text is randomly extracted, a character next to the character is predicted by means of a character language model based on a long- and short-term memory unit (LSTM) recurrent neural network and the word segmentation network is constrained. By modeling a character co-representing relationship in texts in different fields by means of the character language model and transferring information to the word segmentation network by means of embedded representation, the field transfer ability of word segmentation is enhanced, and the method has very huge practical value.

Description

technical field [0001] The invention relates to the field of information processing, in particular to a method for domain migration based on neural network Chinese word segmentation. Background technique [0002] Chinese word segmentation is a basic task in Chinese natural language processing. Its goal is to convert a sequence composed of Chinese characters into a sequence composed of Chinese words. Because Chinese words are the basic unit of Chinese semantic expression, Chinese word segmentation is a very important basic task, and the performance of the word segmentation system will directly affect the upper-level tasks of Chinese natural language processing, such as information retrieval and machine translation. [0003] In the past ten years, there has been a lot of research work on Chinese word segmentation, and many remarkable achievements have been made. On the one hand, many standard data sets for Chinese word segmentation have been established; on the other hand, ma...

AI Technical Summary

Problems solved by technology

In semi-supervised domain transfer, we have a lot of labeled data in the source domain, but we can only get unlabeled data in the target domain

Images