A Method for Segmenting Chinese General Texts Based on Partially Supervised Learning

Abstract

The invention belongs to the technical field of language processing, and discloses a Chinese generalized text segmentation method based on partly supervised learning. The word segmentation task of Chinese short texts is regarded as a two-category or three-category problem, and the main features of the short text are extracted with relatively The contextual feature information of small noises is combined with a partially supervised learning method for word segmentation. Through the control experiments of five groups plus a set of "difficult" data sets, it is not difficult to find that the results of word segmentation of short texts are deeply affected by the length of contextual information, among which binary contextual information can best fit the characteristics of word segmentation of short texts. It can effectively improve the performance of word segmentation; the mixed features of two and three elements can better express the information of each "empty" and its performance is the best, and it will lose performance if it is more or less; the application of partly supervised learning in short text word segmentation is also It can reflect its excellent ability to complete parameters, which can greatly reduce the work of manual labeling and obtain better performance.

Description

technical field [0001] The invention belongs to the technical field of language processing, and in particular relates to a Chinese generalized text segmentation method based on partly supervised learning. Background technique [0002] In natural language processing tasks, the most basic task is to segment a piece of text into blocks containing the most basic semantics. And word can meet the requirement of this task of the present invention just most, the present invention can be extracted by the segmentation that word is easy to be extracted by blank space in the language of self-contained delimiter between similar English this class word, but in Chinese this without separating In the language of symbol, the present invention just needs to carry out a participle task separately. At present, there are two traditional conventional methods. One is based on the matching method, that is, using a manually constructed dictionary to perform word-by-word comparison to verify whether...

AI Technical Summary

Problems solved by technology

[0003] To sum up, the problems existing in the existing technology: relying on large-scale artificial data sets, requiring a lot of manpower and time consumption; the word recognition rate is low; the text cannot be accurately segmented into words with appropriate granularity

Images