A feature-aligned Chinese word segmentation method

Abstract

The present invention claims to protect a feature-aligned Chinese word segmentation method, which includes: 101 extracting features of bigrams from marked data and unmarked data; 102 combining marked data with unmarked data by the Earth Mover's Distance (hereinafter referred to as EMD) method 103 Use the classifier xgboost to train the features of the labeled data after feature alignment, so as to predict the probability of big words in unlabeled data becoming words; 104 Extract some big words and steps from the results of the classifier 101 The bigram integration of labeled data is used as the feature of the conditional random field and trained; 105 The unlabeled data is sequence-labeled and segmented through the established model. The invention mainly performs feature alignment on marked data and unmarked data through EMD, predicts the word formation probability of binary words through classifier learning, and then integrates conditional random fields in a stacking manner to form a new tokenizer.

Description

technical field [0001] The invention belongs to the field of natural language processing, and in particular relates to a feature-aligned Chinese word segmentation method. Background technique [0002] As the most basic unit of language, words play an important role in text analysis tasks. As an indispensable part of natural language processing, Chinese word segmentation has made great progress in recent years and is widely used in various Chinese natural language processing tasks such as information retrieval, knowledge extraction and question answering. Due to the expensive cost of labeling data, the evolution of word usage, and the different needs in different scenarios, the existing Chinese word segmentation methods still have some problems in practical tasks. Although it has high accuracy on regular text, segmentation of low-frequency words is still a challenge in many cases. For example, in the absence of a predefined dictionary, the local word "Gaotang" would be spli...

AI Technical Summary

Problems solved by technology

Of course, the sequence tagging method based on conditional random field is also a commonly used method to deal with Chinese word segmentation. By using enough context features, although good results have been achieved, due to the nature of conditional random field, the parameters will increase exponentially, so It is unwise to directly train CRFs with multiple features

Images