A Chinese word segmentation method with feature alignment

Abstract

A Chinese word segmentation method for feature alignment include extracting feature of binary words from marked data and unmarked data, 101, extracting feature of binary words from marked data and unmarked data, extracting feature of binary words from marked data, 102 align that tagged data with the untagged data by an earth move's distance (hereinafter referred to as EMD) method; 103 train that features of the tag data aft the features are aligned by the classifier xgboost, thereby predict the probability of forming the binary words in the untagged data; 104 extracting a portion of the binarywords from the result of the classifier and integrating the binary words of the tag data of step 101 as features of the conditional random field and performing training; 105,The untagged data are serially tagged and segmented by the established model. The invention mainly features that the marked data and the unmarked data are feature aligned by EMD, the word formation probability of the binary words is predicted by learning the classifier, and then a new word segmenter is formed by integrating the conditional random fields in a stacking manner.

Description

technical field [0001] The invention belongs to the field of natural language processing, and in particular relates to a feature-aligned Chinese word segmentation method. Background technique [0002] As the most basic unit of language, words play a very important role in text analysis tasks. As an indispensable part of natural language processing, Chinese word segmentation has achieved tremendous development in recent years and has been widely used in various Chinese natural language processing tasks such as information retrieval, knowledge extraction and question answering. Due to the high cost of labeling data, the evolution of word usage, and the different needs in different scenarios, existing Chinese word segmentation methods still have some problems in practical tasks. Although it has high accuracy on regular text, the segmentation of low-frequency words is still a challenge in many cases. For example, in the absence of a predefined dictionary, the regional word "Ga...

AI Technical Summary

Problems solved by technology

Of course, the sequence tagging method based on conditional random field is also a commonly used method to deal with Chinese word segmentation. By using enough context features, although good results have been achieved, due to the nature of conditional random field, the parameters will increase exponentially, so It is unwise to directly train CRFs with multiple features

Images