Chinese language lexical analysis method based on linear model

Abstract

The invention provides a Chinese lexical analysis method based on a linear model, comprising the following steps: 1) a Chinese sentence is input and the length of an analysis window is set; 2) the verbatim analysis is carried out to the sentence, the character or a character set of each character in the sentence in a time window is input in a perceptron classifier, thus obtaining the score of a perceptron model which tags the current character as the certain word segmentation tag and part-of-speech tag; at the same time, the character or the character set of the character in the time window is input in the linear lexical analysis model, thus obtaining the score of the linear lexical analysis model which tags the current character as the certain word segmentation tag and the part-of-speech tag; 3) the score of the perceptron model and the score of the linear lexical analysis model are carried out the weighted sum, thus obtaining the comprehensive analysis score, the word segmentation tag and the part-of-speech tag with the highest comprehensive analysis score are taken as the word segmentation tag and the part-of-speech tag of the current character; when the word segmentation tag and the part-of-speech tag of all the characters complete the tagging, the lexical analysis of the Chinese sentence is completed. The Chinese lexical analysis method can significantly improve the accuracy of segmentation and tagging.

Description

technical field [0001] The invention relates to the technical field of statistical natural language processing, in particular statistical Chinese word segmentation and part-of-speech tagging. Background technique [0002] There are two goals of Chinese lexical analysis: word segmentation and part-of-speech tagging. Word segmentation is to divide Chinese sentences that are closely connected between characters into words, so as to convert the sequence of Chinese characters into a sequence of Chinese words; part-of-speech tagging is based on word segmentation, and a part-of-speech tag is added to each Chinese word mark, such as verb VV , noun NN and so on. For a given Chinese sentence, how to perform word segmentation and part-of-speech tagging? There are two strategies: one is to perform word segmentation first, and then perform part-of-speech tagging on the basis of word segmentation; the other is to consider part-of-speech tagging during the word segmentation process. Obv...

AI Technical Summary

Problems solved by technology

However, the existing word segmentation system based on character feature classifiers has a single word segmentation model, and it is difficult to directly use some statistical information obtained from the corpus (for example: how likely is a certain word to be marked as a certain part-of-speech tag? How likely is a part-of-speech tag sequence to appear? How likely is a word sequence to appear?), therefore, the accuracy of segmentation and labeling of the existing word segmentation system based on word feature classifiers needs to be improved

Images