Chinese lexical analysis method

Abstract

The invention discloses a Chinese lexical analysis method. The Chinese lexical analysis method comprises the following steps of (1) obtaining a characteristic function and a weight from a given training corpus; (2) segmenting an input Chinese text: segmenting the input Chinese text into multiple statements, wherein one statement is a word sequence; (3) calculating a conditional probability of all possible lexical information tagging sequences of the word sequences corresponding to the input Chinese text; (4) determining a final lexical information tagging sequence of the word sequences corresponding to the input Chinese text; (5) carrying out Chinese word segmenting, Chinese POS (Part-of-Speech) tagging and Chinese named entity recognizing, and thus obtaining a final Chinese lexical analysis result. According to the Chinese lexical analysis method disclosed by the invention, three subtasks of Chinese lexical analysis are realized by being unified in a word sequence tagging framework, the defects that error is upwards transmitted, amplified and accumulated, and multiple classes of information are difficult to integrate and utilize are overcome, the calculation is simple, and the operation amount is small; a dictionary is not needed at all, and unknown words can also be better segmented and tagged.

Description

technical field [0001] The invention relates to a Chinese lexical analysis method. Background technique [0002] In the field of Chinese information processing, Chinese lexical analysis is one of the important basic research topics. It is not only the basis of deep Chinese information processing such as syntactic analysis, semantic analysis, and text understanding, but also a key link in applications such as machine translation, question answering systems, information retrieval, and information extraction. Chinese lexical analysis mainly includes three sub-tasks of Chinese word segmentation, part-of-speech tagging and named entity recognition. In some related evaluations at home and abroad, they are often evaluated as three independent tasks. In existing research, most scholars are also accustomed to considering the three sub-tasks independently, especially accustomed to processing Chinese word segmentation and part-of-speech tagging in sequence, and then considering part-o...

AI Technical Summary

Problems solved by technology

This method has the disadvantages of complex calculation and large amount of calculation, and does not include the recognition of named entities

Images