Method for processing unknown words in Chinese-language dependency tree banks

Abstract

The invention belongs to the field of processing for natural languages of computational linguistics, and discloses a method for processing unknown words in Chinese-language dependency tree banks. The method includes steps of A, searching all synonyms of the unknown words by the aid of synonym forests; B, computing character pattern similarity degrees among the unknown words and all the synonyms of the unknown words according to character pattern features of Chinese characters; C, extracting mapped words and information quantities of word classes of the mapped words when the character pattern similarity degrees among the unknown words and the multiple synonyms are high, and improving character pattern similarity degree computation models; D, extracting the words with the maximum character pattern similarity degrees as the optimal mapped words of the unknown words and using the extracted words as explanation for the unknown words in the tree banks. The method has the advantages that unit pairs (word classes, word classes) in dependency syntactic analysis can be recovered to unit pairs (word classes, words) or unit pairs (words, word classes) on the premise that the scales of the tree banks are no longer expanded, accordingly, the information granularity can be refined, the problem of data sparseness can be solved, and the dependency syntactic analysis performance can be improved.

Description

technical field [0001] The invention relates to a method for processing unregistered words in a Chinese dependent grammar tree bank, which realizes understanding of unregistered words through known words in the tree bank, and belongs to the field of natural language processing in computational linguistics. Background technique [0002] Syntactic analysis is one of the core issues of natural language processing, and its performance directly affects the correctness and effectiveness of automatic understanding of natural language sentences. Dependency parsing is easier to handle than structured parsing and has received a lot of attention in recent years. At present, many countries are establishing and developing their own language treebanks. As the powerful disambiguation ability of vocabulary itself is gradually discovered, more and more statistical models of dependent syntactic analysis tend to be lexicalized. [0003] Vocabulary is the most discriminative information, and l...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problem of treebank data sparseness in dependency syntactic analysis and the coarse information granularity caused by smoothing part-of-speech information, the present invention provides a processing method for unregistered words in Chinese dependency treebank, which realizes the relationship between unregistered words in treebank and The mapping conversion of known words allows the unit pair to be returned to or to refine the information granularity and alleviate the problem of data sparseness without expanding the data scale. Improved performance of dependency parsing

Images