System and iterative method for lexicon, segmentation and language model joint optimization

Abstract

Provides a method for optimizing the language model, including using the maximum matching technique to establish an initial language model based on the lexicon and segmentation obtained from the received corpus, and by dynamically updating the dictionary and re-segmenting the corpus according to statistical principles, iteratively improves Initialize the language model until a threshold of predictive power is reached.

Description

[0001] This application claims priority to Provisional Patent Application No. 60 / 163850, "An iterative method for lexicon, wordsegmentation and language model joint optimization," filed on November 5, 1999 by the inventors of this application. technical field [0002] The present invention relates to language modeling, more specifically to a system and an iterative method for joint optimization of dictionaries, text segmentation and language models. Background technique [0003] Recent advances in computing power and related technologies have enabled the development of a new generation of powerful application software, including web browsers, word processing and speech recognition applications. For example, the latest generation of web browsers anticipate Uniform Resource Locator (URL) address entry after entering the first two or three characters of a domain name. Word processors offer improved spelling and grammar checking, word prediction, and language conversion. Newer ...

AI Technical Summary

Problems solved by technology

Thus the model cannot accurately predict smaller words contained within a semantically acceptable larger string

[0014] As a result of the above limitations, language models using state-of-the-art dictionaries and segmentation algorithms tend to be error-prone

That is, any errors made in the lexicon or segmentation stages are propagated throughout the language model, limiting the accuracy and predictive properties of the language model

[0015] Finally, restricting the model to a maximum of two prior words of context (as far as 3-gram language models are concerned) is also restrictive, since more context may be required to accurately predict the likelihood of a word

These three limitations of a language model usually lead to poor prediction quality for that language model

Images