Domain dictionary creation

Abstract

Methods, systems, and apparatus, including computer program products, to identify topic words in a document corpus that includes topic documents related to a topic are disclosed. A reference topic word divergence value based on the document corpus and the topic document corpus is determined. A candidate topic word divergence value for a candidate topic word is determined based on the document corpus and the topic document corpus. The candidate topic word is determined to be a topic word if the candidate topic word divergence value is greater than the reference topic word divergence value.

Description

[0001] Cross References to Related Applications [0002] This application claims priority to US Application Nos. 11 / 844,067 and 11 / 844,153, filed August 23,2007. The disclosure of the aforementioned applications should be considered part of (and incorporated by reference) part of the disclosure of the present application. technical field [0003] The present disclosure relates to dictionaries for natural language processing applications such as machine translation, segmentation of non-Roman language words, speech recognition, and input method editors. Background technique [0004] The use of increasingly advanced natural language processing techniques in data processing systems such as speech processing systems, handwriting / optical character recognition systems, automatic translation systems, or in word processing systems for Spelling / grammar checking. These natural language processing techniques may include automatically updating dictionaries for natural language applicat...

AI Technical Summary

Problems solved by technology

Therefore, new words in text cannot be easily identified since new words are compound sequences of characters or existing words

This makes new word detection a difficult task for these languages

Images