System and method for processing text utilizing a suite of disambiguation techniques

Abstract

The invention relates to a system and method for processing natural language text utilizing disambiguation components to identify a disambiguated sense for the text. For the method, it comprises applying a selection of the components to the text to identify a local disambiguated sense for the text. Each component provides a local disambiguated sense of the text with a confidence score and a probability score. The disambiguated sense is determined utilizing a selection of local disambiguated senses. The invention also relates to a system and method for generating sense-tagged text. For the method, it comprises steps of: disambiguating a quantity of documents utilizing a disambiguation component; generating a confidence score and a probability score for a sense identified for a word provided by the component; if the confidence score for the sense for the word is below a set threshold, the sense is ignored; and if the confidence score for the sense for the word is above the set threshold, the sense is added to the sense-tagged text.

Description

RELATED APPLICATION [0001] This application claims the benefit of U.S. Provisional Application No. 60 / 496,681 filed on Aug. 21, 2003.FIELD OF THE INVENTION [0002] The present invention relates to disambiguating natural language text, such as queries to an Internet search engine, web pages and other electronic documents, and disambiguating textual output of a speech to text system. BACKGROUND [0003] Word sense disambiguation is the process of determining the meaning of words in text. For example, the word “bank” can mean a financial institution, an embankment, or an aerial manoeuvre (or several other meanings). When humans listen to or read naturally expressed language, they automatically select the correct meaning of each word based on the context in which it is expressed. A word sense disambiguator is a computer-based system for accomplishing this task, and is a critical component of technology for making naturally expressed language understandable to computers. [0004] A word sense...

AI Technical Summary

Benefits of technology

[0016] After applying the selection to the text, the method may refine a knowledge base of each component in the selection utilizing the disambiguated sense (or senses).

Problems solved by technology

It is accepted by those skilled in the art that, although humans perform word sense disambiguation effortlessly, and this is a critical step in understanding naturally expressed language, no system has yet been developed to accomplish word sense disambiguation of general texts to an accuracy sufficient to permit deployment in such applications.

Even current advanced word sense disambiguation systems may have an accuracy of only approximately 33%, thereby making their results too inaccurate for many applications.

Images