Systems and methods for aggregating related inputs using finite-state devices and extracting meaning from multimodal inputs using aggregation

Abstract

When using finite-state methods to represent inputs, it is beneficial to combine related inputs and to represent combinations of related inputs in the finite-state device. In multimodal recognition systems and methods that integrate different recognized modes, combining related inputs helps the unification process.

Description

[0001] This non-provisional application claims the benefit of U.S. provisional application No. 60 / 313,121, filed on Aug. 15, 2001, which is incorporated herein by reference in its entirety.[0002] 1. Field of Invention[0003] This invention is directed to aggregating related inputs when using finite-state methods.[0004] 2. Description of Related Art[0005] Multimodal interfaces allow input and / or output to be conveyed over multiple different channels, such as speech, graphics, gesture and the like. Multimodal interfaces enable more natural and effective interaction, because particular modes are best-suited for particular kinds of content. Multimodal interfaces are likely to play a critical role in the ongoing migration of interaction from desktop computing to wireless portable computing devices, such as personal digital assistants, like the Palm Pilot.RTM., digital cellular telephones, public information kiosks that are wirelessly connected to the Internet or other distributed networks...

AI Technical Summary

Problems solved by technology

One barrier to adopting such wireless portable computing devices is that they offer limited screen real estate, and often have limited keyboard interfaces, if any keyboard interface at all.

While the technique disclosed in Johnston 1 overcomes many of the limitations of earlier multimodal systems, this technique does not scale well to support multi-gesture utterances, complex unimodal gestures, or other modes and combinations of modes.

However, the unification-based approach disclosed in Johnston 1-Johnston 3 does not allow for tight coupling of multimodal parsing with speech and gesture recognition.

Moreover, multimodal parsing cannot directly influence the progress of either speech recognition or gesture recognition.

The multi-dimensional parsing approach is also subject to significant concerns in terms of computational complexity.

This complexity is manageable when the inputs yield only n-best results for small n. However, the complexity quickly gets out of hand if the inputs are sizable lattices with associated probabilities.

The unification-based approach also runs into significant problems when choosing between multiple competing parses and interpretations.

Systems and methods for recognizing and representing gestural input have been very limited.

However, neither approach allows for efficiently and generically representing arbitrary gestures.

However, it is generally difficult, if not effectively impossible to capture all of different possible sequences of coordinates that occur in a gestural input in order to copy them from the gesture input tape to the meaning output tape of the finite-state automaton.

However, this limits the number of distinct gesture events in a single utterance to no more than the available number of variables, because the number of gestural inputs that can be handled is limited by the number of variables used.

Images