Multi-modal information processing and interaction system

Abstract

The invention relates to a multi-modal information processing and interaction system which is used for solving the problems that in a multi-modal interaction system, a modal fusion mode is simple, and a dialogue mechanism is stiff. The system comprises a multi-modal information cognition module, a multi-modal information fusion module and a multi-modal dialogue management module, the multi-modal information cognition module is used for identifying all modal interaction information of a user, and the multi-modal information fusion module performs intention fusion on all modal interaction information of the user by using a DS evidence theory to obtain a multi-modal dialogue result; a final interaction intention of the user is determined and a formalized instruction which corresponds to the final interaction intention of the user and can be identified by a machine is acquired. the multi-modal dialogue management module adopts a dialogue management model integrating a finite-state machine and an information slot filling method for a multi-modal human-computer interaction scene, and is used for controlling a dialogue process and generating a response; according to the invention, the accuracy of user interaction intention recognition is effectively improved, and natural and flexible man-machine interaction is realized.

Description

technical field [0001] The invention relates to multi-modal information fusion technology, in particular to a multi-modal interactive system that effectively utilizes multi-modal information and can realize human-computer friendly interaction. Background technique [0002] Since the late 20th century, more and more scholars have paid more and more attention to the research on multimodal human-computer interaction. Many university laboratories and scientific research institutions at home and abroad have established relevant scientific research teams, such as the School of Human-Computer Interaction at Carnegie-Mellon University, the Artificial Intelligence Research Center at Stanford University, and the Media Lab at the Massachusetts Institute of Technology. Large companies such as Google and Microsoft have also injected a lot of manpower and material resources into the research of multimodal human-computer interaction. Since multi-modal human-computer interaction has receiv...

AI Technical Summary

Problems solved by technology

[0005] In order to effectively solve the problem of simple modal fusion and rigid dialogue mechanism in the multimodal interactive system, the present invention first establishes a multimodal information fusion model, based on the D-S evidence theory, making full use of multimodal information for intent Fusion, and combine the interaction information of each mode under the intent based on the slot filling method

Images