The invention relates to a gesture and voice fused multi-mode man-
machine interaction
system and method, and belongs to the field of multi-mode man-
machine interaction. The
system comprises an interactive
perception layer, an
interactive analysis layer and an interactive response layer, the interactive
perception layer is used for acquiring multi-
modal interactive information, accessing gesture and voice
modal information acquisition equipment, and acquiring and
processing original signals of the interactive information; the interaction analysis layer is used for defining the meaning of the human-computer interaction information of each channel and analyzing the meaning and instruction intention of the multi-
modal interaction information; and the interaction response layer is used for realizing mapping between the man-
machine interaction intention obtained by the interaction analysis layer and the interaction operation of the application
system, and comprises an instruction-operation conversion module. According to the invention, independent recognition and fusion recognition functions of gesture instructions and voice instructions are provided, the robustness of the man-machine interaction system is improved, a more natural and convenient man-machine
interaction mode is provided, and the development requirements of intelligent man-machine interaction are met.