A multi-mode man-machine interaction system and an interaction method

Abstract

The invention discloses a multi-mode man-machine interaction system. The system comprises a non-contact instruction collection unit and an instruction response unit, wherein the non-contact instruction collection unit comprises a gesture instruction collection module, an eye movement instruction collection module and a voice instruction collection module, and the instruction response unit comprises an interface processing module, an instruction fusion module and a fusion information processing module. In the interaction process, the non-contact instruction collection is carried out, the gesture, eye movement and voice instructions are generated by the instruction fusion module to form a fusion instruction, and the fusion information processing module receives the fusion instruction or thevoice instruction, converts the fusion instruction or the voice instruction into a protocol instruction and outputs the protocol instruction through the interface processing module. According to the present invention, the instruction fusion function of the three non-contact instructions is achieved, the man-machine interaction is natural, and the instruction output is reliable and stable. And theestablishment of the hot zone can provide the reliable instruction interference resistance and predictability, so that the hot zone is more humanized, and the instruction fusion is more accurate. Thesystem is high in integration, is suitable for airborne operation and can meet various man-machine interaction requirements, and the whole interaction is non-contact operation.

Description

technical field [0001] The invention relates to an interactive system, in particular to a multi-mode human-computer interaction system and an interaction method, belonging to the technical field of human-computer interaction. Background technique [0002] At present, human-computer interaction is realized through contact instruction input. Human-computer interaction through instruction buttons or touch screen instructions will disperse part of the operator's energy during the instruction input process. In order to increase the convenience of interaction, the voice command function will be embedded in the human-computer interaction system, so as to realize the cooperative operation of voice commands and contact commands. Voice and touch cooperation can improve the convenience of interaction, but it still cannot realize natural operation. [0003] The patent No. CN104714645A also proposes a multi-modal human-computer interaction system and method. This patent mainly uses mult...

AI Technical Summary

Problems solved by technology

Voice and touch cooperation can improve the convenience of interaction, but it still cannot realize natural operation

[0003] The patent No. CN104714645A also proposes a multi-modal human-computer interaction system and method. This patent mainly uses multi-level artificial neural networks and mutually exclusive information to distinguish and discriminate the instruction information sent by the human body. Response, the main forms of instructions involved include somatosensory, voice and gestures. In actual implementation, this method will convert the parallel instruction form into serial, and does not realize the natural operation in the true sense.

[0004] The patent number is CN103809733A, which proposes an interactive method based on human sight and finger pointing. This method uses images to obtain sight and hand movements, and determines the operation target according to the corresponding gestures and eye indication time. Use specific gestures to start or stop gesture recognition and complete the target operation response. The structure is simple and easy to implement. This operation method does not realize a natural and smooth human-computer interaction method. It is just a combination of two simple operation modes and does not realize real natural interaction

Images