Multi-mode man-machine interaction system and method fusing gestures and voice

Abstract

The invention relates to a gesture and voice fused multi-mode man-machine interaction system and method, and belongs to the field of multi-mode man-machine interaction. The system comprises an interactive perception layer, an interactive analysis layer and an interactive response layer, the interactive perception layer is used for acquiring multi-modal interactive information, accessing gesture and voice modal information acquisition equipment, and acquiring and processing original signals of the interactive information; the interaction analysis layer is used for defining the meaning of the human-computer interaction information of each channel and analyzing the meaning and instruction intention of the multi-modal interaction information; and the interaction response layer is used for realizing mapping between the man-machine interaction intention obtained by the interaction analysis layer and the interaction operation of the application system, and comprises an instruction-operation conversion module. According to the invention, independent recognition and fusion recognition functions of gesture instructions and voice instructions are provided, the robustness of the man-machine interaction system is improved, a more natural and convenient man-machine interaction mode is provided, and the development requirements of intelligent man-machine interaction are met.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal human-computer interaction, and in particular relates to a multimodal human-computer interaction system and method integrating gesture and voice. Background Art

[0002] In recent years, with the development of new technologies such as sensor technology, big data technology, and artificial intelligence technology, human-computer interaction is moving towards intelligent perception, high efficiency, convenience, and people-oriented. Traditional human-computer interaction methods such as keyboards, mice, and touch screens can no longer meet the needs of intelligent human-computer interaction. It is urgent to build a new, efficient, convenient, and intelligent human-computer interaction method to achieve natural and efficient interaction between people and systems.

[0003] With the continuous development of intelligent theories and technologies such as computer vision, speech recognition, and natural language processing, researchers have implanted single-modal, contactless intelligent interaction methods such as voice or gestures into human-computer interaction systems to broaden the information sources and implementation methods of human-computer interaction and improve the convenience of human-computer interaction systems. However, single-modal interaction input methods such as voice and gestures are easily affected by environmental factors such as noise, bumps, and changes in lighting, which makes it impossible for the system to fully and accurately capture the user's interaction instructions, resulting in a misunderstanding of the user's operating intentions, causing the system to respond incorrectly, thereby reducing the accuracy and reliability of human-computer interaction. Therefore, single-modal intelligent human-computer interaction methods are difficult to meet the needs of efficient and accurate human-computer interaction.

[0004] In response to the above problems, multimodal human-computer interaction technology has become a hot research direction and a key application direction. By effectively combining interactive information such as voice and gesture, the information complementarity of each channel can be achieved, thereby improving the accuracy and reliability of intelligent human-computer interaction, improving the efficiency of human-computer interaction, and realizing a more intelligent, natural, harmonious and efficient human-computer interaction method. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is how to provide a multimodal human-computer interaction system and method that integrates gestures and voice, so as to solve the problem that a single-modal intelligent human-computer interaction method is difficult to meet the needs of efficient and accurate human-computer interaction.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention proposes a multimodal human-computer interaction system integrating gesture and voice, the system comprising: an interaction perception layer, an interaction analysis layer and an interaction response layer;

[0009] The interactive perception layer is used to obtain multimodal interactive information. It is responsible for the access of gesture and voice modal information acquisition devices, as well as the acquisition and processing of the original signals of interactive information, including: gesture information acquisition module and voice information acquisition module;

[0010] The interaction analysis layer is the core of the human-computer interaction system. It is used to define the meaning of human-computer interaction information in each channel and analyze the meaning and instruction intention of multimodal interaction information. It includes: interaction instruction definition module, gesture instruction recognition module, voice instruction recognition module, and multimodal instruction fusion module.

[0011] The interactive response layer is the output window of the human-computer interaction system, which is used to realize the interactive operation mapping between the human-computer interaction intention obtained by the interactive analysis layer and the application system, including the instruction-operation conversion module.

[0012] The present invention also provides a multimodal human-computer interaction method integrating gesture and voice, the method comprising the following steps:

[0013] Step S1, interactive information perception: through the interactive perception layer, user gesture and voice interaction information is collected via relevant sensor devices;

[0014] Step S2, single-mode interaction information analysis: the interaction perception layer transmits the collected interaction information to the gesture command recognition module and the voice command recognition module of the interaction analysis layer; the interaction command definition module of the interaction analysis layer defines specific gesture and voice interaction mapping instructions; according to the interaction command definition module, the gesture command recognition module uses a gesture recognition method to recognize the instruction meaning of the user's gesture information, and the voice command recognition module uses a voice recognition method to recognize the instruction meaning of the user's voice information;

[0015] Step S3, multi-modal instruction semantic judgment: Use the multi-modal instruction semantic judgment method to judge whether the instruction information of each modal channel has operational semantics. When there is instruction information of a certain modality that does not have operational semantics, directly output the instruction information with operational intention semantics and jump to step S5. Otherwise, enter step S4;

[0016] Step S4, multimodal information fusion analysis: input the gesture command and the voice command into the multimodal command fusion understanding model, and the multimodal command fusion understanding model integrates the gesture interaction information and the voice interaction information, compares the command definition of the interaction command definition module, and evaluates and analyzes the user's operation intention;

[0017] Step S5, instruction-operation conversion: the user's operation intention is transmitted to the instruction-operation conversion module of the interactive response layer, and the instruction-operation conversion module automatically generates an operation instruction according to the user's operation intention and the GIS map software operation API;

[0018] Step S6, application system response: the operation instruction generated by the instruction-operation conversion module is transmitted to the GIS map software, and the GIS map software responds to realize a closed loop of human-computer interaction operation.

[0019] (III) Beneficial effects

[0020] The present invention proposes a multimodal human-computer interaction system and method integrating gesture and voice, and the beneficial effects of the present invention are embodied in:

[0021] (1) Through multi-channel and multi-modal human-computer interaction methods such as gestures and voice, the human-computer interaction information input capability is fully expanded, providing users with a more natural and convenient human-computer interaction method and accelerating the operational efficiency of human-computer interaction;

[0022] (2) By fusing multimodal interaction information, we can avoid the problem of decreased interaction accuracy when single-channel interaction information is affected by the environment, and improve the robustness of the human-computer interaction method;

[0023] (3) The multimodal human-computer interaction system adopts a modular design approach, which can upgrade module functions and performance through advanced technologies and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a block diagram of a multimodal human-computer interaction system integrating gesture and voice according to the present invention;

[0025] Figure 2 This is a flow chart of the application of the multimodal human-computer interaction system of the present invention;

[0026] Figure 3 is a system block diagram of an embodiment of the present invention;

[0027] Figure 4 This is a structural diagram of a multimodal instruction fusion understanding model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0029] The purpose of the present invention is to provide a multimodal human-computer interaction system and method that integrates gestures and voice, aiming to solve the problem of how to design a more natural and convenient intelligent human-computer interaction system and method through multimodal interaction channels such as gestures and voice, and realize functions such as multimodal interaction information collection, recognition of instructions of each modal interaction information, multimodal instruction fusion, and instruction-operation conversion.

[0030] In order to achieve the above object, the present invention adopts the following technical solutions:

[0031] A multimodal human-computer interaction system integrating gesture and speech Figure 1 As shown, it includes: an interactive perception layer, an interactive parsing layer and an interactive response layer.

[0032] The interactive perception layer is the information source of the human-computer interaction system. It is used to obtain multimodal interactive information. It is responsible for the access of gesture, voice and other modal information acquisition devices, as well as the acquisition and processing of the original signals of the interactive information, including: gesture information acquisition module, voice information acquisition module.

[0033] The interaction analysis layer is the core of the human-computer interaction system. It is used to define the meaning of human-computer interaction information in each channel and analyze the meaning and command intention of multimodal interaction information. It includes: interaction command definition module, gesture command recognition module, voice command recognition module, and multimodal command fusion module.

[0034] The interactive response layer is the output window of the human-computer interaction system, which is used to realize the interactive operation mapping between the human-computer interaction intention obtained by the interactive analysis layer and the application system, including the instruction-operation conversion module.

[0035] The gesture information acquisition module is used to acquire the user's gesture information, and can support acquiring gesture information of different modalities such as image modality gesture information and motion parameter modality gesture information through different sensors such as cameras and smart gloves.

[0036] The voice information acquisition module is used to acquire the user's voice information and can support acquiring voice information through different sensors such as air conduction microphones and bone conduction microphones.

[0037] The interaction instruction definition module is used to define the meaning of human-computer interaction information of each channel and realize the mapping of gesture actions, voice information and related interaction instructions.

[0038] The gesture command recognition module is used to recognize user gesture actions. It can parse image modal gesture information through a gesture recognition method based on machine vision, and can also parse motion parameter modal gesture information through a template matching method or a deep learning method.

[0039] The voice command recognition module is used to recognize user voice information and can implement voice information analysis through deep learning-based voice recognition technology.

[0040] The multimodal command fusion module is used to realize the fusion of multimodal command information such as gesture information and voice information and the recognition of interactive intentions. It adopts a decision-level fusion method to fuse and analyze the instructions of gesture recognition and voice recognition to clarify the user's operation intentions.

[0041] The instruction-operation conversion module is used to define the interactive operation mapping relationship between the human-computer interaction instruction and the target application system, and provide an interactive operation interface with the target application system, so that the target application system makes relevant operation responses according to the user's human-computer interaction intention, thereby realizing the user's operation intention.

[0042] The present invention also provides a human-computer interaction method based on the fusion of multimodal interactive commands of gesture commands and voice commands. Multimodal interactive command fusion is actually the semantic fusion process of multi-channel interactive command information at the decision-making level, that is, based on the command results of gesture recognition and voice recognition, a deep understanding of the user's interactive operation semantics and intentions is achieved. The core is to use natural language processing algorithms to build a multimodal command fusion understanding model, extract complementary or similar operation semantic feature information in voice commands and gesture commands, and achieve a complete analysis of the user's real operation intentions through the fusion analysis of multi-channel operation semantic feature information such as voice and gesture, thereby improving the accuracy of human-computer interaction.

[0043] Furthermore, in order to avoid unnecessary calculation and operation time expenditure caused by calling the multimodal instruction fusion understanding model when a certain channel lacks interaction intention during actual use, the present invention adds a multimodal instruction semantic judgment step before calling the multimodal instruction fusion understanding model. The main principle of the multimodal instruction semantic judgment step is: extract the verb instructions in each modal interaction instruction such as gesture and voice, and compare them with the defined interaction action name. When the input verb instruction matches the defined interaction action name, it means that the modal input instruction has semantic information, otherwise, it does not have semantic information. When and only when both the voice and gesture channel instructions have semantic information, the multimodal instruction fusion understanding model is triggered to perform multimodal instruction fusion analysis, otherwise, the channel instruction with semantic information is directly output.

[0044] In summary, the specific application process of the human-computer interaction application system provided by the present invention is as follows: Figure 2 , including the following steps:

[0045] Step S1, interactive information perception: through the interactive perception layer, user gestures, voice and other interactive information are collected through relevant sensor devices.

[0046] Step S2, unimodal interactive information analysis: The interactive perception layer transmits the collected interactive information to the gesture command recognition module and the voice command recognition module of the interactive analysis layer. The interactive command definition module of the interactive analysis layer defines specific gesture and voice interactive mapping instructions. According to the interactive command definition module, the gesture command recognition module uses a gesture recognition method to identify the instruction meaning of the user's gesture information, and the voice command recognition module uses a voice recognition method to identify the instruction meaning of the user's voice information.

[0047] Step S3, multi-modal instruction semantic judgment: Use the multi-modal instruction semantic judgment method to judge whether the instruction information of each modal channel has operational semantics. When there is instruction information of a certain modality that does not have operational semantics, directly output the instruction information with operational intention semantics and jump to step S5. Otherwise, enter step S4;

[0048] Step S4, multimodal information fusion analysis: The gesture instructions and voice instructions are input into the multimodal instruction fusion understanding model, and the multimodal instruction fusion understanding model integrates the gesture interaction information and the voice interaction information, compares the instruction definition of the interaction instruction definition module, and evaluates and analyzes the user's operation intention.

[0049] Step S5, instruction-operation conversion: the user's operation intention is transmitted to the instruction-operation conversion module of the interactive response layer, and the instruction-operation conversion module automatically generates an operation instruction according to the user's operation intention and the GIS map software operation API.

[0050] Step S6, application system response: the operation instruction generated by the instruction-operation conversion module is transmitted to the GIS map software, and the GIS map software responds to realize a closed loop of human-computer interaction operation.

[0051] Embodiment 1:

[0052] The present invention uses a camera and an air conduction microphone as gesture and voice information collection sensors, and uses GIS (geographic spatial information system) map software as the target application system to construct an embodiment of a human-computer interaction application system. The system components of the embodiment are as follows: Figure 3 As shown, the camera is connected to the multimodal human-computer interaction system via the gesture information acquisition module of the interactive perception layer, providing a source of gesture image information for the human-computer interaction application system. The air conduction microphone is connected to the multimodal human-computer interaction system via the voice information acquisition module of the interactive perception layer, providing a source of voice information for the human-computer interaction application system. The instruction-operation conversion module of the interactive response layer is connected to the application operation software interface (API) provided by the GIS map software, and the user's intention is converted into a corresponding operation instruction through the instruction-operation conversion module, prompting the GIS map software to make a relevant operation response.

[0053] Furthermore, the gesture recognition method adopted by the gesture command recognition module in step S2 of the multimodal human-computer interaction application system is divided into two parts: gesture posture estimation and gesture command recognition. The specific implementation steps are as follows:

[0054] Step S211: Hand gesture estimation: Based on the OpenPose algorithm, the key points of the hand in the image are located and the coordinates of the key points of the hand are obtained.

[0055] Step S212: Gesture command recognition: Based on the time-varying relationship of the coordinates of the key points of the hand, the hand action-gesture command mapping relationship in the interactive command definition module is compared to infer and recognize the command meaning of the current gesture action.

[0056] Furthermore, typical dynamic gesture commands in the hand action-gesture command mapping relationship in the interactive command definition module are shown in Table 1, and static gesture commands are common sign language words, such as "1-9", "OK", etc. The hand action-gesture command mapping relationship can be added, modified or deleted according to actual application requirements.

[0057] Table 1

[0058]

[0059]

[0060] Furthermore, the voice command recognition module in step S2 of the multimodal human-computer interaction application system adopts the ASRFrame open source Chinese speech recognition framework to build a speech recognition system to realize the recognition of user voice information.

[0061] Furthermore, the interactive command definition module defines command keywords in voice information input, such as "left", "right", "up", "down", "translate", "clockwise", "counterclockwise", "rotate", "select", "plot", "mark", "zoom in", "zoom out", etc. Voice command keywords can be added, modified or deleted according to actual application requirements.

[0062] Furthermore, the multimodal instruction fusion understanding model structure in step S4 of the multimodal human-computer interaction application system is as follows: Figure 4 As shown in the figure, the improved design is mainly based on the Transformer network structure, including four parts: information encoding, position encoding, encoder, and decoder.

[0063] (1) Information Coding

[0064] The multimodal command fusion understanding model of the present invention is input with Chinese commands such as gesture command text, voice command text, and defined user command text. To facilitate network learning features, word embedding (Word2Vec) is used to encode the input commands and convert the Chinese commands into low-dimensional vectors (tokens) with a dimension of 300.

[0065] (2) Positional encoding

[0066] After information encoding, the gesture command sequence is G = [ges1, ges2, ..., ges n ], the voice command sequence is V = [Voc1, Voc2, ..., Voc m], the user command sequence defined is T = [Tag1, Tag2…, Tag l ]. In order to keep the original order of each sequence element in the instruction text, the relative position encoding is performed by combining the sine and cosine functions provided in the Transformer.

[0067]

[0068] Among them, pos is the position of the token in the text sequence, d model is the dimension of the token, which is 300 in this embodiment.

[0069] (3) Encoder

[0070] The encoder mainly includes a gesture command text input branch and a voice command text input branch. First, the gesture command information and voice command information that have been information encoded and position encoded are spliced ​​together and retained as shallow features to prevent overfitting and gradient disappearance in the network. Secondly, the gesture command information and voice command information are subjected to multi-head attention and nonlinear feature extraction operations in their respective branches to extract key information from each modal command and obtain normalized features. Finally, the normalized feature information of each modality is spliced ​​together, and the shallow feature information retained is combined. The multi-head attention mechanism is used to extract the correlation between gesture command features and voice command features, thereby realizing the fusion of command information from different modalities.

[0071] (4) Decoder

[0072] The decoder is mainly a defined user instruction text input branch, which adopts a standard Transformer decoder structure, and its input is the basic vocabulary in the defined user instruction text. Among them, the multi-head masked attention mechanism is used to ensure that when the decoder generates the output instruction sequence, the model can only use the information before the current position to prevent the current token from paying attention to the future token. The information leakage caused by the token, thereby ensuring the stability of the model during training. The multi-head attention mechanism in the decoder branch normalizes the instruction fusion feature information encoded by the encoder as K and V to participate in the multi-layer decoding calculation, and uses the normalized feature information obtained by the user instruction text input branch as Q, so as to realize the association calculation with the encoding information of the target instruction text and complete the fusion of multi-modal instruction information. Finally, the feedforward network is used to realize deep feature extraction and mapping, and the fully connected layer and the Softmax layer are used to obtain the output probability matrix of the decoder. The corresponding user instruction vocabulary is selected according to the maximum probability to form the final output instruction.

[0073] Furthermore, the normalization in the multimodal instruction fusion understanding model in this embodiment is implemented by layer normalization, and the cross entropy function is used as the loss function of the model for model training.

[0074] Furthermore, the typical command-operation intention mapping relationship for GIS map software operations is shown in Table 2. The command-operation intention mapping relationship can be added, modified or deleted according to actual application requirements.

[0075] Table 2

[0076]

[0077]

[0078] From the above description, it can be found that the multimodal human-computer interaction system and method of the present invention that integrates gesture and voice have the functions of independent recognition and fusion recognition of gesture commands and voice commands, improve the robustness of the human-computer interaction system, provide a more natural and convenient human-computer interaction method, and meet the development needs of intelligent human-computer interaction.

[0079] The beneficial effects of the present invention are embodied in:

[0080] (1) Through multi-channel and multi-modal human-computer interaction methods such as gestures and voice, the human-computer interaction information input capability is fully expanded, providing users with a more natural and convenient human-computer interaction method and accelerating the operational efficiency of human-computer interaction;

[0081] (2) By fusing multimodal interaction information, we can avoid the problem of decreased interaction accuracy when single-channel interaction information is affected by the environment, and improve the robustness of the human-computer interaction method;

[0082] (3) The multimodal human-computer interaction system adopts a modular design approach, which can upgrade module functions and performance through advanced technologies and has strong scalability.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal human-computer interaction system integrating gesture and voice, characterized in that: The system includes: an interactive perception layer, an interactive analysis layer, and an interactive response layer; The interactive perception layer is used to obtain multimodal interactive information. It is responsible for the access of gesture and voice modal information acquisition devices, as well as the acquisition and processing of the original signals of interactive information, including: gesture information acquisition module and voice information acquisition module; The interaction analysis layer is the core of the human-computer interaction system. It is used to define the meaning of human-computer interaction information in each channel and analyze the meaning and instruction intention of multimodal interaction information. It includes: interaction instruction definition module, gesture instruction recognition module, voice instruction recognition module, and multimodal instruction fusion module. The interactive response layer is the output window of the human-computer interaction system, which is used to realize the interactive operation mapping between the human-computer interaction intention obtained by the interactive analysis layer and the application system, including the instruction-operation conversion module.

2. The multimodal human-computer interaction system integrating gesture and voice according to claim 1, characterized in that: The gesture information acquisition module is used to acquire the user's gesture information, and supports acquiring gesture information of image modality gesture information and motion parameter modality gesture information through a camera or smart gloves; The voice information acquisition module is used to acquire the user's voice information, and supports acquiring voice information through air conduction microphones and bone conduction microphone sensors; The interaction instruction definition module is used to define the meaning of human-computer interaction information of each channel and realize the mapping of gesture actions, voice information and related interaction instructions; The gesture command recognition module is used to recognize user gestures, analyze image modal gesture information through a machine vision-based gesture recognition method, and analyze motion parameter modal gesture information through a template matching method or a deep learning method; The voice command recognition module is used to recognize user voice information and implement voice information analysis through deep learning-based voice recognition technology; The multimodal command fusion module is used to realize the fusion of gesture information, voice information and multimodal command information and the recognition of interactive intentions, and adopts a decision-level fusion method to fuse and analyze the instructions of gesture recognition and voice recognition to clarify the user's operation intentions; The instruction-operation conversion module is used to define the interactive operation mapping relationship between the human-computer interaction instruction and the target application system, and provide an interactive operation interface with the target application system, so that the target application system makes relevant operation responses according to the user's human-computer interaction intention, thereby realizing the user's operation intention.

3. The multimodal human-computer interaction system integrating gesture and voice according to claim 2, characterized in that: The camera is connected to the multimodal human-computer interaction system via the gesture information acquisition module of the interactive perception layer, providing a source of gesture image information for the human-computer interaction application system; the air conduction microphone is connected to the multimodal human-computer interaction system via the voice information acquisition module of the interactive perception layer, providing a source of voice information for the human-computer interaction application system; the instruction-operation conversion module of the interactive response layer is connected to the application operation software interface provided by the GIS map software, and the user's intention is converted into a corresponding operation instruction through the instruction-operation conversion module, prompting the GIS map software to make a relevant operation response.

4. A multimodal human-computer interaction method integrating gesture and voice based on the system according to any one of claims 1 to 3, characterized in that: The method comprises the following steps: Step S1, interactive information perception: through the interactive perception layer, user gesture and voice interaction information is collected via relevant sensor devices; Step S2, single-mode interaction information analysis: the interaction perception layer transmits the collected interaction information to the gesture command recognition module and the voice command recognition module of the interaction analysis layer; the interaction command definition module of the interaction analysis layer defines specific gesture and voice interaction mapping instructions; according to the interaction command definition module, the gesture command recognition module uses a gesture recognition method to recognize the instruction meaning of the user's gesture information, and the voice command recognition module uses a voice recognition method to recognize the instruction meaning of the user's voice information; Step S3, multi-modal instruction semantic judgment: Use the multi-modal instruction semantic judgment method to judge whether the instruction information of each modal channel has operational semantics. When there is instruction information of a certain modality that does not have operational semantics, directly output the instruction information with operational intention semantics and jump to step S5. Otherwise, enter step S4; Step S4, multimodal information fusion analysis: input the gesture command and the voice command into the multimodal command fusion understanding model, and the multimodal command fusion understanding model integrates the gesture interaction information and the voice interaction information, compares the command definition of the interaction command definition module, and evaluates and analyzes the user's operation intention; Step S5, instruction-operation conversion: the user's operation intention is transmitted to the instruction-operation conversion module of the interactive response layer, and the instruction-operation conversion module automatically generates an operation instruction according to the user's operation intention and the GIS map software operation API; Step S6, application system response: the operation instruction generated by the instruction-operation conversion module is transmitted to the GIS map software, and the GIS map software responds to realize a closed loop of human-computer interaction operation.

5. The multimodal human-computer interaction system integrating gesture and voice as claimed in claim 4, characterized in that: In step S2, the gesture recognition method adopted by the gesture command recognition module is divided into two parts: gesture posture estimation and gesture command recognition. The specific implementation steps are as follows: Step S211: Hand gesture estimation: Based on the OpenPose algorithm, the key points of the hand in the image are located and the coordinates of the key points of the hand are obtained; Step S212: Gesture command recognition: Based on the time-varying relationship of the coordinates of the key points of the hand, the hand action-gesture command mapping relationship in the interactive command definition module is compared to infer and recognize the command meaning of the current gesture action.

6. The multimodal human-computer interaction system integrating gesture and voice according to claim 6, characterized in that: In the hand action-gesture command mapping relationship in the interactive command definition module, dynamic gesture commands include: single-finger click, two-finger click, grasp, pinch, open, translate and rotate; static gesture commands include: "1-9" and "OK". The hand action-gesture command mapping relationship is added, modified or deleted according to actual application requirements.

7. The multimodal human-computer interaction system integrating gesture and voice according to claim 4, characterized in that: In step S2, the voice command recognition module uses the ASRFrame Chinese voice recognition framework to build a voice recognition system to realize the recognition of user voice information; The interactive command definition module defines command keywords in voice information input, including: "left", "right", "up", "down", "translate", "clockwise", "counterclockwise", "rotate", "select", "plot", "mark", "zoom in" and "zoom out".

8. The multimodal human-computer interaction system integrating gesture and voice according to any one of claims 4 to 7, characterized in that: The multimodal instruction fusion understanding model in step S4 is improved and designed based on the Transformer network structure, including four parts: information encoding, position encoding, encoder, and decoder: (1) Information Coding The input of the multimodal command fusion understanding model is Chinese commands, including gesture command text, voice command text and defined user command text. The input command is encoded using the word embedding Word2Vec method to convert the Chinese commands into low-dimensional vectors. (2) Positional encoding After information encoding, the gesture command sequence is G = [ges1, ges2, ..., ges n ], the voice command sequence is V = [Voc1, Voc2, ..., Voc m ], the user command sequence defined is T = [Tag1, Tag2…, Tag l ]; In order to keep the original order of each sequence element in the instruction text, the relative position encoding is performed by combining the sine and cosine functions provided in Transformer: Among them, pos is the position of the token in the text sequence, d model is the dimension of token; (3) Encoder The encoder includes a gesture command text input branch and a voice command text input branch; first, the gesture command information and voice command information that have been information encoded and position encoded are spliced ​​and retained as shallow features; second, the gesture command information and voice command information are subjected to multi-head attention and nonlinear feature extraction operations in their respective branches to extract key information from each modal command and obtain normalized features; finally, the normalized feature information of each modality is spliced, and the retained shallow feature information is combined to use the multi-head attention mechanism to extract the correlation between gesture command features and voice command features, thereby realizing the fusion of different modal command information; (4) Decoder The decoder is a defined user instruction text input branch, which adopts a standard Transformer decoder structure, and its input is the basic vocabulary in the defined user instruction text. Among them, the multi-head masked attention mechanism is used to ensure that the model can only use the information before the current position when the decoder generates the output instruction sequence. The multi-head attention mechanism in the decoder branch normalizes the instruction fusion feature information encoded by the encoder and uses it as K and V to participate in multi-layer decoding calculations, and uses the normalized feature information obtained from the user instruction text input branch as Q, thereby realizing the association calculation with the encoding information of the target instruction text and completing the fusion of multi-mode instruction information. Finally, a feedforward network is used to realize deep feature extraction and mapping, and a fully connected layer and a Softmax layer are used to obtain the output probability matrix of the decoder, and the corresponding user instruction vocabulary is selected according to the maximum probability to form the final output instruction.

9. The multimodal human-computer interaction system integrating gesture and voice according to claim 8, characterized in that: Normalization in the multimodal instruction fusion understanding model is implemented by layer normalization.

10. The multimodal human-computer interaction system integrating gesture and voice according to claim 8, characterized in that: The multimodal instruction fusion understanding model uses the cross entropy function as the model's loss function for model training.

Images