Multimodal human-machine interaction system and method based on life support

Abstract

This application provides a multimodal human-computer interaction system and method based on life support. In the system, a multimodal information cognition and fusion module can encode user interaction information across multiple modalities using a multimodal fusion autoencoder model; a natural language understanding module can classify the encoded interaction information to determine the user's interaction intent; a multimodal dialogue management module can use preset emotion tagging optimization slot-filling strategies and a preset dialogue management model to control the dialogue flow with the user and generate response content; a backend decision-making module can use a preset behavior tree decision-making mechanism to control the system's workflow during task execution; and a user interaction module outputs response information to provide feedback to the user based on the response content and workflow. This allows for the generation of personalized responses based on different user emotions, behaviors, and other information, thus improving the user experience of human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of technology, and more specifically, to a multimodal human-computer interaction system and method based on life support. Background Technology

[0002] The aging population is becoming increasingly prominent, posing challenges to socio-economic growth, family-based elder care, and the supply of health and elderly care services. Furthermore, the rapid development of artificial intelligence and robotics has promoted the development of service robots. Among these, voice interaction provides the most convenient and human-centered interaction method between users and robots. With the development of computer and internet technologies, multimodal interaction systems are receiving increasing attention, utilizing the rich semantic information of multimodal systems to achieve more natural and intelligent human-computer interaction. Currently, multimodal interaction systems for life support scenarios still have some problems. For example, the published patent CN110209789B has a relatively simple dialogue mechanism during interaction, only responding mechanically to user input text and visual images of products, and unable to generate personalized answers based on different user emotions, behaviors, and other information. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a multimodal human-computer interaction system and method based on life support, which can improve the problem of not being able to generate personalized answers based on the user's different emotions, behaviors and other information during the human-computer interaction process.

[0004] To achieve the above technical objectives, the technical solution adopted in this application is as follows:

[0005] In a first aspect, embodiments of this application provide a life support multimodal human-computer interaction system, the system comprising: a multimodal information cognition and fusion module, a natural language understanding module, a multimodal dialogue management module, a backend decision-making module, and a user interaction module;

[0006] The multimodal information cognition and fusion module is used to encode the user's multimodal interaction information through a pre-created multimodal fusion autoencoder model to obtain the encoded interaction information.

[0007] The natural language understanding module is used to perform classification tasks on encoded interactive information to determine the user's interaction intent. The classification tasks include emotion recognition and intent recognition.

[0008] When the interaction intent indicates that the user expects to interact, the multimodal dialogue management module is used to use a preset emotion tag to optimize the slot filling strategy and a preset dialogue management model to control the dialogue process with the user and generate response content.

[0009] The backend decision module is used to control the workflow of the system in task execution by adopting a preset behavior tree decision mechanism.

[0010] The user interaction module is used to output response information for feedback to the user based on the response content and the workflow. The response information includes at least one of interactive text, interactive voice, and interactive video.

[0011] In conjunction with the first aspect, in some optional implementations, the multimodal interaction information, including text features, audio features, and visual features, are respectively represented as follows: R represents a two-dimensional array, T represents time, and d represents the data length. The multi-mode fusion autoencoder model includes the Bi-LSTM model, the TextCNN model, and the MSCNN model.

[0012] The multimodal information cognition and fusion module encodes the interaction information of the multiple modalities, including:

[0013] The Bi-LSTM model is used to extract text features from the interaction information of the multiple modalities, resulting in a processed text feature sequence x′. L , is represented as:

[0014] x′ L =f L (BiLSTM(x L )) (1)

[0015] in,

[0016] The TextCNN model is used to extract speech features from the interaction information of the multiple modalities, resulting in a processed speech feature sequence x′. A , is represented as:

[0017] x′ A =f A (TextCNN(x A (2)

[0018] in,

[0019] The MSCNN model is used to extract visual features from the interaction information of the multiple modalities, resulting in a processed visual feature sequence x′. V , is represented as:

[0020] x′ V =f V MSCNN(x) V (3)

[0021] in,

[0022] For the text feature sequence x′ L and speech feature sequence x′ A By concatenating the vectors along the same dimension, we obtain the feature vector x′. α , is represented as:

[0023]

[0024] Among them, operators Indicates a splicing operation;

[0025] For the concatenated feature vector x′ α and visual features x′ β Attention mechanism fusion is performed, in which... d α =d′ L +d′ A ,

[0026] The feature vector x′ is processed by a pre-defined attention module. α and visual features x′ β The fusion, wherein the feature vector x′ α As an index query for querying data, let and the visual feature x′ β As the index key and the data value itself;

[0027] make in, The attention module performs attention mechanism operations to obtain the encoding vector x″ output by the attention module. V , is represented as:

[0028]

[0029] in,

[0030] The text feature x′ L The speech feature x′ A and the encoded vector x″ V By concatenating them along the same dimension, we obtain the encoded vector x″, which serves as the encoded interactive information.

[0031] In conjunction with the first aspect, in some optional implementations, the natural language understanding module is used to perform a classification task of emotion recognition or intent recognition based on the encoded vector x″, through a fully connected layer and a ReLU activation function, to obtain a recognition prediction result. Represented as:

[0032]

[0033] in,

[0034] In conjunction with the first aspect, in some optional implementations, the sentiment tag optimization slot filling strategy includes:

[0035] During interaction with the user, it is in a state of waiting to obtain the user's interaction information;

[0036] When user interaction information is obtained, entity recognition, intent recognition, and emotion recognition are performed on the user interaction information through multiple preset network models. If the result of intent recognition indicates that the user has no interaction intent, the system re-enters the waiting interaction state; if the result of intent recognition indicates that the user has interaction intent, the system enters the preset slot filling state.

[0037] When entering the slot filling state, the corresponding semantic slot information is matched according to the user's identified interaction intent, and the slot information in the user's interaction information is obtained through entity recognition. The slot information is then filled into a preset semantic slot. If the slot information is empty, a preset inquiry state is entered until the slot information is not empty. If the semantic slot is filled, the current round of dialogue ends. If the slot information is not empty and the semantic slot is not filled, a preset emotion judgment state is entered until the semantic slot is filled.

[0038] When entering the query state, the query is repeated according to the default state of the semantic slot to obtain new slot information, and after obtaining the new slot information, it is determined whether the filling is complete.

[0039] When entering the emotional state judgment state, the emotional tag obtained by recognizing the user's interaction information through the natural language understanding module is acquired; when the emotional tag indicates that the user expects to continue the conversation, the inquiry state is entered; if the emotional tag indicates that the user does not expect to continue the conversation, the slot information for filling is obtained from the preset knowledge graph, and the semantic slot is filled completely according to the reverse recommendation algorithm.

[0040] Once the semantic slots are filled, the current round of dialogue ends.

[0041] In conjunction with the first aspect, in some optional implementations, the dialogue management model includes a dialogue tracking module and a dialogue strategy module;

[0042] The dialogue tracking module is used to record the user's historical messages. When the current round of dialogue with the user is completed and the slots of the semantic slots are not filled, it records the information of the slots that have been filled. The slot information is the attribute of a word or phrase that is predefined in the dialogue text.

[0043] The dialogue strategy module is used to predict the user's next interaction action based on the interaction intent obtained by the natural language understanding module, and to generate a response or search for slot information in a preset knowledge graph based on the slot filling status.

[0044] In conjunction with the first aspect, in some optional implementations, the control flow of the behavior tree decision-making mechanism includes:

[0045] The semantic slot information filled during the dialogue is converted into format cards and passed to the preset behavior tree module;

[0046] After receiving the data from the format card, the behavior tree module describes the task that the system needs to perform based on the instruction information after the semantic slots are filled, and creates a behavior tree corresponding to the task, as well as registering all nodes;

[0047] Run the behavior tree to execute the workflow corresponding to the task;

[0048] Once the task is completed, the user is informed of the task's execution result.

[0049] In conjunction with the first aspect, in some optional implementations, the user interaction module includes a voice module and a display module, wherein the voice module is used to output interactive voice in the response information, and the display module is used to output interactive video and / or interactive text in the response information.

[0050] Secondly, embodiments of this application also provide a life support multimodal human-computer interaction method, applied to the above-mentioned system, the method comprising:

[0051] By using a pre-created multimodal fusion autoencoder model, the user's multimodal interaction information is encoded to obtain the encoded interaction information;

[0052] The task of classifying encoded interactive information includes emotion recognition or intent recognition to determine the user's interaction intent.

[0053] When the interaction intent indicates that the user expects to interact, a preset emotion tag optimization slot filling strategy and a preset dialogue management model are used to control the dialogue process with the user and generate response content.

[0054] A pre-defined behavior tree decision-making mechanism is used to control the workflow of the system during task execution;

[0055] Based on the response content and the workflow, response information for feedback to the user is output, including at least one of interactive text, interactive voice, and interactive video.

[0056] The invention employing the above technical solution has the following advantages:

[0057] The technical solution provided in this application includes a multimodal information cognition and fusion module, a natural language understanding module, a multimodal dialogue management module, a backend decision-making module, and a user interaction module. The multimodal information cognition and fusion module can encode user interaction information across multiple modalities using a pre-created multimodal fusion autoencoder model to obtain encoded interaction information. The natural language understanding module can perform classification tasks on the encoded interaction information to determine the user's interaction intent; these classification tasks include emotion recognition and intent recognition. When the interaction intent indicates that the user expects to interact, the multimodal dialogue management module can use a preset emotion tag to optimize slot filling strategies and a preset dialogue management model to control the dialogue flow with the user and generate response content. The backend decision-making module can use a preset behavior tree decision-making mechanism to control the system's workflow during task execution. The user interaction module outputs response information to provide feedback to the user based on the response content and workflow; this response information includes at least one of interactive text, interactive voice, and interactive video. Thus, by combining emotion recognition, intent recognition, and the behavior tree decision-making mechanism to control the system's workflow during task execution, personalized responses can be generated based on the user's different emotions and behavioral actions, which helps improve the user experience of human-computer interaction. Attached Figure Description

[0058] This application can be further illustrated by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0059] Figure 1 One of the block diagrams of a life support-based multimodal human-computer interaction system provided in the embodiments of this application.

[0060] Figure 2 The second block diagram of a life support-based multimodal human-computer interaction system provided in the embodiments of this application.

[0061] Figure 3 This is a schematic diagram of the network structure of the multi-mode fusion autoencoder model provided in the embodiments of this application.

[0062] Figure 4 This is a schematic diagram of the attention module fusion feature provided in an embodiment of this application.

[0063] Figure 5 This is a flowchart illustrating the emotion tag optimization slot filling strategy provided in the embodiments of this application.

[0064] Figure 6 This is a schematic diagram of the behavior tree decision-making process provided in the embodiments of this application.

[0065] Icons: 10-Human-Computer Interaction System; 11-Multimodal Information Recognition and Fusion Module; 12-Natural Language Understanding Module; 13-Multimodal Dialogue Management Module; 14-Backend Decision Module; 15-User Interaction Module. Detailed Implementation

[0066] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0067] First Embodiment

[0068] Please refer to the reference. Figure 1 and Figure 2 This application provides a multimodal human-computer interaction system 10 based on life support, referred to as the human-computer interaction system 10. The human-computer interaction system 10 can be a software system formed by software modules, or a system combining software and hardware. For example, the human-computer interaction system 10 can be deployed on a robot with mobility capabilities, or on other terminal platforms, such as interactive devices in public areas of shopping malls. Electronic devices deploying the human-computer interaction system 10 are equipped with acquisition modules such as cameras, microphones, and touch screens to collect multimodal interaction information such as the user's image, voice, and input text.

[0069] In this embodiment, the human-computer interaction system 10 may include a multimodal information cognition and fusion module 11, a natural language understanding module 12, a multimodal dialogue management module 13, a backend decision-making module 14, and a user interaction module 15. The functions of each module are as follows:

[0070] The multimodal information cognition and fusion module 11 is used to encode the user's multimodal interaction information through a pre-created multimodal fusion autoencoder model to obtain the encoded interaction information;

[0071] Natural Language Understanding Module 12 is used to perform classification tasks on encoded interactive information to determine the user's interaction intent. The classification tasks include sentiment recognition, intent recognition, and entity recognition.

[0072] When the interaction intent indicates that the user expects to interact, the multimodal dialogue management module 13 is used to optimize the slot filling strategy and the dialogue management model with preset emotion tags, control the dialogue process with the user and generate response content.

[0073] Backend decision module 14 is used to control the workflow of the system in task execution by adopting a preset behavior tree decision mechanism;

[0074] The user interaction module 15 is used to output response information for feedback to the user based on the response content and workflow. The response information includes at least one of interactive text, interactive voice, and interactive video.

[0075] In this embodiment, the multi-modal interaction information includes text features, audio features, and visual features. The text features, audio features, and visual features obtained from the same user interaction information are respectively represented as follows: R represents a two-dimensional array, T represents the time of data collection, and d represents the data length. Please refer to... Figure 3 Multimodal fusion autoencoder models can include Bi-LSTM models, TextCNN models, and MSCNN models.

[0076] To ensure that each element of the input feature sequence has sufficient awareness of its neighboring elements, and to combine the different characteristics of each feature in the time domain or frequency domain, the input feature sequence is sequentially fed into the Bi-LSTM model, the TextCNN model, and the MSCNN model for feature extraction.

[0077] For example, based on the temporal continuity of text signals, RNN (Recurrent Neural Network) models can be used to extract key information such as context and n-grams within sentences. Speech signals contain less information in the temporal domain, requiring preprocessing such as AD (Analog-to-Digital), framing, windowing, and Fast Fourier Transform (FFT) to convert them into frequency domain information. Convolution is then used to capture local spectral correlations, and CNN (Convolutional Neural Network) models can be used to extract key information about vocal features, audio frequencies, and timbre. Visual features effectively represent texture, color, and contours in images; multi-scale convolution can be used to capture local image correlations and extract visual features.

[0078] Please refer to the reference. Figure 2 , Figure 3 and Figure 4 In this embodiment, the process by which the multimodal information cognition and fusion module 11 encodes interaction information from multiple modalities may include:

[0079] Using a Bi-LSTM model, text features are extracted from interactive information across multiple modalities, resulting in a processed text feature sequence x′. L , is represented as:

[0080] x′ L =f L (BiLSTM(x L )) (1)

[0081] in,

[0082] The TextCNN model is used to extract speech features from interactive information of multiple modalities, resulting in a processed speech feature sequence x′. A , is represented as:

[0083] x′ A =f A (TextCNN(x A (2)

[0084] in,

[0085] The MSCNN model is used to extract visual features from interactive information of multiple modalities, resulting in a processed visual feature sequence x′. V , is represented as:

[0086] x′ V =f V MSCNN(x)V (3)

[0087] in, T α and T β The subscripts α and β are used to distinguish data and have no other meaning. For example, in and In this context, α is for these two features x′ L and x′ A Adding / splicing along the same dimension, see formula (4) below, and In this context, β indicates that it is not necessary to adjust x″. V To splice;

[0088] Considering text feature x′ L and speech features x′ A Similarity in the time and frequency domains, and complementarity between the two modalities, for text feature sequences x′ L and speech feature sequence x′ A By concatenating the vectors along the same dimension, we obtain the feature vector x′. α , is represented as:

[0089]

[0090] Among them, operators Indicates a splicing operation;

[0091] For the concatenated feature vector x′ α and visual features x′ β Attention mechanism fusion is performed, in which...

[0092] The feature vector x′ is processed through a pre-defined attention module. α and visual features x′ β The fusion, where the feature vector x′ α As an index query for querying data, let And visual feature x′ β As the index key and the data value itself;

[0093] make in, The attention mechanism is applied through the attention module to obtain the encoding vector x″ output by the attention module. V , encoding vector x″ V It can better characterize the impact of local features on global features, and is expressed as:

[0094]

[0095] in,

[0096] Text features x′ L Speech features x′ A and encoding vector x″ V By concatenating them along the same dimension, we obtain the encoded vector x″, which serves as the encoded interactive information.

[0097] In this embodiment, the natural language understanding module 12 is used to perform a classification task of emotion recognition or intent recognition based on the encoding vector x″ through a fully connected layer and the ReLU activation function, so as to obtain the recognition prediction result. Represented as:

[0098]

[0099] in,

[0100] In this embodiment, the multimodal fusion autoencoder model can analyze the characteristics of different features through convolutional neural networks, recurrent neural networks, and attention mechanisms. This model can obtain information about specific features from the original signal and obtain a weighted fused feature vector based on attention-based fusion. This model is an autoencoder model and can be used as a pre-trained model, trained on different samples for various downstream tasks such as intent recognition or sentiment analysis.

[0101] Please refer to Figure 5 In this embodiment, the emotion tag optimization slot filling strategy includes:

[0102] During interaction with the user, it is in a state of waiting to obtain the user's interaction information;

[0103] When user interaction information is obtained, entity recognition, intent recognition and emotion recognition are performed on the user interaction information through multiple preset network models. If the intent recognition result indicates that the user has no interaction intent, the system will re-enter the waiting interaction state; if the intent recognition result indicates that the user has interaction intent, the system will enter the preset slot filling state.

[0104] When entering the slot filling state, the corresponding semantic slot information is matched according to the user's interaction intent identified, and the slot information in the user's interaction information is obtained through entity recognition, and the slot information is filled into the preset semantic slot; if the slot information is empty, the preset inquiry state is entered until the slot information is not empty; if the semantic slot is filled, the current round of dialogue ends; if the slot information is not empty and the semantic slot is not filled, the preset emotion judgment state is entered until the semantic slot is filled.

[0105] When entering the query state, the query is repeated according to the default state of the semantic slot to obtain new slot information, and after obtaining the new slot information, it is determined whether the filling is complete.

[0106] When entering the emotion judgment state, the emotion tag obtained by recognizing the user's interaction information through the natural language understanding module 12 is obtained; if the emotion tag indicates that the user expects to continue the conversation, the inquiry state is entered; if the emotion tag indicates that the user does not expect to continue the conversation, the slot information for filling is obtained from the preset knowledge graph, and the complete semantic slot is filled according to the reverse recommendation algorithm.

[0107] Once the semantic slots are filled, the current round of dialogue ends.

[0108] In this embodiment, the dialogue management model includes a dialogue tracking module and a dialogue policy module;

[0109] The dialogue tracking module is used to record the user's historical messages. When the current dialogue with the user is completed and the semantic slots are not filled, it records the information of the slots that have been filled. The slot information is the attributes of words or phrases that are predefined in the dialogue text.

[0110] The dialogue strategy module is used to predict the user's next interaction action based on the interaction intent obtained by the natural language understanding module 12, and to generate a response or search for slot information in a preset knowledge graph based on the slot filling status.

[0111] Please refer to this again. Figure 5 In this embodiment, for multimodal human-computer interaction scenarios, the multimodal dialogue management module 13 adopts an emotion tag optimization slot filling strategy, and combines a dialogue management model that integrates knowledge graph and semantic slot filling, which can control the dialogue process and generate responses.

[0112] The dialogue management model can consist of a dialogue tracking module and a dialogue strategy module. The dialogue tracking module provides dialogue tracking functionality, recording the user's historical messages. When a dialogue round is completed but slots are not yet filled, it can record information about already filled slots. The dialogue strategy module can predict the next action based on user interaction information obtained from the natural language understanding module 12, and can also directly generate responses based on slot filling status, or search for slot information in the knowledge graph based on sentiment tags, among other operations.

[0113] In this embodiment, the emotion tag optimization slot filling strategy divides the interaction process into 6 states: waiting for interaction, natural language processing, slot filling, asking questions, judging emotions, and ending the dialogue.

[0114] Waiting for interaction: When a user interacts with the system, the first state is waiting to obtain interaction information, which is the waiting for interaction state.

[0115] Natural Language Processing (NLP): This includes entity recognition, intent recognition, and sentiment analysis. After acquiring user interaction information, the system uses different network models to perform entity recognition, intent recognition, and sentiment analysis on the user's interaction information. For example, it uses the TextCNN model for intent recognition, the Bi-LSTM-CRF model for entity recognition, and the multi-modal fusion autoencoder model provided in this application's embodiments for sentiment recognition. Entity recognition can be understood as recognizing objects in images, or objects, locations, people, etc., in text. Intent recognition can include different types of intents, such as the intent to seek help (e.g., getting water for the user) or the intent to chat. If intent recognition fails (e.g., the user has no clear interaction intent), the system re-enters the waiting interaction state; if intent recognition succeeds, it indicates that the user expects human-computer interaction, and the system enters the slot filling state.

[0116] Slot filling: This involves matching the identified user intent with corresponding semantic slot information. Different intents can be matched with different semantic slot information. Slot information from the interaction information obtained through entity recognition is filled into the semantic slots. If a slot is empty, a query state is entered until the slot is not empty. If the semantic slot is filled, the current round of dialogue ends. If the slot is not empty but not filled, the sentiment state is judged until the slot is filled. A semantic slot consists of multiple slots, and the information filled in each slot is called slot information, which can be understood as keywords in a sentence.

[0117] Inquiry: The human-computer interaction system 10 continuously queries according to the default state of the semantic slot to obtain new slot information. After obtaining the new slot information, it then determines whether the slot filling is complete.

[0118] Sentiment Determination: A multi-modal fusion autoencoder model is used to obtain sentiment tags from user interaction information, optimizing the slot filling strategy. Sentiment tags can include labels such as happy and angry. When the sentiment tag is positive / Pos (e.g., happy), it indicates that the user wants to continue the conversation, and the system enters a query state; conversely, (e.g., the sentiment tag is angry), it indicates that the user does not want to continue the conversation. The slot filling information needs to be obtained from the knowledge graph. The knowledge graph is searched based on existing information, and the complete semantic slots are filled according to a reverse-order recommendation algorithm.

[0119] End Dialogue: Once the slot is filled, this dialogue session ends.

[0120] In this embodiment, through semantic slot filling, the human-computer interaction system 10 can retain valid user interaction information and discard invalid information during the interaction process, making the user interaction process unrestricted by the information filling order and closer to a natural interaction state. Optimizing slot filling through sentiment tags can further enhance the system's intelligence and improve the user experience. When the user doesn't want to talk or speaks less, the number of dialogue rounds is reduced, and the machine can still understand the user's needs. Combining semantic information of common household items stored in a knowledge graph makes it more adaptable to human-computer interaction behavior in constantly changing interactive scenarios, enabling natural and flexible human-computer interaction.

[0121] Please refer to Figure 6 In this embodiment, the control flow of the behavior tree decision-making mechanism includes:

[0122] The semantic slot information filled during the dialogue is converted into format cards and passed to the preset behavior tree module;

[0123] After receiving the data from the format card, the behavior tree module describes the task that the system needs to perform based on the instruction information after the semantic slots are filled, and creates a behavior tree corresponding to the task, as well as registering all nodes;

[0124] Run the behavior tree to perform the workflow corresponding to the task. For example, in the scenario of the robot "get me a bottle of water", when the robot's human-computer interaction system 10 recognizes that the user intends to get water and the location is clearly in the kitchen, it registers the water-getting behavior tree, runs the behavior tree nodes, and the robot then performs the workflow according to nodes such as obstacle avoidance navigation, target search, and grasping operation.

[0125] Once the task is completed, the user is informed of the task's execution result, thus concluding the multimodal human-computer interaction process corresponding to that task.

[0126] In this embodiment, a behavior tree mechanism is used to describe the workflow of the backend control / decision robot during task execution.

[0127] The role of backend decision module 14 in the human-computer interaction process includes:

[0128] 1) Control the workflow. Based on the characteristics of robot motion control, the robot's work execution is divided into multiple states. By combining the interface information of multiple modules, the behavior tree is used to control the transition of action states, ensuring that the execution process is reasonable and natural, and improving the robot's autonomy and intelligence.

[0129] 2) Program state management: When a robot performs complex tasks, there are complex logical states. By dividing the entire workflow into execution states and modular components, it is easier to understand and maintain the workflow.

[0130] In this embodiment, during human-computer interaction, user expression information such as voice, gestures, and facial expressions is first collected through devices such as microphones and cameras. Then, the computer processes the received multimodal information, using a multimodal information fusion model to identify user interaction information such as expressed intent and emotion, matching semantic slots with interaction information, and filling slots using an emotion-labeled optimized slot-filling strategy. The multimodal dialogue management module 13 controls the dialogue flow and generates machine response information, obtaining the interacted information for backend task execution. For robot task execution, a behavior tree mechanism is used for control, determining the robot's workflow during task execution. After task execution, the system outputs the interaction information between the system and the user, including dialogue interaction via voice output and application interaction via touchscreen.

[0131] The multimodal human-computer interaction framework constructed in this invention addresses the problem of low efficiency in multimodal fusion by employing a multimodal fusion autoencoder model, optimizes slot-filling strategies through emotion tagging to enhance dialogue flexibility and user experience, and improves robot autonomy during human-computer interaction through a behavior tree decision-making mechanism, thus promoting multimodal intelligent human-computer interaction.

[0132] In this embodiment, the user interaction module 15 includes a voice module and a display module. The voice module is used to output interactive voice in the response information, and the display module is used to output interactive video and / or interactive text in the response information.

[0133] Based on the above design, the multimodal fusion autoencoder model combines Bi-LSTM, TextCNN, and MSCNN encoder networks with an attention mechanism, and can be used as a pre-trained model for various tasks. By optimizing the slot-filling strategy using sentiment tags, personalized responses can be generated based on different user emotions, improving the system's intelligence and user experience. A behavior tree mechanism is employed to control the robot's workflow during task execution, ensuring that the robot executes the correct state based on logical decisions when performing complex tasks, thus realizing the entire workflow. This improves the accuracy of user interaction intent recognition, enhances dialogue flexibility, increases the robot's autonomy in human-computer interaction, and promotes multimodal human-computer intelligent interaction.

[0134] Second Embodiment

[0135] This application also provides a multimodal human-computer interaction method based on life support, referred to as the human-computer interaction method. This human-computer interaction method can be applied to the human-computer interaction system 10 described in the first embodiment. The human-computer interaction method may include the following steps:

[0136] Step 110: Encode the user's multimodal interaction information using a pre-created multimodal fusion autoencoder model to obtain encoded interaction information;

[0137] Step 120: Perform a classification task on the encoded interactive information, the classification task including emotion recognition or intent recognition, to determine the user's interaction intent;

[0138] Step 130: When the interaction intent indicates that the user expects to interact, a preset emotion tag optimization slot filling strategy and a preset dialogue management model are used to control the dialogue process with the user and generate response content.

[0139] Step 140: Use a pre-set behavior tree decision-making mechanism to control the system's workflow during task execution;

[0140] Step 150: Based on the response content and the workflow, output response information for feedback to the user, the response information including at least one of interactive text, interactive voice, and interactive video.

[0141] It should be noted that, for the sake of convenience and brevity, the specific working process of the human-computer interaction method described above can be referred to the execution flow of each module in the aforementioned human-computer interaction system, and will not be elaborated further here.

[0142] Based on the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various implementation scenarios of this application.

[0143] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0144] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multi-modal human-computer interaction system based on life support, characterized in that, The system includes: a multimodal information cognition and fusion module, a natural language understanding module, a multimodal dialogue management module, a backend decision-making module, and a user interaction module; The multimodal information cognition and fusion module is used to encode the user's multimodal interaction information through a pre-created multimodal fusion autoencoder model to obtain the encoded interaction information. The natural language understanding module is used to perform classification tasks on encoded interactive information to determine the user's interaction intent. The classification tasks include emotion recognition and intent recognition. When the interaction intent indicates that the user expects to interact, the multimodal dialogue management module is used to use a preset emotion tag to optimize the slot filling strategy and a preset dialogue management model to control the dialogue process with the user and generate response content. The backend decision module is used to control the workflow of the system in task execution by adopting a preset behavior tree decision mechanism. The user interaction module is used to output response information for feedback to the user based on the response content and the workflow. The response information includes at least one of interactive text, interactive voice, and interactive video. The sentiment tag optimization slot filling strategy includes: During interaction with the user, it is in a state of waiting to obtain the user's interaction information; When user interaction information is obtained, entity recognition, intent recognition, and emotion recognition are performed on the user interaction information through multiple preset network models. If the result of intent recognition indicates that the user has no interaction intent, the system re-enters the waiting interaction state; if the result of intent recognition indicates that the user has interaction intent, the system enters the preset slot filling state. When entering the slot filling state, the corresponding semantic slot information is matched according to the user's identified interaction intent, and the slot information in the user's interaction information is obtained through entity recognition. The slot information is then filled into a preset semantic slot. If the slot information is empty, a preset inquiry state is entered until the slot information is not empty. If the semantic slot is filled, the current round of dialogue ends. If the slot information is not empty and the semantic slot is not filled, a preset emotion judgment state is entered until the semantic slot is filled. When entering the query state, the query is repeated according to the default state of the semantic slot to obtain new slot information, and after obtaining the new slot information, it is determined whether the filling is complete. When entering the emotional state judgment state, the emotional tag obtained by recognizing the user's interaction information through the natural language understanding module is acquired; when the emotional tag indicates that the user expects to continue the conversation, the inquiry state is entered; if the emotional tag indicates that the user does not expect to continue the conversation, the slot information for filling is obtained from the preset knowledge graph, and the semantic slot is filled completely according to the reverse recommendation algorithm. Once the semantic slots are filled, the current round of dialogue ends.

2. The system according to claim 1, characterized in that, The multimodal interaction information includes text features, audio features, and visual features, which are respectively represented as follows: , , R represents a two-dimensional array, T represents time, and d represents the data length. The multi-modal fusion autoencoder model includes the Bi-LSTM model, the TextCNN model, and the MSCNN model. The multimodal information cognition and fusion module encodes the interaction information of the multiple modalities, including: The Bi-LSTM model is used to extract text features from the interaction information of the multiple modalities, resulting in a processed text feature sequence. , is represented as: （1）； wherein ; The TextCNN model is used to extract speech features from the interaction information of the multiple modalities, resulting in a processed speech feature sequence. , is represented as: （2）； in, ; The MSCNN model is used to extract visual features from the interaction information of the multiple modalities, resulting in a processed visual feature sequence. , is represented as: （3）； wherein , footnotes and for distinguishing data, no meaning; concatenate the text feature sequence and the speech feature sequence in the same dimension to obtain a feature vector , denoted as: （4）； wherein the operator denotes a concatenation operation; For the concatenated feature vector and visual features Attention mechanism fusion is performed, in which... , , , ; The feature vector is processed through a pre-defined attention module. and visual features The fusion, wherein the feature vector As an index query for querying data, let and the visual features As the index key and the data value itself; make , ,in, , , The attention module performs attention mechanism operations to obtain the encoding vector output by the attention module. , represented as: （5）； wherein ; The text features The speech features and the encoding vector Concatenating along the same dimension yields the encoded vector. As encoded interactive information.

3. The system according to claim 2, characterized in that, The natural language understanding module is used to perform the encoding vector By using fully connected layers and the ReLU activation function, a classification task of emotion recognition or intent recognition is performed to obtain the recognition prediction results. , represented as: （6）； wherein .

4. The system of claim 1, wherein, The dialogue management model includes a dialogue tracking module and a dialogue strategy module; The dialogue tracking module is used to record the user's historical messages. When the current dialogue with the user is completed and the slots of the semantic slots are not filled, it records the information of the slots that have been filled. The slot information is the attribute of a word or phrase that is predefined in the dialogue text. The dialogue strategy module is used to predict the user's next interaction action based on the interaction intent obtained by the natural language understanding module, and to generate a response or search for slot information in a preset knowledge graph based on the slot filling status.

5. The system of claim 1, wherein, The control process of the behavior tree decision-making mechanism includes: The semantic slot information filled during the dialogue is converted into format cards and passed to the preset behavior tree module; After receiving the data from the format card, the behavior tree module describes the task that the system needs to perform based on the instruction information after the semantic slots are filled, and creates a behavior tree corresponding to the task, as well as registering all nodes; Run the behavior tree to execute the workflow corresponding to the task; Once the task is completed, the user is informed of the task's execution result.

6. The system of claim 1, wherein, The user interaction module includes a voice module and a display module. The voice module is used to output the interactive voice in the response information, and the display module is used to output the interactive video and / or interactive text in the response information.

7. A multi-modal human-computer interaction method based on life support, characterized by, Applied to the system according to any one of claims 1-6, the method comprises: By using a pre-created multimodal fusion autoencoder model, the user's multimodal interaction information is encoded to obtain the encoded interaction information; The task of classifying encoded interactive information includes emotion recognition or intent recognition to determine the user's interaction intent. When the interaction intent indicates that the user expects to interact, a preset emotion tag optimization slot filling strategy and a preset dialogue management model are used to control the dialogue process with the user and generate response content. A pre-defined behavior tree decision-making mechanism is used to control the workflow of the system during task execution; Based on the response content and the workflow, response information for feedback to the user is output, including at least one of interactive text, interactive voice, and interactive video.

Images