Intention understanding method and system of multi-modal fusion human-computer interaction

Abstract

The application discloses a multimodal fusion human-computer interaction intention understanding method and system. The implementation steps are as follows: a microphone captures voice and then converts the voice into text; a camera captures a frame sequence; the text and the frame sequence are judged by a judgment gate to determine whether the input is empty; if the content is text, the text is subjected to a feature extraction module to obtain text features; if the content is text and the frame sequence, the frame sequence is subjected to a gating module to judge a gesture type; if the gesture type is a dynamic gesture, the frame sequence is subjected to a network to extract features, and the text features and the features are subjected to a feature fusion module to be fused; if the gesture type is a static gesture, the static gesture is subjected to a feature extraction module to extract features, and the text features and the features are subjected to a fusion network to be fused; if the input content is a dynamic gesture, the frame sequence is subjected to a time-space feature network; and finally, intention classification is performed. The application forms a complete multimodal fusion network, which is an important process for a robot to understand intention, and can improve the intention understanding accuracy.

Description

Technical Field

[0001] This application relates to the field of multimodal fusion image and text processing technology, specifically to a multimodal human-computer interaction intent understanding method and system. Background Technology

[0002] In the context of elderly care applications, acquiring and recognizing information from single modalities such as voice, gestures, and body language has become mainstream to enable robots to better understand the needs of the elderly. However, when elderly people express their intentions, differences in hand shape between the elderly and younger people often lead to inaccurate gestures, resulting in unclear instructions. Furthermore, the inability to accurately recognize dialects can also hinder the robot's understanding of the elderly's intentions, leading to ambiguous commands. Multimodal fusion can aggregate data from multiple sources, making the model's learned representations more complete. This effectively compensates for the shortcomings of single-modal data in human-computer interaction, providing accurate data support for subsequent situational awareness and understanding of interaction intentions.

[0003] There are many methods based on statistical learning to solve the intention understanding problem, such as Bayesian methods, random forests, and hidden Markov models. MI et al. used Bayesian probability to recover the fuzzy feature representation of the target, successfully solving the intention understanding problem with incomplete information. However, the Bayesian algorithm is suitable for small samples; if the number of intentions is increased, misjudgments are prone to occur. Chen et al. used fuzzy C-means clustering to divide high-dimensional emotional features into different subclasses and used multiple two-layer random forests to identify different emotional states. Random forests consume a lot of space and time during training, and their performance is poor under noisy input conditions. The above statistical probability-based intention recognition methods are highly interpretable, but the initial probability selection of the probabilistic model has certain errors and subjectivity.

[0004] With the development of technology, machine learning has demonstrated absolute advantages in processing modal data such as speech and vision, providing new solutions for intent recognition. Techniques such as convolutional neural networks and recurrent neural networks are gradually being used in intent classification research. Currently, multimodal fusion intent extraction methods can be categorized into three types based on their fusion methods: data-level fusion, feature-level fusion, and decision-level fusion.

[0005] Data fusion first fuses information from all sensor observations. In most cases, data fusion depends only on the sensor type, not on user requirements. Chen et al. proposed a bidirectional representation that can generate novel descriptions from images and visual representations from descriptions. The key to both tasks is a novel representation that dynamically captures the visual aspects of a described scene. That is, as words are generated or read, the visual representation is updated to reflect the new information contained in the words. Data fusion, being the lowest level of information fusion, is less robust to interference, impacting accuracy.

[0006] Feature layer fusion effectively improves the recognition rate in fields such as object detection and image recognition. This method extracts features from multiple modalities and maps them into a high-dimensional feature vector as input information. Through model training and learning, it classifies the user's final behavioral intent. Its advantage is that it can analyze the relevant features of different modalities to improve the accuracy of intent understanding. However, it suffers from high time complexity in finding the most important features. Bai et al. proposed an improved single-shot multibox detector (SSD) that uses deep feature fusion between the object detection layer and its adjacent feature layers, which effectively improves object detection accuracy. However, feature layer data fusion cannot well explain causal relationships at the cognitive level; in addition, adjusting and training network parameters due to changes in the number of intents is also a time-consuming and labor-intensive process. Therefore, many researchers are currently focusing on research on decision layer fusion.

[0007] The advantage of the decision-level fusion approach is that each information processing process is independent, without needing to consider factors such as the data format and feature quantity of each modality. It also does not require the collection of a large amount of data for annotation, making it easier to build an intent understanding model. Furthermore, the information of each channel is processed separately, which improves the efficiency and accuracy of intent extraction.

[0008] The methods described above highlight the influence of each modality on the target during the fusion process, improving the intent understanding rate to some extent. However, the weight setting during fusion is somewhat subjective. Therefore, more objective information reflected by the data itself should be considered in the modality fusion process. Traditional statistical intent recognition relies on the selection of initial probabilities, which is subjective. Neural network intent recognition has poor anti-interference capabilities at the data level and may introduce noise. Feature-level fusion is fast but has low accuracy and is complex. Summary of the Invention

[0009] The purpose of this invention is to address the human-computer interaction intent understanding scenario in elderly care by proposing a multimodal fusion method. This method solves the problem of single-modal absence or two modalities jointly determining a task in different environments, providing support for intent understanding in human-computer interaction. The approach to implementing this invention is as follows: First, input information is captured via microphone and camera. The information undergoes data preprocessing to determine if the input data is empty. If the frame sequence is not empty, a gating module classifies it into static or dynamic gestures. If not empty, it is processed according to four input modes: When only text is input, the text features are extracted using a Text Encoder for prediction. When only dynamic gestures are input, the data frame is processed through a CRNN network to obtain spatial and temporal features for prediction. When static gestures and text are input, the text features are extracted using a Text Encoder, and the static gesture features are extracted using a DCN and attention pooling layer. The two features are then processed separately through a Transformer Encoder with shared parameters to obtain fused features. The two features are then concatenated by channel, and the concatenated features are fused and predicted using a multimodal fusion module. Input dynamic gestures and text. Text features are extracted using a Text Encoder, and gesture features are extracted using a CRNN. ​​These two features are then passed through a Transformer Encoder with shared parameters to obtain fused features. The two features are then concatenated by channel separation, and the concatenated features are fused and predicted using a multimodal fusion module. Finally, the predicted instructions are obtained, and tasks are assigned and executed.

[0010] The specific steps of this invention are as follows:

[0011] Step 1: Label the text, static gesture, and dynamic gesture datasets.

[0012] Categorize text and images;

[0013] Write all possible texts to form a JSON file containing only the text and its corresponding instructions;

[0014] Write a JSON file that combines text and static gestures to generate commands;

[0015] Write a JSON file that combines text and dynamic gestures to generate commands;

[0016] The image dataset was divided into a 70% training set, a 10% validation set, and a 20% test set.

[0017] Step 2, preprocess the text:

[0018] The text is processed using the bert-base-chinese model, and text vectors are obtained by corresponding to the dictionary.

[0019] Use a tokenizer to encode the text.

[0020] Step 3, preprocess the image:

[0021] Firstly, only the first and last frames of data are taken as keyframes.

[0022] Then, the images in each batch are preprocessed by random cropping and random flipping, and the images are adjusted to a size of 224*224.

[0023] Step 4: Read the corresponding text and commands from the image:

[0024] Create a dictionary mapping tags to indexes and load the corresponding JSON file;

[0025] Organize the mapping of gesture categories to labels, and handle the case where multiple articles correspond to the same label but have different commands;

[0026] Load the image paths for all gesture categories, and randomly select a text and label from the JSON file.

[0027] Step 5: Determine if the text or data frame is empty by checking the gate.

[0028] The gate module determines whether the input text or data frame contains data.

[0029] The result can be either text only, data frames only, or both text and data frames contain data.

[0030] Step 6: Determine the gesture type via the gating module.

[0031] The gating module consists of an RNN network and a linear layer;

[0032] If data exists in the data frame, the data is input into the gating module. The input data passes through the RNN network and outputs the hidden layer feature hn. hn can well reflect the temporal change state between the first and last frames of the data frame.

[0033] The binary classification score is obtained, and 0.5 is used as the threshold. If the score is greater than 0.5, it is considered a static gesture, and if the score is less than 0.5, it is considered a dynamic gesture.

[0034] By training the gating module, its optimal parameters are obtained and saved locally for later use. When training the complete multimodal model, the data frame is directly tested on the model and the gesture type is returned.

[0035] Step 7: Select different paths based on different inputs to train the model for single-modal recognition or multi-modal fusion:

[0036] Different circuits are selected based on the judgment of the door and the door control module;

[0037] If there is only text, the Transformer Text Encoder is used to extract text features, and the output command is obtained through the MLP layer and executed.

[0038] For dynamic gestures only, the CRNN network captures spatial and temporal features, inputs the feature vectors into the MLP layer to obtain the output command, and executes the corresponding command.

[0039] The dataset includes text and static gestures. Both text and images are preprocessed, with only one frame of the image retained. Text features (f1) are extracted using a Transformer Text Encoder, while edge features are extracted from the static gesture image using a DCN network. Attention pooling is then applied, fixing the size to the length of the text vector to obtain feature vector f2. These two feature vectors are then passed through a Transformer Encoder with shared parameters, fusing features from both modalities and avoiding reliance on a single modality. The two output features are then concatenated channel by channel. The concatenated features are then passed through a ResNet layer to obtain feature vector f3. To preserve original information, the extracted text and image features are combined using a bidirectional attention mechanism to obtain text-to-image attention features (f4) and image-to-text attention features (f5). f3 and f4 are then concatenated channel by channel, comprehensively utilizing the relationships between multiple modalities while preserving unique details and interdependent contextual information in each modality. The concatenated feature f6 = [f3, f4] is then fused across channels using a 1x1 convolution. Then, f6 and f5 are concatenated to further compress and enhance information, resulting in feature f7. f7 is then passed through a self-attention layer to capture long-range cross-channel dependencies, enhancing the global interaction of features. Finally, the obtained features are input into an MLP for final classification.

[0040] Similar to the above model, the text and dynamic gesture model preprocesses the text and the concatenated frame sequence. The text is processed by a Transformer Encoder to obtain text features, and the frame sequence is processed by a CRNN network to obtain spatiotemporal features. The two feature vectors are concatenated by passing them through a Transformer Encoder with shared parameters, and then classified by a multimodal fusion module. Attached Figure Description

[0041] Figure 1 This is a flowchart based on multimodal fusion human-computer interaction intent understanding.

[0042] Figure 2 This is a structural diagram of the module of the present invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings.

[0044] See attached document Figure 1 The steps of the present invention will be described in further detail below.

[0045] Step 1: Label and preprocess the dataset.

[0046] Write JSON files containing text, dynamic gestures, text and static gestures, and text and dynamic gestures. The content of the JSON file should include the gesture category (whether it is a static or dynamic gesture), gesture labels (e.g., one, two, three), text content, and instructions.

[0047] The image dataset was divided into training, validation, and test sets in a ratio of 7:1:2.

[0048] The text is processed using the bert-base-chinese model, and text vectors are obtained from the corresponding dictionary. The text is then encoded using a tokenizer.

[0049] The frame sequence first extracts keyframes, taking only the first and last frames as keyframes. Then, it performs preprocessing such as random cropping and random flipping on the images in each batch, and adjusts the images to a size of 224*224.

[0050] Step 2: Read the corresponding text and commands from the image:

[0051] Create a dictionary mapping labels to indexes, and load the JSON file corresponding to the image and text. The file should contain Type, gesture_label, text, and order.

[0052] Organize the mapping from gesture categories to gesture_labels to handle multiple cases where the same label corresponds to different commands in this article;

[0053] Load the image paths for all gesture categories, and randomly select a text and label from the text field of the corresponding JSON file for training.

[0054] Step 3: Determine if the text or data frame is empty by checking the gate:

[0055] The gate module determines whether the input text or data frame contains data.

[0056] The result can be either text only, data frames only, or both text and data frames contain data.

[0057] Step 4: Determine the gesture type via the gating module:

[0058] The gating module consists of an RNN network and a linear layer;

[0059] If data exists in the data frame, the data is input into the gating module. The input data passes through the RNN network and outputs the hidden layer feature hn. hn can well reflect the temporal change state between the first and last frames of the data frame.

[0060] The binary classification score is obtained, and 0.5 is used as the threshold. If the score is greater than 0.5, it is considered a static gesture, and if the score is less than 0.5, it is considered a dynamic gesture.

[0061] By training the gating module, its optimal parameters are obtained and saved locally for later use. When training the complete multimodal model, the data frame is directly tested on the model and the gesture type is returned.

[0062] Step 5: Select different paths based on different inputs to train the model for single-modal recognition or multi-modal fusion:

[0063] Different circuits are selected based on the judgment of the door and the door control module;

[0064] If there is only text, the Transformer Text Encoder is used to extract text features, and the output command is obtained through the MLP layer and executed.

[0065] For dynamic gestures only, the CRNN network captures spatial and temporal features, inputs the feature vectors into the MLP layer to obtain the output command, and executes the corresponding command.

[0066] Includes text and static gestures:

[0067] The first step involves preprocessing both text and images. Only one frame of the image is retained. The text features f1 are extracted using Transformer TextEncoder, and the static gesture image has its edge features extracted using DCN network. Attention pooling is then performed, and the size is fixed to the length of the text vector to obtain the feature vector f2.

[0068] The second step involves passing the two feature vectors through a Transformer Encoder with shared parameters. This fuses the features from both modalities, avoiding reliance on a single modality. The two output features are then concatenated by channel separation to obtain f. concat1 .

[0069] f1′=TransformerEncoder(f1)

[0070] f2′=TransformerEncoder(f2)

[0071] f concat1 =[f1′,f2′]

[0072] The third step is to pass the combined features through a ResNet network to obtain the feature vector f3. In order not to lose the original information, the extracted text features and image features are passed through a bidirectional attention mechanism to obtain the text-to-image attention feature f4 and the image-to-text attention feature f5.

[0073] f3=ResNet(f concat1 )

[0074] f4 = Attention(f1, f2, f2)

[0075] f5 = Attention(f2, f1, f1)

[0076] The fourth step is to concatenate the channels of f3 and f4, which can make comprehensive use of the relationships between the multimodalities while preserving the unique details and interdependent contextual information of each modality.

[0077] f6=[f3,f4]

[0078] Fifth, the concatenated features f6=[f3, f4] are fused across channels using a 1x1 convolution. Then, f6 and f5 are concatenated to further compress and enhance the information, resulting in feature f7.

[0079] f6′=Conv1x1(f6)

[0080] f7=[f6′,f5]

[0081] The sixth step is to pass f7 through a self-attention mechanism layer to capture long-distance cross-channel dependencies and enhance the global interaction of features.

[0082] f8 = SelfAttention(f7)

[0083] The seventh step is to input the obtained features into the MLP for final classification;

[0084] y^=MLP(f8)

[0085] Similar to the above model, the text and dynamic gesture model preprocesses the text and the concatenated frame sequence. The text is then processed by a transformer encoder to obtain text features, and the frame sequence is processed by a CRNN network to obtain spatiotemporal features. The two feature vectors are then concatenated by passing them through a Transformer Encoder with shared parameters, and finally classified by a multimodal fusion module.

[0086] A complete model of multimodal fusion robot intent understanding, such as Figure 2 As shown, the input consists of text and frame sequences, and the output is the final command for classification after unimodal or multimodal fusion. The robot executes this task after obtaining human-computer interaction commands by collecting gestures and speech.

[0087] The software environment for this invention was selected on a server running Ubuntu 20.04.4, with a GeForce RTX 3090 Ti GPU and 24GB of video memory. Python 3.9 and Pytroch-gpu 1.12.0 were used.

[0088] Similar or identical parts between the various embodiments in this application specification can be referred to mutually. In particular, for the method embodiments, since the methods therein are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to the description in the method embodiments. This application has only been described in detail with reference to preferred embodiments. Those skilled in the art should understand that any changes, modifications, additions, or substitutions made by those skilled in the art within the scope of this application do not depart from the spirit of this application and should also fall within the protection scope of the claims of this application.

Claims

1. A multimodal fusion method for understanding human-computer interaction intent, characterized in that, The method includes: Data preprocessing, and inputting the preprocessed data into the decision gate; If the frame sequence content is not empty, input the frame sequence into the gating module to determine the gesture type; Dynamic gestures are extracted using a feature extraction network to capture the temporal and spatial information of dynamic gestures. Static gestures have features extracted using an edge feature extraction network; Text features are extracted using Transformer Text Encoder; The obtained features are fused using a multimodal fusion module, and the intent is output via an MLP; specifically including: If the input is determined to consist only of dynamic gestures, the obtained spatiotemporal features are passed to the MLP for intent prediction. If the input consists only of text, the obtained text features are passed to the MLP for intent prediction. If the input contains dynamic gestures and text, the gesture features and text features are respectively passed through a Transformer Encoder with shared parameters to perform multimodal semantic alignment, thereby enhancing the interaction between the two modalities. Then, the output features are concat channels and the concatenated features are input into a multimodal fusion network for prediction. The input is determined to contain static gestures and text. Text features (f1) are extracted using a Transformer Text Encoder, and edge features of the static gestures are extracted using a DCN network. These are then combined using attention pooling to obtain feature f2. Text features f1 and f2 are fused using a Transformer Encoder with shared parameters, and then channel concatenation is performed. The concatenated features are processed using a ResNet network to obtain feature f3. Text features f1 and f2 are then processed using a bidirectional cross-attention mechanism to obtain text-to-image attention features f4 and image-to-text attention features f5. Features f3 and f4 are concatenated using a channel concatenation mechanism to obtain feature f6. Feature f6 is then fused across channels using a 1x1 convolution and concatenated with feature f5 using a channel concatenation mechanism to obtain feature f7. Feature f7 is processed through a self-attention mechanism layer and then input into an MLP for classification. Finally, the predicted instructions are passed to the robot to execute the corresponding task.

2. The method according to claim 1, characterized in that, The data preprocessing and inputting the preprocessed data into the decision gate includes: Voice input is acquired via microphone and converted into text via API. The depth video of the gesture is obtained through a binocular camera and then converted into a frame sequence. The captured text and frame sequence are input into the decision gate to determine whether the input content is empty; There are three possibilities for input: only text, only frame sequence, or a combination of frame sequence and text.

3. The method according to claim 1, characterized in that, If the frame sequence content is not empty, the frame sequence is input to the gating module to determine the gesture type, including: If the input is a frame sequence, the frame sequence is passed through a gating module composed of RNN and linear layers, and the feature is determined by extracting time information from the hidden layer of RNN; The result is either a dynamic gesture or a static gesture, and the gesture sequence and gesture type are passed to the corresponding network.

4. The method according to claim 1, characterized in that, The dynamic gestures are extracted using a feature extraction network to obtain their temporal and spatial information, including: If the input is a frame sequence and is recognized as a dynamic gesture by the gating module, the dynamic gesture is input into the CRNN network. The CNN extracts spatial features and the RNN extracts temporal features to obtain the spatiotemporal features of the dynamic gesture.

5. The method according to claim 1, characterized in that, The static gesture extracts features through an edge feature extraction network, including: If the input contains a frame sequence and is identified as a static gesture by the gating module, the static gesture is input into the DCN network, and the DCN network extracts edge features. Edge features are fed into the attention pooling layer, which converts the features into a fixed-length feature vector.

6. The method according to claim 1, characterized in that, The text features are extracted using Transformer Text Encoder, including: If the input is text, the text will be preprocessed using Bert-base-chinese to convert it into a vector. The resulting vectors are used to extract text features using Transformer Text Encoder.

Images