Multi-modal fusion man-machine interaction intention understanding method and system
Through the multimodal fusion human-computer interaction intention understanding method, combined with gated modules and multiple network models, the problem of insufficient single-modal data is solved, the communication accuracy and efficiency between the elderly and the robot is improved, and the communication accuracy and efficiency are enhanced. Anti-interference ability achieves a more accurate understanding of intentions.
Patent Information
- Application Number
- CN202411269806.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-11
AI Technical Summary
In the prior art In human-computer interaction, insufficient or inaccurate single-modal data leads to difficulties in understanding intentions. Especially in the communication between the elderly and robots, there are misjudgment and noise problems in gesture and speech recognition, and multimodal fusion. The method has insufficient interference resistance and accuracy.
Using multimodal fusion human-computer interaction intention understanding method, capture information through microphone and camera, use gated module to determine gesture type, and combine networks such as Text Encoder, CRNN, DCN and Transformer Encoder to perform multimodal fusion of data and features , predict and task allocation through multimodal fusion modules to improve the accuracy and efficiency of intention understanding.
It effectively solves the problem of insufficient single-modal data, improves the accuracy and efficiency of human-computer interaction intention understanding, reduces the impact of noise, and enhances the interpretability and anti-interference ability of information on different modes.
Smart Images

Figure CN119068554A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimodal fusion image and text processing, and specifically to a multimodal human-computer interaction intention understanding method and system. Background Art
[0002] In the context of elderly care, in order for robots to better understand the needs of the elderly, obtaining information and recognition in single modalities such as voice, gestures, and body posture has become the mainstream. However, when the elderly express their intentions, they cannot make standard gestures because of the differences between their palms and those of young people, resulting in inaccurate gesture commands, or the dialect cannot be accurately recognized, causing the robot to be unable to correctly understand the elderly's intentions, resulting in fuzzy commands and other problems. Multimodal fusion can aggregate multi-source data information to make the representation learned by the model more complete, effectively make up for the shortcomings of single modal data in human-computer interaction, and provide accurate data support for subsequent situational awareness and understanding of interactive intentions.
[0003] There are many methods to solve the problem of intent understanding based on statistical learning, such as Bayes, random forest, hidden Markov, etc. MI et al. used Bayesian probability to restore the fuzzy feature expression of the target and successfully solved the intent understanding of incomplete information. However, the Bayesian algorithm is suitable for smaller samples. If the number of intents is increased, misjudgment is prone to occur. Chen et al. used the fuzzy C-means clustering algorithm to divide high-dimensional emotional features into different subclasses, and used multiple two-layer random forests to identify different emotional states. The space and time cost of random forests during training is relatively large, and the effect is not good under the condition of relatively noisy input. The above intention recognition methods based on statistical probability are highly interpretable, but the initial probability selection of the probability model has certain errors and subjectivity.
[0004] With the development of science and technology, machine learning has shown absolute advantages in processing modal data such as speech and vision, providing a new solution for intent recognition. Technologies such as convolutional neural networks and recursive neural networks are gradually being used in intent classification research. At present, the methods of multimodal fusion intent extraction are divided into three categories according to the fusion method: data layer fusion, feature layer fusion and decision layer fusion.
[0005] Data layer fusion first fuses the information observed by all sensors. In most cases, data fusion depends only on the sensor type and not on user needs. Chen et al. proposed a bidirectional representation that can both generate novel descriptions from images and generate visual representations from descriptions. The key to these two tasks is a novel representation that dynamically captures the visual aspects of the scene that has been described. That is, as words are generated or read, the visual representation is updated to reflect the new information contained in the words. Data fusion is the lowest level of information fusion, which has poor anti-interference performance and affects accuracy.
[0006] Feature layer fusion effectively improves the recognition rate in the fields of target detection and image recognition. This method can extract features from multiple modalities and map them into a high-dimensional feature vector as input information through an algorithm. Through model training and learning, the user's final behavioral intention can be classified. Its advantage is that it can analyze the relevant features of different modal information to improve the accuracy of intent understanding, but there is a problem of high time complexity in the process of finding the most important features. Bai et al. proposed an improved single shot multibox detector (SSD), which uses deep feature fusion between the target detection layer and its adjacent feature layer. This method effectively improves the accuracy of target detection. However, the fusion of feature layer data cannot well explain the causal relationship at the cognitive level; in addition, the adjustment and training of network parameters caused by the change in the number of intentions is also a time-consuming and labor-intensive process. Therefore, many researchers currently focus on the research of decision layer fusion.
[0007] The advantage of the decision-layer fusion method is that each information processing process is independent, and there is no need to consider factors such as the data format of each modality and the number of features. It also does not require the collection of a large amount of data for annotation. It is easier to build an intent understanding model, and each channel information is processed separately, which improves the efficiency and accuracy of intent extraction.
[0008] The above research methods highlight the influence of each modality on the target during the fusion process, and improve the intention understanding rate to a certain extent, but the setting of weights in the fusion process is somewhat subjective. Therefore, more objective information that can be reflected by the data itself should be considered in the modal fusion process. Traditional statistical intent recognition relies on the selection of initial probability and is subjective. Neural network intent recognition has poor anti-interference ability at the data level and may introduce noise. Fusion at the feature level is fast, but the accuracy is not high and the process is complicated. Summary of the invention
[0009] The purpose of the present invention is to propose a multimodal fusion human-computer interaction intention understanding method for assisting the elderly and accompanying the elderly, solve the problem of single mode missing or two modes jointly determining a task in different environments, and provide support for the intention understanding of human-computer interaction. The idea of implementing the present invention is: first, the input information is captured by a microphone and a camera, the information is preprocessed, and the input data is judged whether it is empty. When the frame sequence is not empty, it is divided into static gestures or dynamic gestures by a gating module. If it is not empty, it is divided into four input modes for processing: when only text is input, the text is extracted through the Text Encoder for feature prediction. When only dynamic gestures are input, the data frame is passed through the CRNN network to obtain spatial and temporal features for prediction. Static gestures and text are input, and text features are extracted through the Text Encoder. Static gestures extract gesture features through the DCN and attention pooling layer. The two features are respectively passed through the Transformer Encoder with shared parameters to obtain fused features, and then the two features are spliced through channels. The spliced features are fused and predicted through the multimodal fusion module. Dynamic gestures and text are input, and the text features are extracted through the Text Encoder, and the dynamic gesture features are extracted through the CRNN. The two features are respectively passed through the Transformer Encoder with shared parameters to obtain the fused features, and then the two features are spliced by channels. The spliced features are fused and predicted through the multimodal fusion module. Finally, the predicted instructions are obtained and tasks are assigned and executed.
[0010] The specific steps of the present invention are as follows:
[0011] Step 1: Annotate text, static gestures, and dynamic gestures datasets.
[0012] Categorize text and images;
[0013] Write all possible texts to form a json file containing only texts and corresponding instructions;
[0014] Write a json file that combines text and static gestures to get commands;
[0015] Write a json file that combines text and dynamic gestures to get commands;
[0016] The image dataset is divided into 70% training set, 10% validation set and 20% test set.
[0017] Step 2: Preprocess the text:
[0018] The text is processed by the bert-base-chinese model, and the text vector is obtained from the corresponding dictionary;
[0019] Use a tokenizer to encode the text.
[0020] Step 3: Preprocess the image:
[0021] First, only the first and last frames of data are taken as key frames;
[0022] Then, the images in each batch are preprocessed by random cropping, random flipping, etc., and the images are resized to 224*224.
[0023] Step 4: Read the corresponding text and commands through the image:
[0024] Create a mapping dictionary from labels to indices and load the corresponding json file;
[0025] Organize the mapping from gesture categories to labels, and handle the situation where multiple texts correspond to the same label but different commands;
[0026] Load the image paths of all gesture categories and randomly select a text and label in the json file.
[0027] Step 5: Use the judgment gate to determine whether the text or data frame is empty:
[0028] Through the judgment gate module, it is judged whether the input text or data frame has data;
[0029] The result is either only text, only data frame, or both text and data frame.
[0030] Step 6: Determine the gesture type through the gate control module:
[0031] The gating module consists of an RNN network and a linear layer;
[0032] If there is data in the data frame, the data is input into the gating module. The input data passes through the RNN network and outputs the hidden layer feature hn. hn can well reflect the temporal change state between the first frame and the last frame of the data frame.
[0033] Get the binary classification score, take 0.5 as the threshold, if the score is greater than 0.5, it is considered a static gesture, and if the score is less than 0.5, it is considered a dynamic gesture;
[0034] By training the gating module, its optimal parameters are obtained and saved locally for calling. When training the complete multimodal model, the data frame is directly tested on the model and the gesture type is returned.
[0035] Step 7: Select different paths according to different inputs to perform single-modal recognition or multi-modal fusion training models:
[0036] Select different branches according to the judgment gate and gate control module;
[0037] If there is only text, the Transformer Text Encoder is used to extract text features, and the output command is obtained through the MLP layer and executed;
[0038] Only dynamic gestures use the CRNN network to capture spatial and temporal features, input the feature vector into the MLP layer to obtain the output command, and execute the corresponding command;
[0039] There are text and static gestures. Both text and pictures are preprocessed. Only one frame of the picture is retained. The text is extracted through the Transformer Text Encoder to extract text features f1, and the static gesture picture is extracted through the DCN network to extract edge features. Then attention pooling is performed, and the size is fixed to the vector length of the text to obtain the feature vector f2. Then the two feature vectors are passed through the Transformer Encoder with shared parameters, so that the features of the two modalities can be integrated to avoid dependence on a single modality. Then the two output features are spliced by channel. Then the spliced features are passed through a layer of ResNet network to obtain the feature vector f3. In order not to lose the original information, the extracted text features and image features are passed through the bidirectional cross attention mechanism Bidirectional Attention Mechanism to obtain the text-to-image attention features f4 and the image-to-text attention features f5. Then f3 and f4 are spliced by concat channels, which can comprehensively utilize the relationship between multiple modalities while retaining the unique details and interdependent contextual information in each modality. The spliced features f6=[f3, f4] are cross-channel information fused through 1x1 convolution. Then concatenate f6 and f5 to further compress and enhance information to obtain feature f7. Then pass f7 through the self-attention mechanism layer to capture long-distance cross-channel dependencies and enhance the global interaction of features. Finally, the obtained features are input into MLP for final classification.
[0040] The model with text and dynamic gestures is similar to the above model. After preprocessing the text and the spliced frame sequence, the text is passed through the Transformer Encoder to obtain text features, and the frame sequence is passed through the CRNN network to obtain spatiotemporal features. The two feature vectors are respectively passed through the Transformer Encoder with shared parameters and then concat channel spliced, and then classified through the multimodal fusion module. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of human-computer interaction intention understanding based on multimodal fusion.
[0042] Figure 2 It is a structural diagram of the module of the present invention. DETAILED DESCRIPTION
[0043] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0044] Refer to the attached Figure 1 , the steps of the present invention are further described in detail.
[0045] Step 1: Label and preprocess the dataset.
[0046] Write json files of text, dynamic gestures, text and static gestures, and text and dynamic gestures. The content of the json file should include the gesture category, whether it is a static gesture or a dynamic gesture; gesture labels, such as one, two, three, etc.; text content and instructions.
[0047] The image dataset is divided into training set, validation set and test set in a ratio of 7:1:2;
[0048] The text is processed by the bert-base-chinese model, the corresponding dictionary obtains the text vector, and the text is encoded using the tokenizer;
[0049] For the frame sequence, the key frames are first extracted, and only the first and last frames are taken as key frames. Then, the images in each batch are preprocessed by random cropping, random flipping, etc., and the images are resized to 224*224.
[0050] Step 2: Read the corresponding text and commands through the image:
[0051] Create a mapping dictionary from labels to indexes, and load the json file corresponding to the image and text. The file should contain Type, gesture_label, text, and order.
[0052] Organize the mapping from gesture category to gesture_label to handle the situation where multiple documents have the same label but different commands;
[0053] Load the image paths of all gesture categories and randomly select a text and label in the text in the corresponding json file for training.
[0054] Step 3: Use the judgment gate to determine whether the text or data frame is empty:
[0055] Through the judgment gate module, it is judged whether the input text or data frame has data;
[0056] The result is either only text, only data frame, or both text and data frame.
[0057] Step 4: Determine the gesture type through the gate control module:
[0058] The gating module consists of an RNN network and a linear layer;
[0059] If there is data in the data frame, the data is input into the gating module. The input data passes through the RNN network and outputs the hidden layer feature hn. hn can well reflect the temporal change state between the first frame and the last frame of the data frame.
[0060] Get the binary classification score, take 0.5 as the threshold, if the score is greater than 0.5, it is considered a static gesture, and if the score is less than 0.5, it is considered a dynamic gesture;
[0061] By training the gating module, its optimal parameters are obtained and saved locally for calling. When training the complete multimodal model, the data frame is directly tested on the model and the gesture type is returned.
[0062] Step 5: Select different paths according to different inputs to perform single-modal recognition or multi-modal fusion training models:
[0063] Select different branches according to the judgment gate and gate control module;
[0064] If there is only text, the Transformer Text Encoder is used to extract text features, and the output command is obtained through the MLP layer and executed;
[0065] Only dynamic gestures use the CRNN network to capture spatial and temporal features, input the feature vector into the MLP layer to obtain the output command, and execute the corresponding command;
[0066] With text and static gestures:
[0067] In the first step, both the text and the picture are preprocessed. Only one frame of the picture is retained. The text feature f1 is extracted from the text through Transformer TextEncoder, and the edge feature is extracted from the static gesture picture through the DCN network. Then, attention pooling is performed and the size is fixed to the vector length of the text to obtain the feature vector f2.
[0068] In the second step, the two feature vectors are passed through the Transformer Encoder with shared parameters, which can fuse the features of the two modes and avoid relying on a single mode. Then the two output features are spliced into channels to obtain f concat1 .
[0069] f1′=TransformerEncoder(f1)
[0070] f2′=TransformerEncoder(f2)
[0071] f concat1 =[f1′,f2′]
[0072] In the third step, the combined features are passed through a layer of ResNet network to obtain the feature vector f3. In order not to lose the original information, the extracted text features and image features are passed through the bidirectional cross attention mechanism to obtain the text-to-image attention features f4 and the image-to-text attention features f5.
[0073] f3=ResNet(f concat1 )
[0074] f4=Attention(f1,f2,f2)
[0075] f5=Attention(f2,f1,f1)
[0076] In the fourth step, f3 and f4 are concatenated through the concat channel, which can comprehensively utilize the relationship between multiple modalities while retaining the unique details and interdependent contextual information in each modality.
[0077] f6=[f3,f4]
[0078] Step 5: The concatenated feature f6 = [f3, f4] is fused across channels through 1x1 convolution. Then f6 and f5 are concat-channeled to further compress and enhance the information to obtain feature f7.
[0079] f6′=Conv1x1(f6)
[0080] f7=[f6′,f5]
[0081] In the sixth step, f7 is then passed through a self-attention mechanism layer to capture long-range cross-channel dependencies and enhance the global interaction of features.
[0082] f8=SelfAttention(f7)
[0083] In the seventh step, the obtained features are finally input into MLP for final classification;
[0084] y^=MLP(f8)
[0085] The model with text and dynamic gestures is similar to the above model. After preprocessing the text and the spliced frame sequence, the text is passed through the transformer encoder to obtain text features, and the frame sequence is passed through the CRNN network to obtain spatiotemporal features. The two feature vectors are respectively passed through the Transformer Encoder with shared parameters and then concat channel spliced, and then classified through the multimodal fusion module.
[0086] The complete multimodal fusion robot intention understanding model is as follows Figure 2 As shown in the figure, the input is text and frame sequence, and the output is the final command after single-mode or multi-modal fusion and classification. The robot performs the task after collecting gestures and voice to get the command of human-computer interaction.
[0087] The software environment of the present invention is selected on a server with Ubuntu 20.04.4 system, a GeForceRTX3090 Ti GPU, and 24G video memory. Python 3.9 and pytroch-gpu 1.12.0 are used.
[0088] The same and similar parts between the various embodiments in the specification of this application can be referred to each other. In particular, for the method embodiment, since the method therein is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment. This application is only described in detail in conjunction with and with reference to the preferred implementation methods. It should be understood by ordinary technicians in this field that the changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the substantive scope of this application do not deviate from the purpose of this application and should also fall within the scope of protection of the claims of this application.
Claims
1. A multimodal fusion human-computer interaction intention understanding method, characterized in that: The method comprises: Preprocess the data and input the preprocessed data into the judgment gate; If the frame sequence content is not empty, the frame sequence is input into the gate control module to determine the gesture type; Dynamic gestures extract temporal and spatial information of dynamic gestures through feature extraction network; Static gestures are characterized by edge feature extraction network; The text features are extracted through Transformer Text Encoder; The obtained features are fused through the multimodal fusion module, and the intent is output through MLP.
2. The method according to claim 1, characterized in that The data preprocessing and inputting the preprocessed data into the judgment gate include: Get voice input through the microphone and convert the voice into text through the API; Acquire the depth video of the gesture through the binocular camera and convert the depth video into a frame sequence; Input the captured text and frame sequence into the judgment gate to determine whether the input content is empty; There are three possibilities for input: text only, frame sequence only, and frame sequence and text.
3. The method according to claim 2, characterized in that If the frame sequence content is not empty, the frame sequence is input into the gate control module to determine the gesture type, including: If the input contains a frame sequence, the frame sequence is passed through a gating module consisting of an RNN and a linear layer, and the feature is judged by extracting the temporal information from the hidden layer of the RNN; The result may be a dynamic gesture or a static gesture, and the gesture sequence and gesture type are passed to the corresponding network.
4. The method according to claim 3, characterized in that The dynamic gesture extracts temporal and spatial information of the dynamic gesture through a feature extraction network, including: If the input contains a frame sequence and is recognized as a dynamic gesture by the gating module, the dynamic gesture is input into the CRNN network, the spatial features are extracted by CNN, and the temporal features are extracted by RNN to obtain the spatiotemporal features of the dynamic gesture.
5. The method according to claim 4, characterized in that The static gesture extracts features through an edge feature extraction network, including: If the input contains a frame sequence and is recognized as a static gesture by the gating module, the static gesture is input into the DCN network, and the DCN network extracts edge features; The edge features are passed into the attention pooling layer to convert the features into a fixed-length feature vector.
6. The method according to claim 5, characterized in that The text is extracted through Transformer Text Encoder, including: If the input is text, the text is preprocessed by Bert-base-chinese and converted into a vector; The obtained vector is used to extract text features through Transformer Text Encoder.
7. The method according to claim 6, characterized in that The obtained features are fused through a multimodal fusion module, and the intent is output through MLP, including: It is determined that the input is only dynamic gestures, and the obtained spatiotemporal features are passed to MLP for intention prediction; It is determined that the input is only text, and the obtained text features are passed to MLP for intent prediction; Determine whether the input contains dynamic gestures and text, pass the gesture features and text features through the Transformer Encoder with shared parameters, perform multimodal semantic alignment, enhance the interaction between the two modalities, then perform concat channel splicing on the output features, and input the spliced features into the multimodal fusion network for prediction; Determine whether the input contains static gestures and text, pass the obtained static gesture features and text through the Transformer Encoder with shared parameters respectively, then perform concat channel splicing on the output features, and input the spliced features into the multimodal fusion network for prediction; Finally, the predicted instructions are passed to the robot to execute the corresponding tasks.
Citation Information
Patent Citations
Robot man-machine interaction method and device based on gesture and voice recognition
CN105468145A
Gesture recognition method for robot control
CN110046558A
Gesture recognition system fusing bioelectrical impedance information and myoelectricity information
CN111553307A
Video emotion content analysis method based on multi-modal fusion and Transform network
CN116453022A
Information processing method and device, electronic equipment and computer storage medium
CN116468525A
Cited By
Medical data processing-based depression assessment system
CN120236744A
AI multi-technology fusion intelligent dialogue model construction method and system for old people
CN120317381A
AI voice interaction-based pension service scheduling method and system
CN120748370A
