Emotional interaction decision-making method and system for humanoid robot

Abstract

The invention provides an emotional interaction decision-making method and system for a humanoid robot. The emotional interaction decision-making method comprises the following steps: acquiring multi-modal data, acquired by a multi-modal acquisition system, of a user interacting with the humanoid robot; preprocessing the collected multi-modal data to obtain a multi-modal emotion data set; constructing an improved multi-modal fusion network model, and training the improved multi-modal fusion network model by using the multi-modal emotion data set to obtain an emotion-behavior mapping decision model; performing sentiment analysis on the real-time input of the user through the sentiment-behavior mapping decision model, and generating a robot interaction behavior instruction or a service response strategy matched with the sentiment state of the user; the method has the following beneficial effects that the multi-modal emotion of the user can be accurately recognized, and the humanoid robot can be driven to make physical behavior feedback with temperature and emotional quotients in real time, so that the naturalness, affinity and emotion connection experience of man-machine interaction are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an emotional interaction decision-making method and system for humanoid robots. Background Technology

[0002] In today's era of rapid development in artificial intelligence and robotics, humanoid robots are gradually moving from laboratories into real-world scenarios such as homes, healthcare, education, and commercial services. Their core value lies in providing a human-like, warm, and natural interactive experience. However, most current humanoid robots remain at the level of mechanical "command-response" interaction, severely lacking the ability to understand and adapt to the user's real-time emotional state. This results in stiff interactions, a lack of empathy, and difficulty in establishing deep emotional connections. For example, if a robot continues to promote products or perform irrelevant tasks in a cheerful tone when a user is tired and has a low voice, it will greatly reduce the user experience; conversely, if a robot reacts indifferently when a user is excited, it will miss the opportunity to establish positive interaction. The user's emotional state (such as joy, frustration, anxiety, or calmness) should be the core basis for driving the robot to adjust its voice, facial expressions, actions, and service strategies, but existing systems generally neglect this crucial dimension.

[0003] Currently, emotion perception technologies in the robotics field are mostly limited to a single modality, such as judging emotions solely through voice tone or facial expressions. However, human emotion expression is highly multimodal—a simple "I'm fine" (text) may be accompanied by downcast eyes (image) and a trembling voice (audio). Single-modal analysis is easily misled or misses crucial information, resulting in poor robustness. Although some studies have attempted to introduce multimodal deep learning, the modality fusion stage often employs simple feature concatenation or weighted averaging, failing to effectively model the deep semantic relationships and dynamic complementarity between modalities, leading to low accuracy in emotion recognition.

[0004] Furthermore, existing humanoid robot systems suffer from real-time performance bottlenecks. The entire decision-making chain, from perceiving user emotions to generating and executing matching behaviors (such as switching facial expressions, adjusting speech rate, and making gestures), exhibits excessive latency, failing to meet the fluency requirements of natural dialogue. Simultaneously, there is often a disconnect between emotion analysis results and the robot's behavior execution module, lacking an end-to-end closed-loop mechanism from "emotion recognition" to "behavior mapping" and then to "physical execution," making it difficult to truly implement emotional intelligence. Therefore, there is an urgent need for an emotionally-driven interactive decision-making system specifically designed for humanoid robots, based on multimodal emotion analysis, capable of capturing user emotions accurately and in real-time, and directly driving the robot to generate matching voice, facial expressions, actions, and service responses, thereby achieving truly "emotionally intelligent" human-computer interaction. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an emotional interaction decision-making method and system for humanoid robots, which addresses the problem that most humanoid robots currently remain at the mechanical interaction level of "command-response", seriously lacking the ability to understand and adapt to the user's real-time emotional state, resulting in stiff interaction, lack of empathy, and difficulty in establishing deep emotional connections.

[0006] To achieve the above and other related objectives, the present invention provides the following technical solution:

[0007] An emotion-based interaction decision-making method for humanoid robots includes the following steps: acquiring multimodal data of users interacting with the humanoid robot collected by a multimodal acquisition system, wherein the multimodal data includes text data, image data, and audio data; preprocessing the collected multimodal data to obtain a multimodal emotion dataset based on the preprocessing results, and performing data augmentation on the multimodal emotion dataset; constructing an improved multimodal fusion network model, and training the improved multimodal fusion network model using the multimodal emotion dataset, obtaining an emotion-behavior mapping decision model based on the training results; performing emotion analysis on the user's real-time input through the emotion-behavior mapping decision model, and generating robot interaction behavior instructions or service response strategies that match the user's emotional state based on the analysis results.

[0008] An emotional interaction decision-making system for humanoid robots includes: a data acquisition module for acquiring multimodal data of users interacting with the humanoid robot collected by a multimodal acquisition system, wherein the multimodal data includes text data, image data, and audio data; a preprocessing module for preprocessing the acquired multimodal data, obtaining a multimodal emotion dataset based on the preprocessing results, and performing data augmentation on the multimodal emotion dataset; a network model training module for constructing an improved multimodal fusion network model, training the improved multimodal fusion network model using the multimodal emotion dataset, and obtaining an emotion-behavior mapping decision model based on the training results; and an emotion analysis module for performing emotion analysis on the user's real-time input through the emotion-behavior mapping decision model, and generating robot interaction behavior instructions or service response strategies that match the user's emotional state based on the analysis results.

[0009] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the emotional interaction decision-making method for humanoid robots as described above.

[0010] In one embodiment of the present invention, the multimodal acquisition system includes a user interaction interface, a multimodal sensor, and a data integration unit; wherein, the user interaction interface is used to receive text and images input by the user; the multimodal sensor is used to capture the user's facial expressions and audio signals; and the data integration unit is used to synchronously integrate data from different modalities to ensure accurate alignment of data in time and context.

[0011] In one embodiment of the present invention, the collected multimodal data is preprocessed to obtain a multimodal sentiment dataset based on the preprocessing results. This includes: cleaning the collected multimodal data, removing noise and invalid samples, and retaining data containing sentiment expressions; automatically extracting features and performing preliminary sentiment annotation on the cleaned multimodal data; and manually reviewing the automatic annotation results, using specific rules to ensure the accuracy of sentiment labels, thereby generating a high-quality multimodal sentiment dataset.

[0012] In one embodiment of the present invention, constructing the improved multimodal fusion network model includes: constructing a basic multimodal network model, wherein the basic multimodal network model includes a feature extraction module, a fusion module, and a behavior decision head; optimizing the feature extraction module by using a dynamic attention mechanism to adaptively adjust the focus of attention based on the feature importance of each modality, thereby more effectively capturing subtle differences in emotion; adding a cross-modal Transformer module to the fusion module and optimizing the fusion module by using an emotion-behavior gating fusion unit for modal integration; and combining the optimized feature extraction module and fusion module with the behavior decision head to obtain the improved multimodal fusion network model based on the combination result.

[0013] In one embodiment of the present invention, the dynamic attention mechanism adaptively adjusts the focus of attention by calculating the weight of each modality feature to better capture subtle differences in emotion. The calculation formula for the dynamic attention is as follows: ,in, For query vector, For key vectors, For value vectors, is the dimension of the key vector.

[0014] In one embodiment of the present invention, the cross-modal Transformer module extracts deep interaction information from different modalities by computing multiple attention heads in parallel, thereby enhancing the robustness of emotion representation and contextual understanding capabilities. The output of the cross-modal Transformer module is: ,in, This indicates modal feature splicing.

[0015] In one embodiment of the present invention, optimizing the fusion module by using an emotion-behavior gating fusion unit for modal integration includes: dynamically selecting modal contribution using a gating mechanism and controlling the fusion weights through a sigmoid activation function, thereby improving resistance to noisy modalities. The gating fusion formula is as follows: ;in, , , for The calculated gating weights are used not only for modality fusion but also as input parameters for subsequent behavioral decisions. They are used to quantify the influence of different emotional dimensions on the final interactive behavior. The output vector of the emotion-behavior gating fusion unit is directly input to the behavior decision head to drive the humanoid robot to perform speech tone, facial expressions, or body movements that match the dominant emotion.

[0016] In one embodiment of the present invention, during the process of training the improved multimodal fusion network model using the multimodal sentiment dataset and obtaining the sentiment-behavior mapping decision model based on the training results, a composite loss function is used to calculate the error between the predicted sentiment and the true label. The composite loss function includes sentiment classification loss, modality consistency loss, and recommendation accuracy loss; wherein, the overall loss function is as follows: , , and These represent the classification loss weight, consistency loss weight, and recommendation loss weight, respectively.

[0017] The sentiment classification loss uses the cross-entropy loss function to evaluate the prediction accuracy of multi-class sentiment labels, and the modality consistency loss uses KL divergence to measure the consistency of sentiment representation distributions across different modalities. The formula for calculating the KL divergence is as follows: , and These represent the probability distributions of the two modes, respectively.

[0018] As described above, the present invention provides an emotional interaction decision-making method and system for humanoid robots, which has the following beneficial effects: The present invention collects multimodal data of users interacting with humanoid robots and preprocesses the collected multimodal data to obtain a multimodal emotion dataset; then, it performs data augmentation on the dataset; next, it constructs an improved multimodal fusion network model and trains the model using the multimodal emotion dataset to obtain an emotion-behavior mapping decision model; finally, it performs real-time emotion analysis on user input through the emotion-behavior mapping decision model to generate robot interaction behavior instructions or service response strategies that match the emotional state. Therefore, the present invention can not only accurately identify users' multimodal emotions, but also drive humanoid robots to make warm and emotionally intelligent physical behavioral feedback (such as tone changes, facial expression management, and soothing actions) in real time, thereby significantly improving the naturalness, affinity, and emotional connection experience of human-computer interaction. Attached Figure Description

[0019] Figure 1 This is a flowchart of the emotional interaction decision-making method for humanoid robots according to the first embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of an emotional interactive decision-making system for humanoid robots according to the second embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of an electronic device according to the third embodiment of the present invention;

[0022] Figure 4 This is an example diagram of multimodal data fusion in this invention;

[0023] Figure 5 This is a structural diagram of the improved multimodal fusion network model in this invention;

[0024] Figure 6 This is a block diagram of the dynamic attention mechanism in this invention;

[0025] Figure 7 This is a flowchart of the cross-modal Transformer module in this invention;

[0026] Figure 8 This is a schematic diagram of the gating fusion unit in this invention;

[0027] Figure 9 This is a graph showing the training curve of the loss function in this invention;

[0028] Figure 10 This is a framework diagram of the humanoid robot's emotion-behavior decision-making closed loop in this invention;

[0029] Figure 11 This is an experimental comparison chart of different modalities of the sentiment analysis algorithm in this invention;

[0030] Figure 12 This is a comparison chart of the different modalities and categories of the sentiment analysis algorithm in this invention. Detailed Implementation

[0031] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0032] The first embodiment of the present invention relates to an emotional interaction decision-making method for humanoid robots, the process of which is as follows: Figure 1 As shown, the details are as follows:

[0033] Step 101: Acquire multimodal data of users interacting with the humanoid robot collected by the multimodal acquisition system.

[0034] Specifically, the multimodal acquisition system includes a user interface, multimodal sensors integrated into the humanoid robot body, and a data integration unit. The user interface is used to receive text and images input by the user. The multimodal sensors include a high-definition camera and a microphone array to capture the user's facial expressions and audio signals. The data integration unit is used to synchronously integrate data from different modalities to ensure accurate alignment of data in time and context, and to avoid timing deviations between different modalities.

[0035] More specifically, a multimodal acquisition system is constructed, comprising a user interface, multimodal sensors, and a data integration unit. The user interface receives text (such as comments) and images (such as selfies); the multimodal sensors capture audio (such as voice feedback); the data integration unit uses a timestamp synchronization algorithm (such as NTP) to ensure modal alignment; the system is integrated into mobile or web platforms, supporting real-time acquisition and avoiding the limitations of single modality. The multimodal approach combines visual expressions, voice intonation, and text semantics to improve the comprehensiveness of emotion capture. In emotion analysis, multi-view imaging and audio recording ensure timeline alignment, reduce offset bias, and support the construction of large-scale datasets, such as benchmark datasets.

[0036] Step 102: Preprocess the collected multimodal data and obtain a multimodal sentiment dataset based on the preprocessing results.

[0037] Specifically, the collected multimodal data is first cleaned to remove noise and invalid samples, while retaining data containing sentiment expressions. Then, pre-trained models such as BERT and CLIP are used to automatically extract features and perform preliminary sentiment annotation on the cleaned multimodal data. Finally, the automatic annotation results are manually reviewed, and specific rules are used to ensure the accuracy of sentiment labels (positive, neutral, negative), ensuring the high quality and consistency of the final dataset, thereby generating a high-quality multimodal sentiment dataset.

[0038] More specifically, such as Figure 4 As shown, noise and invalid samples are first removed; then BERT is used to extract text features, ResNet to extract image features, and Wav2Vec to extract audio features; preliminary labels are then generated using an automatic annotation tool and manually reviewed to form a dataset; the sentiment labels include positive, neutral, and negative; text processing includes word segmentation, stop word removal, and normalization; image cropping and contrast enhancement to highlight expressions; audio denoising, Mel spectrum conversion, and endpoint detection; BERT generates high-dimensional embeddings, ResNet captures local patterns, Wav2Vec adapts to speech variations, automatic annotation uses CLIP for preliminary classification, and manual review ensures 95% accuracy; modality alignment uses the DTW algorithm to synchronize sequences.

[0039] Step 103: Perform data augmentation on the multimodal sentiment dataset.

[0040] Specifically, it implements text synonym replacement, image flipping, and audio noise addition; adds random occlusion and perturbation to enhance robustness; synthesizes mixed-modal samples, such as audio overlaying images; uses EDA technology to randomly insert / delete words or back-translate text; performs geometric transformations and color jittering on images to simulate lighting; adds audio speed perturbation, pitch shifting, and background noise; uses TokenMixup to generate new samples from mixed-modal tokens to alleviate sparsity issues; Orthogonal Synthesis to reduce cluttered features and improve generalization; and cross-modal enhancements such as Disentanglement Translation to separate features and translate new combinations, support few-shot learning, scale datasets, and improve performance in noisy environments.

[0041] Step 104: Construct an improved multimodal fusion network model.

[0042] Specifically, firstly, a basic multimodal network model is constructed, which includes a feature extraction module, a fusion module, and a behavior decision head. Then, the feature extraction module is optimized by employing a dynamic attention mechanism to adaptively adjust the focus based on the feature importance of each modality, thereby more effectively capturing subtle differences in emotion. Next, a cross-modal Transformer module is added to the fusion module to model long-range dependencies between modalities, and the fusion module is optimized using an emotion-behavior gating fusion unit for modal integration. Finally, the optimized feature extraction module and fusion module are combined with the behavior decision head, and the improved multimodal fusion network model is obtained based on the combination result.

[0043] More specifically, the dynamic attention mechanism adaptively adjusts its focus by calculating the weights of each modality feature to better capture subtle emotional nuances. The formula for calculating dynamic attention is as follows: ,in, For query vector, For key vectors, For value vectors, The dimension of the key vector allows the model to focus on the modal features most critical to emotion recognition;

[0044] The cross-modal Transformer module extracts deep interaction information from text, image, and audio modalities by computing multiple attention heads in parallel, enhancing the robustness of sentiment representation and contextual understanding capabilities. The output of this cross-modal Transformer module is as follows: ,in, This indicates modal feature concatenation;

[0045] The fusion module is optimized by using an emotion-behavior gating fusion unit for modal integration. The specific process involves dynamically selecting modal contributions using a gating mechanism and controlling the fusion weights through a sigmoid activation function, thereby improving resistance to noisy modalities. The formula for this gating fusion is as follows: ;in, , , for The calculated gating weights are used not only for modality fusion but also as input parameters for subsequent behavioral decisions, quantifying the influence of different emotional dimensions on the final interactive behavior. The output vector of the emotion-behavior gating fusion unit is directly input to the behavior decision head to drive the humanoid robot to perform speech tone, facial expressions, or body movements that match the dominant emotion.

[0046] Furthermore, in the development of multimodal decision-making systems, the fusion of information from different modalities (such as text, images, and audio) is crucial. Therefore, this invention designs an improved multimodal fusion network model, aiming to maximize the complementarity between modalities through multi-level feature extraction, cross-modal interaction, and advanced fusion strategies. The overall process is as follows: Figure 5 As shown, the specific optimization process is as follows:

[0047] S41: Building a Traditional Model: In the first step, a traditional multimodal decision-making model was built. This model's architecture consists of three main parts: Feature Extraction Module: This module is responsible for extracting valuable features from the input multimodal data. Specifically, text features are extracted using CNN (Convolutional Neural Network), image features using image CNN, and audio features using RNN (Recurrent Neural Network). These features contain key information from the text, image, and audio data, which is helpful for subsequent fusion and analysis. Fusion Module: In the traditional model, a simple concatenation method is used to combine features from different modalities. This method is intuitive but fails to fully utilize the potential correlations between different modalities. Nevertheless, it provides a basic framework for subsequent optimization. Behavioral Decision Head: Finally, the concatenated features are processed using a Multilayer Perceptron (MLP) to generate the final recommendation result. The MLP model, through multilayer nonlinear transformations, can uncover more complex correlations between features, thereby improving the accuracy of the recommendation.

[0048] S42: Optimizing the Feature Extraction Module: With the initial completion of the model architecture, optimization of the feature extraction module begins. To this end, a dynamic attention mechanism is introduced. This mechanism dynamically adjusts the importance of each feature based on the input features. In traditional feature extraction, all features are treated equally. However, in practice, some features may be more critical in certain scenarios. The dynamic attention mechanism assigns different weights to each feature, allowing the model to focus more on key information, thereby improving the efficiency and accuracy of feature extraction. Figure 6 As shown, the specific formula for calculating attention is: ,in, It is a query vector. It is a key vector. It is a value vector. It is the dimension of the key vector. Through this mechanism, the model can effectively focus on the important parts of the input data and reduce its reliance on redundant information.

[0049] S43: Introduction of Cross-Modal Transformer: To better capture long-distance dependencies between modalities, this invention introduces a cross-modal Transformer into the model. Transformer models excel at handling sequential data, particularly in long-distance dependency modeling. Introducing the Transformer not only strengthens the interaction between modalities but also helps the model better understand the relationships between different modalities. For example, the semantic information of text and images may be closely related, while audio and text may reflect a correlation in emotional information. The cross-modal Transformer can capture these complex cross-modal relationships through its self-attention mechanism, further improving the fusion effect. Figure 7 The structure of the module is shown;

[0050] S44: Optimized Fusion Module: Next, this invention further optimizes the fusion module by employing a gated fusion mechanism to dynamically adjust the weights of each modality. For details, please refer to [link to relevant documentation]. Figure 8 Simple splicing often fails to consider redundant or irrelevant information between modes, which may lead to information mixing. By using a gating mechanism, an importance weight can be assigned to each mode, thereby enhancing the contribution of the main mode and filtering out noise and redundant information. The specific fusion formula is as follows: In embodiments of the invention applied to humanoid robots, these gating weights , , It has dual functions:

[0051] The first function, as mentioned above, is to dynamically fuse multimodal features to generate a high-quality comprehensive sentiment representation vector; the second function is to serve as a meta-parameter for behavioral decisions, guiding the robot's behavior execution module, for example, if... The image modality weight is significantly higher than other weights, indicating that the user's current emotion is mainly conveyed through facial expressions. The behavior decision head will prioritize activating the "facial expression-driven module" to generate a smiling expression that matches the predicted emotion (such as "happy"). Similarly, if If (audio modality weights) are dominant, then the intonation parameters of the speech synthesis module will be adjusted first.

[0052] S45: Combining and Obtaining an Improved Model: After optimizing the above modules, they are combined into an improved multimodal fusion network model. This model achieves significant performance improvements through the following core features: 1) Adaptive Attention Mechanism: By dynamically adjusting attention weights, the influence of noise is reduced, making the model more focused on useful information; 2) Cross-Modal Transformer: Enhances the interaction between different modalities, captures long-distance dependencies, and improves the model's comprehensive understanding ability; 3) Gated Fusion Mechanism: By dynamically weighting and fusing features from various modalities, redundant information is filtered out, ensuring the high quality of fused features; 4) Residual Transformer: This module enhances the model's attention to salient information through residual connections, enabling the model to better extract and retain key information, thereby providing more accurate recommendation results in the fusion stage. Experimental results show that the Residual Transformer significantly improves the fusion accuracy by 10-15%.

[0053] Step 105: Train the improved multimodal fusion network model using the multimodal sentiment dataset, and obtain the sentiment-behavior mapping decision model based on the training results.

[0054] Specifically, in the process of training the improved multimodal fusion network model using a multimodal sentiment dataset and obtaining the sentiment-behavior mapping decision model based on the training results, a composite loss function is used to calculate the error between the predicted sentiment and the real label. The composite loss function includes sentiment classification loss, modality consistency loss and recommendation accuracy loss.

[0055] The overall loss function is shown below: , , and These represent the classification loss weight, consistency loss weight, and recommendation loss weight, respectively.

[0056] The sentiment classification loss uses the cross-entropy loss function to evaluate the prediction accuracy of multi-class sentiment labels, while the modality consistency loss uses KL divergence to measure the consistency of sentiment representation distributions across different modalities. The KL divergence formula is as follows: , and Let represent the probability distributions of the two modalities respectively; the recommendation accuracy loss uses the inverse function of the NDCG (Normalized Discounted Cumulative Gain) metric to optimize the recommendation ranking.

[0057] More specifically, training an improved multimodal fusion network model using a multimodal sentiment dataset to obtain an emotional interaction decision model includes the following steps:

[0058] S51: The dataset is divided into training, validation, and test sets in an 8:1:1 ratio;

[0059] S52: Training uses the Adam optimizer, and the loss function is as follows: Figure 9 As shown, Where CE is the cross-entropy loss, used to evaluate the accuracy of sentiment classification. KL divergence ensures consistency across different modal representations, while MSE (mean squared error) measures the deviation between predicted and target behavioral parameters. Weights are set to =0.5, =0.3, =0.2;

[0060] S53: Verify and adjust hyperparameters, and select the best model;

[0061] S54: Test and confirm performance, use CE to evaluate classification, KL to ensure consistency, and NDCG to optimize ranking; integrate modal encoders with multi-loss fusion; learn discriminative features using multi-layer features and metric losses, use Adam combined with Cosine Annealing to prevent overfitting, train for 50-100 epochs, batch size 32-128; adjust... Weights (e.g.) =0.5), test metrics: accuracy, F1, and MAP.

[0062] Step 106: Perform sentiment analysis on the user's real-time input using the sentiment-behavior mapping decision model, and generate robot interaction behavior instructions or service response strategies that match the user's emotional state based on the analysis results.

[0063] Specifically, please refer to Figures 10 to 12 This invention supports real-time natural interaction with humanoid robots. Users can interact with the robot through natural dialogue, facial expressions, or body gestures. For example, they can say "I'm so tired today" (text), make a dejected expression (image), or make a low, slow sigh (audio). The robot's high-definition camera, microphone array, and optional tactile sensors will simultaneously collect multimodal data and ensure the precise spatiotemporal alignment of each modality of data through a data integration unit (such as a synchronization algorithm based on timestamps or event triggers).

[0064] Feature Extraction and Fusion: This invention utilizes pre-trained models for feature extraction—the BERT model parses the sentiment tendency in text semantics, the ResNet model captures facial key points and facial muscle movements, and the Wav2Vec model analyzes the fundamental frequency, speech rate, and energy changes of speech to identify intonation and emotion; subsequently, the feature vectors are fed into a cross-modal Transformer module for deep interaction and integrated through an emotion-behavior gating fusion unit; this gating unit not only generates the fused vector, but also outputs gating weights ( , , This will be used as a meta-parameter to quantify the contribution of text, visual, and auditory modalities to the current emotion, providing a basis for subsequent behavioral decisions.

[0065] Sentiment Analysis and Behavioral Decision-Making: The fused high-dimensional sentiment vector is input into the classification layer, outputting a probability distribution of the user's sentiment state (e.g., [0.1 positive, 0.2 neutral, 0.7 negative]). This probability distribution, along with the gating weights, is input into the behavior decision head, which is a multi-branch neural network or rule engine. Based on the sentiment category and dominant modality, it retrieves and generates specific interaction instructions from a pre-defined "sentiment-behavior mapping library." For example, when a "negative" sentiment is detected and the visual modality (…),… When the weight is highest: the robot will drive the facial display to generate a "concerned" or "comforting" expression; simultaneously, the speech synthesis module will adjust to a soft, slow tone, saying, "You sound like you've had a tough day. How about I play some light music for you, or tell you a joke to help you relax?"; and optionally, it will perform a slight forward lean to express concern; when a "positive" emotion is detected and the audio modality ( When the weight is highest: the robot will display a "bright smile" expression, responding in a cheerful and upbeat tone, "That's great! Sharing happiness doubles the joy!", and may perform a celebratory wave or high-five; when a "neutral / inquiry" emotion is detected: the robot will maintain a standard smiling expression, providing information or services in a clear and steady tone, such as "Okay, I'll check the weather for you right away" or "Do you need me to get you anything?";

[0066] Service Response and Execution: The generated interactive behavior commands are sent to the robot's underlying execution controller, driving its speech synthesis system, facial expression engine, and limb motion planning module, respectively. The system is designed with a response latency of less than 1 second to ensure smooth and natural interaction. Furthermore, the system supports higher-level "service response strategies." For example, if multiple interactions consistently detect a user in a "sad" state, the robot can proactively initiate a caring dialogue: "I've noticed you seem a bit down lately. Would you like me to contact your family or schedule a listener for you?" This step directly maps the emotion analysis results to physical-world interactive behaviors, enabling the humanoid robot to achieve true "reading between the lines" and "emotional resonance," thereby greatly enhancing the friendliness, naturalness, and user satisfaction of human-computer interaction.

[0067] The second embodiment of the present invention relates to an emotional interactive decision-making system for humanoid robots. Please refer to [link to relevant documentation]. Figure 2 ,include:

[0068] The data acquisition module is used to acquire multimodal data of users interacting with the humanoid robot collected by the multimodal acquisition system. The multimodal data includes text data, image data, and audio data.

[0069] The preprocessing module is used to preprocess the collected multimodal data, obtain a multimodal sentiment dataset based on the preprocessing results, and perform data augmentation on the multimodal sentiment dataset.

[0070] The network model training module is used to build an improved multimodal fusion network model and train the improved multimodal fusion network model using a multimodal sentiment dataset. Based on the training results, an sentiment-behavior mapping decision model is obtained.

[0071] The sentiment analysis module is used to perform sentiment analysis on the user's real-time input through a sentiment-behavior mapping decision model, and generate robot interaction behavior instructions or service response strategies that match the user's emotional state based on the analysis results.

[0072] It is not difficult to see that this embodiment is a system implementation corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0073] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0074] The third embodiment of the present invention relates to an electronic device; please refer to [link / reference]. Figure 3 ,include:

[0075] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described emotional interaction decision-making method for humanoid robots;

[0076] It should be noted that the electronic device also includes a multimodal sensor interface and a behavior execution controller interface; the memory is used to store the processor's executable instructions and the emotion-behavior mapping model; the multimodal sensor interface is used to receive real-time input data from the robot's camera and microphone array; the behavior execution controller interface is used to output the generated interactive behavior instructions to the robot's speech synthesis module, facial expression driving module, and limb motion control module to drive the humanoid robot to perform anthropomorphic responses that match the user's emotional state.

[0077] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0078] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0079] The fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method embodiments.

[0080] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0081] In summary, this invention collects multimodal data from users interacting with a humanoid robot and preprocesses the collected multimodal data to obtain a multimodal emotion dataset. Then, it performs data augmentation on the dataset, constructs an improved multimodal fusion network model, and trains the model using the multimodal emotion dataset to obtain an emotion-behavior mapping decision model. Finally, it performs real-time emotion analysis on user input through the emotion-behavior mapping decision model to generate robot interaction behavior instructions or service response strategies that match the emotional state. Therefore, this invention can not only accurately identify users' multimodal emotions but also drive the humanoid robot to provide warm and emotionally intelligent physical behavioral feedback (such as tone changes, facial expression management, and soothing gestures) in real time, thereby significantly improving the naturalness, affinity, and emotional connection experience of human-computer interaction.

[0082] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. All equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this invention should still be covered by the claims of this invention.

Claims

1. An emotional interaction decision-making method for humanoid robots, characterized in that, Includes the following steps: The system acquires multimodal data of users interacting with a humanoid robot, collected by a multimodal acquisition system. The multimodal data includes text data, image data, and audio data. The collected multimodal data is preprocessed, and a multimodal sentiment dataset is obtained based on the preprocessing results. The multimodal sentiment dataset is then augmented. An improved multimodal fusion network model is constructed, and the improved multimodal fusion network model is trained using the multimodal sentiment dataset. Based on the training results, an sentiment-behavior mapping decision model is obtained. The emotion-behavior mapping decision model performs emotion analysis on the user's real-time input and generates robot interaction behavior instructions or service response strategies that match the user's emotional state based on the analysis results.

2. The emotional interaction decision-making method for humanoid robots according to claim 1, characterized in that: The multimodal acquisition system includes a user interface, multimodal sensors, and a data integration unit; The user interaction interface is used to receive text and images input by the user; the multimodal sensor is used to capture the user's facial expressions and audio signals; and the data integration unit is used to synchronously integrate data from different modalities to ensure accurate alignment of data in time and context.

3. The emotional interaction decision-making method for humanoid robots according to claim 1, characterized in that: The process of preprocessing the collected multimodal data to obtain a multimodal sentiment dataset includes: The collected multimodal data is cleaned to remove noise and invalid samples, while retaining data containing sentiment expression. Automatic feature extraction and preliminary sentiment annotation are performed on the cleaned multimodal data; The automatic annotation results are manually reviewed, and specific rules are used to ensure the accuracy of sentiment labels, thereby generating a high-quality multimodal sentiment dataset.

4. The emotional interaction decision-making method for humanoid robots according to claim 1, characterized in that: The construction of the improved multimodal fusion network model includes: Construct a basic multimodal network model, wherein the basic multimodal network model includes a feature extraction module, a fusion module, and an action decision head; The feature extraction module is optimized by using a dynamic attention mechanism to adaptively adjust the focus of attention based on the feature importance of each modality, thereby more effectively capturing subtle differences in emotions. A cross-modal Transformer module is added to the fusion module, and the fusion module is optimized, using an emotion-behavior gating fusion unit for modal integration; The optimized feature extraction module and fusion module are combined with the behavior decision head, and an improved multimodal fusion network model is obtained based on the combination result.

5. The emotional interaction decision-making method for humanoid robots according to claim 4, characterized in that: The dynamic attention mechanism adaptively adjusts the focus of attention by calculating the weight of each modality feature to better capture subtle differences in emotion. The formula for calculating dynamic attention is as follows: ； in, For query vector, For key vectors, For value vectors, is the dimension of the key vector.

6. The emotional interaction decision-making method for humanoid robots according to claim 4, characterized in that: The cross-modal Transformer module extracts deep interaction information from different modalities by computing multiple attention heads in parallel, enhancing the robustness of sentiment representation and contextual understanding capabilities. The output of the cross-modal Transformer module is: ； in, This indicates modal feature splicing.

7. The emotional interaction decision-making method for humanoid robots according to claim 4, characterized in that: The optimization of the fusion module, using an emotion-behavior gating fusion unit for modal integration, includes: A gating mechanism is used to dynamically select modal contributions, and the fusion weights are controlled by a sigmoid activation function, thereby improving resistance to noisy modes. The formula for the gating fusion is as follows: ； in, , , for The calculated gating weights are used not only for modality fusion but also as input parameters for subsequent behavioral decisions. They are used to quantify the influence of different emotional dimensions on the final interactive behavior. The output vector of the emotion-behavior gating fusion unit is directly input to the behavior decision head to drive the humanoid robot to perform speech tone, facial expressions, or body movements that match the dominant emotion.

8. The emotional interaction decision-making method for humanoid robots according to claim 1, characterized in that: In the process of training the improved multimodal fusion network model using the multimodal sentiment dataset and obtaining the sentiment-behavior mapping decision model based on the training results, a composite loss function is used to calculate the error between the predicted sentiment and the true label. The composite loss function includes sentiment classification loss, modality consistency loss, and recommendation accuracy loss. The overall loss function is as follows: , , and These represent the classification loss weight, consistency loss weight, and recommendation loss weight, respectively. The sentiment classification loss uses the cross-entropy loss function to evaluate the prediction accuracy of multi-class sentiment labels, and the modality consistency loss uses KL divergence to measure the consistency of sentiment representation distributions across different modalities. The formula for calculating the KL divergence is as follows: , and These represent the probability distributions of the two modes, respectively.

9. An emotional interactive decision-making system for humanoid robots, characterized in that: include: The data acquisition module is used to acquire multimodal data of users interacting with the humanoid robot collected by the multimodal acquisition system, wherein the multimodal data includes text data, image data and audio data; The preprocessing module is used to preprocess the collected multimodal data, obtain a multimodal sentiment dataset based on the preprocessing results, and perform data augmentation on the multimodal sentiment dataset. The network model training module is used to construct an improved multimodal fusion network model, and to train the improved multimodal fusion network model using the multimodal sentiment dataset, and to obtain a sentiment-behavior mapping decision model based on the training results; The sentiment analysis module is used to perform sentiment analysis on the user's real-time input through the sentiment-behavior mapping decision model, and generate robot interaction behavior instructions or service response strategies that match the user's emotional state based on the analysis results.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the emotional interaction decision-making method for humanoid robots as described in any one of claims 1 to 8.

Images