A robot interaction understanding and behavior adjustment method based on a multi-modal emotion layer

Abstract

The application provides a robot interaction understanding and behavior adjustment method based on a multi-modal emotion layer, belongs to the technical field of embodied intelligent robot control, and comprises the following steps: a cognitive layer extracts multi-modal environment perception data input, attention weighted fusion and semantic scene graph construction, a unified cognitive state is generated in combination with historical memory, and state transition prediction and multi-step generative task planning are performed through a world model; an emotion layer extracts multi-modal interaction observation data, adopts a dynamic weighted fusion mechanism to perform robust multi-modal emotion and intention joint modeling, combines context perception and time sequence state update to output an emotion modulation signal; the cognitive layer uses a task stage and an execution result to perform context constraint and feedback correction on emotion judgment, the emotion layer uses an emotion state to dynamically adjust the feature dimension importance and decision weight of cognitive representation through a gating mechanism, and adjusts a robot control instruction. The environment cognition and user state are closed-loop cooperated, and the continuity of task planning and the naturalness of behavior response of the robot in complex embodied interaction are improved.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligent robot control technology, specifically relating to a method for robot interaction understanding and behavior regulation based on a multimodal emotional layer. Background Technology

[0002] Embossed intelligent robots need to interact with humans naturally and efficiently in complex and dynamic real-world environments, and complete long-term, multi-stage physical operation tasks. Such systems not only require robots to have the ability to perceive and understand multimodal information such as vision, depth, language, and proprioception, but also to dynamically adjust their behavior according to the user's emotions, intentions, and interaction context, thereby achieving safe, reliable, and socially intelligent human-robot collaboration.

[0003] Existing systems typically treat user emotion recognition as an independent emotion classification task, outputting only discrete emotion labels (such as happy, angry, and sad) without relating these emotional states to the current task stage, interaction history, and robot execution results. This leads to the robot's inability to understand user emotions and the task-related reasons behind them. Furthermore, the collected environmental data is processed separately and then fed into the decision-making module through simple feature concatenation or post-processing. When faced with long-term, partially observable, or dynamically changing tasks, the robot is prone to losing task context, resulting in repetitive planning, reduced efficiency in robot behavior adjustment, and decreased naturalness of interaction and user trust. Summary of the Invention

[0004] To address the shortcomings of existing technologies in understanding and regulating the interaction of embodied intelligent robots, this invention provides a method for understanding and regulating the interaction of robots based on a multimodal emotion layer.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for robot interaction understanding and behavior regulation based on a multimodal emotion layer includes the following steps: The system acquires multimodal environmental perception data, including environmental RGB images, environmental 3D spatial structure information, and robot body state; encodes the multimodal environmental perception data separately, and uses attention-weighted fusion of the multimodal encoding results to obtain a multimodal fusion representation; constructs a semantic scene graph based on the multimodal environmental perception data; maps the multimodal fusion representation and semantic scene graph to a latent space to obtain a cognitive state; acquires the user's target instruction, and combines historical memory variables and the cognitive state to obtain a task context representation; Acquire multimodal interaction observation data including visual, speech, and language; extract visual facial expression and posture features, speech prosodic features, and language semantic sentiment features respectively; perform weighted fusion of the facial expression and posture features, prosodic features, and semantic sentiment features to obtain multimodal fusion features; perform nonlinear mapping on the multimodal fusion features to generate a potential emotional representation; jointly encode the potential emotional representation with the task context representation to obtain an emotional state; map the emotional state into a modulation signal, and output the emotional state and the modulation signal. The cognitive state and the emotional state are fused to obtain a cognitive-emotional synergistic state; the robot is then controlled and constrained based on the cognitive-emotional synergistic state and the modulated signal.

[0006] Preferably, the memory state variable of the previous moment and the cognitive state are input into the memory update function to obtain the historical memory variable; the memory state variable of the previous moment is obtained by compressing and encoding the historical observation scene features, completed sub-tasks, execution errors and environmental change information stored inside the robot.

[0007] Preferably, the construction of the semantic scene graph specifically involves: identifying the categories of all objects in the scene based on the collected multimodal environmental perception data using a visual encoder, and calculating the three-dimensional position and physical size of the objects using depth information to form a set of nodes in the graph; combining the category attributes of the objects to infer the interaction relationships to form a set of edges in the graph; and using the target object and action intent extracted from the language instructions by the language encoder as prior constraints to obtain the semantic scene graph; wherein, the nodes of the semantic scene graph represent entities in the environment, and the edges of the semantic scene graph represent the semantic relationships between the nodes.

[0008] Preferably, a dynamic weighted fusion mechanism is used to weight and fuse the facial expression and posture features, prosodic features, and semantic sentiment features to obtain multimodal fusion features. Specifically, the facial expression and posture features, prosodic features, and semantic sentiment features are concatenated and input into a lightweight neural network. The network output is normalized by the Softmax function to obtain a weight distribution with a sum of 1. The confidence of each modality feature is evaluated, and weights are assigned based on the confidence of different modalities. The facial expression and posture features, prosodic features, and semantic sentiment features are weighted and summed based on the weights corresponding to the modalities to obtain the final multimodal fusion features.

[0009] Preferably, the method further includes generating a gating weight vector with the same dimension as the cognitive features based on the emotional state, using the gating weight vector to gating the cognitive state, and fusing the modulated cognitive state with the emotional state to input the input world model to drive the next round of state prediction and task planning.

[0010] This invention also provides a robot interaction understanding and behavior regulation system based on a multimodal emotion layer, specifically including: The cognitive processing module is used to acquire multimodal environmental perception data, including environmental RGB images, environmental 3D spatial structure information, and robot body state; encode the multimodal environmental perception data separately, and use attention-weighted fusion of the multimodal encoding results to obtain a multimodal fusion representation; construct a semantic scene graph based on the multimodal environmental perception data; map the multimodal fusion representation and semantic scene graph to the latent space to obtain the cognitive state; acquire the user's target instruction, and combine historical memory variables and the cognitive state to obtain a task context representation.

[0011] The emotion processing module is used to acquire multimodal interactive observation data including visual, speech, and language; extract visual facial expression and posture features, speech prosodic features, and language semantic sentiment features respectively; perform weighted fusion of the facial expression and posture features, prosodic features, and semantic sentiment features to obtain multimodal fusion features; perform nonlinear mapping on the multimodal fusion features to generate a potential emotional representation; jointly encode the potential emotional representation with the task context representation to obtain an emotional state; map the emotional state into a modulation signal, and output the emotional state and the modulation signal.

[0012] The behavior regulation module is used to fuse the cognitive state and the emotional state to obtain a cognitive-emotional coordinated state; and to control and constrain the robot based on the cognitive-emotional coordinated state and the modulation signal.

[0013] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute steps in a method for understanding and regulating robot interaction based on a multimodal emotion layer.

[0014] The robot interaction understanding and behavior regulation method based on a multimodal emotion layer provided by this invention has the following beneficial effects: This invention integrates attention-weighted fusion of RGB, 3D spatial structure information, and ontological state, mapping them to a unified latent space. Within this unified latent space, semantic scene graph reasoning captures the correlation information of various modal environmental data, avoiding the insufficient information interaction between features caused by traditional simple concatenation. Multimodal fusion features extracted from vision, speech, and language are used to obtain emotional states through joint task context representation, ensuring coherent emotional judgments that conform to real-world physical situations. Cognitive-emotional collaborative state constraint control, obtained by fusing cognitive and emotional states, avoids rigid judgments based solely on the actual environment that lose emotional depth, ensuring decisions align with both objective environmental logic and user subjective expectations. This enables the robot to understand both the task and the user, resulting in more natural, consistent, and reliable responses in complex human-computer interaction scenarios. Attached Figure Description

[0015] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a diagram of the mathematical model architecture of the cognitive layer in an embodiment of the present invention.

[0017] Figure 2 This is a diagram of the mathematical model architecture for the emotion layer in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of information interaction between cognitive and emotional channels in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0020] Example This invention provides a method for robot interaction understanding and behavior regulation based on a multimodal emotion layer, specifically as follows: Figure 1 As shown, it includes the following steps: Step 1: Acquire four types of input data: environmental RGB images, depth observations, language commands, and robot body states. The data is time-aligned and then preprocessed. RGB images undergo scale normalization, distortion correction, and illumination enhancement. Depth maps are processed with hole filling, outlier filtering, and depth range truncation to reduce errors caused by reflections, occlusion, and sensor noise. Language commands are segmented, extracted, and parsed using task slots, transforming them into a structured description of "target object, action type, and constraints." Robot state vectors are standardized to map pose, joint angles, velocity, and gripper states to a uniform numerical range.

[0021] Step 2: At time t, the cognitive layer receives four types of input: RGB image, depth observation, language commands, and robot body state. The RGB image is primarily used for target category, texture appearance, and scene semantic recognition; depth observation is used to estimate 3D spatial structure information, including target distance, 3D position, and reachable areas; language commands provide the target object, action intention, and constraints; and robot body state describes the robot's current execution capabilities and action reachability. To ensure consistency across different modalities in the temporal dimension, the system synchronizes RGB, depth, and state data based on the perception timestamp. Observations with time differences exceeding a threshold are discarded or compensated using the most recent valid frame, thereby reducing the impact of sensor latency on scene representation.

[0022] The preprocessed four types of modal data are input into their respective encoders to extract their respective feature vectors: ; in, , , and These represent the visual encoder, depth encoder, speech encoder, and state encoder, respectively. , , and These represent the feature representations of the four modalities.

[0023] A fusion representation is formed using attention weighting: ; in, Indicates the first Modal features, Indicates the corresponding weight. This is a fused multimodal scene representation. The mechanism can dynamically adjust the importance of different modalities according to task requirements. For example, when the target's color and appearance are relatively clear, visual features have a higher weight; when the target is occluded or interfered with by similar objects, depth information and linguistic constraints can provide supplementary information; when the robot approaches the target and enters the operation phase, the importance of the body state and the end effector pose is further enhanced.

[0024] In the semantic scene representation layer, a three-layer modeling approach of "object, relation, and task" is adopted. Object detection networks or segmentation models are used to characterize the object categories, attributes, and locations in the scene at the object layer; the relation layer describes the topological, distance, and interactive relationships between objects; and the task layer introduces language goals and behavioral constraints to form a task-oriented scene semantic graph. If the scene is represented as a graph structure:

[0025] ; in, Represents a collection of objects or regions. The set of edges representing spatial and semantic relationships. This represents the constraints formed by language instructions and task objectives. Through this structure, the robot can complete target localization, object relationship analysis, operability judgment, and task-related information filtering at the cognitive level. Compared with simple feature stitching, scene graph representation can preserve the relationship structure between objects, enabling subsequent world models and task planning modules not only to know "what is in the scene," but also to further determine "which objects are related to the task" and "which actions are executable in the current state."

[0026] Task-related semantic scene graph and multimodal fusion representation These signals are collectively mapped to a unified latent space, and together with the four types of input signals received at time t—RGB image, depth observation, language commands, and robot body state—form a cognitive state used for world model prediction and task planning. .

[0027] ; in, This represents a multimodal coding network. To unify the latent states, the network can consist of a visual encoder, a deep encoder, a language encoder, and a state mapping layer, and achieves semantic alignment between modalities through a cross-attention mechanism.

[0028] The visual encoder extracts target and scene appearance features, the depth encoder extracts target distance, spatial hierarchy, and reachability information, the language encoder extracts task semantics and constraints, and the state mapping layer maps the robot's current pose, velocity, and actuator state to the same latent space. These four types of features are then fused by the fusion module to form a unified cognitive state, enabling the system to simultaneously consider "what's in the environment," "where the target is," "what the user wants to do," and "whether the robot can currently perform the task." Furthermore, to enhance historical dependency and task continuity, historical memory variables can be introduced. This forms a recursive representation:

[0029] ; in, This represents a temporal memory update function (such as GRU). This memory variable can be used to record historical observations, completed subtasks, failed actions, and environmental changes, enabling the cognitive layer to perform continuous reasoning by combining the task execution process with information from the current frame, rather than relying solely on the information in the current frame.

[0030] World models are used to learn the evolution of latent states under the influence of actions. Their basic form is as follows: ; in, For the current action, For state transition model, The potential state for the next time step as predicted by the model. Historical memory is introduced. Therefore, it is possible to make predictions using historical context in partially observable or long-term tasks. If task objectives or action rewards are further considered, a prediction function can be defined:

[0031] ; in, Indicates action The estimation of task benefit or goal completion rate corresponding to the current cognitive state. The role of the world model in this invention is mainly reflected in three aspects: First, predicting the changes in the environmental state and robot state after the current action is executed; second, estimating the impact of candidate actions on the probability of task completion and safety risks; third, providing internal extrapolation of the state for several future steps for long-term task planning. Through the multi-step state prediction and candidate action evaluation mechanism based on the world model, the cognitive layer can compare the task benefits and safety risks of different candidate action sequences before actual execution, and select the action sequence that better meets the task goal and safety constraints, reducing the risks caused by blind trial and error. During training, the loss function is usually constructed by minimizing the prediction error:

[0032] ; in, For the model to predict the state, To be observed from the next moment The latent states obtained through the same coding network, For task consistency loss, These are the weighting coefficients. Primarily used to constrain the consistency between cognitive states and task objectives, it can consist of goal matching loss, sub-task classification loss, or action feasibility judgment loss. Backpropagation updates the parameters of the multimodal coding network and state transition model based on the loss calculation. The joint representation and world model construction process is represented by the pseudocode shown in Algorithm 1:

[0033] In this process, These are not labels added manually, but rather labels observed in the next moment. The latent state obtained through the same coding network; This refers to the potential state predicted by the world model based on the current memory state and action. By minimizing the error between the two, the model can gradually learn the transition relationship between "current cognitive state, action, and future state".

[0034] like Figure 1 As shown, the basic implementation logic of the cognitive layer is as follows: First, features are encoded for vision, depth, language, and robot state respectively. Then, a unified latent representation is generated through a fusion module, and combined with historical memory to form a continuous cognitive state. Based on this, the world model predicts future latent states according to the current state and actions, and then the network parameters are jointly optimized by prediction error and task loss. This process demonstrates the complete computational chain from multimodal input and joint representation learning to world model prediction, and also shows that the robot's cognitive layer is not a static perception module, but a dynamic cognitive system with temporal memory, state inference, and task adaptation capabilities.

[0035] The cognitive layer has obtained the cognitive state through multimodal coding and world model. and historical memory status In long-term task planning, the system further integrates the current cognitive state, historical memory state, and task objective into a task context representation:

[0036] ; in, Indicates the current task context state. This represents the task context encoding function. This represents the historical trajectory from the start of the task to the current moment, including historical observations, executed actions, and execution feedback. Compared to making decisions directly using single-frame observations, task context representation can preserve task execution progress and stage dependencies, enabling the system to avoid redundant planning, missed steps, or ignoring historical failure actions in long-sequence tasks. Based on this, the generative programming model learns the conditional distribution: .

[0037] A three-tiered generation mechanism of "task objective, sub-task sequence, and action plan" is adopted. First, the cognitive layer parses the task objective based on language semantics and scene representation, outputting the target object, action type, target location, task constraints, and completion conditions. The task objective serves as input for subsequent sub-task decomposition and is also passed to the emotion layer, providing contextual information such as the current task content and task stage for emotion and intent recognition.

[0038] Subsequently, the cognitive layer generates an ordered sequence of subtasks based on the task objective and the predictions of state transitions from the world model. It outputs the execution order, preconditions, completion status, and exception handling conditions for each subtask. For example, the object delivery task is decomposed into subtasks such as target localization, path approach, grasping preparation, grasping execution, object transfer, and target delivery. The subtask sequence is used to determine the current task stage and provides it to the emotion layer as contextual constraints, enabling the emotion layer to adjust the user's emotions and interaction intentions based on whether the robot is currently in the approach, grasping, or delivery stage. Simultaneously, the subtask sequence is passed to the control layer to determine which motion control or interaction execution modules need to be invoked.

[0039] Finally, the cognitive layer transforms the current subtask into an action plan or set of candidate actions that can be invoked by the control layer, outputting the action type, target pose, execution order, and initial constraints such as speed, safety distance, and action amplitude. The emotion layer modulates the initial constraints based on the user's emotional state and interaction intent, generating behavioral adjustment parameters such as task priority, explanation feedback frequency, speed adjustment, safety distance, and pause confirmation. The control layer generates specific control commands such as chassis movement, robotic arm operation, gripper opening and closing, voice feedback, or local replanning based on the modulated action plan, and feeds back the execution results, trajectory deviations, and safety events to the cognitive and emotion layers for updating task stages, emotion judgments, and subsequent action plans.

[0040] Therefore, the three-level generation mechanism forms a continuous chain of "task objective determination, sub-task decomposition, action plan generation, emotion modulation, control execution, and feedback update," enabling the robot to maintain the continuity of planning, interactive adaptability, and execution safety in complex tasks. The corresponding optimization objective can be expressed as:

[0041] ; The first term is the action generation loss. To account for planning consistency loss, These are the corresponding weighting coefficients.

[0042] Step 3: Divide the emotional layer input into four categories: visual, speech, language, and contextual information. The visual modality primarily reflects the user's outward facial expressions, gaze direction, head posture, and body movement changes; the speech modality primarily reflects speech rate, intensity, pauses, intonation, and prosodic fluctuations; the language modality primarily reflects the semantics of user commands, emotional words, negation expressions, and interaction intentions; and the contextual state includes task stage, historical dialogue, execution results, and the relative state between the user and the robot. Let the complete emotional layer input be:

[0043] ; in, , , , Let visual, speech, language, and context states be represented respectively. Then, the joint state of emotion and intention can be represented as: ; in, This represents the potential state of the emotional layer. This is a multimodal sentiment encoding function. The sentiment layer output... It is not directly equivalent to robot action commands, but is further transformed into modulation variables for the cognitive and control layers. In other words, the emotional layer mainly provides human-centered constraints for task planning in the cognitive layer and safe execution in the control layer, while the final behavior still needs to be generated in combination with environmental semantics, task objectives and underlying safety constraints.

[0044] The emotion layer is divided into four sub-modules: multimodal emotion perception, intention reasoning, emotion state modulation, and interactive feedback output. The multimodal emotion perception module is responsible for extracting emotional cues from visual, auditory, and linguistic signals; the intention reasoning module estimates the user's current goal, demand intensity, and interaction preferences by combining contextual information; the emotion state modulation module maps emotions and intention states to policy biases or behavioral constraints; and the interactive feedback output module generates corresponding language responses, action styles, and interaction rhythm adjustments based on the modulation results. The emotion state modulation process can be represented as:

[0045] ; in, For the modulated signals sent to the cognitive and control layers, and These are the mapping weights and bias terms, respectively. This modulation signal primarily affects interfaces such as task priority, interpretation feedback frequency, action speed boundaries, safety distance, and replanning trigger conditions.

[0046] Multimodal emotion and intent feature modeling is fundamental to the robot's emotion layer's ability to understand user states. Its core objective is to extract discriminative and complementary emotional cues from visual, speech, language, and interactive contexts, and to construct a unified emotion and intent representation space. Because different modalities exhibit significant differences in temporal resolution, semantic granularity, and noise distribution, this process requires not only single-modal feature extraction but also addressing issues such as cross-modal temporal alignment, feature scale uniformity, and differences in modal reliability.

[0047] In the visual modality, the system primarily focuses on explicit emotional cues such as facial expressions, eye movement direction, head posture, and body movements, which can be recorded as visual features. In speech modalities, paralinguistic features such as fundamental frequency, energy, speech rate, pauses, formants, and prosodic variations are extracted to form speech representations. In language modalities, intention and attitude information are extracted from user commands, semantic dependencies, affective polarity, and contextual information to obtain language features. The three modalities mentioned above achieve time alignment within the same interactive window. Among them, visual and speech modalities are better suited for capturing continuous emotional changes, while the speech modalities are better suited for expressing clear task intentions and user needs.

[0048] To distinguish between complete emotional layer input and multimodal observation input, the multimodal observation portion without context is denoted as: ; Its joint emotional expression can be written as: ; in, It is a multimodal fusion mapping function, specifically a concatenation layer with masks or attention weights followed by a fully connected mapping layer (MLP). Indicates time The emotional potential features are described. Only the three observable modalities of vision, speech, and language are described, while the contextual state is not. This will be introduced separately in subsequent intent inference and contextual constraints. This symbolic distinction avoids confusing modality feature extraction with contextual reasoning at the same level, making the data flow within the sentiment layer clearer. To enhance information interaction between different modalities, a weighted fusion and attention allocation mechanism is adopted:

[0049] ; in, Indicates the first Class modality at time The encoding result, This represents the contribution weight of the corresponding mode. This is a multimodal fusion feature. The weight is not a fixed constant, but is dynamically adjusted based on modal quality, recognition confidence, and the current interaction environment.

[0050] For example, when there is significant background noise in the speech or the confidence level of the speech recognition result is low, the system reduces the weight of the speech modality and increases the influence of visual expression and linguistic semantics; when the user's face is obscured or off-camera, the system relies more on speech prosody and linguistic content. Through this dynamic allocation method, the emotion layer can maintain relatively stable emotion judgment under conditions of unbalanced modal quality. In specific implementation, Emotional latent representations can be further generated through a mapping layer or a classification head. This serves as the input for subsequent intent inference and emotion modulation. Building upon this, intent modeling further integrates emotional state and task context to form a joint inference function: ;

[0051] in, This contextual information mainly includes the current task stage, historical interaction records, robot execution status, and the most recent action feedback; This refers to the user's intent category or interaction goal. Therefore, the emotion layer can not only identify the user's current emotion but also further determine whether the user wants to continue the task, request an explanation, correct the robot's behavior, or terminate the current interaction, thus providing a more complete human-centered state representation for subsequent cognitive modulation and behavior generation. To more clearly describe the program implementation process of the above method, the multimodal emotion and intent feature modeling process can be represented as pseudocode, as in Algorithm 2.

[0052] Basic implementation logic: First, visual, speech, and language information are temporally aligned, and single-modal sentiment features are extracted using independent encoders. Then, fusion weights are assigned based on modality confidence and contextual consistency to construct a unified latent sentiment representation. Finally, intent inference is completed by combining contextual information. The overall process is consistent with the formulas mentioned earlier.

[0053] ; This consistency reflects the computational chain of multimodal input, unified representation, contextual constraints, and intentional output. This process provides a basic computational framework for online estimation of sentiment and intention states at the sentiment layer.

[0054] Context-aware multimodal emotion recognition and robustness enhancement. The emotion layer introduces a context-aware multimodal emotion recognition method, which jointly models explicit cues such as visual, speech, and language with dialogue history, task stage, environmental state, and execution feedback to improve the stability and context consistency of emotion recognition.

[0055] Set time Multimodal input is The context information is The emotional state is: ; in, To integrate the current observation with the emotional state based on context, the contextual information here is not an additional emotion label, but rather a conditional variable used to interpret the current emotional cue. For example, after a robot fails to perform a grasping task, a user's silence, frown, or repeated questioning is more likely to correspond to confusion or dissatisfaction; while at the beginning of the task, the same short pause might simply be the user waiting for the robot to respond. Therefore, introducing contextual information can reduce the dependence of emotion recognition on single-frame facial expressions or single-sentence speech, making the emotion layer output more consistent with the current task context. To characterize the continuous changes in emotion over time, a temporal state update mechanism is introduced:

[0056] ; in, Indicates time The implicit state of emotional temporal sequence can preserve information on the evolution of emotions in historical interactions, thus avoiding oversensitivity to anomalous signals at a single moment. In implementation, the emotion layer focuses more on trend changes within an interaction window than on emotional labels at isolated moments. For example, the system only gradually increases the confidence level of anxiety or dissatisfaction when a user repeatedly exhibits increased speech speed, repeated urging, physical retreat, or negative expressions; if the anomalous signal appears only once and subsequently recovers smoothly, the system will not immediately trigger the emphasis strategy. To address issues such as modality loss, noise pollution, and inconsistent quality of heterogeneous information, a dynamic weighted fusion mechanism is adopted to improve robustness, which takes the form of:

[0057] ; in, For the first Modal feature representation, For the corresponding weights, This represents the robustly fused sentiment representation. The weights are determined by modality confidence, contextual consistency, and historical stability, enabling the system to automatically enhance the effectiveness of other modalities when one modality degenerates. Robustness here is not achieved simply by increasing model complexity, but rather by reducing misjudgments through modality quality assessment, historical state smoothing, and contextual constraints. Multimodal features are concatenated and input into a lightweight neural network (such as an MLP). The network output is normalized using a Softmax function, resulting in a weight distribution with a sum of 1. The quality or confidence of each modality feature is evaluated using a noise-aware uncertainty estimator. Higher quality (lower uncertainty) results in a larger weight. For example, when the speech signal is affected by environmental noise, the system reduces the speech modality weight; when the text content is too short or lacks explicit sentiment words, the system compensates using visual expressions and historical interaction states; when visual information is occluded, the system increases the relative contribution of the speech and language modalities. Furthermore, a joint loss function can be constructed:

[0058] ; in, The emotion classification loss is used to constrain emotion category prediction; Context consistency constraints are used to reduce conflicts between emotion judgments and task stages and linguistic semantics; This is a robustness regularization term used to enhance the model's adaptability to noise perturbations and modality loss. Compared to a simple sentiment classification loss, this joint loss allows the model to simultaneously focus on classification accuracy, contextual plausibility, and modality stability during training, making it more suitable for real-world embodied interaction scenarios.

[0059] The ultimate goal of emotion recognition is not merely to obtain user emotion labels, but to further map emotional states into executable interaction strategies and behavioral regulation signals, enabling the emotional layer to substantially participate in the decision-making process of the cognitive layer and the control process of the control layer. In other words, the system needs to transform the user's emotional intensity, intention preferences, contextual pressure, and historical interaction states into adjustable parameters such as response tone, action amplitude, execution speed, interaction distance, and task priority, thereby achieving emotion-driven embodied behavior adaptation.

[0060] Let the emotional state vector output by the emotional layer be... The context state is The interaction strategy can then be defined as: ; in, The interaction strategy vector at the current moment can include parameters such as language style, feedback frequency, action compliance, and response intensity. Furthermore, to achieve continuous modulation of the execution layer, behavior modulation signals can be constructed: ;

[0061] in, and For the mapping matrix, For bias terms, This is the adjustment signal sent to the cognitive planning and motor control module. It does not directly replace the planning results of the cognitive layer or the control instructions of the control layer, but rather acts as a modulating variable on interfaces such as task priority, language interpretation frequency, motion speed boundaries, safe distance thresholds, and replanning trigger conditions. This preserves the safety constraints of the underlying control while allowing the robot's behavior to reflect responses to the user's state. This is reflected in the unfolding form under emotional state and contextual conditions, where... Indicates the current emotional expression, This represents the interaction context, and together they determine the direction and intensity of the modulated signal.

[0062] In the strategy generation process, such as Figure 2 As shown, a three-tiered mapping mechanism of "emotional state, strategy parameters, and execution constraints" is employed. First, emotion category and intensity are mapped to response strategy biases. For example, when the user is calm, the system maintains a normal response pace; when the user shows confusion, the system adds explanation and confirmation; when the user shows anxiety, the system slows down the action pace and explains the subsequent actions in advance; when the user shows dissatisfaction or resistance, the system pauses high-risk actions and reconfirms the task requirements. Second, intentional state is mapped to task priority adjustment factors. For example, when the user explicitly requests "stop for a moment," the priority of pausing or waiting safely should be higher than continuing the original task. Finally, a moderating factor is introduced in the execution layer. Correct the control output:

[0063] ; in, These are the original control commands. The adjusted control output, This indicates element-wise modulation. This process mainly affects continuous control quantities such as velocity, acceleration, end effector amplitude, and approach distance, making the robot's movements smoother, more stable, and more consistent with the current interaction context.

[0064] Step 4: The information interaction and modulation mechanism of the cognitive and emotional dual channels.

[0065] The cognitive and affective layers are not independent but rather achieve continuous information interaction and state modulation through a dual-channel coupling mechanism. The cognitive channel is primarily responsible for environmental understanding, task reasoning, and state prediction, while the affective channel is mainly responsible for user emotion perception, intent recognition, and interaction preference estimation. Together, they influence the robot's decision-making and execution processes, enabling the system to understand both the task and the user's state, thus achieving more natural embodied interaction.

[0066] From the perspective of information flow, the scene semantic state output by the cognitive layer The emotional layer provides task context, environmental constraints, and behavioral stage information; the emotional state output by the emotional layer... This, in turn, acts on the cognitive layer to regulate task priority, decision weights, and response methods. Here, it's important to distinguish between two easily confused variables: This indicates the interaction context used within the emotional layer, such as historical dialogues, task phases, and performance feedback. This represents the cognitive state output by the cognitive layer, encompassing environmental semantics, task objectives, and planning results. While related in content, they exist at different levels. Therefore, the joint state of the two channels can be represented as:

[0067] ; in, Indicates a cross-layer fusion function. This represents a state of cognitive and emotional coordination. This state includes not only knowledge of the external environment but also prior knowledge of the user's internal state, serving as a crucial basis for subsequent strategy generation. This can be obtained from the aforementioned multimodal robust fusion representation and its temporal state update, and is used to represent the emotional state output from the emotion layer to the cognitive layer. To describe the modulating effect of emotion on cognition, a gating mechanism can be further constructed:

[0068] ; in, For activation function, This represents the cognitive state modulated by emotion. This mechanism demonstrates that emotional states can dynamically influence the importance of different feature dimensions in cognitive representations. For example, when the user is calm or actively cooperating, the system can maintain high task performance efficiency; when the user is confused, anxious, or dissatisfied, the system can increase the weight of explanatory feedback, safe distance, and conservative actions, while reducing the tendency to directly perform high-risk actions. This approach prevents the robot from mechanically performing the original task when the user is clearly uncomfortable or their intentions are unclear. Correspondingly, the cognitive-emotional feedback modulation can also be represented as:

[0069] ; This involves using task phases, environmental risks, and execution results to correct current emotional judgments, avoiding isolated emotion recognition detached from context. For example, when a user speaks faster but the task is in a time-sensitive phase, the system should not simply interpret it as a negative emotion, but rather explain it in conjunction with the urgency of the task. When a user becomes silent, frowns, or repeatedly asks questions after the robot fails to grasp repeatedly, the system should increase the confidence level of the confused or dissatisfied state by combining the execution failure records, and proactively provide explanations or reconfirm the need. Thus, cognitive state provides contextual constraints for emotional judgment, and emotional state, in turn, regulates cognitive decision-making, forming a closed loop. After performing an action, changes in the physical world (success, failure, or external interference) are re-inputted into the brain as new observations (step one). Simultaneously, the user's reaction to the new action (such as emotional calming or increased impatience) is re-inputted into the heart. If the execution result does not conform to the world model prediction (excessive error), or the emotional state deteriorates sharply (the user issues a "stop" command), the system triggers dynamic replanning, interrupting the current action sequence, returning to a safe state, or generating a new sub-task plan.

[0070] like Figure 3 As shown, the information interaction between the cognitive and emotional channels is essentially a bidirectional, dynamically updated joint reasoning process. Cognition provides the semantic basis and task context for emotion, while emotion provides humanistic constraints and strategy biases for cognition. The two form a closed-loop modulation mechanism through fusion, gating, and feedback.

[0071] This invention also provides a robot interaction understanding and behavior regulation system based on a multimodal emotion layer, comprising: The cognitive processing module is used to acquire multimodal environmental perception data, including environmental RGB images, environmental 3D spatial structure information, and robot body state; encode the multimodal environmental perception data separately, and use attention-weighted fusion of the multimodal encoding results to obtain a multimodal fusion representation; construct a semantic scene graph based on the multimodal environmental perception data; map the multimodal fusion representation and semantic scene graph to the latent space to obtain the cognitive state; acquire the user's target command, and combine historical memory variables and cognitive state to obtain the task context representation.

[0072] The emotion processing module is used to acquire multimodal interactive observation data including visual, speech, and language. It extracts visual facial expression and posture features, speech prosodic features, and language semantic sentiment features, respectively. It then performs weighted fusion of these features to obtain multimodal fusion features, performs nonlinear mapping on them to generate a latent emotional representation, and jointly encodes the latent emotional representation with the task context representation to obtain the emotional state. Finally, it maps the emotional state to a modulation signal and outputs the emotional state and the modulation signal.

[0073] The behavior regulation module is used to fuse cognitive and emotional states to obtain a cognitive-emotional coordinated state; and to control and constrain the robot based on the cognitive-emotional coordinated state and the modulation signal.

[0074] The modules in the aforementioned robot interaction understanding and behavior regulation system based on a multimodal emotion layer can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0075] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a robot interaction understanding and behavior regulation method based on a multimodal emotion layer. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0076] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a robot interaction understanding and behavior adjustment method based on a multimodal emotion layer. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0077] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] It should be noted that the above specific embodiments enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for robot interaction understanding and behavior regulation based on a multimodal emotion layer, characterized in that, Includes the following steps: The system acquires multimodal environmental perception data, including environmental RGB images, environmental 3D spatial structure information, and robot body state; encodes the multimodal environmental perception data separately, and uses attention-weighted fusion of the multimodal encoding results to obtain a multimodal fusion representation; constructs a semantic scene graph based on the multimodal environmental perception data; maps the multimodal fusion representation and semantic scene graph to a latent space to obtain a cognitive state; acquires the user's target instruction, and combines historical memory variables and the cognitive state to obtain a task context representation; Acquire multimodal interaction observation data including visual, speech, and language; extract visual facial expression and posture features, speech prosodic features, and language semantic sentiment features respectively; perform weighted fusion of the facial expression and posture features, prosodic features, and semantic sentiment features to obtain multimodal fusion features; perform nonlinear mapping on the multimodal fusion features to generate a potential emotional representation; jointly encode the potential emotional representation with the task context representation to obtain an emotional state; map the emotional state into a modulation signal, and output the emotional state and the modulation signal. The cognitive state and the emotional state are fused to obtain a cognitive-emotional synergistic state; Control constraints on robots are based on cognitive-emotional co-state and modulated signals.

2. A method for robot interaction understanding and behavior regulation based on a multimodal emotion layer according to claim 1, characterized in that, The memory state variable from the previous moment and the cognitive state are input into the memory update function to obtain the historical memory variable. The memory state variable from the previous moment is obtained by compressing and encoding the historical observation scene features, completed sub-tasks, execution errors, and environmental change information stored inside the robot.

3. A method for robot interaction understanding and behavior regulation based on a multimodal emotion layer according to claim 1, characterized in that, The construction of the semantic scene graph specifically involves: identifying the categories of all objects in the scene through a visual encoder based on the collected multimodal environment perception data, and using depth information to calculate the three-dimensional position and physical size of the objects to form a set of nodes in the graph; By combining the category attributes of objects, the interaction relationships are inferred, forming a set of edges in the graph; the target object and action intent extracted from the language instructions by the language encoder are used as prior constraints to obtain the semantic scene graph; where the nodes of the semantic scene graph represent entities in the environment, and the edges of the semantic scene graph represent the semantic relationships between nodes.

4. A method for robot interaction understanding and behavior regulation based on a multimodal emotion layer according to claim 1, characterized in that, A dynamic weighted fusion mechanism is employed to weight and fuse the facial expression and posture features, prosodic features, and semantic sentiment features to obtain multimodal fusion features. Specifically, the facial expression and posture features, prosodic features, and semantic sentiment features are concatenated and input into a lightweight neural network. The network output is normalized by the Softmax function to obtain a weight distribution with a sum of 1. The confidence level of each modality feature is evaluated, and weights are assigned based on the confidence levels of different modalities. The facial expression and posture features, prosodic features, and semantic sentiment features are weighted and summed based on the weights corresponding to the modalities to obtain the final multimodal fusion features.

5. A method for robot interaction understanding and behavior regulation based on a multimodal emotion layer according to claim 1, characterized in that, It also includes generating a gating weight vector with the same dimension as the cognitive features based on the emotional state, using the gating weight vector to gating the cognitive state, and fusing the modulated cognitive state with the emotional state to input the world model to drive the next round of state prediction and task planning.

6. A robot interaction understanding and behavior regulation system based on a multimodal emotion layer, characterized in that, include: The cognitive processing module is used to acquire multimodal environmental perception data, including environmental RGB images, environmental 3D spatial structure information, and robot body state. Multimodal environment perception data are encoded separately, and attention-weighted fusion of the multimodal encoding results is used to obtain a multimodal fusion representation; a semantic scene graph is constructed based on the multimodal environment perception data; the multimodal fusion representation and the semantic scene graph are mapped to the latent space to obtain the cognitive state; the user's target instruction is obtained, and the task context representation is obtained by combining historical memory variables and the cognitive state; The emotion processing module is used to acquire multimodal interaction observation data of the user, including visual, speech, and language; extract facial expression and posture features from the visual representation, prosodic features from the speech representation, and semantic emotion features from the language representation; perform weighted fusion of the facial expression and posture features, prosodic features, and semantic emotion features to obtain multimodal fusion features; perform nonlinear mapping on the multimodal fusion features to generate a potential emotional representation; jointly encode the potential emotional representation with the task context representation to obtain an emotional state; map the emotional state into a modulation signal, and output the emotional state and the modulation signal. The behavior regulation module is used to fuse the cognitive state and the emotional state to obtain a cognitive-emotional synergistic state; Control constraints on robots are based on cognitive-emotional co-state and modulated signals.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes a computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method of any one of claims 1 to 5.

Images