A multi-modal emotion recognition closed-loop system and method based on ontology motion perception and execution feedforward
By introducing proprioceptive motion perception and execution feedforward into the multimodal emotion recognition system, the self-disturbance problem caused by robot proprioceptive motion is solved, the robustness of recognition and the stability of user interaction are achieved, a closed-loop control of perception-decision-execution is constructed, and the accuracy and continuity of robot interaction are improved.
Patent Information
- Application Number
- CN202611283198.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-24
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal emotion recognition systems, based on robot platforms with active movement capabilities, neglect the impact of self-disturbance caused by the robot's own movement on the visual channel recognition quality. This leads to distorted confidence assessment and incorrect fusion results. Furthermore, the system architecture is an open-loop structure, and feedback actions cause jitter in recognition quality, resulting in a broken user interaction experience.
By introducing proprioceptive motion state variables such as neck posture and angular velocity into the confidence assessment, a multimodal emotion recognition closed-loop system is realized. The fusion weights are adjusted using an execution feedforward mechanism to construct a closed-loop control of perception-decision-execution, predict and adjust the interference of feedback actions on the visual channel, and form a complete closed-loop system.
It effectively suppresses recognition jitter caused by feedback actions, improves recognition robustness, reduces response latency, constructs a complete closed loop of perception-decision-execution, and enhances the user interaction experience.
Smart Images

Figure CN123116138A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-computer interaction and emotion computing technology, and more specifically to a multimodal emotion recognition closed-loop system and method based on ontology motion perception and execution feedforward. Background Technology
[0002] Currently, multimodal emotion recognition is a core capability for service robots to achieve natural human-computer interaction. The mainstream technical approach usually integrates the recognition results from three channels: vision (facial expressions), speech (prosodic features), and semantics (text content). The final emotion judgment output is obtained through weighted fusion or attention mechanisms, which drives the robot to respond in the form of screen expressions, voice replies, neck movements, etc.
[0003] Technological Evolution of Existing Emotion Recognition and Head Interaction Solutions
[0004] In order of technological evolution, current solutions for emotion recognition and head interaction in companion / service robots mainly fall into the following four categories: 1. Single-modal emotion recognition solution: This solution identifies the user's emotional state based solely on vision (facial expressions) or speech (tone, semantics), outputs the emotion category, and then drives the robot to provide a corresponding response. Representative products include early desktop robots such as Jibo and Vector. This solution is highly susceptible to quality fluctuations from a single channel and lacks robustness.
[0005] 2. Multimodal fixed-weight fusion scheme: This scheme simultaneously acquires visual, speech, and semantic signals, with each channel outputting its own emotion recognition result. Decision-level or feature-level fusion is then performed using preset fixed weights. This scheme cannot adapt to dynamic changes in channel quality.
[0006] 3. Multimodal dynamic fusion schemes encompassing equal weighting, weighted average, and attention-based approaches: Building upon Scheme 2, the fusion weights of each modality are dynamically adjusted based on the quality of external signal acquisition. This is currently the mainstream research direction in multimodal emotion recognition, and can be further subdivided into three sub-schemes: Equal-weight fusion sub-scheme: The output probability distributions of the three channels are directly averaged to obtain the final sentiment category; The weighted fusion sub-scheme based on external quality factors dynamically adjusts the weights of each channel according to external environmental quality indicators such as ambient light intensity, face detection confidence, speech signal-to-noise ratio, and image blur. Attention mechanism fusion sub-scheme: Adaptively learn the relative importance of different modalities on different samples through neural networks.
[0007] 4. Head-Motion Follow-up + Facial Feedback Solution: This solution uses multi-degree-of-freedom neck motors to achieve real-time tracking of the user's face, while simultaneously displaying facial feedback via a screen or mechanical structure. Representative products include LOONA and EMO. In this solution, the neck tracking module, facial feedback module, and emotion recognition module typically operate independently, without considering the coupling effects between them.
[0008] Significant shortcomings of existing technologies on robotic platforms with active movement capabilities
[0009] When the aforementioned existing technologies are deployed on robotic platforms with active motion capabilities (especially robots with neck / head degrees of freedom), they reveal significant shortcomings in the following two aspects: 1. The sensory input dimension is incomplete, and the robot's own self-disturbance is ignored.
[0010] Existing multimodal confidence assessment systems only consider external environmental factors (lighting, occlusion, blurring, noise, etc.), completely ignoring the self-disturbance factors caused by the robot's own motion, which have a greater impact on recognition quality. Cameras are typically mounted on the robot's head, which is driven by a neck motor: a turn of the neck causes the face to shift out of center of the image; a lower neck leaves only the user's chin in the frame; a rapid neck rotation causes motion blur; facial expressions and empathetic movements also cause the viewpoint to deviate from the optimal area for capturing the user's face, resulting in problems such as face deviation, motion blur, and viewpoint distortion.
[0011] These self-perturbations are internal factors, and their impact on the visual channel recognition quality is far greater than changes in external factors such as ambient light. However, current technologies do not incorporate them into the confidence assessment system. The system still performs fusion according to the original weights, directly leading to inflated confidence levels in visual emotion recognition, erroneous fusion results, or jitter.
[0012] 2. The system architecture is an open-loop structure, and feedback actions negatively impact the recognition quality.
[0013] Traditional recognition-decision-execution linkages employ a unidirectional open-loop architecture. The workflow is as follows: the recognition module outputs user emotional information, the decision module generates corresponding feedback actions based on this information, and the execution module drives the motor to complete the action. In this architecture, the three modules operate in a serial manner. The recognition module cannot obtain the real-time operating status of the execution module; simultaneously, changes in the robot's state caused by feedback actions (such as facial expressions or neck movements) cannot be transmitted back to the recognition module, resulting in a failure to establish a closed-loop control system among tracking, feedback, and recognition.
[0014] This open-loop architecture is prone to typical negative scenarios: when the robot detects that the user is in a sad state, the decision-making module generates an empathetic head tilt as feedback, which then activates the neck motor, causing the camera's field of view to shift and significantly reducing the recognition quality of the visual modality. In this situation, the multimodal fusion result will frequently fluctuate between sad and neutral states, and the robot's on-screen facial expressions and voice feedback will also become chaotic, ultimately causing an interruption and fragmentation of the user experience.
[0015] To address this issue, existing remedies include waiting for visual stability before making a decision, performing temporal smoothing on the fusion results, and adjusting weights after detecting a decline in visual quality. These are all post-event compensations: jitter has already occurred, misidentification has already occurred, and additional response delays have been introduced.
[0016] Therefore, how to provide a predictive weight adjustment mechanism for ontology actions to fundamentally compensate for the instability of recognition performance in dynamic interaction is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0017] In view of this, the present invention provides a multimodal emotion recognition closed-loop system and method based on ontological motion perception and execution feedforward, aiming to solve the following three interrelated technical problems: First, the existing multimodal confidence assessment system only considers external environmental factors and completely ignores the self-disturbance caused by the robot's own motion, resulting in the problem of confidence assessment distortion and incorrect fusion weight allocation in the visual channel during the robot's own movement; Second, the existing recognition-decision-execution architecture is a unidirectional open-loop link, and the recognition module is completely unaware of the execution intention and execution state of the feedback action, resulting in the problem of severe jitter in the fusion result between adjacent emotion categories during the execution of the feedback action, the screen expression and voice feedback changing erratically, and the user interaction experience being broken; Third, the existing remedies for the above jitter are all post-compensation in nature, with inherent defects such as large response delay and the inability to eradicate jitter from the mechanism.
[0018] To address the aforementioned issues, this invention, on the one hand, explicitly introduces proprioceptive motion state variables such as neck posture, angular velocity, and orientation deviation into the confidence assessment function, upgrading the confidence assessment of the visual channel from solely perceiving the external environment to considering both internal and external factors, thereby eliminating confidence distortion caused by proprioceptive self-perturbation at its source. On the other hand, by transmitting action parameters back to the fusion decision module in real time before the actual execution of the feedback action, predictive adjustment of the fusion weights for the next moment is achieved, upgrading traditional post-event compensation to pre-event pre-adjustment, eliminating the interference of feedback actions on recognition quality from a mechanistic perspective, and ultimately realizing a true closed loop between perception, decision-making, and execution.
[0019] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal emotion recognition closed-loop system based on ontology motion perception and execution feedforward includes a multimodal perception module, an ontology state perception module, a confidence assessment module, a multimodal fusion decision module, a feedback action generation module, a pre-execution predictive feedback module, and an execution module. The multimodal perception module is used to acquire visual, speech, and semantic perception signals in parallel, and output the probability distribution of the emotion category for each corresponding channel. The body state perception module is used to collect the motion parameters of the robot's neck actuator and the body posture parameters in real time. After kinematic calculation, it outputs a set of body motion states consisting of the complete head posture, head angular velocity, and user orientation deviation. The input of the confidence assessment module is connected to the multimodal perception module and the ontology state perception module respectively, and is used to calculate the real-time confidence of each channel based on the external quality factor and the ontology motion penalty factor. The input of the multimodal fusion decision module is connected to the confidence evaluation module, which is used to normalize the confidence of each channel into fusion weights, perform weighted fusion of the three sentiment probability distributions, and output the final sentiment judgment result. The input end of the feedback action generation module is connected to the multimodal fusion decision module, and is used to generate a set of feedback action parameters based on the emotion judgment result. The set of feedback action parameters includes at least the action type, target angle, start time, duration and maximum angular velocity. The input end of the pre-execution predictive feedback module is connected to the feedback action generation module, and the output end is fed forward to the confidence evaluation module. It is used to calculate the interference prediction value of the visual channel based on the action parameter set before the actual execution of the feedback action, and to make predictive pre-adjustment of the channel fusion weight at the next moment based on the interference prediction value. The input terminal of the execution module is connected to the feedback action generation module, and is used to synchronously drive the neck actuator, voice feedback unit and display feedback unit to perform corresponding feedback actions at a preset start time.
[0020] Optionally, the set of motion states of the body specifically includes:
[0021] In the formula, These are the head yaw, pitch, and roll angles, respectively. To correspond to the three-axis angular velocities, For the fuselage attitude angle and fuselage angular velocity, For the target angle of the action, The remaining execution time of the action; the head posture is calculated by aligning the neck encoder data and the body IMU data with timestamps and then solving them through chained coordinate transformation.
[0022] Optionally, the confidence assessment module calculates the confidence score of the visual channel using the following formula:
[0023] in, The external quality factor is obtained based on illumination intensity, face detection confidence, and image blur. The ontological motion penalty factor, taking values in the range (0,1), is a monotonically decreasing function of head motion intensity, used to characterize the degree of attenuation of visual recognition quality caused by ontological self-perturbation. The confidence level of the speech channel is jointly determined by the speech signal-to-noise ratio, speech activity detection results, and acoustic energy; the confidence level of the semantic channel is jointly determined by the speech recognition confidence level, text length, and semantic completeness. Neither of these incorporates the ontological motion penalty factor. Optionally, the ontological motion penalty factor... Using exponential function form:
[0024] in, For the head's synthesized angular velocity, The angle of deviation between the head orientation and the user's orientation. The pitch and rotation angle, For the tumble rotation angle, - The penalty coefficient is obtained by fitting the calibration dataset. Optionally, the pre-execution predictive feedback module performs the following operations: Constructing a visual interference prediction function based on action parameter set ,Place The value range is [0,1], and it is positively correlated with the amplitude of the movement and the maximum angular velocity. Within the advance window before the start of the action, according to Feedforward adjustment of the weights for each channel:
[0025] in, , , For baseline fusion weights, , , This is the adjustment coefficient; The adjusted weights, after normalization, are used as the fusion weights during action execution. Optionally, after the action execution is completed, the confidence assessment module smoothly restores the weights of each channel from the pre-adjusted value to the baseline value using a monotonic transition function; the monotonic transition function includes a cosine transition function, an exponential decay function, or a linear ramp function. Optionally, the neck actuator is a servo drive structure with 2 to 6 degrees of freedom, employing an independent drive method or a multi-servo parallel coupling drive method.
[0026] A multimodal emotion recognition closed-loop method based on ontology motion perception and execution feedforward includes: S1. Parallel acquisition of visual, speech, and semantic signals, and outputting the probability distribution of sentiment categories for each channel respectively; S2. Real-time acquisition of neck encoder data and body IMU data, kinematic calculation and coordinate transformation to obtain head posture, head angular velocity and user orientation deviation, and output the body motion state set; S3. Integrate external quality factors and ontological motion penalty factors to calculate the real-time confidence scores of the visual, speech, and semantic channels; if there is an execution feedforward signal, perform predictive pre-adjustment of the channel weights based on the interference prediction value. S4. Normalize the confidence of each channel into a fusion weight, weight and fuse the three sentiment probability distributions, and output the final sentiment judgment result. S5. Generate a set of feedback action parameters, including action type, target angle, and execution sequence, based on the emotion judgment result; S6. Feed the feedback action parameter set to the confidence assessment stage, calculate the visual channel interference prediction value, complete the predictive adjustment of the weights at the next moment, and send the action parameters into the execution queue. S7. Synchronously drive the neck actuator, voice feedback unit and display feedback unit to perform feedback actions at the preset start time; S8. Repeatedly execute S1~S7. During action execution, the prediction deviation is dynamically corrected based on the real-time ontology state. After the action ends, the channel weights are smoothly restored to the baseline value. Optionally, in S6, when there are multiple consecutive actions to be executed, the weight adjustment intervals corresponding to each action are marked sequentially. When the intervals overlap, the interference prediction values are superimposed, and the visual weight lower limit protection constraint is maintained. When a feedforward action is canceled, the corresponding weight adjustment is revoked, causing the weights to roll back to the state before the action was triggered. As can be seen from the above technical solution, compared with the prior art, this invention discloses a multimodal emotion recognition closed-loop system and method based on ontology motion perception and execution feedforward, which has the following beneficial effects: (1) Suppress recognition jitter caused by feedback actions from a mechanistic perspective By implementing a pre-predictive feedback pathway, the fusion decision module adjusts channel weights before the feedback action is triggered. The fusion result during the feedback action is mainly based on the output of the high-confidence channel, thus suppressing recognition jitter caused by temporary quality degradation of the visual channel from a mechanistic perspective. In a typical empathic head tilt scenario, the solution of this invention significantly reduces the number of switching times of the fused emotion category during the action and the fluctuation amplitude of the emotion probability output compared to traditional solutions.
[0027] (2) Achieving a pre-compensation response
[0028] Traditional post-event compensation schemes require waiting for visual quality degradation to be detected before compensation is initiated, resulting in a response delay on the order of hundreds of milliseconds. The weight adjustment of this invention occurs before the start of the action, exhibiting negative delay characteristics (i.e., compensation is completed before the disturbance occurs; the lead time is determined by the prediction lead of M6, typically 50-100 ms in this embodiment), effectively avoiding the compensation lag problem of traditional schemes.
[0029] (3) Significantly improves recognition robustness during active motion.
[0030] By incorporating the body's motion state into the confidence assessment, the visual channel's confidence assessment can reflect the decline in image quality caused by head movement, thus making the allocation of multimodal fusion weights more reasonable. Compared to baseline schemes that only use external quality factors (illumination, blurriness, detection confidence, etc.), this invention significantly improves the accuracy of emotion recognition in interactive scenarios where the robot has active movement, and the greater the movement intensity, the more significant the improvement.
[0031] (4) Construct a complete closed loop of perception-decision-execution-feedback perception.
[0032] This invention couples the originally independently operating recognition, decision-making, and execution modules into a closed-loop collaborative system through the execution feedforward path (M6→M3) and the ontology state feedback path (M2→M3). This breaks the limitation of the recognition module being unaware of the execution state under the traditional open-loop architecture, and lays the architectural foundation for more complex embodied interactions (such as active following, attention switching, multi-user switching, etc.).
[0033] (5) Low project implementation cost
[0034] The new input information required for this invention (neck motor encoder readings, body IMU data, and feedback motion parameters) are all existing data within the robot system with active motion capabilities, without the need to add additional hardware sensors. At the software architecture level, only the addition of a feedforward data path (M6→M3) and a confidence evaluation function for the body motion factors is required to achieve the above-mentioned technical effects, demonstrating good engineering feasibility. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0036] Figure 1 This is a diagram illustrating the overall architecture of the closed-loop system provided by the present invention. Figure 2 This is a schematic diagram of the sensor layout provided by the present invention; Figure 3 This is a closed-loop timing diagram provided by the present invention; Figure 4 The overall flowchart of the method provided by this invention; Figure 5 The predictive feedback weighting curve provided by this invention. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] The robot hardware platform upon which this invention is based comprises seven hardware functional units (distinguished from the software modules M1~M7 described below): a vision acquisition unit, a voice acquisition unit, a neck actuator unit, a body posture perception unit, a display feedback unit, a voice feedback unit, and a main control computing unit. The preferred parameters for each unit are as follows: Visual acquisition unit: Uses RGB or RGB-D camera with a resolution ≥1280×720 (720P), frame rate ≥30fps, and horizontal field of view ≥60°; Voice acquisition unit: adopts an N-channel microphone array (N≥4), sampling rate ≥16 kHz, signal-to-noise ratio ≥60 dB; Neck actuator: Multi-degree-of-freedom servo motor group, configured in this embodiment as three degrees of freedom: Yaw / Pitch / Roll, with each servo motor having an angle resolution ≤1°, a repeatability accuracy ≤1°, and position feedback; Body attitude perception unit: It consists of the position feedback interface of the neck actuator and the inertial measurement unit (IMU) of the fuselage. After fusion, it outputs the head attitude, with an update frequency of ≥50 Hz and an angle accuracy of ≤1°. Display feedback unit: An LCD or OLED screen located on the front of the head; Voice feedback unit: speaker; Main control computing unit: An embedded platform that supports multi-threaded scheduling and local AI inference (e.g., NPU computing power ≥ 4 TOPS).
[0039] It should be noted that the above parameters are preferred examples. The scope of protection of this invention is not limited to: (a) specific values such as resolution, frame rate, number of channels, and sampling rate; (b) specific device models; (c) the number of neck degrees of freedom and driving method; (d) the specific model and computing power of the main control unit; and (e) the installation location of the IMU (body / head / dual IMU redundancy). All equivalent alternatives that can achieve the same function are within the scope of protection of this invention.
[0040] like Figure 2 (Sensor layout diagram) As shown, the visual acquisition unit of the present invention is rigidly fixed to the front end of the robot head, and its optical axis direction changes strictly synchronously with the head posture (this rigid coupling relationship is a necessary prerequisite for the present invention to introduce the head motion state into the visual confidence assessment); the head is connected to the body through a three-degree-of-freedom neck actuator, and the neck joint includes three degrees of freedom: horizontal rotation (Yaw), pitch rotation (Pitch), and roll rotation (Roll).
[0041] In this embodiment, the neck adopts a three-servo motor drive architecture: one servo motor (denoted as S1) independently drives the yaw degree of freedom; the other two servos (denoted as S2 and S3) are coupled in parallel to jointly drive the pitch and roll degrees of freedom. Compared with independent drive, the parallel coupling structure can reduce one servo motor, reduce head weight and wiring complexity, and is a common mechanical structure for companion robots.
[0042] Let the output angles of S2 and S3 be respectively 1. 2. Then the head pitch and roll angles are determined by a pre-calibrated mapping function. f ( 1, 2) The solution is obtained. Its physical mechanism is: when the two servos rotate in the same direction, they drive the Pitch; when they rotate in opposite directions, they drive the Roll. A typical differential solution relationship is: .in The coupling coefficient is determined by the transmission ratio of the mechanical linkage, and its typical value ranges from 0.5 to 1. ≤1.0, this embodiment takes =1.0 (i.e., the two servo linkages are arranged symmetrically).
[0043] For asymmetric mechanisms, the above equation can be generalized to a general linear mapping: Where the matrix Determined by offline calibration of the mechanical structure. This invention does not limit the specific form of this mapping; anything that can establish (…). 1 , 2 )arrive All methods for solving the mapping relationships fall within the protection scope of this invention.
[0044] To reduce the weight and wiring complexity of the head mechanism, this embodiment mounts the IMU inside the fuselage (other IMU placement methods are detailed in the alternative solutions section below). Since the IMU directly measures the fuselage attitude, while visual acquisition requires the head's attitude in the world coordinate system, the head attitude needs to be obtained through the following chained coordinate transformation: .in, Output directly from the in-body IMU; From the neck three servo encoder f ( 1 , 2 The encoder and IMU are obtained by mapping. Since the encoder and IMU come from different acquisition paths, they need to be aligned with hardware timestamps before chaining transformation (see the motion state parameter definition section in the following text for the alignment scheme).
[0045] The above spatial layout creates the following geometric relationship: When the angle of the neck joint changes Δ θ( Δ When θ is the angular change of any degree of freedom or combination thereof in Yaw / Pitch / Roll, the optical axis direction of the camera and the orientation of the imaging plane also change synchronously, causing the imaging position of the user's face in the image to shift (caused by Yaw / Pitch) or even the image to rotate (caused by Roll). When the angular velocity of the neck joint ω When the displacement velocity of the imaging points within the image frame exceeds a certain threshold, motion blur occurs. When the direction of the neck deviates too much from the user's actual position, the user's face deviates from the center of the image, and the confidence of face detection decreases significantly. When the neck tilt angle is too large, the camera's viewpoint tilts up or down, resulting in perspective distortion of the captured face or only showing local areas such as the chin / forehead, which greatly reduces the accuracy of expression recognition.
[0046] The above geometric relationship is the physical basis for incorporating the body's motion state into the confidence assessment of the visual channel in this invention.
[0047] like Figure 1As shown in the system architecture diagram, the multimodal emotion recognition closed-loop system proposed in this invention includes the following seven modules: M1 Multimodal Perception Module, which collects and identifies emotional features from visual, speech, and semantic channels, and outputs the probability distribution of emotion categories for each channel; M2 Body State Perception Module, which reads the encoder data of the three servo motors in the neck and the IMU data of the fuselage in real time, and outputs the complete head posture and fuselage posture after kinematic calculation and coordinate transformation; M3 Confidence Evaluation Module, which fuses external quality factors and body motion factors to calculate the confidence of each channel; M4 Multimodal Fusion Decision Module, which outputs the emotion judgment based on the weighted fusion of the three channels according to the confidence; M5 Feedback Action Generation Module, which generates feedback action parameters (including action type, target angle, start and end time, duration, maximum angular velocity, and amplitude level) according to the emotion judgment; M6 Pre-Execution Predictive Feedback Module, which feeds the action parameters forward to the confidence evaluation module before the actual execution of the action for weight pre-adjustment; M7 Execution Module, which drives the motor, voice, and screen to execute the feedback action.
[0048] The outputs of M1 and M2 are connected to M3; the output of M3 is connected to M4; the output of M4 is connected to M5; the output of M5 is connected to M6 and M7 in parallel; the output of M6 is fed forward to M3, forming a closed-loop data path of perception-decision-execution-feedback perception.
[0049] The body motion state described in this invention refers to the set of robot's own kinematic parameters that directly affect the recognition quality of the visual channel:
[0050] The first three axis angles are obtained from the neck servo encoder and kinematic calculations, and the corresponding angular velocities are obtained from angle difference calculations; the fuselage attitude and angular velocity are provided by the IMU, and all the above parameters are collected at ≥50 Hz. The target angle and remaining time are given by the parameters of the M5 feedback motion generation module when the motion is triggered. The head world attitude is obtained by chaining transformation after aligning the timestamps of the encoder and IMU data; the user azimuth angle is calculated by face detection combined with head attitude; the three-source data (encoder, IMU, camera) are unified to the visual frame timestamp using hardware timestamps and software interpolation, with an alignment error ≤5ms. Among them, the first 8 parameters (head / fuselage attitude and angular velocity) are used for real-time confidence evaluation by the M3 confidence evaluation module, and the last 2 parameters ( Used for feedforward interference prediction in the M6 pre-execution predictive feedback module.
[0051] This invention explicitly incorporates the proprioceptive motion state into visual confidence assessment, achieving a balance between internal and external factors. Visual confidence is defined as follows: Among them, external factors The ontology penalty factor is determined by factors such as illumination, detection confidence, and ambiguity. It is a monotonically decreasing function of the motion state. A typical form is... The parameters were penalized for synthesized head angular velocity, orientation deviation, pitch deviation, and roll deviation, respectively, with each coefficient determined through least-squares fitting of the calibration dataset. Speech confidence. Determined by SNR, VAD, and acoustic energy, without introducing ontological factors (the impact of motion on microphone acquisition is negligible); semantic confidence. The fusion weights are determined by ASR confidence, text length, and semantic completeness. The three-channel confidence scores are normalized to obtain the fusion weights. Then calculate the fusion probability. And take the category with the highest probability as the sentiment judgment. .
[0052] The feedback action parameter set (including type, target angle, start and end times, duration, maximum angular velocity, and amplitude level) output by the M5 feedback action generation module is not directly sent to the M7 execution module. Instead, it is first fed back to the M3 confidence assessment module via the M6 pre-execution predictive feedback module. The M6→M3 end-to-end delay is ≤10ms; the feedback window is... ,in ms, meaning M3 must be at the start of the action. Weight pre-tuning is completed within the first 50-100ms. M6 calculates the action's range within the execution interval based on a preset motion parameter-visual quality mapping model. The interference prediction function of the internal visual channel is passed to M3 as feedback content.
[0053]
[0054] in The mapping function is positively correlated with the amplitude of motion and angular velocity, and can be used with... Consistent form or independent calibration, A larger value indicates a stronger expected interference. Within the previous lead window, the M3 confidence assessment module was based on Feedforward adjustment of the weights for each channel:
[0055] in This is a lower limit protection value for visual weights (typically 10% of the baseline, to prevent complete failure of the visual channel), and after normalization, it is based primarily on speech / semantics. Significantly greater than The M7 execution module is in Actions are executed continuously, and the fused output is unaffected by temporary visual lapses. After the action is completed, the weights are smoothed using a cosine or exponential function. Restores to baseline within milliseconds to avoid sudden jitter.
[0056] Complete process of this invention ( Figure 4 A typical interaction cycle consists of 8 steps: (1) Multimodal perception (M1): Parallel acquisition of visual / speech / semantic three-channel emotion probabilities at 30Hz; (2) Ontology state acquisition (M2): Reading the encoder and IMU at ≥50Hz to calculate head pose and angular velocity; (3) Confidence assessment (M3): Fusion of external quality factors and ontology penalty factors. and feedforward interference prediction Calculate the overall confidence level of each channel; (4) Fusion decision (M4): Normalized weighted fusion output sentiment judgment (5) Feedback Action Generation (M5): Based on (6) Predictive feedback before execution (M6): Feed A forward to M3 for weight prediction and simultaneously send it to the execution queue; (7) Action execution (M7): In (8) Closed-loop continuous operation: The above steps are executed in a loop, and M2 continuously outputs the actual motion state, and the prediction deviation of M6 is corrected in real time.
[0057] Figure 3 (Closed-loop sequence diagram) shows the sequence of key events in the above process on the timeline, where t 3 (Weight pre-adjustment) strictly earlier than t 5 (actual visual quality decline), which is the core of the visualization of the present invention that pre-adjusts before the disturbance.
[0058] It is important to note that: (1) The present invention does not limit the number of neck degrees of freedom and the driving method, and can be applied to neck / gimbal structures with 2 to 6 degrees of freedom; the driving method can be an independent drive of one servo motor with one degree of freedom, or a parallel coupling drive of multiple servos.
[0059] (2) The present invention does not limit the installation location of the IMU. The IMU can be installed on the fuselage, or on the head, or one IMU can be arranged on the head and one on the fuselage to form dual IMU redundancy; accordingly, the head attitude can be obtained by a combination of encoder + fuselage IMU chain calculation, direct measurement by head IMU, or fusion of the two.
[0060] (3) It can be implemented using an exponential function, a piecewise linear function, a sigmoid function, or a lookup table method; the specific form does not affect the validity of this invention.
[0061] (4) It can be obtained by real-time function calculation or by a lookup table pre-built according to action type and amplitude level, to adapt to low-computing embedded platforms.
[0062] (5) This invention can be extended to more modalities (such as physiological signals and posture recognition) or degenerated to bimodal (visual + speech) applications, while the core mechanism remains unchanged.
[0063] (6) Alternative camera installation locations: This invention is applicable to various layouts in which the camera is installed on the head, gimbal, or other follow-up mechanisms.
[0064] (7) Platform form replacement: This invention is applicable to any platform with a moving camera, such as desktop companion robots, wheeled service robots, humanoid robots, and pet-like robots; for mobile robots, the chassis motion state and motion commands can be further incorporated into the input range of M2 and M6 to achieve unified modeling of all body motion sources.
[0065] (8) The key timing parameters in this invention (end-to-end delay from M6 to M3, prediction lead time of M3, and weight recovery time after action) are all calibrable parameters. The specific value range can be adjusted according to the platform's computing power and response requirements. Typical values are ≤10 ms, 50~100 ms, and 300~500 ms, respectively.
[0066] (9) The weight smoothing recovery function after the action ends can be a cosine transition function, an exponential decay function, a linear ramp function or other monotonic transition functions. As long as the weight smoothly transitions from the pre-adjusted value to the baseline value, it is within the scope of protection of this invention.
[0067] The present invention will be further illustrated by the following specific embodiments.
[0068] Example 1: Complete Implementation Process of Companion Robot Empathic Interaction Scenarios
[0069] This embodiment selects a companion robot equipped with a motion chassis, possessing autonomous following capabilities, and having three degrees of freedom in its neck—yaw, pitch, and roll—as the carrier to fully describe the specific implementation scheme of the present invention. The robot's overall hardware platform is divided into seven core functional modules, each matched with dedicated hardware components and set standardized optimal performance indicators. Through collaborative cooperation, they complete the entire human-computer interaction process. The specific correspondence between the overall hardware configuration and each module is as follows: An RGB-D camera is mounted on the forehead, with measured hardware specifications of 1280×720 resolution and 30 fps. This device belongs to the visual acquisition module (corresponding to the visual input source of the M1 module). The general technical specifications of this module require a resolution of no less than 720P, a frame rate of no less than 30 fps, and a horizontal field of view of no less than 60°. Its core function is to acquire visual images of the user's face, providing visual data support for subsequent interactive functions such as face tracking.
[0070] The device features a multi-microphone array mounted on its chest, corresponding to the voice acquisition module (the voice input source for the M1 module). The module's standard parameters are at least four channels, a sampling rate ≥16 kHz, and a signal-to-noise ratio (SNR) ≥60 dB. It is primarily used to acquire raw user voice signals in real time, ensuring the clarity and integrity of the acquired signals during voice interaction.
[0071] The body attitude perception unit consists of encoder feedback from three servo motors in the neck and an IMU sensing component installed inside the body. The servo encoders output real-time angles and angular velocities for the three degrees of freedom (Yaw / Pitch / Roll) of the neck, while the body IMU outputs the robot's overall attitude and angular velocity in the world coordinate system. The data collected by both units is processed through coordinate transformation and timestamp alignment, and then fused using a data fusion algorithm to obtain the complete head attitude in the world coordinate system. This unit updates attitude information at a frequency of no less than 50Hz and has an angle measurement accuracy of ≤1°, providing real-time feedback data on head attitude and angular velocity to the M2 module.
[0072] It should be noted that the above hardware specifications are only preferred configurations for this embodiment, and the scope of protection of this invention is not limited to the specific device model, parameter indicators, and neck degree of freedom configuration and driving method.
[0073] Step 1: Multimodal Sensing (M1)
[0074] The system acquires three data streams in parallel at a frequency of 30Hz: (1) Visual channel: The camera captures the user's facial image, and the face detection and expression recognition model outputs the probability distribution of 7 basic emotions (anger, disgust, fear, happiness, sadness, surprise, and neutrality). ; (2) Voice channel: The microphone collects the user's voice signal, and the probability distribution of the corresponding 7 types of emotions is output by the acoustic feature extraction and voice emotion recognition model. ; (3) Semantic channel: The ASR module converts speech into text, and the text sentiment classification model outputs the probability distribution of the seven sentiment categories. .
[0075] Step 2: Real-time reading of the entity's status (M2)
[0076] The M2 module performs the following operations at a frequency of 50Hz: Read the encoder data of the neck three servo motors ; The head's attitude relative to the fuselage was obtained through kinematic calculations: ; The attitude of the host in the world frame is read from the IMU output of the host. ; via chain transformation Obtain the complete head posture; The three-degree-of-freedom angular velocity is obtained by the angle difference between two adjacent samples. ; The user's azimuth angle in the world coordinate system is calculated from the face detection results. Calculate the head orientation deviation angle .
[0077] Step 3: Confidence Assessment (M3)
[0078] The M3 module calculates the three-channel confidence level at a frequency of 30Hz: Visual confidence level:
[0079] in External quality factor (determined by illumination, detection confidence, image blur, etc.); The calibration coefficients, in this embodiment, are obtained by fitting a pre-collected calibration dataset using the least squares method, with a typical order of magnitude of [missing value]. ; Voice confidence The result is determined by SNR, VAD, and acoustic energy. Semantic confidence The value is determined by ASR confidence, text length, and semantic integrity.
[0080] Step 4: Multimodal Fusion Decision (M4)
[0081] Three-channel confidence scores are normalized to form fusion weights: Fusion probability distribution:
[0082] Pick As the final emotional judgment output.
[0083] Step 5: Feedback Action Generation (M5)
[0084] Suppose M4 determines that the user's current emotion is sadness, and M5 generates feedback action instruction A based on a preset empathy strategy. An example of its parameters is shown below (the specific values are determined by the empathy strategy library): A = { Empathy tilts head. Current yaw +15° Current pitch -5° Current Roll +10° Current time + 100 ms 800 ms 25° / s : middle } Simultaneously generate synchronized voice feedback (such as "I'm here with you") and on-screen emoticons (concerned emoticons).
[0085] Step Six: Pre-implementation Predictive Feedback (M6)
[0086] After M5 generates A, A is not directly sent to M7 for execution. Instead, it first goes through M6 for feedforward. M6 performs the following operations: (a) Calculate the interference prediction function:
[0087] in , To predict the calibration coefficients (and the measured calibration coefficients of M3) Different meanings Describe the predicted intensity-interference intensity mapping. Describes the measured motion volume-confidence decay mapping. In this example, By typical magnitude Take the value and substitute it into the parameter in this example ( =25° / s, After (=15°), we can obtain It is approximately on the order of 0.6.
[0088] (b) Weight pre-adjustment Completed within the previous 100 ms lead time (this lead time is determined by the prediction lead time of M6). Take the adjustment coefficient , , Substitute The result was: , , After normalization, the visual weights were reduced to approximately 10% of the baseline, while the speech and semantic weights increased accordingly. The specific values were adjusted from... The value is determined by the given value.
[0089] (c) Send A to the execution queue and wait. Triggered at any time.
[0090] (d) Weight smoothing recovery after the action ends: After the action ends ( Using a cosine transition function with a typical duration of approximately 500ms, the weights of each channel are smoothly restored to the baseline value:
[0091] in The recovery time is typically 500 ms in this embodiment. .
[0092] The evolution of the three-channel weights over time described in items (a) to (d) above is as follows: Figure 5 As shown. Figure 5 The text clearly shows three key sections: The system consists of a pre-weighting zone before the action, a weighting plateau zone during the action execution, and a cosine smoothing recovery zone after the action ends. The visual channel weight is reduced to about 10% of the baseline, while the speech and semantic channel weights are increased accordingly. After the action ends, the system recovers to the baseline value after a 500 ms cosine transition.
[0093] Step 7: Action Execution (M7)
[0094] M7 At any given moment, parameter A synchronously drives the three servos in the neck (including the parallel coupling solution of Pitch / Roll), performs speech synthesis, and executes on-screen facial expression feedback actions. At this point, because the channel weights in M3 are already... After the previous adjustments, the M4 fusion output is mainly dominated by the speech and semantic channels throughout the entire action execution, and the impact of the temporary quality degradation of the visual channel on the final fusion output is significantly suppressed.
[0095] Step 8: Closed-loop continuous operation
[0096] Steps one through seven above are executed continuously in a loop. When a deviation occurs in the motor's operation (e.g., a discrepancy between the actual angular velocity and the predicted value, or the presence of external disturbances), M2 senses it in real time. Changes, through Function to dynamically adjust visual confidence This allows for immediate correction of prediction deviations. This dual correction mechanism, consisting of M6 prediction pre-adjustment and M2 measured correction, enhances the system's robustness under non-ideal operating conditions.
[0097] Example 2: Continuous Closed-Loop Operation in Multi-Turn Dialogue Scenarios
[0098] This embodiment illustrates the implementation of the present invention in a continuous multi-turn dialogue scenario, focusing on the stability of the closed-loop mechanism when continuous actions are triggered.
[0099] Scenario Description: Users engage in continuous, multi-round conversations with a companion robot. Within a short conversation window, the robot needs to perform multiple feedback actions in sequence, including nodding (agreement), shaking its head (disagreement), tilting its head (empathy), and looking up (surprise). The duration and interval of each action can vary, with a typical duration on the order of several hundred milliseconds.
[0100] Key points for implementation: (1) Feedforward pre-ordering of the action queue. When M5 generates multiple actions to be executed consecutively... At that time, M6 feeds forward each action parameter to M3 sequentially. M3 maintains a weight adjustment time axis, marking the weight downsampling interval and recovery interval for each action on the time axis. If the time intervals of two adjacent actions overlap, the interference prediction value within the overlapping period is... The weights are superimposed, and a lower bound protection constraint is applied to the visual weights:
[0101] in This is the lower limit coefficient for visual weight (in this embodiment, it is set to 0.1 to prevent the visual channel from being completely blocked when continuous actions are superimposed).
[0102] (2) Weight rollback for action cancellation / interruption: If an action that has been fed forward but not yet executed is canceled / interrupted, the weight rollback will be affected. t start If the action was previously canceled by the upper-level strategy (e.g., due to a sudden interruption by the user or a change in the upper-level intent), M6 will notify M3 to revoke the weight adjustment corresponding to the action. The weight will be restored to the state before the action was triggered in the next M3 calculation cycle, ensuring that the weight trajectory is synchronized with the actual action.
[0103] (3) Utilization of the temporal continuity of emotion states. Taking advantage of the temporal continuity of emotion recognition results, M4 introduces first-order temporal smoothing in the fusion decision-making process: Typical This further ensures the stability of the fused output within a long time window.
[0104] in This represents the original fusion probability distribution obtained by weighted fusion of the three channels at the current moment. This is the output after first-order timing smoothing; ∈[0.3,0.5] is the smoothing coefficient ( The larger the value, the closer it is to the current observation; the weaker the smoothing. (Initialization) = After applying temporal smoothing, the final sentiment judgment is changed to... Output.
[0105] Example 3: Lightweight Implementation of Low-Computing-Power Embedded Platform
[0106] For embedded deployment scenarios with limited computing power (such as mobile robot main controllers using low-to-mid-range SoCs / MCUs), the core modules of this invention, such as M3 and M6, can be implemented in a lightweight manner to reduce computational overhead while maintaining the core closed-loop mechanism: (1) Table lookup implementation of the interference prediction function The calculation is changed from real-time function operation to table lookup: an interference prediction lookup table is pre-established offline according to different action types (nodding, shaking, tilting, raising, etc.) and different amplitude levels (small, medium, large). When feeding forward, M6 directly looks up the table by {action type, amplitude level} to obtain the interference value sequence, avoiding the overhead of floating-point operations such as exponential functions, which can significantly reduce the single calculation time of M6.
[0107] (2) Piecewise linear approximation of the self-perturbation penalty function
[0108] Will The exponential function is approximated by a piecewise linear function: when hour, ;when hour, ;when hour, .in , The segmentation threshold, , For piecewise slopes, The lower confidence limit protection value (to prevent) (The visual channel is completely blocked when the visual signal decays to zero under intense motion). This approximation can be replaced by a lookup table and a single multiply-accumulate operation, further reducing the computational burden on embedded platforms.
[0109] (3) Channel confidence calculation frequency downsampling
[0110] The confidence evaluation frequency of M3 is reduced from 30 Hz to 10 Hz (or other low-frequency levels suitable for computing power), and the channel weights at intermediate moments are obtained through linear interpolation. Given that the temporal rate of change in human emotional states is typically much lower than the M3 computation frequency, the impact of this downsampling on the system response is negligible, but the computational overhead of M3 can be reduced proportionally to the frequency.
[0111] (4) Quantitative deployment of multimodal models
[0112] By unifying the visual, speech, and semantic recognition models to INT8 precision (or using other low-bit formats such as INT4 / FP16 depending on platform support), and in conjunction with hardware acceleration units such as NPU / DSP, the total time for single-frame multimodal perception can be controlled within M1 sampling period, meeting the requirements for real-time perception.
[0113] Example 4: Implementation methods under different neck degree-of-freedom configurations
[0114] The core mechanism of this invention (predictive feedback before execution + proprioceptive motion state feedback) is independent of the specific number and drive configuration of the neck degrees of freedom. The following examples illustrate implementation differences under several typical neck structures to demonstrate the structural independence of this invention and support the corresponding dependent claims.
[0115] (1) 2-DOF neck (Yaw + Pitch): Roll-related terms are omitted, and the proprioceptive motion penalty factor is simplified to:
[0116] in The kinematic solution for M2 is simplified to a two-degree-of-freedom form, while the procedures for M4 to M7 remain completely unchanged.
[0117] (2) 3-DOF independent drive (each degree of freedom corresponds to an independent servo): The kinematic solution in M2 is simplified from the parallel coupling solution in Example 1 to a one-to-one mapping:
[0118] The processing flow of the remaining modules (M1, M3~M7) is completely consistent with that of Embodiment 1, demonstrating the compatibility of the present invention with different neck drive methods.
[0119] (3) Gimbal-mounted 2-DOF (Yaw + Pitch, camera mounted on gimbal): The head concept in this invention is equivalently replaced by a gimbal. Replace with The remaining mechanisms are completely identical. This implementation method is applicable to application scenarios with motion camera platform structures, such as security patrol robots, live streaming robots, drone pods, and vehicle-mounted gimbal monitoring.
[0120] (4) High-DOF humanoid robot neck (≥4 DDOF, such as humanoid multi-joint neck): The body motion penalty factor is expanded to a weighted sum of all DDOFs:
[0121] in The total number of degrees of freedom of the neck. For the first The current angle of one degree of freedom, These are the corresponding calibration coefficients. The kinematic solution of M2 is correspondingly extended to an N-degree-of-freedom chain transformation, while the remaining mechanisms of M3~M7 remain unchanged.
[0122] The above four types of neck structure variations are all equivalent implementations of the core mechanism of this invention and all fall within the protection scope of this invention. Those skilled in the art can select the appropriate M2 kinematics calculation method based on the specific mechanical structure of the robot platform. The parameterized form is used, and the processing flow of M1, M3~M7 does not require any changes.
[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0124] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multi-modal affective recognition closed-loop system based on body motion perception and execution of feedforward, characterized in that, It includes a multimodal perception module, an ontology state perception module, a confidence assessment module, a multimodal fusion decision module, a feedback action generation module, a pre-execution predictive feedback module, and an execution module; The multimodal perception module is used to acquire visual, speech, and semantic perception signals in parallel, and output the probability distribution of the emotion category for each corresponding channel. The body state perception module is used to collect the motion parameters of the robot's neck actuator and the body posture parameters in real time. After kinematic calculation, it outputs a set of body motion states consisting of the complete head posture, head angular velocity, and user orientation deviation. The input of the confidence assessment module is connected to the multimodal perception module and the ontology state perception module respectively, and is used to calculate the real-time confidence of each channel based on the external quality factor and the ontology motion penalty factor. The input of the multimodal fusion decision module is connected to the confidence evaluation module, which is used to normalize the confidence of each channel into fusion weights, perform weighted fusion of the three sentiment probability distributions, and output the final sentiment judgment result. The input end of the feedback action generation module is connected to the multimodal fusion decision module, and is used to generate a set of feedback action parameters based on the emotion judgment result. The set of feedback action parameters includes at least the action type, target angle, start time, duration and maximum angular velocity. The input end of the pre-execution predictive feedback module is connected to the feedback action generation module, and the output end is fed forward to the confidence evaluation module. It is used to calculate the interference prediction value of the visual channel based on the action parameter set before the actual execution of the feedback action, and to make predictive pre-adjustment of the channel fusion weight at the next moment based on the interference prediction value. The input terminal of the execution module is connected to the feedback action generation module, and is used to synchronously drive the neck actuator, voice feedback unit and display feedback unit to perform corresponding feedback actions at a preset start time.
2. The multi-modal affective recognition closed-loop system based on body motion perception and execution feedforward according to claim 1, wherein, The specific set of body motion states is as follows: wherein, are the head yaw, pitch, roll three-axis angles, respectively, are the corresponding three-axis angular velocities, are the body attitude angles and body angular velocities, is the action target angle, is the action remaining execution time; the head attitude is obtained by chain coordinate transformation after the neck encoder data and the body IMU data are timestamped.
3. The multi-modal affect recognition closed loop system based on body motion perception and execution feedforward according to claim 2, wherein, The formula for calculating the confidence score of the visual channel by the confidence assessment module is as follows: wherein, is an external quality factor based on the illumination intensity, the face detection confidence, and the image blurriness; is a body motion penalty factor with a value range of (0, 1) and used to represent the attenuation degree of the body self-disturbance on the visual recognition quality; the voice channel confidence is determined by the voice signal-to-noise ratio, the voice activity detection result, and the acoustic energy; the semantic channel confidence is determined by the voice recognition confidence, the text length, and the semantic integrity, and neither of the two introduces the body motion penalty factor.
4. A multimodal emotion recognition closed-loop system based on ontology motion perception and execution feedforward as described in claim 3, characterized in that, The body motion penalty factor Using exponential function form: in, For the head's synthesized angular velocity, The angle of deviation between the head orientation and the user's orientation. The pitch and rotation angle, For the tumble rotation angle, - The penalty coefficient is obtained by fitting the calibration dataset.
5. A multimodal emotion recognition closed-loop system based on ontology motion perception and execution feedforward as described in claim 1, characterized in that, The pre-execution predictive feedback module performs the following operations: Constructing a visual interference prediction function based on action parameter set The The value range is [0,1], and it is positively correlated with the amplitude of the movement and the maximum angular velocity. Within the advance window before the start of the action, according to Feedforward adjustment of the weights for each channel: in, , , For baseline fusion weights, , , This is the adjustment coefficient; The adjusted weights, after normalization, are used as the fusion weights during action execution.
6. A multimodal emotion recognition closed-loop system based on ontology motion perception and execution feedforward according to claim 5, characterized in that, After the action is completed, the confidence assessment module smoothly restores the channel weights from the preset values to the baseline values through a monotonic transition function; the monotonic transition function includes a cosine transition function, an exponential decay function, or a linear ramp function.
7. A multimodal emotion recognition closed-loop system based on ontology motion perception and execution feedforward as described in claim 1, characterized in that, The neck actuator is a servo drive structure with 2 to 6 degrees of freedom, and adopts an independent drive mode or a multi-servo parallel coupling drive mode.
8. A multimodal emotion recognition closed-loop method based on ontology motion perception and execution feedforward, applied to the multimodal emotion recognition closed-loop system based on ontology motion perception and execution feedforward as described in any one of claims 1-7, characterized in that, include: S1. Parallel acquisition of visual, speech, and semantic signals, and outputting the probability distribution of sentiment categories for each channel respectively; S2. Real-time acquisition of neck encoder data and body IMU data, kinematic calculation and coordinate transformation to obtain head posture, head angular velocity and user orientation deviation, and output the body motion state set; S3. Integrate external quality factors and ontological motion penalty factors to calculate the real-time confidence scores of the visual, speech, and semantic channels; if there is an execution feedforward signal, perform predictive pre-adjustment of the channel weights based on the interference prediction value. S4. Normalize the confidence of each channel into a fusion weight, weight and fuse the three sentiment probability distributions, and output the final sentiment judgment result. S5. Generate a set of feedback action parameters, including action type, target angle, and execution sequence, based on the emotion judgment result; S6. Feed the feedback action parameter set to the confidence assessment stage, calculate the visual channel interference prediction value, complete the predictive adjustment of the weights at the next moment, and send the action parameters into the execution queue. S7. Synchronously drive the neck actuator, voice feedback unit and display feedback unit to perform feedback actions at the preset start time; S8. Repeat S1~S7. During the execution of the action, the prediction deviation is dynamically corrected by the real-time ontology state. After the action is completed, the channel weights are smoothly restored to the baseline value.
9. A multimodal emotion recognition closed-loop method based on ontology motion perception and execution feedforward according to claim 8, characterized in that, In step S6, when there are multiple consecutive actions to be executed, the weight adjustment intervals corresponding to each action are marked in sequence. When the intervals overlap, the interference prediction values are superimposed, and the visual weight lower limit protection constraint is maintained. When the feedforward action is canceled, the corresponding weight adjustment is revoked, and the weight is rolled back to the state before the action was triggered.
