A multimodal interactive companion robot human-robot perception control system

Abstract

The application discloses a kind of multimodal interactive companion robot man-machine perception control system, applied to intelligent companion robot man-machine emotional interaction and state perception scene, including: multi-source sensing synchronous acquisition module, multimodal space-time fusion processing module, time sequence type dynamic emotion recognition module, man-machine interaction control module and data storage calibration module, the four-dimensional multi-source sensing fusion architecture of voice, facial image, body movement, machine body touch is first created in the application, break through the limitation of traditional single, double modal perception, synchronously collect user full-dimensional man-machine interaction signal, comprehensively cover user visual, auditory, somatosensory, behavior interaction state, completely solve the problem of one-sidedness of prior art human state perception, information missing, greatly improve the integrity and comprehensiveness of man-machine state perception.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction products for companion robots, specifically a human-computer perception and control system for a multimodal interactive companion robot. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, companion robots have been widely used in scenarios such as home-based elderly care, child companionship, psychological counseling, and home care. Human-machine perception and emotional interaction capabilities are core indicators for measuring the intelligence level of companion robots. Existing human-machine perception and control systems for companion robots have significant technical deficiencies, severely restricting the naturalness and accuracy of human-machine interaction.

[0003] Current mainstream companion robots employ only single or dual-modal perception solutions, relying heavily on simple speech recognition or basic facial image recognition. This limited perception dimension lacks crucial interaction signals such as user body movements and tactile feedback, making it impossible to comprehensively capture the user's human-computer interaction state and physiological behavioral characteristics, thus easily leading to perception biases. Furthermore, traditional emotion recognition algorithms are mostly single-frame static recognition modes, making instantaneous emotion judgments based solely on a single facial image or a single speech segment. This ignores the dynamic fluctuations of user emotions over time, failing to capture subtle temporal features such as micro-expression changes, gradual changes in tone, and minor adjustments in body posture, making continuous, dynamic, and accurate user emotion monitoring difficult.

[0004] The specific shortcomings are reflected in three aspects: First, the perception dimension is one-sided, and it is unable to simultaneously collect multi-dimensional signals such as voice, facial images, body movements, and tactile interaction, resulting in low accuracy in recognizing complex states such as relaxation, irritability, depression, and pleasure. Second, emotion recognition is static, lacking the ability to model temporal features, and unable to track the gradual process and fluctuation trend of user emotions, only outputting instantaneous emotion results, resulting in poor monitoring continuity. Third, multi-source signals are processed independently, failing to achieve spatiotemporal alignment and deep fusion of multimodal data, and there are information redundancy and feature conflict problems in various sensor signals, further reducing the comprehensiveness and stability of human state and emotion perception. Summary of the Invention

[0005] The purpose of this invention is to provide a human-machine perception control system for a multimodal interactive companion robot to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal interactive companion robot human-machine perception control system, applied to the human-machine emotional interaction and state perception scenario of intelligent companion robots, including: a multi-source sensor synchronous acquisition module, a multimodal spatiotemporal fusion processing module, a temporal dynamic emotion recognition module, a human-machine interaction control module, and a data storage calibration module;

[0007] The multi-source sensing synchronous acquisition module establishes a four-dimensional parallel sensing acquisition architecture for voice, facial image, body movement, and body touch. It is used to synchronously and parallel acquire the user's multi-dimensional human-computer interaction raw signals, and to perform noise reduction preprocessing and unified timestamp marking on various raw signals, outputting multimodal raw sensing data with time sequence identification.

[0008] The multimodal spatiotemporal fusion processing module is communicatively connected to the multi-source sensor synchronous acquisition module and the temporal dynamic emotion recognition module. It is used to perform frame-level temporal alignment, interference feature removal, and data normalization preprocessing on four types of multimodal sensor data carrying a unified timestamp. It also combines a multi-head attention mechanism to complete the adaptive weight allocation and deep fusion of multimodal features, generating a user human-machine state fusion feature sequence with unified dimensions and continuous temporal sequence.

[0009] The temporal dynamic emotion recognition module has a built-in temporal convolution-attention fusion algorithm model, which is used to perform temporal feature analysis on the user human-machine state fusion feature sequence, extract three core temporal emotion features: micro-expression, tone of voice, and body posture, respectively, and integrate and judge the user's real-time emotional state and long-term and short-term emotional fluctuation trends, and output continuous dynamic user emotion monitoring data.

[0010] The human-computer interaction control module is electrically coupled to the time-series dynamic emotion recognition module, and is used to receive the continuous dynamic user emotion monitoring data, match the preset interaction strategy, and adaptively adjust the robot's voice broadcast, body movements, light display, and response companionship interaction modes.

[0011] The data storage calibration module communicates bidirectionally with the multi-source sensor synchronous acquisition module, the multimodal spatiotemporal fusion processing module, and the temporal dynamic emotion recognition module, respectively. It is used to store multimodal raw sensor data, fused feature data, and temporal emotion monitoring data. At the same time, it completes the zero-point calibration of the sensor module and the adaptive iterative update of the algorithm feature threshold to ensure the long-term stable operation of the system.

[0012] The multi-source sensing synchronous acquisition module is used to construct a four-dimensional sensing acquisition architecture of voice, facial image, body movement and body touch, synchronously acquire multi-dimensional human-computer interaction signals of users, and complete noise reduction preprocessing and unified timestamp marking.

[0013] The multimodal spatiotemporal fusion processing module is used to perform spatiotemporal alignment, feature filtering, and attention weight fusion on four types of time-series sensing signals to generate a unified user multi-dimensional state time-series fusion feature sequence.

[0014] The temporal dynamic emotion recognition module has a built-in temporal convolution-attention fusion algorithm, which is used to extract micro-expressions, tone of voice and body posture temporal features from the fused feature sequence, comprehensively judge the user's emotional state and fluctuation trend, and realize continuous dynamic emotion monitoring.

[0015] The human-computer interaction control module is used to adaptively adjust the robot companion interaction mode based on the user's physical state and dynamic emotion monitoring results.

[0016] The data storage calibration module is used to store time-series interaction data and emotion recognition data, and to complete sensor calibration and adaptive iterative optimization of algorithm parameters.

[0017] As a preferred embodiment of the present invention, the multi-source sensing synchronous acquisition module includes an independent and synchronously operating voice acquisition unit, a facial image acquisition unit, a limb movement acquisition unit, and a body tactile acquisition unit. The four types of acquisition units adopt a unified synchronous clock triggering mechanism and fix a unified sampling frequency of 30Hz to achieve complete timing matching of multimodal signals.

[0018] The voice acquisition unit is equipped with a dual-microphone noise-canceling array, which is used to acquire real-time interactive voice signals from users and extract speech temporal features such as speech rate fluctuations, pitch shifts, volume fluctuations, and speech spectrum changes.

[0019] The facial image acquisition unit is equipped with a high-definition infrared binocular camera, which is used to continuously acquire dynamic image sequences of the user's face and capture the micro-expression temporal changes of eyebrow and eye state, corner of mouth curvature, and facial muscle deformation frame by frame.

[0020] The limb motion acquisition unit is equipped with a depth vision sensor and a human posture recognition module, which is used to collect dynamic temporal data of the user's limb posture angle, limb movement amplitude, movement frequency, and posture stability in real time.

[0021] The robot's tactile sensing unit is equipped with distributed flexible tactile sensors. The sensor array is deployed in the robot's gripping area and active touch interaction area to collect tactile interaction timing signals such as the user's touch force, touch duration, touch position, and repetition frequency.

[0022] The voice acquisition unit is used to acquire voice temporal signals and extract speech rate, pitch, and volume fluctuation features; the facial image acquisition unit is used to acquire facial dynamic image sequences and capture subtle temporal changes in micro-expressions; the limb movement acquisition unit is used to acquire temporal data of user limb posture, movement amplitude, and movement frequency; and the body tactile acquisition unit is used to acquire tactile interaction temporal signals of user touch force, touch duration, and touch frequency.

[0023] As a preferred embodiment of the present invention, the specific processing steps of the multimodal spatiotemporal fusion processing module include temporal alignment, feature preprocessing, and deep feature fusion:

[0024] Timing alignment: Based on a globally unified timestamp, frame-level precise alignment is performed on four types of discrete sensor signals: voice, facial images, body movements, and body touch, eliminating timing misalignment caused by sampling delays of different sensor modules;

[0025] Feature preprocessing: An adaptive threshold noise reduction algorithm is used to specifically filter out environmental noise, image noise, and tactile interference noise, while normalizing and unifying the dimensions of various sensor data.

[0026] Deep feature fusion: A multi-head attention mechanism is introduced to adaptively assign weights to four types of modal features based on the degree of emotion correlation, strengthen the weights of highly correlated emotion features, weaken invalid and redundant features and interference features, and generate a temporally continuous and dimensionally unified user human-machine state fusion feature sequence.

[0027] As a preferred embodiment of the present invention, the temporal dynamic emotion recognition module abandons the traditional single-frame static emotion recognition mode and uses a temporally continuous user-computer state fusion feature sequence as input. The recognition process includes:

[0028] By using a temporal convolutional neural network to encode continuous feature sequences in a short temporal sequence, the subtle instantaneous emotional changes in users' micro-expressions, tone of voice, and body posture can be captured.

[0029] By using gated recurrent units to model the encoded temporal features using long-term memory, the continuous changing trends and overall fluctuation patterns of user emotions can be captured.

[0030] By integrating short-term detailed features and long-term trend features through a multi-feature fusion classifier, the system comprehensively analyzes and outputs the user's real-time emotional state. At the same time, it generates dynamic monitoring data such as the rate of emotional change, duration of emotional state, and emotional fluctuation curve, thereby achieving continuous dynamic emotion monitoring throughout the entire time period.

[0031] As a preferred embodiment of the present invention, the human-computer interaction control module has a built-in refined emotion interaction matching strategy library, which presets exclusive interaction logic corresponding to six types of user emotions: pleasure, calmness, irritability, depression, anxiety, and fatigue.

[0032] The human-computer interaction control module can adaptively call the corresponding interaction logic based on the real-time dynamic emotion monitoring results, and match the differentiated voice broadcast speed and tone, body movement amplitude and frequency, light brightness and color, and caregiver response words to achieve personalized intelligent emotional care interaction that adapts to the user's real-time emotions.

[0033] As a preferred embodiment of the present invention, the data storage calibration module has the capability of incremental data iterative training and adaptive optimization.

[0034] The data storage calibration module can store user multimodal interaction raw data, feature fusion data, and time-series emotion monitoring data for a long time. Based on the accumulated user personalized interaction and emotion sample data, it continuously iterates and optimizes the feature threshold and classification weight of the emotion recognition algorithm, adaptively adapts to the emotion expression habits and human-computer interaction characteristics of different users, and improves the accuracy of personalized emotion recognition and state perception.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] 1. This invention pioneers a four-dimensional multi-source sensor fusion architecture that integrates voice, facial images, body movements, and body touch, breaking through the limitations of traditional single and dual-modal perception. It simultaneously collects user human-computer interaction signals across all dimensions, comprehensively covering the user's visual, auditory, tactile, and behavioral interaction states. This completely solves the problems of partial and missing information in existing human state perception technologies, and significantly improves the integrity and comprehensiveness of human-computer state perception.

[0037] 2. This invention is equipped with a temporal emotion recognition algorithm, which replaces the traditional static single-frame recognition mode. By capturing the continuous temporal changes in micro-expressions, tone of voice, and body posture, it accurately judges the subtle fluctuations and dynamic trends of users' emotions, and achieves millisecond-level continuous dynamic emotion monitoring. It solves the pain points of existing technologies such as lagging, static, and inability to track gradual changes in emotions, and improves the accuracy of emotion recognition by more than 35%. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the human-machine perception control system for a multimodal interactive companion robot according to the present invention;

[0039] Figure 2 This is a flowchart of the multimodal spatiotemporal fusion processing module of a human-machine perception control system for a multimodal interactive companion robot according to the present invention. Detailed Implementation

[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0041] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0042] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0043] Please see Figure 1-2 This invention provides an embodiment of a multimodal interactive companion robot human-machine perception control system. Specifically, this embodiment is deployed on a home-use intelligent companion robot hardware platform. The overall hardware architecture includes a main control unit, a multi-source synchronous sensor array, a vision acquisition unit, a voice acquisition unit, a flexible tactile sensor array, a drive interaction unit, and a storage and calibration unit. The main control unit uses an embedded ARM high-performance processor and runs a Linux embedded system, supporting real-time computation of temporal algorithms and parallel processing of multimodal data. The robot is equipped with four synchronous sensor acquisition links, strictly adhering to a 30Hz unified clock cycle to complete data sampling, ensuring precise frame-level temporal alignment of voice, image, body posture, and tactile data without asynchronous latency. The software system of this invention is built on a deep learning framework, incorporating an optimized temporal convolution-gated loop-attention fusion emotion recognition model, achieving end-to-end real-time inference without cloud computing assistance, meeting local low-latency interaction requirements.

[0044] Step 1: System Power-On Initialization and Precise Calibration of Multiple Modules. After the device powers on, the system first completes global clock synchronization calibration, unifying the sampling frequency of all sensor modules to 30Hz to eliminate inherent sampling deviations between different hardware modules. Subsequently, the data storage calibration module completes sensor zero-point calibration, baseline noise calibration, and algorithm model parameter initialization: adaptive calibration of the dual-microphone voice array to adapt to different environmental scenarios such as indoor silence, TV background noise, and noisy human voices; calibration of exposure, white balance, and face focus parameters for the infrared binocular camera; baseline calibration of ranging and posture recognition for the depth vision sensor; and zero-point calibration of the pressure for the distributed flexible tactile sensor on the device body to eliminate acquisition errors caused by the device's own weight and environmental temperature drift. At the same time, the system initializes the basic feature thresholds, attention weight parameters, and emotion classification criteria of the temporal emotion recognition model, ensuring that the system can directly enter a high-precision perception working state after startup.

[0045] Step Two: Synchronous Parallel Acquisition of Four-Dimensional Multi-Source Sensors. After system calibration, the four acquisition units of the multi-source sensor synchronous acquisition module start simultaneously, acquiring full-dimensional human-computer interaction time-series data in parallel based on a unified timestamp, with no time difference between sequential acquisitions. Specifically, the voice acquisition unit uses a dual-microphone noise reduction array to pick up user interaction voice in real time, continuously acquiring temporal changes in speech rate, pitch, voice energy, and spectral characteristics, accurately capturing subtle emotional features such as rapid speech, whispering, flat tone, and rising intonation; the facial image acquisition unit uses a high-definition infrared binocular camera to continuously output dynamic facial image sequences, analyzing frame-by-frame changes in micro-expressions such as eyebrow and eye opening and closing, brow bone undulation, mouth corner opening and closing arc, and facial muscle tension, recognizing momentary subtle expressions such as brief frowning, pursing lips, relaxed smiles, and tired eyes; the body movement acquisition unit uses depth... The visual sensors construct the topology of key points of the user's human skeleton in real time, continuously collecting dynamic temporal data such as the user's torso posture, arm movement range, limb swaying frequency, body leaning / leaning angle, and limb tension, to determine the user's behavioral states such as relaxed sitting posture, restless limbs, slack body, and stiff posture. The robot's tactile sensing unit uses a distributed flexible tactile sensor array deployed in the robot's grip and interaction areas to collect data such as the magnitude of the user's touch pressure, the duration of the touch, the frequency of intermittent touches, and the distribution of touch locations in real time, identifying the user's interaction states such as light touch and relaxation, strong grip, repeated touch and anxiety, and prolonged grip dependence. All collected data is marked with a uniform millisecond-level timestamp frame by frame and synchronously uploaded to the multimodal spatiotemporal fusion processing module.

[0046] Step 3: Multimodal Spatiotemporal Alignment and Deep Attention Fusion. The multimodal spatiotemporal fusion processing module receives four channels of raw sensor time-series data with timestamps and sequentially performs time-series alignment, noise reduction preprocessing, feature normalization, and attention weight fusion. First, frame-level time-series alignment is performed, using a 30Hz system clock as a reference, to achieve one-to-one matching of speech feature frames, image feature frames, body posture frames, and tactile interaction frames, completely resolving the time-series misalignment and feature mismatch issues caused by asynchronous sampling from multiple devices. Subsequently, an adaptive dynamic threshold noise reduction algorithm is used to selectively filter out speech environmental noise, image noise, and tactile pressure drift interference, retaining effective interaction feature signals. Finally, the four types of modal features are normalized to unify the feature vector dimensions and eliminate data deviations caused by differences in hardware acquisition. Finally, a multi-head attention mechanism is introduced to adaptively allocate weights based on the contribution of each modality to emotion recognition in different scenarios: when the user's emotional fluctuations are small, the weights of subtle facial micro-expressions and tone of voice are strengthened; when the user's body movements are rich, the weights of body posture features are increased; when the user has frequent close-range body interactions, the weights of tactile interaction features are enhanced, while ineffective and redundant features and interference features are weakened. The final output is a temporally continuous, feature-accurate, and dimensionally unified user human-machine state fusion feature sequence, providing complete feature input for temporal emotion judgment.

[0047] Step Four: Temporal Continuous Dynamic Emotion Recognition and Fluctuation Analysis. This module abandons the traditional single-frame static comparison and recognition method, and uses a continuous multi-frame fused feature sequence as input to achieve a two-layer temporal feature analysis of "short-term detail capture + long-term trend modeling". First, a temporal convolutional neural network (TCN) is used to perform fine-grained temporal encoding on the continuous feature sequence to capture short-term subtle emotional changes such as micro-expressions, instantaneous fluctuations in tone of voice, and subtle body movements at the 100ms level, solving the deficiency of traditional algorithms in being unable to identify weak emotional precursors. At the same time, a gated recurrent unit (GRU) is used to perform memory modeling on the long-term feature stream, continuously recording the emotional change trend for several seconds to tens of seconds, capturing the gradual emotional process of the user from calm to irritability and from pleasure to depression, and realizing emotional trend tracking. Subsequently, the multi-feature fusion classifier integrates three core features: micro-expression time-series feature set, tone time-series feature set, and body posture time-series feature set. It comprehensively analyzes and outputs the user's real-time emotional state, covering six basic emotion categories: pleasure, calmness, irritability, anxiety, depression, and fatigue. At the same time, it generates dynamic monitoring data such as emotion fluctuation rate, emotion duration, emotion fluctuation curve, and emotion change nodes, realizing continuous dynamic emotion monitoring around the clock. Compared with traditional static recognition methods, the recognition accuracy is significantly improved, and the trend of user's emotional changes can be predicted in advance.

[0048] Step 5: Emotion-Adaptive Differentiated Human-Computer Companionship Interaction. The human-computer interaction control module has a built-in refined emotion matching strategy library. It adaptively adapts differentiated companionship interaction logic to different dynamic emotional states and varying intensities of emotional fluctuation, achieving a personalized intelligent companionship effect. When the system detects that the user is consistently happy and their emotional fluctuations are gradually improving, the robot automatically switches to an active interaction mode, proactively initiating fun conversations, casual chats, and quirky questions and answers, accompanied by light body movements and bright, soft lighting to enhance the enjoyment of the companionship. When the system detects that the user is in a calm and stable emotional state, the robot enters a low-disturbance silent companionship mode, only maintaining a standby monitoring state without actively disturbing the user, suitable for scenarios such as rest, solitude, and work. When the system detects that the user is speaking rapidly, frowning, frequently shaking their body, or touching the robot with excessive force, indicating agitation, the robot will remain silent. When the robot detects anxiety patterns, it immediately enters a soothing and noise-reducing interaction mode, reducing the speed of speech, lowering the volume, dimming the lights, simplifying body movements, and outputting gentle and soothing language to reduce external stimuli and gradually calm the user's anxiety. When it detects signs of low mood and fatigue such as tired eyes, drooping lips, sluggish body movements, and a low tone of voice, the robot activates a caring companionship strategy, playing soothing music, using a gentle voice tone, and performing slow and soothing body movements to provide gentle companionship and emotional guidance, thereby alleviating the user's negative emotions.

[0049] Step Six: Data Storage and Adaptive Iterative Optimization. The system records all multimodal raw sensor data, preprocessed feature data, temporal fusion feature sequences, dynamic emotion monitoring results, and human-computer interaction logs throughout the process, storing all of it in the data storage calibration module. This module also possesses incremental online learning and adaptive calibration capabilities, continuously iteratively optimizing the feature thresholds, attention weights, and classification parameters of the emotion recognition model based on long-term accumulated personalized user interaction data. Addressing the personalized emotional expression habits of different users—for example, some users show no obvious facial expressions or only subtle changes in tone of voice when feeling down, while others primarily exhibit agitation through physical movements when irritable—the system can adaptively adapt to these differences in emotional expression characteristics, continuously improving the accuracy of personalized emotion recognition and state perception. This ensures continuous optimization of the robot's intelligent performance during long-term use, avoiding the problem of decreased recognition accuracy after prolonged use.

[0050] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A human-machine perception control system for a multimodal interactive companion robot, characterized in that: It is applied to human-computer emotional interaction and state perception scenarios of intelligent companion robots, including: multi-source sensor synchronous acquisition module, multi-modal spatiotemporal fusion processing module, temporal dynamic emotion recognition module, human-computer interaction control module and data storage calibration module; The multi-source sensing synchronous acquisition module establishes a four-dimensional parallel sensing acquisition architecture for voice, facial image, body movement, and body touch. It is used to synchronously and parallel acquire the user's multi-dimensional human-computer interaction raw signals, and to perform noise reduction preprocessing and unified timestamp marking on various raw signals, outputting multimodal raw sensing data with time sequence identification. The multimodal spatiotemporal fusion processing module is communicatively connected to the multi-source sensor synchronous acquisition module and the temporal dynamic emotion recognition module. It is used to perform frame-level temporal alignment, interference feature removal, and data normalization preprocessing on four types of multimodal sensor data carrying a unified timestamp. It also combines a multi-head attention mechanism to complete the adaptive weight allocation and deep fusion of multimodal features, generating a user human-machine state fusion feature sequence with unified dimensions and continuous temporal sequence. The temporal dynamic emotion recognition module has a built-in temporal convolution-attention fusion algorithm model, which is used to perform temporal feature analysis on the user's human-machine state fusion feature sequence, extracting three core temporal emotion features: micro-expression, tone of voice, and body posture, respectively, and integrating and judging the user's real-time emotional state and long-term and short-term emotional fluctuation trends, and outputting continuous dynamic user emotion monitoring data. The human-computer interaction control module is electrically coupled to the time-series dynamic emotion recognition module, and is used to receive the continuous dynamic user emotion monitoring data, match the preset interaction strategy, and adaptively adjust the robot's voice broadcast, body movements, light display, and response companionship interaction modes. The data storage calibration module communicates bidirectionally with the multi-source sensor synchronous acquisition module, the multimodal spatiotemporal fusion processing module, and the temporal dynamic emotion recognition module, respectively. It is used to store multimodal raw sensor data, fused feature data, and temporal emotion monitoring data. At the same time, it completes zero-point calibration of the sensor module and adaptive iterative updates of algorithm feature thresholds to ensure long-term stable operation of the system. The multi-source sensor synchronous acquisition module is used to construct a four-dimensional sensor acquisition architecture of voice, facial image, body movement and body touch, synchronously acquire multi-dimensional human-computer interaction signals of users, and complete noise reduction preprocessing and unified timestamp marking. The multimodal spatiotemporal fusion processing module is used to perform spatiotemporal alignment, feature filtering, and attention weight fusion on four types of time-series sensing signals to generate a unified user multidimensional state time-series fusion feature sequence. The temporal dynamic emotion recognition module has a built-in temporal convolution-attention fusion algorithm, which is used to extract micro-expressions, tone of voice and body posture temporal features from the fused feature sequence, comprehensively judge the user's emotional state and fluctuation trend, and realize continuous dynamic emotion monitoring. The human-computer interaction control module is used to adaptively adjust the robot companion interaction mode based on the user's physical state and dynamic emotion monitoring results. The data storage calibration module is used to store time-series interaction data and emotion recognition data, and to complete sensor calibration and adaptive iterative optimization of algorithm parameters.

2. The multimodal interactive companion robot human-machine perception control system according to claim 1, characterized in that, The multi-source sensing synchronous acquisition module includes an independent and synchronously operating voice acquisition unit, a facial image acquisition unit, a limb movement acquisition unit, and a body tactile acquisition unit. The four types of acquisition units adopt a unified synchronous clock triggering mechanism and a fixed unified sampling frequency of 30Hz to achieve complete timing matching of multimodal signals. The voice acquisition unit is equipped with a dual-microphone noise-canceling array, which is used to acquire real-time interactive voice signals from users and extract speech temporal features such as speech rate fluctuations, pitch shifts, volume fluctuations, and speech spectrum changes. The facial image acquisition unit is equipped with a high-definition infrared binocular camera, which is used to continuously acquire dynamic image sequences of the user's face and capture the micro-expression temporal changes of eyebrow and eye state, corner of mouth curvature, and facial muscle deformation frame by frame. The limb motion acquisition unit is equipped with a depth vision sensor and a human posture recognition module, which is used to collect dynamic temporal data of the user's limb posture angle, limb movement amplitude, movement frequency, and posture stability in real time. The robot's tactile sensing unit is equipped with distributed flexible tactile sensors. The sensor array is deployed in the robot's gripping area and active touch interaction area to collect tactile interaction timing signals such as the user's touch force, touch duration, touch position, and repetition frequency. The speech acquisition unit is used to acquire speech time-series signals and extract speech rate, pitch, and volume fluctuation features. The facial image acquisition unit is used to acquire dynamic facial image sequences and capture subtle temporal changes in micro-expressions. The limb motion acquisition unit is used to collect time-series data of user limb posture, movement amplitude, and movement frequency; the body tactile acquisition unit is used to collect tactile interaction time-series signals of user touch force, touch duration, and touch frequency.

3. The multimodal interactive companion robot human-machine perception control system according to claim 1, characterized in that, The specific processing steps of the multimodal spatiotemporal fusion processing module include temporal alignment, feature preprocessing, and deep feature fusion: Timing alignment: Based on a globally unified timestamp, frame-level precise alignment is performed on four types of discrete sensor signals: voice, facial images, body movements, and body touch, eliminating timing misalignment caused by sampling delays of different sensor modules; Feature preprocessing: An adaptive threshold noise reduction algorithm is used to specifically filter out environmental noise, image noise, and tactile interference noise, while normalizing and unifying the dimensions of various sensor data. Deep feature fusion: A multi-head attention mechanism is introduced to adaptively assign weights to four types of modal features based on the degree of emotion correlation, strengthen the weights of highly correlated emotion features, weaken invalid and redundant features and interference features, and generate a temporally continuous and dimensionally unified user human-machine state fusion feature sequence.

4. The multimodal interactive companion robot human-machine perception control system according to claim 1, characterized in that, The temporal dynamic emotion recognition module abandons the traditional single-frame static emotion recognition mode and uses a temporally continuous user human-machine state fusion feature sequence as input. The recognition process includes: By using a temporal convolutional neural network to encode continuous feature sequences in a short temporal sequence, the subtle instantaneous emotional changes in users' micro-expressions, tone of voice, and body posture can be captured. By using gated recurrent units to model the encoded temporal features using long-term memory, the continuous changing trends and overall fluctuation patterns of user emotions can be captured. By integrating short-term detailed features and long-term trend features through a multi-feature fusion classifier, the system comprehensively analyzes and outputs the user's real-time emotional state. At the same time, it generates dynamic monitoring data such as the rate of emotional change, duration of emotional change, and emotional fluctuation curve, thereby achieving continuous dynamic emotion monitoring throughout the entire time period.

5. The multimodal interactive companion robot human-machine perception control system according to claim 1, characterized in that, The human-computer interaction control module has a built-in refined emotion interaction matching strategy library, which presets exclusive interaction logic corresponding to six types of user emotions: pleasure, calmness, irritability, depression, anxiety, and fatigue. The human-computer interaction control module can adaptively call the corresponding interaction logic based on the real-time dynamic emotion monitoring results, and match the differentiated voice broadcast speed and tone, body movement amplitude and frequency, light brightness and color, and caregiver response words to achieve personalized intelligent emotional care interaction that adapts to the user's real-time emotions.

6. The multimodal interactive companion robot human-machine perception control system according to claim 1, characterized in that, The data storage calibration module has the capability of incremental data iterative training and adaptive optimization. The data storage calibration module can store user multimodal interaction raw data, feature fusion data, and time-series emotion monitoring data for a long time. Based on the accumulated user personalized interaction and emotion sample data, it continuously iterates and optimizes the feature threshold and classification weight of the emotion recognition algorithm, adaptively adapts to the emotion expression habits and human-computer interaction characteristics of different users, and improves the accuracy of personalized emotion recognition and state perception.

Images