A robot with emotion recognition and monitoring function

Abstract

The application discloses a kind of robots with emotional recognition and monitoring function, including robot platform multimodal perception module, emotional evaluation module, intelligent control module, execution intervention module, remote communication module, man-machine interaction module, judge emotional state and trigger intervention measures by real-time detection user multimodal physiological indicators;Periodically assess the emotional state of user, adjust intervention measures according to state change, if the emotion continues to see improvement, then execute emergency operation;It also includes health monitoring module, can assess the current health status of user, and take corresponding measures.The application effectively solves the emotional care and health monitoring problem of children and the elderly in the family when no one accompanies, provides intelligent, humanized companionship and security guarantee.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a robot with emotion recognition and monitoring functions, especially a home service robot with emotion recognition, health monitoring and emergency assistance functions. Background Technology

[0002] With the increasing aging of society and the prevalence of the "empty nest" phenomenon, adults are unable to spend constant time with children and the elderly at home due to work and other reasons, making it difficult to detect their emotional fluctuations and health abnormalities in a timely manner. Existing home service robots mostly focus on cleaning, security, or simple entertainment functions, and generally suffer from the following shortcomings:

[0003] 1) Lack of proactive identification and tiered response capabilities to user emotions;

[0004] 2) The health monitoring function is limited, usually only supporting the passive reception of data from external devices, and lacking active data collection methods;

[0005] 3) Emotional care, health monitoring and basic service functions are disconnected from each other, and the level of intelligence is insufficient.

[0006] Therefore, there is an urgent need for a family companion robot that can comprehensively realize emotion recognition, multimodal health data collection, dynamic interactive response, and emergency assistance. Summary of the Invention

[0007] The purpose of this invention is to provide a robot with emotion recognition and monitoring functions.

[0008] To achieve the above objectives, this application provides the following solution:

[0009] A robot with emotion recognition and monitoring capabilities includes:

[0010] The robot platform has autonomous navigation, dynamic following and attitude stabilization control functions, and has a built-in power supply module and automatic charging unit.

[0011] The multimodal sensing module, including an electrocardiogram (ECG) sensor, a skin conductance sensor, an electroencephalogram (EEG) sensor, and a sweat chemistry sensor, is a wearable device that communicates with the robot platform via a remote communication module. It is used to periodically collect the user's ECG signals, skin conductance signals, EEG signals, and sweat chemistry signals.

[0012] The emotion assessment module, integrated into the embedded processor of the robot platform, is equipped with a cross-modal emotion recognition model constrained by physiological mechanisms. It is used to preprocess and extract features from the signals collected by the multimodal perception module, construct a high-dimensional feature representation, and output the emotion state assessment result.

[0013] The intelligent control module, an embedded processor integrated into the robot platform, is used to control the robot platform to start and follow the user based on the evaluation results of the emotion assessment module, control the execution intervention module to execute intervention measures, and adjust the intervention measures according to the periodic emotion state evaluation of the emotion assessment module to achieve closed-loop control.

[0014] The intervention module, an embedded processor integrated into the robot platform, is used to implement intervention measures based on instructions from the intelligent control module.

[0015] The remote communication module, located on the robot platform, supports Wi-Fi / Bluetooth / 5G communication and is used to enable communication between the ECG sensor, skin conductance sensor, EEG sensor, and sweat chemistry sensor in the multimodal perception module and the robot platform.

[0016] The human-computer interaction module includes a camera, a touch screen, a voice control unit, and a communication unit, which are used to realize human-computer interaction, including but not limited to system settings, voice control, and video calls with emergency contacts.

[0017] The user wears ECG sensors, skin conductance sensors, EEG sensors, and sweat chemistry sensors. The multimodal perception module collects the user's ECG, skin conductance, EEG, and sweat chemistry signals in real time. The emotion assessment module evaluates the user's emotional state. The intelligent control module controls the robot platform to navigate autonomously and dynamically follow the user, and controls the execution intervention module to perform daily greetings and emotional interventions based on the user's emotional state. The system continuously and periodically assesses the user's emotional state in real time, adjusts intervention measures according to changes in the state, and executes emergency assistance if the emotional state does not improve.

[0018] Furthermore, the robot also includes a health monitoring module, an embedded processor integrated into the robot platform, used to determine the user's current health status and assign a rating based on signals collected by the multimodal perception module, and send the results to the intelligent control module; the intelligent control module controls the intervention module to execute intervention measures based on the results sent by the health monitoring module.

[0019] The beneficial effects of this invention are:

[0020] 1) Actively collect user physiological indicators to achieve multimodal, 24 / 7 health monitoring;

[0021] 2) An emotion-level response mechanism is proposed, which significantly improves the humanization and intelligence of companionship;

[0022] 3) Deeply integrate emotional care, health monitoring, and the transportation of goods and autonomous navigation to form a complete solution for unattended family scenarios. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall working process of the present invention.

[0024] Figure 2 A schematic diagram of a cross-modal emotion recognition model constrained by physiological mechanisms. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This embodiment provides a robot with emotion recognition and monitoring functions, including a robot platform multimodal perception module, an emotion assessment module, an intelligent control module, an execution intervention module, a remote communication module, and a human-computer interaction module; specifically:

[0027] The robot platform described above has autonomous navigation, dynamic following and attitude stabilization control functions, and has a built-in power supply module and automatic charging unit. Optionally, the mobile robot body can be a quadruped robot, a wheeled robot or a tracked robot. In this embodiment, a robot dog is selected. In other embodiments, the robot platform is also equipped with a storage compartment for storing and transporting items such as medicines, water cups, etc.

[0028] The multimodal sensing module includes an electrocardiogram (ECG) sensor, an electrical conductance of skin (EDA) sensor, an electrical electroencephalogram (EEG) sensor, and a sweat chemistry sensor. It is a wearable device and communicates with the robot platform through a remote communication module. It is used to collect the user's ECG signals, EDA signals, EEG signals, and sweat chemistry signals at fixed intervals.

[0029] The emotion assessment module is integrated into the embedded processor of the robot platform and is equipped with a cross-modal emotion recognition model constrained by physiological mechanisms. It is used to preprocess and extract features from the signals collected by the multimodal perception module, construct a high-dimensional feature representation, and output the emotion state assessment result.

[0030] In one exemplary embodiment, the cross-modal emotion recognition model includes a preprocessing module, a feature extraction module, a multi-directional cross-attention module, and an output head module, such as... Figure 2As shown; the preprocessing module is used to preprocess the signals of each modality and output the preprocessed signals; the encoder module is used to output the feature sequences of each modality signal; the multi-directional cross-attention module is used for feature fusion, modeling and regulating relationships and introducing attention weights, and outputting the fused features; the output head module is used to output the emotion state classification and confidence score. Specifically:

[0031] The preprocessing module performs preprocessing on each modal signal: for the electrocardiogram (ECG) signal, it sequentially performs power frequency noise removal, baseline drift correction, abnormal amplitude truncation, and bandpass filtering to remove respiratory and low-frequency drift interference; for the electrodermal signal, it sequentially performs noise reduction, normalization, abnormal fluctuation smoothing, and low-pass filtering to suppress high-frequency noise; for the electroencephalogram (EEG) signal, it sequentially performs artifact suppression, bandpass filtering, and channel normalization to reduce electrooculogram (EOG) and electromyogram (EMG) interference; for the sweat chemical signal, it sequentially performs drift correction, concentration normalization, and outlier suppression to obtain a stable chemical time series; and the preprocessed modal signals are synchronized and aligned on the time axis to ensure that the multimodal signals reflect the physiological and psychological state at the same moment.

[0032] The feature extraction module includes a first feature extraction module, a second feature extraction module, a third feature extraction module, and a fourth feature extraction module arranged in parallel. Each of these modules includes a signal encoding module and a Transformer encoder. The signal encoding module consists of a multi-head attention layer with a jump connection structure and a feedforward propagation layer. The feedforward propagation layer uses a multilayer perceptron (MLP) network with fused features.

[0033] The preprocessed ECG signal is input to the signal encoding module in the first feature extraction module. After cosine position encoding, the ECG word sequence input is obtained to the ECG Transformer encoder. The high-dimensional temporal representation reflecting heart rate variability and parasympathetic nervous system regulation features is extracted through a multi-layer self-attention mechanism to obtain the ECG feature sequence. The first feature extraction module enables the multimodal emotion recognition model to implicitly learn heart rate rhythm changes and autonomic nervous system regulation information without explicitly calculating the HRV index.

[0034] The preprocessed ESC signal is input into the signal encoding module of the second feature extraction module, which automatically decomposes it into tonic components reflecting the slowly varying baseline level and phasic components reflecting the transient arousal response. Based on the tonic and phasic components, an ESC term sequence is constructed and input into the corresponding EDA Transformer encoder. The ESC feature sequence reflecting the intensity and persistence of sympathetic nerve activation is extracted through a self-attention mechanism. The second feature extraction module enables the ESC signal to explicitly distinguish the baseline state and transient response of the sympathetic nerve during the temporal modeling process.

[0035] The preprocessed EEG signal is input into the signal encoding module in the third feature extraction module to perform time-frequency representation modeling on the EEG signal and extract EEG activity components corresponding to multiple frequency bands. The frequency bands include at least the β band (12-30 Hz), α band (8-12 Hz), θ band (4-8 Hz), and δ band (0.5-4 Hz). Based on the multi-frequency band activity, an EEG Transformer encoder corresponding to the input EEG word sequence is constructed. Through the self-attention mechanism, EEG feature sequences reflecting the cortical activity patterns of emotion regulation, cognitive load, and anxiety are extracted, thereby obtaining a high-dimensional representation at the central nervous system level. The EEG features specifically include complexity features such as power of each frequency band, Higuchi fractal dimension, correlation dimension, approximate entropy, Lyapunov exponent, and detrended fluctuation analysis (DFA).

[0036] The preprocessed sweat chemical signal is input into the signal encoding module in the fourth feature extraction module. The sweat term sequence is constructed based on the rate and duration of chemical component change and is input into the corresponding Transformer encoder. The sweat feature sequence is extracted through the self-attention mechanism. The sweat features specifically refer to molecular biomarker features, including the absolute concentration, time change rate, normalized concentration change amplitude, and multi-molecule cooperative change pattern features of glucose, lactic acid, uric acid, sodium ions, potassium ions, and ammonium ions. Among them, glucose, lactic acid, and uric acid reflect changes in metabolism and energy demand, while sodium ions, potassium ions, and ammonium ions reflect the electrolyte regulation state of sweat glands caused by sympathetic nerve activation.

[0037] The multi-directional cross-attention module consists of a feedforward network layer and a multi-head cross-attention layer. ECG and skin conductance features are input into the feedforward network layer and processed by MLP to obtain ECG and skin conductance fusion features. In the multi-head cross-attention layer, the ECG features are used as query vectors and the ECG and skin conductance fusion features are used as key value vectors to model the regulatory relationship between the central nervous system and the autonomic nervous system, and the first fusion feature is obtained. At the same time, the ECG and skin conductance fusion features are used as query vectors and sweat features are used as key value vectors to model the correlation between short-term neural response and long-term stress load, and the second fusion feature is obtained. Finally, a gating function based on the physiological priors of the autonomic nervous system and the central nervous system is introduced, and the first fusion feature and the second fusion feature are passed through a three-layer fully connected network to obtain the final fusion feature.

[0038] The output head module consists of an MLP layer and a softmax classification layer. The final fused features are input into the output head module, and after being mapped by the MLP layer, a classification vector is output. The classification vector is mapped to three categories: mild anxiety, high anxiety, and calmness. After passing through the softmax classification layer, the confidence level of each category is obtained. The category with the highest confidence level is taken as the final category, and the confidence level range is 0-100%.

[0039] Optionally, the prediction distribution can be estimated by random deactivation, multiple inferences, or Bayesian modeling to obtain the confidence information output by the output head module. Specifically, in this embodiment, Bayesian modeling is used to estimate the confidence information.

[0040] The training process of the cross-modal emotion recognition model is as follows:

[0041] 1) Data preparation:

[0042] Each training sample consisted of inputs of the test subjects' electrocardiogram (ECG), skin conductance (SC), electroencephalogram (EEG), and sweat chemical signals, along with their corresponding real-world labels. The real-world labels were determined jointly by the test subjects' online self-reporting and the clinical physician's assessment.

[0043] 2) Training process:

[0044] The training samples are input into the emotion recognition model, which outputs anxiety state ratings and confidence scores, and calculates cross-entropy loss: the cross-entropy loss between the anxiety state classification ratings output by the model and the true labels is calculated and used for backpropagation and parameter updates of the emotion recognition model. The model is optimized through iterative training.

[0045] The intelligent control module, integrated into the embedded processor of the robot platform, is used to control the robot platform to start and follow the user based on the evaluation results of the emotion assessment module, control the execution intervention module to execute intervention measures, and adjust the intervention measures according to the periodic emotion state evaluation of the emotion assessment module to achieve closed-loop control.

[0046] The aforementioned intervention module is integrated into the embedded processor of the robot platform and is used to perform daily greetings, emotional interventions, or emergency assistance operations according to the instructions of the intelligent control module.

[0047] In one exemplary implementation, if the emotion recognition module determines that the user's emotion is in a state requiring intervention, the intelligent control module controls the intervention module to execute the corresponding instruction based on the emotion state rating:

[0048] When the emotional state assessment result is "normal", the intelligent control module controls the robot platform to start and follow the user, and then controls the execution intervention module to perform daily greetings; after the daily greetings are completed, the robot platform returns to standby mode and continues to periodically monitor the user's emotional state.

[0049] When the emotional state assessment result is "mild anxiety", the intelligent control module controls the robot platform to start and follow the user, and then controls the intervention module to issue caring voice messages; the multimodal perception module continuously and periodically collects the user's electrocardiogram signals, skin conductance signals, electroencephalogram signals, and sweat chemical signals; the emotional assessment module assesses the user's current emotional state in real time. If the user's emotional state returns to "normal" and continues to reach the preset number of assessment cycles (e.g., 10 consecutive assessment cycles), the robot platform returns to standby mode; if the user's emotional state does not return to "normal" and continues to reach the preset number of assessment cycles (e.g., 10 consecutive assessment cycles), an emergency help request is triggered to initiate a video call with the emergency contact.

[0050] When the emotional state assessment result is "high anxiety", the intelligent control module controls the robot platform to start and follow the user, and triggers an emergency help request to initiate a video call to the emergency contact.

[0051] The aforementioned remote communication module, located on the robot platform, supports Wi-Fi / Bluetooth / 5G communication and is used to enable communication between the electrical sensors, skin conductance sensors, electroencephalogram sensors, and sweat chemical sensors in the multimodal perception module and the robot platform.

[0052] The human-computer interaction module includes a camera, a touch screen, and a voice control module, which are used to realize human-computer interaction.

[0053] The robot's operation is as follows: the user wears an ECG sensor, skin conductance sensor, brainwave sensor, and sweat chemistry sensor. A multimodal perception module collects the user's ECG, skin conductance, brainwave, and sweat chemistry signals in real time. An emotion assessment module evaluates the user's emotional state. An intelligent control module controls the robot platform to navigate autonomously and dynamically follow the user, and controls the intervention module to perform daily greetings and emotional interventions based on the user's emotional state. The robot continuously and periodically assesses the user's emotional state in real time, adjusting intervention measures according to changes in state. If the emotional state does not improve, an emergency assistance operation is initiated. Figure 1 As shown.

[0054] In other embodiments, the robot also includes a health monitoring module, an embedded processor integrated into the robot platform, used to determine the user's current health status and assign a rating based on signals collected by the multimodal perception module, and send the results to the intelligent control module; based on the results sent by the health monitoring module, if the intelligent control module detects that the user is in an unhealthy state, it controls the execution intervention module to take corresponding measures such as inquiring about the user's condition and prompting the user to take medication, and if the user does not respond or indicates that the condition is serious, it initiates a video call to the emergency contact.

[0055] In other embodiments, the robot also has the ability to generate health monitoring reports and voice wake-up control functions.

[0056] The above description represents a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A robot with emotion recognition and monitoring capabilities, characterized in that, include: The robot platform has autonomous navigation, dynamic following and attitude stabilization control functions, and has a built-in power supply module and automatic charging unit. The multimodal sensing module, including an electrocardiogram (ECG) sensor, a skin conductance sensor, an electroencephalogram (EEG) sensor, and a sweat chemistry sensor, is a wearable device that communicates with the robot platform via a remote communication module. It is used to periodically collect the user's ECG signals, skin conductance signals, EEG signals, and sweat chemistry signals. The emotion assessment module, integrated into the embedded processor of the robot platform, is equipped with a cross-modal emotion recognition model. It is used to preprocess and extract features from the signals collected by the multimodal perception module, construct a high-dimensional feature representation, and output the emotion state assessment result. The intelligent control module, an embedded processor integrated into the robot platform, is used to control the robot platform to start and follow the user based on the evaluation results of the emotion assessment module, control the execution intervention module to execute intervention measures, and adjust the intervention measures according to the periodic emotion state evaluation of the emotion assessment module to achieve closed-loop control. The intervention module, an embedded processor integrated into the robot platform, is used to implement intervention measures based on instructions from the intelligent control module. The remote communication module, located on the robot platform, enables communication between the ECG sensor, skin conductance sensor, EEG sensor, and sweat chemistry sensor in the multimodal perception module and the robot platform. The human-computer interaction module is used to implement human-computer interaction. The cross-modal emotion recognition model includes a preprocessing module, a feature extraction module, a multi-directional cross-attention module, and an output head module; The preprocessing module is used to preprocess each modal signal and output the preprocessed signal; the encoder module is used to output the feature sequence of each modal signal; the multi-directional cross-attention module is used for feature fusion, modeling and regulating relationships and introducing attention weights, and outputting the fused features. The output header module is used to output the emotional state classification and confidence level; The multi-directional cross-attention module fuses ECG and skin conductance features to obtain ECG-skin conductance fusion features. Using EEG features as query vectors and ECG-skin conductance fusion features as key-value vectors, it models the regulatory relationship between the central nervous system and the autonomic nervous system to obtain the first fusion feature. Simultaneously, using ECG-skin conductance fusion features as query vectors and sweat features as key-value vectors, it models the correlation between short-term neural responses and long-term stress load to obtain the second fusion feature. Finally, the first fusion feature and the second fusion feature are fused to obtain the final fusion feature.

2. The robot with emotion recognition and monitoring function according to claim 1, characterized in that, In the aforementioned cross-modal emotion recognition model, The preprocessing module sequentially performs power frequency noise removal, baseline drift correction, abnormal amplitude truncation, and bandpass filtering on the ECG signal; denoising, normalization, abnormal fluctuation smoothing, and low-pass filtering on the skin conductance signal; artifact suppression, bandpass filtering, and channel normalization on the EEG signal; drift correction, concentration normalization, and outlier suppression on the sweat chemical signal; and synchronizes the preprocessed modal signals on the time axis. The feature extraction module includes a first feature extraction module, a second feature extraction module, a third feature extraction module, and a fourth feature extraction module arranged in parallel. The preprocessed ECG signal is input into the first feature extraction module. After cosine position encoding, the ECG word sequence is obtained. A high-dimensional temporal representation reflecting heart rate variability and parasympathetic regulation characteristics is extracted through a multi-layer self-attention mechanism to obtain the ECG feature sequence. The preprocessed ESC signal is input into the second feature extraction module, which decomposes it to obtain ESC term sequences that reflect the slow-varying baseline level and transient arousal response. ESC feature sequences that reflect the intensity and persistence of sympathetic nerve activation are extracted through the self-attention mechanism. The preprocessed EEG signals are input into the third feature extraction module to construct EEG word sequences. EEG feature sequences reflecting emotion regulation, cognitive load, and anxiety-related cortical activity patterns are extracted through a self-attention mechanism. The preprocessed sweat chemical signals are input into the fourth feature extraction module to construct sweat word sequences based on the rate and duration of chemical component changes. Sweat feature sequences are extracted through a self-attention mechanism. The output head module maps the final fused features into three categories: mild anxiety, high anxiety, and calmness, and obtains the confidence score for each category. The category with the highest confidence score is taken as the final category.

3. A robot with emotion recognition and monitoring functions according to claim 1 or 2, characterized in that, The user wears ECG sensors, skin conductance sensors, brain conductance sensors, and sweat chemistry sensors. The multimodal perception module collects the user's ECG signals, skin conductance signals, brain conductance signals, and sweat chemistry signals in real time. The emotion assessment module evaluates the user's emotional state. The intelligent control module controls the robot platform to navigate autonomously and dynamically follow the user, and controls the execution intervention module to execute corresponding instructions based on the user's emotional state.

4. A robot with emotion recognition and monitoring functions according to claim 3, characterized in that, Specifically, the intervention module that controls the execution of corresponding instructions based on the user's emotional state executes the following: When the emotional state assessment result is "normal", the intelligent control module controls the robot platform to start and follow the user, and then controls the execution intervention module to perform daily greetings; after the daily greetings are completed, the robot platform returns to standby mode and continuously monitors the user's emotional state periodically. When the emotional state assessment result is "mild anxiety", the intelligent control module controls the robot platform to start and follow the user, and then controls the execution intervention module to issue caring voice; the multimodal perception module continuously and periodically collects the user's electrocardiogram signal, skin conductance signal, electroencephalogram signal, and sweat chemical signal; the emotional assessment module assesses the user's current emotional state in real time. If the user's emotional state returns to "normal" and continues to reach the preset number of assessment cycles, the robot platform returns to standby mode. If the user's emotional state does not return to "normal" and continues to reach the preset number of assessment cycles, an emergency call will be triggered to initiate a video call with the emergency contact. When the emotional state assessment result is "high anxiety", the intelligent control module controls the robot platform to start and follow the user, and triggers an emergency help request to initiate a video call to the emergency contact.

5. A robot with emotion recognition and monitoring functions according to claim 1 or 2, characterized in that, The robot also includes a health monitoring module, an embedded processor integrated into the robot platform, which is used to determine the user's current health status and rate it based on the signals collected by the multimodal perception module, and send the results to the intelligent control module. Based on the results sent by the health monitoring module, the intelligent control module controls the intervention module to execute intervention measures.

6. A robot with emotion recognition and monitoring functions according to claim 3, characterized in that, The robot also includes a health monitoring module, an embedded processor integrated into the robot platform, which is used to determine the user's current health status and rate it based on the signals collected by the multimodal perception module, and send the results to the intelligent control module. Based on the results sent by the health monitoring module, the intelligent control module controls the intervention module to execute intervention measures.

7. A robot with emotion recognition and monitoring functions according to claim 1 or 2, characterized in that, The robot platform is also equipped with a storage compartment for storing and transporting items.

Images