Emotion accompanying robot system based on multi-dimensional perception and interaction method

Abstract

The invention relates to an emotion accompanying robot system based on multi-dimensional perception and an interaction method, and belongs to the technical field of intelligent robots. The system comprises a sensing module which captures micro-expression, voice, body temperature, touch and other multi-source data through a biological radar array, multispectral imaging and a touch sensing fabric; the decision-making module quantifies the emotion intensity by using an emotion state calculation engine, constructs a user personalized emotion file in combination with the dynamic knowledge graph, and generates a dynamic emotion graph; and the execution module realizes anthropomorphic emotion expression through bionic face driving, joint compliance control and thermal feedback. The interaction method comprises the steps of data capture, emotion file construction, dynamic graph generation, interactive execution and feedback adjustment, definition of an emotion intensity quantification formula, a graph edge weight model and the like. According to the method, physiological signals, environment situations and bionic expression are fused, multi-modal precise emotion perception and safe interaction are achieved, man-machine naturalness is improved, the method is suitable for elderly accompanying, psychological counseling and other scenes, and the emotion accompanying effect is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent robotics technology and relates to an emotional companion robot system and interaction method based on multi-dimensional perception. Specifically, it relates to an emotional companion robot system and interaction method that integrates physiological signal monitoring, environmental context understanding, and bionic emotional expression. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and robotics, emotional companion robots have been widely used in areas such as elderly care and psychological counseling. However, existing technologies still have some shortcomings in terms of emotional perception, feedback expression, human-machine empathy, and interactive safety.

[0003] Most existing systems utilize only a single modality, such as speech analysis, and are unable to fully capture physiological indicators like micro-expressions and changes in body temperature, resulting in low emotion recognition accuracy. Furthermore, some robots use fixed action templates for feedback, lacking the adaptability to personalized emotional expression. Furthermore, due to the lack of long-term memory models, existing systems struggle to achieve sustained human-machine empathy and exhibit fragmented characteristics. Furthermore, existing tactile interaction modules lack dynamic feedback control of contact force, posing certain safety risks.

[0004] Several invention patents have been developed to address the challenges of enabling emotional companion robots to achieve multimodal interaction, accurately perceive users' emotional states, and express emotions in an anthropomorphic manner. For example, CN1128832A uses only single-modal speech analysis, which is unable to capture physiological indicators such as micro-expressions and changes in body temperature, resulting in limited emotional perception. The robot proposed in JP2020156782A uses fixed action templates, lacks adaptability to personalized emotional expression, and suffers from feedback distortion. The system proposed in US20220345871A1 fails to establish a long-term memory model, resulting in fragmented emotional interaction and barriers to human-machine empathy. The tactile interaction module proposed in CN110696056A lacks dynamic feedback control of contact force, resulting in safety flaws.

[0005] To address these issues, it is necessary to develop an emotional companion robot system based on multidimensional perception. This system needs to integrate technologies such as bio-radar, multispectral imaging, and tactile sensing to capture multimodal physiological data such as micro-expressions, voice, and body temperature in real time. It also needs to construct a dynamic knowledge graph submodule based on environmental contextual information to more accurately perceive the user's emotional state. Furthermore, the system needs to achieve anthropomorphic emotional expression and safe human-machine interaction through mechanisms such as bionic facial actuation, thermal feedback submodule adjustment, and joint compliance control submodule, thereby enhancing the user experience and emotional companionship effectiveness. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide an emotional companion robot system and interaction method based on multi-dimensional perception. The system captures a variety of physiological and behavioral data of users in real time through a perception module, including micro-expressions, voice characteristics, body temperature changes, and tactile interaction data. Based on these multi-source data, the system uses the emotional state calculation engine and dynamic knowledge graph submodule at the decision-making layer to build a personalized emotional profile of the user, and generates a dynamic emotional map in combination with environmental context information. At the execution layer, the system realizes anthropomorphic emotional expression and safe human-computer interaction through the bionic facial drive submodule, thermal feedback submodule, and joint compliance control submodule.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] An emotional companion robot system based on multi-dimensional perception, comprising:

[0009] The perception module captures multi-source data including user micro-expressions, voice characteristics, body temperature changes, and tactile interactions in real time;

[0010] The decision module acquires multi-source data transmitted by the perception module, builds a personalized user emotion profile, and generates a dynamic emotion map based on environmental context information;

[0011] The execution module uses the dynamic emotion map generated by the decision module to perform anthropomorphic emotional expression and safe human-computer interaction, achieving accurate perception and response to user emotions;

[0012] The perception module specifically includes:

[0013] The bio-radar array submodule accurately captures the user's emotional-related physiological characteristics, such as facial micro-expressions and vocal cord vibrations;

[0014] The multispectral imaging submodule captures facial micro-expression features and detects changes in blood oxygen saturation and capillary blood flow caused by emotional fluctuations, providing a basis for emotion recognition.

[0015] The tactile sensing fabric submodule detects pressure and skin impedance to sense the tactile interaction force and skin condition between the user and the robot;

[0016] The decision module specifically includes:

[0017] The emotional state calculation engine quantifies and analyzes physiological signals and environmental context information to calculate the intensity of emotion;

[0018] The dynamic knowledge graph submodule builds a personalized user emotion profile and generates a dynamic emotion graph based on real-time environmental context information, providing a basis for emotion state assessment and prediction for subsequent emotional responses and interactions.

[0019] The execution module specifically includes:

[0020] The bionic facial driver module accurately simulates human facial expressions and presents corresponding emotional feedback based on the emotional state analyzed in the dynamic emotion map;

[0021] The joint compliance control submodule uses the intensity of emotion to calculate the specific forward and side tilt angles of the neck, driving the robot's neck joints to tilt forward and the head to tilt sideways to simulate a human's listening posture.

[0022] The thermal feedback submodule simulates human body temperature to enhance emotional empathy based on the analysis results of the dynamic emotion map.

[0023] Furthermore, the bio-radar array submodule also includes: a motion compensation unit to ensure that emotional indicators such as heart rate variability are obtained in a dynamic interactive environment.

[0024] Furthermore, the tactile sensing fabric submodule includes:

[0025] The surface sensing layer is composed of a flexible capacitive tactile unit array, which is used to detect the contact pressure distribution and touch trajectory characteristics in real time;

[0026] The middle functional layer integrates PVDF piezoelectric film for dynamic pressure detection and skin impedance monitoring;

[0027] The bottom control layer has a built-in temperature compensation circuit and is used to realize data transmission through the bus.

[0028] Furthermore, the user's physiological response, emotional state and event information are integrated into a triple node to form the personalized user emotion profile.

[0029] Furthermore, the dynamic emotion map is updated in real time according to trigger conditions to form a personalized profile containing the user's emotion evolution rules and physiological response characteristics, providing decision support for anthropomorphic interaction.

[0030] Furthermore, the system monitors the user's reaction to the robot's emotional expression in real time and adjusts subsequent emotional interactions and behavioral responses accordingly.

[0031] A method for interacting with an emotional companion robot based on multi-dimensional perception includes the following steps:

[0032] S1. Use the perception layer to capture the user's micro-expressions, voice features, body temperature changes, and tactile interaction data in real time;

[0033] S2, based on the multi-source data obtained in S1, uses the emotional state and dynamic knowledge graph to build a personalized emotional profile of the user at the decision-making level, and combines the environmental context information to generate a dynamic emotional graph;

[0034] S3, based on the dynamic emotion map generated by S2, performs anthropomorphic emotion expression and safe human-computer interaction at the execution layer, achieving accurate perception and response to user emotions;

[0035] S4. Monitor the user's response to the robot's emotional expression in real time, and adjust the robot's subsequent emotional interaction and behavioral response accordingly.

[0036] Furthermore, the S1 specifically includes the following steps:

[0037] S11. Combine the micro-tremor analysis algorithm based on wavelet packet decomposition and Hilbert-Huang transform to capture the physiological characteristics of user's facial micro-expressions and vocal cord vibrations, and obtain heart rate variability emotion indicators in a dynamic interactive environment;

[0038] S12. Use deep learning algorithms to capture facial micro-expression features in real time, monitor subcutaneous hemodynamic parameters to detect changes in blood oxygen saturation and capillary blood flow caused by emotional fluctuations, and perform spatiotemporal synchronous analysis of expression-physiological signals.

[0039] S13, detecting pressure and skin impedance to sense the tactile interaction force and skin state between the user and the robot;

[0040] The S2 specifically includes the following steps:

[0041] S21. Quantitatively analyze physiological signals and environmental context information using a preset emotional state algorithm to calculate emotional intensity;

[0042] S22. Through graph neural networks, the user's physiological response, emotional state, and current events are integrated into triple nodes to build a personalized user emotional profile.

[0043] S23. Combine real-time environmental context information to generate a dynamic emotion map, providing a basis for emotional state assessment and prediction for subsequent emotional responses and interactions;

[0044] The S3 specifically includes the following steps:

[0045] S31. By driving the robot's neck joint forward and its head sideways, it simulates a human listening posture, demonstrating its attention and understanding of the user.

[0046] S32, driving the robot to simulate human facial expressions and present corresponding emotional feedback based on the emotional state analyzed in the dynamic emotional map;

[0047] S33. Drive the robot to simulate human body temperature and enhance emotional empathy.

[0048] Furthermore, the emotion intensity in S21 is quantified as follows:

[0049]

[0050] Where E represents Emotion Intensity, α represents the comprehensive weight coefficient of physiological signals, β represents the comprehensive weight coefficient of situational signals, and w i represents the normalized weight of the i-th type of physiological signal, f i (physio) represents the physiological signal feature extraction function, g(context_score) represents the context quantification function;

[0051] The dynamic emotion graph in S23 includes three types of nodes:

[0052] Event nodes record the time, type, and context of interactions;

[0053] Emotion nodes store emotion intensity, emotion label, and duration;

[0054] Physiological nodes store raw sensor data;

[0055] The three nodes are connected by weighted edges, where the event-emotion edge weight is the rate of change of emotion intensity, and the emotion-physiology edge weight is the correlation between physiological signals and emotion intensity;

[0056] The event-emotion edge weight formula is expressed as:

[0057]

[0058] Where ΔE is the change in emotion intensity, and Δt is the change in time;

[0059] The emotion-physiology edge weight formula is:

[0060] w m→p =ρ(S,E)

[0061] Where ρ(S,E) is the correlation coefficient between the physiological signal S and the emotional signal E.

[0062] Furthermore, the step S31 specifically includes the following steps:

[0063] Design a two-degree-of-freedom neck motion control algorithm. The specific tilt angle calculation formula is as follows:

[0064]

[0065] θ y =5°×min(1,1.5E)(E∈[0,1])

[0066] Where E represents the emotional intensity, θ xrepresents the forward tilt angle of the neck, θ y Indicates the neck tilt angle;

[0067] The S32 specifically includes the following steps:

[0068] Using a shape memory alloy actuator array, the shape memory alloy actuator response driving model is expressed as the following formula:

[0069] θ face =θ0+k×E(θ0∈[10°,15°],k∈[0,1])

[0070] Where θ face is the angle of facial expression change, θ0 represents the basic angle, k is the proportional coefficient, which represents the influence of emotion intensity on the angle change, and E represents the emotion intensity;

[0071] The S33 specifically includes the following steps:

[0072] The temperature control model based on the Peltier effect is expressed as the following formula:

[0073]

[0074] Where T(t) represents the temperature at time t, T0 represents the initial temperature, t represents time, τ represents the time constant, and E represents the emotion intensity.

[0075] The beneficial effects of the present invention are:

[0076] By integrating physiological signal monitoring, environmental context understanding, and biomimetic emotional expression, this robot achieves multimodal interaction for emotional companionship. Compared to existing technologies, this invention not only captures physiological indicators such as micro-expressions and body temperature changes, but also builds a personalized user emotional profile through a dynamic knowledge graph submodule, allowing for more accurate perception of the user's emotional state.

[0077] At the execution layer, the bionic facial drive submodule, thermal feedback submodule, and joint compliance control submodule achieve anthropomorphic emotional expression and safe human-computer interaction. This method can effectively enhance the naturalness and safety of human-computer interaction, and is particularly suitable for service scenarios requiring emotional interaction, such as elderly care and psychological counseling, effectively improving user experience and the effectiveness of emotional care.

[0078] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0080] Figure 1 This is a flow chart of an emotional companion robot interaction method based on multi-dimensional perception according to an embodiment of the present invention;

[0081] Figure 2 This is a schematic diagram of an emotional companion robot system based on multi-dimensional perception according to an embodiment of the present invention;

[0082] Figure 3 This is a schematic diagram of multimodal signal acquisition and output and emotional state calculation according to an embodiment of the present invention;

[0083] Figure 4 This is a schematic diagram of outputting execution content based on emotional state according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0085] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0086] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0087] See also Figure 1, which is a flow chart of an emotional companion robot interaction method based on physiological-contextual multi-dimensional perception according to an embodiment of the present invention; please refer to Figure 2 Schematic diagram of an emotional companion robot system based on physiological-situational multi-dimensional perception according to an embodiment of the present invention.

[0088] The present invention proposes an emotional companion robot system based on multi-dimensional perception, which mainly includes:

[0089] The perception module uses the bio-radar array submodule, multispectral imaging submodule, and tactile sensing fabric submodule to capture the user's micro-expressions, voice characteristics, body temperature changes, and tactile interaction data in real time at the perception layer;

[0090] The decision module, based on the multi-source data obtained in step S1, uses the emotion state calculation engine and the dynamic knowledge graph submodule at the decision layer to build a personalized emotion profile of the user, and generates a dynamic emotion graph in combination with the environmental context information;

[0091] The execution module, based on the dynamic emotion map generated in step S2, realizes anthropomorphic emotional expression and safe human-computer interaction through the bionic facial drive submodule, thermal feedback submodule and joint compliance control submodule at the execution layer, and completes the accurate perception and response to the user's emotions.

[0092] In view of the system provided by the present invention, the present invention also proposes an emotional companion robot interaction method based on multi-dimensional perception, which mainly includes the following steps:

[0093] S1. Use the perception layer to capture the user's micro-expressions, voice features, body temperature changes, and tactile interaction data in real time;

[0094] S2, based on the multi-source data obtained in S1, uses the emotional state and dynamic knowledge graph to build a personalized emotional profile of the user at the decision-making level, and combines the environmental context information to generate a dynamic emotional graph;

[0095] S3, based on the dynamic emotion map generated by S2, performs anthropomorphic emotion expression and safe human-computer interaction at the execution layer, achieving accurate perception and response to user emotions;

[0096] S4. Monitor the user's response to the robot's emotional expression in real time, and adjust the robot's subsequent emotional interaction and behavioral response accordingly.

[0097] S1 specifically includes the following steps:

[0098] The S1.1 bio-radar array submodule uses adaptive beamforming technology (focusing accuracy ±2°) implemented by 16-channel MIMO antennas, combined with a micro-tremor analysis algorithm (resolution 0.01Hz) based on Wavelet Packet Decomposition (WPD) and Hilbert-Huang Transform (HHT), to accurately capture the user's facial micro-expressions, vocal cord vibrations and other emotional-related physiological characteristics; the motion compensation unit (IMU correction, range ±15cm) ensures stable acquisition of emotional indicators such as heart rate variability (HRV) in a dynamic interactive environment, and its millimeter wave phase analysis formula is Submillimeter displacement detection (Δd<0.1mm) is used to quantify subtle limb tremors caused by emotional fluctuations in users, providing a high signal-to-noise ratio physiological data basis for emotional state recognition.

[0099]

[0100] Where, represents the change in the phase of the received signal, λ represents the wavelength of the electromagnetic wave, and x(τ) represents the instantaneous displacement of the target face. represents the system phase noise.

[0101] Vibration signal signal-to-noise ratio optimization:

[0102]

[0103] Where, SNR enh It represents the signal-to-noise ratio after beamforming, H K (f) represents the beamforming weight of the Kth channel, S xx (f) Power spectral density of the vibration signal, N0 represents the noise power spectral density, and B represents the system bandwidth.

[0104] The S1.2 multispectral imaging submodule uses dual-channel collaborative sensing technology. The visible light channel (400-700nm) is equipped with a high-resolution Sony IMX585 sensor. Through deep learning algorithms, it captures 52 facial micro-expression features in real time, including periocular muscle movement and changes in the corners of the mouth, with an accuracy of 0.02mm / pixel. The near-infrared channel (850-940nm) monitors subcutaneous hemodynamic parameters based on the PPG principle, accurately detecting changes in blood oxygen saturation and capillary blood flow caused by emotional fluctuations. The dual-channel data is fused at the hardware level to achieve spatiotemporal synchronization analysis of expression and physiological signals, providing a multi-dimensional basis for emotion recognition. In the multispectral imaging submodule, the data of the optical channel and the near-infrared channel usually need to be fused. The fusion formula based on weighted average is:

[0105] F fucsed =α×Fvisible +(1-α)×F NIR

[0106] Where, F fucsed represents the fused features, F visible Represents the characteristics of the visible light channel, F NIR represents the feature of the near-infrared channel, and α represents the fusion weight.

[0107] S1.3 tactile sensing fabric submodule is used to detect contact pressure distribution and touch trajectory characteristics in real time; it adopts a three-layer composite structure design, including a surface sensing layer composed of a flexible capacitive tactile unit array (unit density 4 / cm 2 , detection range 0.1-50N, positioning accuracy ±0.5mm), an intermediate functional layer of integrated polyvinylidene fluoride (PVDF) piezoelectric film (for dynamic pressure detection and skin impedance monitoring), and a bottom control layer with built-in temperature compensation circuit (operating temperature range -20℃ to 60℃). The fabric has a sampling frequency of 1kHz and realizes data transmission via a CAN with Flexible Data-Rate (CANFD) bus, forming a multimodal tactile perception system. The dynamic pressure detection formula of the above intermediate functional layer is:

[0108] Q=A×d 33 ×σ zz

[0109] Where Q represents the charge generated by the piezoelectric effect, A represents the sensitive area of ​​the PVDF piezoelectric film, and d 33 represents the piezoelectric constant, σ zz Indicates the pressure on the film surface.

[0110] This invention utilizes three technologies: bioradar, multispectral imaging, and a tactile sensing fabric submodule to capture the user's physiological and behavioral data in real time. The bioradar utilizes beamforming technology and a micro-tremor algorithm to accurately monitor facial microexpressions and vocal cord vibrations. The multispectral imaging submodule uses visible light and near-infrared dual channels to capture facial expressions and monitor blood flow changes. The tactile sensing fabric submodule utilizes a three-layer composite structure to detect pressure and skin impedance, adapting to varying temperature environments.

[0111] The sensing module constructed by the present invention can capture a variety of physiological and behavioral data of users in real time and accurately. Specifically, the bio-radar array submodule operates in the 60GHz frequency band with a resolution of up to 0.05mm. 3, which can detect facial micro-tremors with a frequency range of 0.1-5Hz, and can capture subtle changes in facial muscles during emotional fluctuations. The multispectral imaging submodule contains dual channels of visible light (400-700nm) and near-infrared (850-940nm), which can synchronously capture the user's expression and blood flow changes, providing richer physiological signals for emotional analysis. The tactile sensing fabric submodule is embedded with a capacitive-piezoresistive composite sensor with a sampling rate of up to 1kHz, which can detect contact pressure (0.1-50N) and skin impedance, thereby sensing the tactile interaction force and skin condition between the user and the robot. Through these advanced sensor technologies, the perception module provides the emotional care robot system with comprehensive and accurate user physiological and behavioral information, enabling it to more accurately perceive and understand the user's emotional state, thereby achieving more considerate emotional care services.

[0112] In S2, the system enters the decision layer for in-depth processing based on the multi-source data obtained in step S1, including the user's micro-expressions, voice characteristics, body temperature changes, and tactile interaction data. Figure 3 , which is a schematic diagram of multimodal signal acquisition and output and emotional state calculation according to an embodiment of the present invention; the emotional state calculation engine uses a preset algorithm to perform quantitative analysis on physiological signals and environmental context information to calculate the emotional intensity.

[0113] The emotional state calculation engine and dynamic knowledge graph submodule of the decision layer specifically include the following steps:

[0114] The emotional state calculation engine of the present invention realizes accurate quantification of emotional intensity through multi-source data fusion technology. The specific expression of the emotional intensity quantification algorithm is as follows:

[0115]

[0116] Where, Emotion Intensity is the emotional intensity, α is the comprehensive weight coefficient of physiological signals, β is the comprehensive weight coefficient of situational signals, and w i represents the normalized weight of the i-th type of physiological signal, f i (physio) represents the physiological signal feature extraction function, and g(context_score) represents the context quantification function.

[0117] w i The optimized weights (∑ i w i =1), f i(physio) is the normalized physiological characteristic value, β (β = 1-α) represents the contextual factor weight, and g (context_score) is the context analysis score after Sigmoid normalization (including environmental parameters and semantic analysis results);

[0118] When α=0.9,β=0.1,f i Including the changes in facial expressions and tone characteristics, the emotion intensity quantification algorithm is as follows:

[0119] E=0.7×(ΔFacial)+0.2×(ΔVoice)+0.1×(ΔContext)

[0120] In the formula, ΔFacial represents the change in facial expression, ΔVoice represents the change in voice features, and ΔContext represents the change in environmental context. When E is greater than 0.65, the system will trigger the three-level response strategy.

[0121] Furthermore, the dynamic knowledge graph submodule uses a graph neural network (GNN) to integrate information such as the user's physiological reactions, emotional state, and current events into triple nodes, building a personalized user emotional profile. Combined with real-time contextual information, the system generates a dynamic emotional graph, providing accurate emotional state assessment and prediction for subsequent emotional responses and interactions.

[0122] A dynamic knowledge graph submodule is constructed using a graph neural network. The graph contains three types of nodes: event nodes record the time, type, and context of an interaction; emotion nodes store emotion intensity, emotion label, and duration; and physiological nodes store raw sensory data. Nodes are connected by weighted edges. The weight of the event-emotion edge is the rate of change of emotion intensity, and the weight of the emotion-physiology edge is the correlation between physiological signals and emotion intensity.

[0123] The event-emotion edge weight formula is:

[0124]

[0125] Where ΔE is the change in emotion intensity, and Δt is the change in time.

[0126] The emotion-physiology edge weight formula is:

[0127] w m→p =ρ(S,E)

[0128] Where ρ(S,E) is the correlation coefficient between the physiological signal S and the emotional signal E.

[0129] The system uses multiple trigger conditions, such as sudden changes in emotion intensity (over 0.2 per minute), emotion duration exceeding three minutes, and special event detection, to update the knowledge graph in real time. Ultimately, the system forms a personalized profile encompassing the user's emotional evolution patterns and physiological response characteristics, providing decision support for anthropomorphic interactions.

[0130] In S3, the system implements anthropomorphic emotional expression and safe human-computer interaction through the bionic facial drive submodule, thermal feedback submodule, and joint compliance control submodule mechanism based on the dynamic emotional map generated in step S2. Specifically, the system first drives the neck joint forward θ through the joint compliance control submodule mechanism. x , head tilt θ y , simulating the human listening posture, showing attention and understanding to users.

[0131] The implementation of anthropomorphic emotional interaction through a multimodal collaborative control mechanism at the execution layer specifically includes the following steps:

[0132] S3.1 Joint compliance control submodule:

[0133] Considering the impact of emotion intensity on posture, a two-degree-of-freedom neck motion control algorithm is designed. The specific tilt angle calculation formula is as follows:

[0134]

[0135] θ y =5°×min(1,1.5E)(E∈[0,1])

[0136] Where E represents the emotional intensity, θ x represents the forward tilt angle of the neck, θ y Indicates the neck tilt angle.

[0137] S3.2 Bionic facial driver submodule:

[0138] The bionic facial drive submodule uses 42 sets of shape memory alloy (SMA) actuators to accurately simulate human facial expressions and present corresponding emotional feedback based on the emotional state analyzed in the dynamic emotion map. The 42 sets of shape memory alloy (SMA) actuator arrays are used. The shape memory alloy actuator response drive model is expressed as the following formula:

[0139] θ face =θ0+k×E(θ0∈[10°,15°], k∈[0,1])

[0140] Where θ faceis the angle of facial expression change, θ0 represents the basic angle, k is the proportional coefficient, which represents the influence of emotion intensity on the angle change, and E represents the emotion intensity.

[0141] S3.3 Thermal feedback submodule:

[0142] The thermal feedback submodule uses the Peltier effect to simulate human body temperature and enhance emotional empathy. These modules work together to enable the robot to accurately perceive and respond to user emotions based on the analysis results of the dynamic emotion map. The temperature control model based on the Peltier effect is expressed as the following formula:

[0143]

[0144] Where T(t) represents the temperature at time t, T0 represents the initial temperature of 34°, t represents time, τ represents the time constant, and E represents the emotion intensity.

[0145] Through built-in feedback sensors, the S4 system monitors the user's reaction to the robot's emotional expression in real time, adjusts subsequent emotional output accordingly, and completes closed-loop control of emotional interaction, thereby achieving accurate perception and response to user emotions.

[0146] For an example of a grief response process, see Figure 4 , which is a schematic diagram of outputting execution content based on emotional state according to an embodiment of the present invention, describes in detail a method for an emotional care robot to detect and respond to a user's sadness. The specific implementation steps are as follows:

[0147] S1. Multimodal physiological signal acquisition and preprocessing

[0148] Collect and process the following physiological characteristic signals in real time:

[0149] 1. Millimeter-wave radar array detects facial microtremor signals

[0150] Characteristic frequency extraction formula:

[0151] f m (t)=arg max(FFT{R(t)}), R(t)∈[0.1, 5]Hz

[0152] Test results: The tremor frequency of the glabellar muscles was significantly reduced from the baseline value of 1.2 Hz to 0.3 Hz (Δf = -75%).

[0153] 2. Multispectral imaging submodule calculates venous blood flow velocity:

[0154] Blood flow velocity calculation formula:

[0155]

[0156] Test results: The test showed that the blood flow velocity decreased by 18% (ρ<0.01).

[0157] 3. The speech analysis module extracts the fundamental frequency standard deviation feature:

[0158] Fundamental frequency standard deviation calculation formula:

[0159]

[0160] Test results: The test value dropped from 45 Hz to 28 Hz (Δσ=-37.8%).

[0161] S2. Quantitative Analysis and Determination of Emotional State

[0162] Constructing a sentiment deviation index model in the cloud-based sentiment computing engine:

[0163] E=0.7×(ΔFacial)+0.2×(ΔVoice)+0.1×(ΔContext)

[0164] in:

[0165]

[0166] ΔContext=β×Env(t)

[0167] The response is triggered when the following conditions are met:

[0168] E>0.65 and t d >30s

[0169] S3. Multimodal Emotional Response Execution

[0170] The anthropomorphic response is achieved through the following control algorithms:

[0171] 1. Neck posture control:

[0172] Control algorithm:

[0173]

[0174] θ y =5°×min(1,1.5E)(E∈[0,1])

[0175] 2. Thermal feedback submodule control:

[0176] Temperature gradient formula:

[0177]

[0178] Where T(t) represents the temperature at time t, T0 represents the initial temperature of 34°, t represents time, τ is 1.5s, and E represents the emotion intensity.

[0179] 3. Voice interaction generation:

[0180] Formula for generating empathy sentences:

[0181] u(t)=LSTM(E(t),H(t-1))

[0182] Output content: Contains the detected change in tactile interaction characteristics (ΔF = -15%) and uses an open-ended question sentence format.

[0183] The present invention discloses an emotional companion robot system and interaction method based on multi-dimensional perception. The system includes three modules: perception, decision-making, and execution: the perception module captures micro-expressions, voice, body temperature, touch and other multi-source data through a biological radar array (including motion compensation), multi-spectral imaging, and tactile sensing fabrics (divided into surface / middle / bottom layers); the decision-making layer uses an emotional state calculation engine to quantify emotional intensity, and combines dynamic knowledge graphs to build user personalized emotional profiles (physiological-emotional-event triples), generating dynamic emotional graphs containing event / emotional / physiological nodes; the execution layer realizes anthropomorphic emotional expression through bionic facial drive, joint compliance control (two-degree-of-freedom algorithm), and thermal feedback (Peltier model). The interaction method includes data capture (micro-tremor analysis / deep learning), emotional profile construction (graph neural network), dynamic graph generation and interactive execution and feedback adjustment, and defines the emotional intensity quantification formula, graph edge weight model, etc. The present invention integrates physiological signals, environmental context and bionic expression to achieve multimodal precise emotional perception and safe interaction, enhance the naturalness of man and machine, and is suitable for scenarios such as elderly care and psychological counseling, enhancing the effect of emotional care.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. An emotional companion robot system based on multi-dimensional perception, characterized in that: include: The perception module captures multi-source data including user micro-expressions, voice characteristics, body temperature changes, and tactile interactions in real time; The decision module acquires multi-source data transmitted by the perception module, builds a personalized user emotion profile, and generates a dynamic emotion map based on environmental context information; The execution module uses the dynamic emotion map generated by the decision module to perform anthropomorphic emotional expression and safe human-computer interaction, achieving accurate perception and response to user emotions; The perception module specifically includes: The bio-radar array submodule accurately captures the user's emotional-related physiological characteristics, such as facial micro-expressions and vocal cord vibrations; The multispectral imaging submodule captures facial micro-expression features and detects changes in blood oxygen saturation and capillary blood flow caused by emotional fluctuations, providing a basis for emotion recognition. The tactile sensing fabric submodule detects pressure and skin impedance to sense the tactile interaction force and skin condition between the user and the robot; The decision module specifically includes: The emotional state calculation engine quantifies and analyzes physiological signals and environmental context information to calculate the intensity of emotion; The dynamic knowledge graph submodule builds a personalized user emotion profile and generates a dynamic emotion graph based on real-time environmental context information, providing a basis for emotion state assessment and prediction for subsequent emotional responses and interactions. The execution module specifically includes: The bionic facial driver module accurately simulates human facial expressions and presents corresponding emotional feedback based on the emotional state analyzed in the dynamic emotion map; The joint compliance control submodule uses the intensity of emotion to calculate the specific forward and side tilt angles of the neck, driving the robot's neck joints to tilt forward and the head to tilt sideways to simulate a human's listening posture. The thermal feedback submodule simulates human body temperature to enhance emotional empathy based on the analysis results of the dynamic emotion map.

2. The multi-dimensional perception-based emotional companion robot system according to claim 1, characterized in that: The bioradar array submodule also includes a motion compensation unit to ensure that emotional indicators such as heart rate variability are obtained in a dynamic interactive environment.

3. The multi-dimensional perception-based emotional companion robot system according to claim 1, characterized in that: The tactile sensing fabric submodule includes: The surface sensing layer is composed of a flexible capacitive tactile unit array, which is used to detect the contact pressure distribution and touch trajectory characteristics in real time; The middle functional layer integrates PVDF piezoelectric film for dynamic pressure detection and skin impedance monitoring; The bottom control layer has a built-in temperature compensation circuit and is used to realize data transmission through the bus.

4. The multi-dimensional perception-based emotional companion robot system according to claim 1, characterized in that: The user's physiological response, emotional state and event information are integrated into a triple node to form the personalized user emotion profile.

5. The multi-dimensional perception-based emotional companion robot system according to claim 1, characterized in that: The dynamic emotion map is updated in real time according to trigger conditions to form a personalized profile containing the user's emotion evolution rules and physiological response characteristics, providing decision support for anthropomorphic interaction.

6. The multi-dimensional perception-based emotional companion robot system according to claim 1, characterized in that: The system monitors the user's reaction to the robot's emotional expression in real time and adjusts subsequent emotional interaction and behavioral responses accordingly.

7. A method for interacting with an emotional companion robot based on multi-dimensional perception, characterized in that: The following steps are involved: S1. Use the perception layer to capture the user's micro-expressions, voice features, body temperature changes, and tactile interaction data in real time; S2, based on the multi-source data obtained in S1, uses the emotional state and dynamic knowledge graph to build a personalized emotional profile of the user at the decision-making level, and combines the environmental context information to generate a dynamic emotional graph; S3, based on the dynamic emotion map generated by S2, performs anthropomorphic emotion expression and safe human-computer interaction at the execution layer, achieving accurate perception and response to user emotions; S4. Monitor the user's response to the robot's emotional expression in real time, and adjust the robot's subsequent emotional interaction and behavioral response accordingly.

8. The method for interacting with an emotional companion robot based on multi-dimensional perception according to claim 7, characterized in that: The S1 specifically includes the following steps: S11. Combine the micro-tremor analysis algorithm based on wavelet packet decomposition and Hilbert-Huang transform to capture the physiological characteristics of user's facial micro-expressions and vocal cord vibrations, and obtain heart rate variability emotion indicators in a dynamic interactive environment; S12. Use deep learning algorithms to capture facial micro-expression features in real time, monitor subcutaneous hemodynamic parameters to detect changes in blood oxygen saturation and capillary blood flow caused by emotional fluctuations, and perform spatiotemporal synchronous analysis of expression-physiological signals. S13, detecting pressure and skin impedance to sense the tactile interaction force and skin state between the user and the robot; The S2 specifically includes the following steps: S21. Quantitatively analyze physiological signals and environmental context information using a preset emotional state algorithm to calculate emotional intensity; S22. Through graph neural networks, the user's physiological response, emotional state, and current events are integrated into triple nodes to build a personalized user emotional profile. S23. Combine real-time environmental context information to generate a dynamic emotion map, providing a basis for emotional state assessment and prediction for subsequent emotional responses and interactions; The S3 specifically includes the following steps: S31. By driving the robot's neck joint forward and its head sideways, it simulates a human listening posture, demonstrating its attention and understanding of the user. S32, driving the robot to simulate human facial expressions and present corresponding emotional feedback based on the emotional state analyzed in the dynamic emotional map; S33. Drive the robot to simulate human body temperature and enhance emotional empathy.

9. The method for interacting with an emotional companion robot based on multi-dimensional perception according to claim 8, characterized in that: The emotional intensity in S21 is quantified as follows: Where E represents Emotion Intensity, α represents the comprehensive weight coefficient of physiological signals, β represents the comprehensive weight coefficient of situational signals, and w i represents the normalized weight of the i-th type of physiological signal, f i (physio) represents the physiological signal feature extraction function, g(context_score) represents the context quantification function; The dynamic emotion graph in S23 includes three types of nodes: Event nodes record the time, type, and context of interactions; Emotion nodes store emotion intensity, emotion label, and duration; Physiological nodes store raw sensor data; The three nodes are connected by weighted edges, where the event-emotion edge weight is the rate of change of emotion intensity, and the emotion-physiology edge weight is the correlation between physiological signals and emotion intensity; The event-emotion edge weight formula is expressed as: Where ΔE is the change in emotion intensity, and Δt is the change in time; The emotion-physiology edge weight formula is: w m→p =ρ(S,E) Where ρ(S,E) is the correlation coefficient between the physiological signal S and the emotional signal E.

10. The method for interacting with an emotional companion robot based on multi-dimensional perception according to claim 8, characterized in that: The S31 specifically includes the following steps: Design a two-degree-of-freedom neck motion control algorithm. The specific tilt angle calculation formula is as follows: i y =5°×min(1,1.5E) Where E represents the emotional intensity, E∈[0,1], θ x represents the forward tilt angle of the neck, θ y Indicates the neck tilt angle; The S32 specifically includes the following steps: Using a shape memory alloy actuator array, the shape memory alloy actuator response driving model is expressed as the following formula: i face =θ0+k×E Where θ face is the angle of facial expression change, θ0 represents the basic angle, θ0∈[10°,15°], k is the proportional coefficient, k∈[0,1], represents the influence of emotion intensity on angle change, E represents emotion intensity; The S33 specifically includes the following steps: The temperature control model based on the Peltier effect is expressed as the following formula: Where T(t) represents the temperature at time t, T0 represents the initial temperature, t represents time, τ represents the time constant, and E represents the emotion intensity.

Images