Old people accompanying robot system based on multi-mode emotion cognition reinforcement learning

Abstract

The invention discloses an elderly accompanying robot system based on multi-mode emotion cognition reinforcement learning. The system comprises a robot body, a multi-source data acquisition and feature extraction unit, a multi-data-source dynamic feature fusion unit, an emotional state estimation and modeling unit and a personalized interaction strategy generation unit. The multi-source data acquisition and feature extraction unit is used for extracting voice signals, physiological signals and behavior signals; the multi-data-source dynamic feature fusion unit is used for realizing cross-modal coupling through a learnable attention mechanism; the emotional state estimation and modeling unit is used for constructing a three-dimensional emotional vector space, predicting an emotional state through a mode of fusing time sequence characteristics and reinforcement learning, and converting a multi-modal signal into a quantifiable emotional index; and the personalized interaction strategy generation unit is used for generating a personalized interaction strategy for controlling the robot body.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a companion robot system for the elderly based on multimodal emotional cognitive reinforcement learning. Background Technology

[0002] In existing technologies, user emotion recognition and interaction strategy generation systems based on multimodal data have been initially explored, but they generally suffer from the following drawbacks: **Single-dimensional emotion recognition:** Traditional solutions often rely on single-modal data (such as voice analysis or facial expression recognition) for emotion judgment, lacking effective fusion of multimodal information. This single-dimensional analysis struggles to capture complex user emotional states, especially when multimodal features are inconsistent (such as forced smiles), leading to potential misjudgments. **Rigid interaction strategies:** Existing systems often use preset rule bases or static decision trees to generate interaction strategies, lacking adaptability to dynamic user emotional states. When a specific emotion is detected, the system often mechanically triggers a fixed response template, failing to achieve personalized interactions tailored to each user, and unable to address the need for strategy adjustments during continuous emotional evolution. **Insufficient cross-modal data collaboration:** Although some solutions attempt to integrate multimodal data, they generally lack effective feature coupling mechanisms. Different modalities of data (such as speech spectra, facial action units, and heart rate variability) naturally differ in temporal scale and feature dimensions. Existing technologies have failed to address the spatiotemporal alignment problem of multi-source heterogeneous data, resulting in limited learning effectiveness of cross-modal emotion representations. Lack of predictive intervention capabilities: Existing systems are mostly limited to emotional responses at the current moment, failing to establish temporal emotion prediction models. Due to the lack of modeling of emotional evolution patterns, the system only initiates intervention when the user's emotions fluctuate significantly, missing the optimal opportunity to provide preventative guidance before emotions worsen. Lack of dynamic weighting mechanisms: In the process of multimodal feature fusion, existing methods typically employ fixed weight allocation strategies (such as assigning 40% weight to speech, 30% to facial expressions, and 30% to physiological signals to all users), ignoring the differences in dominant modalities among different user groups and in different scenarios. For example, the weight of physiological signals should be increased for users with language impairments, while the weight of facial expression analysis needs to be strengthened in video call scenarios. These shortcomings lead to problems such as high emotional misjudgment rates, stiff interactive experiences, and insufficient personalization in practical applications of existing systems. Therefore, there is an urgent need in this field for a technical solution that can achieve deep multimodal coupling, dynamic strategy generation, and emotion prediction capabilities. Summary of the Invention

[0003] This invention provides an elderly companion robot system based on multimodal emotion cognition reinforcement learning. The system includes: a robot body, a multi-source data acquisition and feature extraction unit, a dynamic feature fusion unit from multiple data sources, an emotion state estimation and modeling unit, and a personalized interaction strategy generation unit.

[0004] The multi-source data acquisition and feature extraction unit is used to extract speech signals, physiological signals, and behavioral signals;

[0005] The dynamic feature fusion unit from multiple data sources is used to achieve cross-modal coupling through a learnable attention mechanism;

[0006] The sentiment state estimation and modeling unit is used to construct a three-dimensional sentiment vector space, predict sentiment state by fusing temporal features and reinforcement learning, and transform multimodal signals into quantifiable sentiment indicators.

[0007] The personalized interaction strategy generation unit is used to generate personalized interaction strategies for controlling the robot body.

[0008] Furthermore, speech signal extraction includes extracting speech feature vectors from the original speech signal using the Mel-Cepstral Coefficients (MFCC) method. :

[0009]

[0010]

[0011] in, For speech feature vectors; The MFCC coefficient number; For the serial number MFCC characteristics at that time; For time-domain framing, it represents the t-th frame after segmentation of the original speech signal; The Fourier transform represents the conversion of a time-domain signal into a frequency-domain signal. This indicates power spectrum calculation, which calculates the energy distribution of the spectrum; N is the number of points in the FFT, which determines the frequency resolution; The number of Mel filters controls the fineness of the frequency band division; Frequency index; This is the index of the filter; For the first Each filter at frequency Weight at each location.

[0012] Furthermore, physiological signal extraction specifically includes monitoring heart rate signals and normalizing them:

[0013]

[0014]

[0015] in, For physiological signal feature vectors, for The normalized value of the heart rate signal at any given time. for Heart rate value at any time This represents the historical average heart rate. The historical heart rate standard deviation, The time interval is the dimension of the physiological signal feature vector.

[0016] Furthermore, behavior signal extraction includes using pose estimation algorithms to continuously detect limb keypoints in the video sequence, calculating the Euclidean distance change of limb keypoint positions between adjacent frames, and when within a time unit... For example, in a frame, if the displacement of a keypoint exceeds a preset threshold, then an action is considered to have occurred. ,otherwise ;

[0017] Within the specified time window Within that time period, count the total number of action events that occurred, and then divide by the time. The average action frequency was obtained. :

[0018]

[0019] in, The average frequency of limb movements. For time unit Detected action events, For time windows.

[0020] Furthermore, achieving cross-modal coupling through learnable attention mechanisms specifically includes:

[0021] Get Time Input speech feature vector Physiological signal feature vector Average frequency of limb movements Projecting the features of all modalities onto a unified dimension. After the projection 3D feature vector , , :

[0022]

[0023]

[0024]

[0025] in, , These are the projection matrix parameters. The dimension is Used to convert scalars Convert to dimensional vector;

[0026] Through attention The unit generates dynamic weights for each modality at the current time step based on the projected features and emotional state of the previous time step.

[0027]

[0028]

[0029]

[0030] in, Indicates a splicing operation; yes At any moment The hidden state and cell state; It is the emotional state vector of the previous moment, as The initial state input; It is The weight matrix, for The hidden layer dimension is used to... Mapped to a 3D logits vector; For the weighting coefficients, satisfying ;

[0031] Using weighted coefficients, the projection features of each modality at the current time are weighted and summed to obtain the final fused feature vector. :

[0032] .

[0033] Furthermore, by fusing temporal features and reinforcement learning to predict sentiment states, multimodal signals are transformed into quantifiable sentiment indicators. Specifically, this involves using recurrent neural networks to model the temporal dynamics of sentiment, with the current multimodal fusion features as input. The output is the estimated sentiment state. ,

[0034]

[0035] in, for exist The hidden state at any given moment;

[0036] This allows emotional states to be quantified in a continuous three-dimensional space.

[0037]

[0038] in, Let be the emotional state vector at time t. Let t represent the positive or negative emotional tendency. Let t be the intensity of emotion. Let t represent the degree of emotional dominance.

[0039] Furthermore, personalized interaction strategies for controlling the robot body are generated, specifically including:

[0040] Define a Markov decision process:

[0041]

[0042] Where S is the state space, i.e., the emotional state vector. A is used to quantify the user's real-time emotional state and provide input to the decision network; A is the action space. ,For example =“Voice comfort” ="Play soothing music", =“I suggest we play a little game” ="Contact nursing staff", as the output of the decision network; R is the reward function, which at time step Output The definition of is:

[0043]

[0044] in, Represents the time step Instant rewards received , indicating from arrive The change in valence at time t. , indicating from arrive Changes in arousal level over time. represent User engagement at all times These are weighting coefficients used to balance the importance of different reward items. It is a positive constant, serving as the upper and lower bounds of the clipping range. It is the natural logarithm function; For discount factor parameters, It is used to calculate the present value of future rewards.

[0045] Furthermore, the generation of personalized interaction strategies for controlling the robot body also includes:

[0046] The expected reward of taking action a in state s is updated through a double Q-network, and the risk of overestimation is reduced by using two independent critic networks to minimize the reward.

[0047]

[0048] in, This represents the Q-value of the state-action pair in the current critic network. The current critic network parameters can be optimized. This allows the critic network to more accurately evaluate the value of actions. The Q-value of the target critic network. Here are the parameters for the target critic network, where i=1,2 indicates a dual network. For the next emotional state, The next action generated by the target policy network. Here, R is the discount factor parameter, and R is the instant reward. For expectation operators;

[0049] The optimized policy network directly optimizes consecutive actions through deterministic policy gradients, generating actions that maximize the Q-value.

[0050]

[0051] in, Actions generated based on the current state; Policy network parameters, by adjusting maximize Choose the action that maximizes the Q value; The Q-value generated by the first critic network, The expectation of the sampled state s in the experience replay pool D, () represents the Q-value evaluation of the state-action pair by the first critic network. The action generated by the policy network in state s.

[0052] This invention combines multimodal user data, including voice, physiological, and behavioral signals, to more accurately understand user emotions. Based on user profiles and real-time emotional states, it dynamically generates adaptive dialogue / action strategies through reinforcement learning. It predicts emotional needs using Markov models to achieve preventative emotional intervention; for example, it triggers specific intervention measures when a user is detected to be in a depressed mood and have an abnormal heart rate. This solution proposes a dynamic emotional weight allocation mechanism and a cross-modal feature coupling algorithm, solving the problems of single-dimensional emotion recognition and rigid interaction strategies in existing technologies. Attached Figure Description

[0053] Figure 1 This is a structural block diagram of an elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to the present invention. Detailed Implementation

[0054] The present invention will now be further described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention. It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention.

[0055] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0056] Embodiment 1 of the present invention relates to an elderly companion robot system based on multimodal emotion cognition reinforcement learning. The system includes a robot body, a multi-source data acquisition and feature extraction unit, a dynamic feature fusion unit of multiple data sources, an emotion state estimation and modeling unit, and a personalized interaction strategy generation unit.

[0057] The multi-source data acquisition and feature extraction unit is used to extract speech signals, physiological signals, and behavioral signals.

[0058] Speech signal extraction includes extracting speech feature vectors from the original speech signal using the Mel-Cepstral Coefficients (MFCC) method. :

[0059]

[0060]

[0061] in, For speech feature vectors; The MFCC coefficient number; For the serial number MFCC characteristics at that time; For time-domain framing, it represents the t-th frame after segmentation of the original speech signal; The Fourier transform represents the conversion of a time-domain signal into a frequency-domain signal. This indicates power spectrum calculation, which calculates the energy distribution of the spectrum; N is the number of points in the FFT, which determines the frequency resolution; The number of Mel filters controls the fineness of the frequency band division; Frequency index; This is the index of the filter; For the first Each filter at frequency Weight at each location.

[0062] Temporal framing refers to dividing the speech signal into short, overlapping frames, each 20ms-40ms long, with a 50% overlap; power spectrum calculation is used to obtain the energy distribution of the signal at different frequencies. MFCC divides the power spectrum through a set of... A Mel filter, the Mel scale is a non-linear frequency scale that focuses more on the low-frequency region, which is more sensitive to the human ear.

[0063] Physiological signal extraction specifically includes monitoring heart rate signals and normalizing them:

[0064]

[0065]

[0066] in, For physiological signal feature vectors, for The normalized value of the heart rate signal at any given time. for Heart rate value at any time This represents the historical average heart rate. The historical heart rate standard deviation, The time interval is the dimension of the physiological signal feature vector.

[0067] Behavioral signal extraction involves using pose estimation algorithms to continuously detect limb keypoints in a video sequence, calculating the Euclidean distance change between adjacent frames of the limb keypoint positions, and performing this calculation within a time unit. For example, in a frame, if the displacement of a keypoint exceeds a preset threshold, then an action is considered to have occurred. ,otherwise Within the specified time window Within that time period, count the total number of action events that occurred, and then divide by the time. The average action frequency was obtained. :

[0068]

[0069] in, The average frequency of limb movements. For time unit Detected action events, For time windows.

[0070] The dynamic feature fusion unit from multiple data sources is used to achieve cross-modal coupling through a learnable attention mechanism.

[0071] Get Time Input speech feature vector Physiological signal feature vector Average frequency of limb movements Project the features of all the above modalities onto a unified dimension. After the projection 3D feature vector , , :

[0072]

[0073]

[0074]

[0075] in, , These are the projection matrix parameters. The dimension is Used to convert scalars Convert to Dimensional vector.

[0076] Through an attention The unit generates dynamic weights for each modality at the current time step based on the projected features and emotional state of the previous time step.

[0077]

[0078]

[0079]

[0080] in, Indicates a splicing operation; yes At any moment The hidden state and cell state; It is the emotional state vector of the previous moment, as The initial state input; It is The weight matrix, for The hidden layer dimension is used to... Mapped to a 3D logits vector; For the weighting coefficients, satisfying .

[0081] Using weighted coefficients, the projection features of each modality at the current time are weighted and summed to obtain the final fused feature vector. :

[0082] .

[0083] The sentiment state estimation and modeling unit constructs a three-dimensional sentiment vector space and predicts sentiment states by fusing temporal features and reinforcement learning, transforming multimodal signals into quantifiable sentiment indicators. Specifically, it uses recurrent neural networks to model the temporal dynamics of sentiment, with the current multimodal fusion features as input. The output is the estimated sentiment state. ,

[0084]

[0085] in, for exist The hidden state at any given moment.

[0086] This allows emotional states to be quantified in a continuous three-dimensional space.

[0087]

[0088] in, Let be the emotional state vector at time t. Let t represent the positive or negative emotional tendency. Let t be the intensity of emotion. Let t represent the degree of emotional dominance.

[0089] The personalized interaction strategy generation unit is used to generate personalized interaction strategies for controlling the robot body. Specifically, it includes:

[0090] Define a Markov decision process:

[0091]

[0092] Where S is the state space, i.e., the emotional state vector. A is used to quantify the user's real-time emotional state and provide input to the decision network; A is the action space. ,For example =“Voice comfort” ="Play soothing music", =“I suggest we play a little game” ="Contact nursing staff", as the output of the decision network; R is the reward function, which at time step Output The definition of is:

[0093]

[0094] in, Represents the time step Instant rewards received , indicating from arrive The change in valence at time t. , indicating from arrive Changes in arousal level over time. represent User engagement at all times These are weighting coefficients used to balance the importance of different reward items. It is a positive constant, serving as the upper and lower bounds of the clipping range. It is the natural logarithm function; For discount factor parameters, For example, 0.99 is used to calculate the present value of future rewards.

[0095] The expected reward (action value function) for taking action a in state s is updated using a dual-Q network, and the risk of overestimation is reduced by using two independent critic networks to minimize the value.

[0096]

[0097] in, This represents the Q-value of the state-action pair in the current critic network. The current critic network parameters can be optimized. This allows the critic network to more accurately evaluate the value of actions. The Q-value of the target critic network. Here are the parameters for the target critic network, where i=1,2 indicates a dual network. For the next emotional state, The next action generated by the target policy network. Here, R is the discount factor parameter, R is the instant reward, and E[] is the expected operator.

[0098] The optimized policy network directly optimizes consecutive actions through deterministic policy gradients, generating actions that maximize the Q-value.

[0099]

[0100] in, Actions generated based on the current state; Policy network parameters, by adjusting maximize Choose the action that maximizes the Q value; The Q-value generated by the first critic network, The expectation of the sampled state s in the experience replay pool D (the experience replay pool stores historical interaction data), () represents the Q-value evaluation of the state-action pair by the first critic network. The action generated by the policy network in state s.

[0101] An example of using the above-mentioned interactive strategy of the present invention to control a service robot could be an elderly user in a public area of ​​a nursing home, where the service robot's task is to maintain the user's positive mood.

[0102] First, the robot collects user data through sensors, and a dynamic feature fusion unit from multiple data sources calculates the current emotional state. At the initial moment hour, This indicates that the user is in a slightly negative, calm, and moderately controlled emotional state. The Q network evaluates all available actions. Q value Action space The calculated Q value is assumed to be: , , , Choose the action with the highest Q value: .

[0103] Control the robot to perform actions (Plays nostalgic music). In the next moment... Monitor the user again and calculate the new emotional state. . Participation also increased from Rise to .

[0104] Calculate the execution action Instant rewards received afterward .

[0105]

[0106] in, .

[0107] Substitute specific values ​​and assume ,

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] Calculate the reward:

[0114]

[0115] Store this interactive experience in the experience replay pool.

[0116] Through extensive learning in this way, the robot will eventually master the optimal interaction strategy tailored to a specific user.

[0117] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A companion robot system for the elderly based on multimodal emotional cognitive reinforcement learning, characterized in that, The system includes: a robot body, a multi-source data acquisition and feature extraction unit, a dynamic feature fusion unit from multiple data sources, an emotion state estimation and modeling unit, and a personalized interaction strategy generation unit. The multi-source data acquisition and feature extraction unit is used to extract speech signals, physiological signals, and behavioral signals; The dynamic feature fusion unit from multiple data sources is used to achieve cross-modal coupling through a learnable attention mechanism; The sentiment state estimation and modeling unit is used to construct a three-dimensional sentiment vector space, predict sentiment state by fusing temporal features and reinforcement learning, and transform multimodal signals into quantifiable sentiment indicators. The personalized interaction strategy generation unit is used to generate personalized interaction strategies for controlling the robot body.

2. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 1, characterized in that, Speech signal extraction includes extracting speech feature vectors from the original speech signal using the Mel-Cepstral Coefficients (MFCC) method. : ； ； in, For speech feature vectors; The MFCC coefficient number; For the serial number MFCC characteristics at that time; For time-domain framing, it represents the t-th frame after segmentation of the original speech signal; The Fourier transform represents the conversion of a time-domain signal into a frequency-domain signal. This indicates power spectrum calculation, which calculates the energy distribution of the spectrum; N is the number of points in the FFT, which determines the frequency resolution; The number of Mel filters controls the fineness of the frequency band division; Frequency index; This is the index of the filter; For the first Each filter at frequency Weight at each location.

3. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 1, characterized in that, Physiological signal extraction specifically includes monitoring heart rate signals and normalizing them: ； ； in, For physiological signal feature vectors, for The normalized value of the heart rate signal at any given time. for Heart rate value at any time This represents the historical average heart rate. The historical heart rate standard deviation, The time interval is the dimension of the physiological signal feature vector.

4. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 1, characterized in that, Behavioral signal extraction involves using pose estimation algorithms to continuously detect limb keypoints in a video sequence, calculating the Euclidean distance change between adjacent frames of the limb keypoint positions, and performing this calculation within a time unit. For example, in a frame, if the displacement of a keypoint exceeds a preset threshold, then an action is considered to have occurred. ,otherwise ; Within the specified time window Within that time period, count the total number of action events that occurred, and then divide by the time. The average action frequency was obtained. : ； in, The average frequency of limb movements. For time unit Detected action events, For time windows.

5. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 1, characterized in that, Achieving cross-modal coupling through learnable attention mechanisms specifically includes: Get Time Input speech feature vector Physiological signal feature vector Average frequency of limb movements Projecting the features of all modalities onto a unified dimension. After the projection 3D feature vector , , : ； ； ； in, , These are the projection matrix parameters. The dimension is Used to convert scalars Convert to dimensional vector; Through attention The unit generates dynamic weights for each modality at the current time step based on the projected features and emotional state of the previous time step. ； ； ； in, Indicates a splicing operation; yes At any moment The hidden state and cell state; It is the emotional state vector of the previous moment, as The initial state input; It is The weight matrix, for The hidden layer dimension is used to... Mapped to a 3D logits vector; For the weighting coefficients, satisfying ; Using weighted coefficients, the projection features of each modality at the current time are weighted and summed to obtain the final fused feature vector. : 。 6. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 1, characterized in that, This approach predicts sentiment states by fusing temporal features and reinforcement learning, transforming multimodal signals into quantifiable sentiment indicators. Specifically, it uses recurrent neural networks to model the temporal dynamics of sentiment, with the current multimodal fusion features as input. The output is the estimated sentiment state. , ； in, for exist The hidden state at any given moment; This allows emotional states to be quantified in a continuous three-dimensional space. ； in, Let be the emotional state vector at time t. Let t represent the positive or negative emotional tendency. Let t be the intensity of emotion. Let t represent the degree of emotional dominance.

7. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 1, characterized in that, Generate personalized interaction strategies for controlling the robot body, specifically including: Define a Markov decision process: ； Where S is the state space, i.e., the emotional state vector. A is used to quantify the user's real-time emotional state and provide input to the decision network; A is the action space. ,For example =Voice comforting, "Play soothing music", "I suggest we play a little game." ="Contact nursing staff", as the output of the decision network; R is the reward function, which is applied at time step... Output The definition of is: ； in, Represents the time step Instant rewards received , indicating from arrive The change in valence at time t. , indicating from arrive Changes in arousal level over time. represent User engagement at all times These are weighting coefficients used to balance the importance of different reward items. It is a positive constant, serving as the upper and lower bounds of the clipping range. It is the natural logarithm function; For discount factor parameters, It is used to calculate the present value of future rewards.

8. The elderly companion robot system based on multimodal emotional cognitive reinforcement learning according to claim 7, characterized in that, The generation of personalized interaction strategies for controlling the robot body also includes: The expected reward of taking action a in state s is updated through a double Q-network, and the risk of overestimation is reduced by using two independent critic networks to minimize the reward. ； in, This represents the Q-value of the state-action pair in the current critic network. The current critic network parameters can be optimized. This allows the critic network to more accurately evaluate the value of actions. The Q-value of the target critic network. Here are the parameters for the target critic network, where i=1,2 indicates a dual network. For the next emotional state, The next action generated by the target policy network. Here, R is the discount factor parameter, and R is the instant reward. For expectation operators; The optimized policy network directly optimizes consecutive actions through deterministic policy gradients, generating actions that maximize the Q-value. ； in, Actions generated based on the current state; Policy network parameters, by adjusting maximize Choose the action that maximizes the Q value; The Q-value generated by the first critic network, The expectation of the sampled state s in the experience replay pool D, () represents the Q-value evaluation of the state-action pair by the first critic network. The action generated by the policy network in state s.

Images