Emotion recognition method, emotion accompanying method and emotion accompanying system applied to old-age nursing

Through multimodal feature fusion and behavioral decision-making model, the problem of insufficient emotional care for middle-aged and elderly people in the existing technology is solved, accurate emotional recognition and personalized emotional care are achieved, which relieves the loneliness of the elderly and improves the quality of life.

CN120260834APending Publication Date: 2025-07-04ESTUN MEDICAL TECH (NANJING) CO LTD

Patent Information

Application Number
CN202510237643.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-03-03
Publication Date
2025-07-04

AI Technical Summary

Technical Problem

The existing elderly care system fails to meet the actual needs of the elderly in terms of personalized emotional interaction, emotional monitoring and psychological support, especially inability to provide accurate emotional care plans, resulting in the elderly living in a lonely and lack of companionship for a long time.

Method used

The multimodal feature fusion method is adopted to extract user facial expressions, speech and physiological signal characteristics, build an emotion recognition model, combine the Markov-like decision-making process to perform behavioral selection, and provide personalized emotional care solutions.

Benefits of technology

It has achieved five types of accurate identification of user emotions, provided personalized emotional care, helped the elderly to relieve loneliness, improve the quality of life, and continuously optimized care plans through iterative learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120260834A_ABST
    Figure CN120260834A_ABST
Patent Text Reader

Abstract

The invention discloses an emotion recognition method, an emotion accompanying method and an emotion accompanying system applied to old-age nursing. The emotion accompanying method comprises the following steps: extracting facial expressions, voice features and physiological signal features of a user; and constructing an emotion recognition model, inputting facial expressions, voices and physiological signal features of the user, and outputting an emotion classification result through multi-modal feature fusion, the emotion classification result being a five-classification vector. And an accompanying behavior decision model is constructed, behavior selection is performed by using a Markov-like decision process, and the accompanying behavior decision model comprises a state set and a behavior set. According to the emotion recognition method provided by the invention, the emotion of the user can be recognized, and the emotion of the user is classified into five types, so that more accurate emotion recognition is realized; according to the emotion accompanying method provided by the invention, a personalized emotion accompanying scheme can be provided according to the recognized emotion of the user, emotion accompanying is realized, the elderly is helped to relieve loneliness, and the life quality of the elderly is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the technical field of elderly care, and particularly to an emotion recognition method, an emotional companionship method and a system applied to elderly care. Background Art

[0002] With the intensification of the aging society, more and more elderly people live in an environment of loneliness and lack of company for a long time, resulting in a worrying mental health status. Although the existing technologies provide many elderly care systems, most of them focus on the physical care of the elderly or the help with daily life, and lack effective solutions in emotional companionship. Especially in terms of personalized emotional interaction, emotion monitoring and psychological support, the current technologies have not yet met the actual needs of the elderly. Therefore, there is an urgent need for an intelligent and accurate emotion recognition function and a personalized emotional companionship solution to help the elderly relieve loneliness and improve their quality of life.

[0003] Since emotions are strongly correlated with physiological indicators, fusing physiological parameters can more accurately judge the emotion category of the elderly. Patent CN 113139525 A fuzzily processes the emotion category and only simplifies the recognition process with positive, neutral and negative, and cannot achieve a reasonable emotional companionship plan. Summary of the Invention

[0004] On the one hand, this application provides an emotion recognition method applied to elderly care, which has the advantage of being able to recognize the emotions of users and classify the emotions of users into five categories, realizing more accurate emotion recognition.

[0005] The scheme is as follows. An emotion recognition method applied to elderly care includes the following steps:

[0006] S1: Extract the facial expression features of the user;

[0007] S2: Extract the voice features of the user;

[0008] S3: Extract the physiological signal features of the user;

[0009] S4: Construct an emotion recognition model, input the facial expression, voice and physiological signal features of the user, and output an emotion classification result through multi-modal feature fusion. The emotion classification result is a five-class vector;

[0010] S5: Construct an emotion classifier and output the emotion with the highest probability according to the emotion classification result.

[0011] Further, the step S1 includes the following steps:

[0012] Obtain the facial image of the user;

[0013] Identify the user's facial expression from the user's facial image using a Convolutional Neural Network (CNN), and output the facial expression feature vector F c ;

[0014] The Convolutional Neural Network (CNN) includes two Convolution-ReLU-Pooling modules and one fully connected layer.

[0015] Further, step S2 includes the following steps:

[0016] Obtain the user's audio signal;

[0017] Use a simplified Gated Recurrent Unit (GRU) to extract the temporal features in the audio signal, and the output result passes through a fully connected layer to output the emotion feature F s , where the facial expression feature vector F c and the emotion feature F s have the same dimension.

[0018] Further, step S3 includes the following steps:

[0019] Collect the user's heart rate data;

[0020] Perform Min-Max normalization on the collected heart rate data within a defined time period, and then calculate the mean, standard deviation, maximum and minimum values, and change rate respectively to form a 5-dimensional vector Fh.

[0021] Further, in step S4, the emotion recognition model uses the combination of CNN and Gated Recurrent Unit (GRU) network. Through multi-modal feature fusion, finally, a fully connected layer with 5 units and a Softmax activation function are used to complete emotion classification;

[0022] Weightedly fuse the facial expression feature Fc and the speech feature Fs, and then splice them with the physiological signal Fh to form a joint feature vector F fused :

[0023] F fused =[α·F c +β·F s ,F h

[0024] where α and β are weights;

[0025] The emotion classification result P emotion is:

[0026] P emotion =Softmax(F fused ·W + b)

[0027] The emotion classification result P emotion ​Expressed as a five-category vector [p(L), p(A), p(H), p(S), p(N)];

[0028] In step S5, the emotion classifier outputs the emotion Emotion with the highest probability among the four emotions p(L), p(A), p(H), p(S), p(N), which is expressed as:

[0029] Emotion = argmax{p(L), p(A), p(H), p(S), p(N)}

[0030] Where L, A, H, S, and N respectively represent the loneliness, anger, happiness, sadness, and neutral emotions of the user.

[0031] On the other hand, the present application provides an emotional companionship method applied to elderly care. Its advantage is that according to the identified user emotions, a personalized emotional companionship plan is provided to achieve emotional companionship, help the elderly relieve loneliness, and improve their quality of life.

[0032] The technical solution is as follows. An emotional companionship method applied to elderly care includes the following steps:

[0033] Identify the user's emotions according to the above-mentioned emotion recognition method;

[0034] Construct a companionship behavior decision model, and use a class Markov decision process to select behaviors. The companionship behavior decision model includes a state set S and a behavior set A corresponding to the state set S;

[0035] The state set S contains five emotional states {L, A, H, S, N}, and the behavior set A includes several companionship behaviors;

[0036] Determine the companionship behavior according to the identified user emotions and the behavior set A.

[0037] Furthermore, it also includes the steps:

[0038] Construct a reward function R (s,a) Used to evaluate the effect after taking a certain action a in a certain state s:

[0039] R (s,a) = w1·ΔE + w2·ΔF - w3·ΔT

[0040] Where ΔE is the improvement amount of emotion change, obtained according to the user's emotions before and after the companionship behavior; ΔF is the improvement amount of the elderly's feedback on the companionship measures, defined in three levels: -0.8 for poor, 0.1 for average, and 0.5 for good; ΔT is the change amount of the companionship time cost, that is, the importance of the duration cost of the companionship measures, expressed by the ratio between the duration of the companionship measures and the improvement amount w1, w2, and w3 are weight factors, with values of 0.6, 0.3, and 0.1 respectively;

[0041] Based on the state space S and the action space A, calculate the transition probability P(S t after taking action A t in each state S t+1 |S t , A t ), and the reward R(S t , A t ) for each action, and obtain the optimal policy through the iterative method.

[0042] Furthermore, it further includes the steps of:

[0043] Construct a database, and update the database according to the results of each emotion detection, including emotion records, behavior selections, and result feedback. The database is represented as:

[0044]

[0045] where E i : the emotional state at the i-th time; T i : the timestamp; B i : the behavior plan taken for E i ; R i : the effect score of the behavior plan.

[0046] On the other hand, the present application provides an elderly care system, including a processor and a memory. When the computer program stored in the memory is called and executed by the processor, the above-mentioned emotion recognition method or emotional escort method is implemented.

[0047] On the other hand, the present application provides a computer-readable medium. When the computer program stored in the computer-readable medium is called and executed by the computer, the above-mentioned emotion recognition method or emotional escort method is implemented.

[0048] In summary, the beneficial effects of the present application are as follows:

[0049] 1. The emotion recognition method provided by the present application can recognize the emotions of users, classify the emotions of users into five categories, and achieve more accurate emotion recognition;

[0050] 2. The emotional escort method provided by the present application can provide personalized emotional escort plans according to the recognized emotions of users, achieve emotional escort, help the elderly relieve loneliness, and improve their quality of life;

[0051] 3. The emotional escort method provided by the present application can continuously learn and iterate, so as to be able to propose the optimal emotional escort plan for improving the emotions of users. Brief Description of the Drawings

[0052] Figure 1 is a schematic flowchart of the emotional companionship method of the present application. Detailed Embodiment

[0053] The following will describe in detail the specific embodiments of the present application with reference to the accompanying drawings.

[0054] A specific embodiment of the present application provides an emotion recognition method applied to elderly care, including the following steps:

[0055] S1: Extract the facial expression features of the user; the step S1 includes the following steps:

[0056] Obtain the facial image of the user;

[0057] Use a convolutional neural network CNN to identify the facial expression of the user from the facial image of the user and output the facial expression feature vector F c ;

[0058] The convolutional neural network CNN includes 2 convolutional-ReLU-pooling modules and a fully connected layer.

[0059] Assume that the input is a facial image with a size of H×W×C, where H is the image height, W is the width, and C is the number of channels (usually 3, representing an RGB color image). The present invention uses a model composed of 2 "convolutional-ReLU-pooling" modules and a fully connected layer to obtain the facial expression feature Fc.

[0060] (1) Convolutional-pooling module

[0061] Denote the input image as the symbol The convolutional layer selects 32 filters with a convolutional kernel of 3x3, a stride of 1, and a padding of 1 (to keep the image size unchanged).

[0062] F1 = Conv2D(X f , filters = 32, kenerl_size = 3, strides = 1, padding = 1)

[0063] The size of the output tensor is H×W×32.

[0064] The convolutional layer extracts local features (such as edges, textures, etc.). Adding the ReLU activation function after the convolutional layer can increase the non-linear expression ability of the model and help the model capture more complex and abstract features. Therefore, using ReLU after the convolutional layer is to perform a non-linear transformation on the extracted features, thereby improving the expression ability of the model.

[0065] A1 = ReLU(F1)

[0066] After the convolutional layer, the ReLU activation function helps the network introduce non-linear feature mapping, while the pooling layer helps compress the spatial dimension and reduce the computational complexity. The pooling layer can reduce the output size of each layer in the network, thereby improving computational efficiency, and by discarding some unimportant local details, it emphasizes the more representative features in the image. Finally, a 2×2 max pooling is set to reduce the spatial dimension and retain important features.

[0067] P1 = MaxPool(A1, pool_size = 2, strides = 2)

[0068] The size of the output tensor is H / 2 × W / 2 × 32.

[0069] In the second convolutional-pooling module, the number of filters is set to 64. Others are the same as those in the first convolutional-pooling module; so the output tensor size is H / 4 × W / 4 × 64.

[0070] (2) Fully connected layer

[0071] Flatten the pooling result and input it into the fully connected layer. Through the fully connected layer, map the features to a smaller space and use ReLU activation. Here, the units parameter of the fully connected layer is set to 128, indicating the number of neurons in the fully connected layer (i.e., the dimension of the output vector). Therefore, the output dimension after passing through the fully connected layer is also 128.

[0072] F c = Dense(F flattened , units = 128, activation = ReLU) to obtain the facial expression feature vector F c .

[0073] S2: Extract the user's speech features; Step S2 includes the following steps:

[0074] Obtain the user's audio signal;

[0075] Adopt a simplified gated recurrent unit GRU to extract the temporal features in the audio signal, and the output result passes through a fully connected layer to output the emotion feature F s , where the facial expression feature vector F c and the emotion feature F s have the same dimension.

[0076] Assume the input is the MFCC feature matrix of the speech signal where T is the number of time steps and D is the feature dimension.

[0077] To fuse image features and speech features in one dimension, the dimension of the hidden state is set to 128 here, and then the output features of the GRU are input into a fully connected layer with 128 units.

[0078] h t = GRU(X s , h t-1 )

[0079] F s = Dense(h T , units = 128, activation = ReLU)

[0080] The GRU finally outputs the emotion feature F s .

[0081] S3: Extract user physiological signal features; the step S3 includes the following steps:

[0082] Collect the user's heart rate data;

[0083] Perform Min - Max normalization on the collected heart rate data within a defined time period (such as 500 ms), and then calculate the mean, standard deviation, maximum and minimum values, and change rate respectively to form a 5 - dimensional vector Fh.

[0084] S4: Construct an emotion recognition model, input the user's facial expression, speech, and physiological signal features, and output the emotion classification result through multi - modal feature fusion. The emotion classification result is a five - classification vector;

[0085] In the step S4, the emotion recognition model combines a CNN and a gated recurrent unit GRU network. Through multi - modal feature fusion, finally, a fully connected layer with 5 units and a Softmax activation function are used to complete emotion classification;

[0086] Weight - fuse the facial expression feature Fc and the speech feature Fs, and then splice them with the physiological signal Fh into a joint feature vector F fused :

[0087] F fused = [α·F c + β·F s , F h

[0088] Where α and β are weights; if the speech signal is missing or has low energy, then α takes a relatively large value, such as 0.8, and the weight β of the speech feature is 0.2. When the speech feature is significant and the speech emotion is very obvious (such as anger, happiness), then take α = 0.9 and β = 0.1.

[0089] The emotion classification result P emotion ​is:

[0090] P emotion = Softmax(F fused ·W + b)

[0091] The emotion classification result P emotion is represented as a five - classification vector [p(L), p(A), p(H), p(S), p(N)];

[0092] S5: Construct an emotion classifier, and output the emotion with the highest probability according to the emotion classification result.

[0093] The emotion classifier outputs the emotion Emotion with the highest probability among the four emotions p(L), p(A), p(H), p(S), p(N), which is represented as:

[0094] Emotion = argmax{p(L), p(A), p(H), p(S), p(N)}

[0095] where L, A, H, S, and N respectively represent the user's loneliness, anger, happiness, sadness, and neutral emotions.

[0096] Another specific embodiment of this application provides an emotional companionship method applied to elderly care, including the following steps:

[0097] Identify the user's emotion according to the emotion recognition method described above;

[0098] Construct a companion behavior decision model, which includes a state set S and a behavior set A corresponding to the state set S;

[0099] The state set S contains five emotion states {L, A, H, S, N}, and the behavior set A includes several companion behaviors; typically, the personalized behavior sets for the emotions L, A, and S that require companionship are:

[0100] A L = {Chat, play chess, video accompany}

[0101] A A = {Voice comfort, talk about happy things, outdoor activities}

[0102] A S = {Tell stories, tell jokes, chat for comfort}

[0103] Determine the companion behavior according to the identified user emotion and the behavior set A.

[0104] Construct a reward function R (s,a)It is used to evaluate the effect after taking an action a in a certain state s, representing the degree of emotional change and the effect of companionship. The goal of the reward function is to enable the elderly to transform from a negative emotional state (L, A, S) to a more positive state H or a stable state N by selecting appropriate actions.

[0105] R (s,a) = w1·ΔE + w2·ΔF - w3·ΔT

[0106] Among them, ΔE is the improvement amount of emotional change, obtained based on the emotions of the user before and after the companionship behavior, such as the improvement from "angry" to "happy"; ΔF is the improvement amount of the elderly's feedback on the companionship measures, defined in three levels: -0.8 for poor, 0.1 for average, and 0.5 for good; ΔT is the change amount of the companionship time cost, that is, the importance of the duration cost of the companionship measures, represented by the ratio between the duration of the companionship measures and the improvement amount. w1, w2, and w3 are weight factors, taking values of 0.6, 0.3, and 0.1 respectively.

[0107] Specifically, for the lonely state L, if the companionship measures can significantly reduce the sense of loneliness, then ΔE is positive and the reward function is larger. For the angry state A, if the voice soothing measures taken can effectively relieve the emotional fluctuations, the reward is greater. For the sad state S, by telling jokes or stories to relieve sadness, the reward value increases.

[0108] A typical score of ΔE is shown in the following table:

[0109]

[0110]

[0111] Based on the state space S and the action space A, calculate the transition probability P(S t |S t ), A t+1 |S t , A t ) and the reward R(S t , A t ) of each action, and obtain the optimal strategy through the iterative method.

[0112] Assume that the current state is S t = L (lonely state), and the possible actions are A1 (chat), A2 (play chess), and A3 (video companionship). Calculate the corresponding rewards as R(L, A1) = 0.8, R(L, A2) = 0.5, R(L, A3) = 0.6; the system selects the action with the highest reward, that is, chat (A1), because it can most effectively relieve the loneliness of the elderly.

[0113] It further includes steps: constructing a database, and updating the database according to each emotion detection result, including emotion records, behavior selections, and result feedbacks. The database is represented as:

[0114]

[0115] where E i : the emotional state at the i-th time; T i : timestamp; B i : the behavior plan taken for E i ; R i : the effect score of the behavior plan (such as the emotional changes obtained through facial expressions and voice feedback).

[0116] Another specific embodiment of the present application provides a pension care system, including a processor and a memory. When the computer program stored in the memory is called and executed by the processor, it implements the emotion recognition method or the emotional escort method as described above.

[0117] Another specific embodiment of the present application provides a computer-readable medium. When the computer program stored in the computer-readable medium is called and executed by the computer, it implements the emotion recognition method or the emotional escort method as described above.

[0118] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the creative concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application.

Claims

1. An emotion recognition method applied to elderly care, characterized in that, It includes the following steps: S1: Extract the facial expression features of the user; S2: Extract the voice features of the user; S3: Extract the physiological signal features of the user; S4: Construct an emotion recognition model, input the facial expression, voice and physiological signal features of the user, and through multi-modal feature fusion, output the emotion classification result, and the emotion classification result is a five-classification vector; S5: Construct an emotion classifier, and output the emotion with the highest probability according to the emotion classification result.

2. The emotion recognition method applied to elderly care according to claim 1, wherein The step S1 includes the following steps: Obtain the facial image of the user; Use a Convolutional Neural Network (CNN) to recognize the user's facial expression from the user's facial image and output a facial expression feature vector F c ; The convolutional neural network CNN includes 2 convolutional-ReLU-pooling modules and a fully connected layer.

3. The emotion recognition method applied to elderly care according to claim 1, wherein The step S2 includes the following steps: Obtain the audio signal of the user; A simplified gated recurrent unit (GRU) is used to extract the temporal features in the audio signal, and the output result passes through a fully connected layer to output the emotion feature F s , where the facial expression feature vector F c and the emotion feature F s have the same dimension.

4. The emotion recognition method for elderly care according to claim 1, characterized in that The step S3 includes the following steps: Collect the heart rate data of the user; Perform Min-Max normalization on the collected heart rate data within the defined time period, and then calculate the mean, standard deviation, maximum and minimum values and change rate respectively to form a 5-dimensional vector Fh.

5. The emotion recognition method for elderly care according to claim 1, characterized in that In the step S4, the emotion recognition model adopts the combination of CNN and gated recurrent unit GRU network, through multi-modal feature fusion, and finally uses a fully connected layer with units of 5 and a Softmax activation function to complete emotion classification; The facial expression feature Fc and the voice feature Fs are weighted and fused, and then concatenated with the physiological signal Fh to form a joint feature vector F fused : F fused = [α·F c + β·F s , F h ​ Where α and β are weights; Emotion classification result P emotion is as follows: P emotion = Softmax(F fused ·W + b) Emotion classification result P emotion Expressed as a five-classification vector [p(L), p(A), p(H), p(S), p(N)]; In step S5, the emotion classifier outputs the emotion Emotion with the highest probability among the four types of emotions p(L), p(A), p(H), p(S), p(N), which is expressed as: Emotion = argmax{p(L), p(A), p(H), p(S), p(N)} Where L, A, H, S, and N respectively represent the loneliness, anger, happiness, sadness and neutral emotions of the user.

6. An emotional companionship method applied to elderly care, characterized in that, It includes the following steps: Identify the emotion of the user according to the emotion recognition method described in any one of claims 1-5; Construct a companion behavior decision-making model, and use a class Markov decision-making process to perform behavior selection. The companion behavior decision-making model includes a state set S and a behavior set A corresponding to the state set S; The state set S contains five emotion states {L, A, H, S, N}, and the behavior set A includes several companion behaviors; Determine the companion behavior according to the identified user emotion and the behavior set A.

7. The emotional companionship method for elderly care according to claim 6, wherein It also includes the step: Construct the reward function R (s,a) Used to evaluate the effect after taking an action a in a certain state s: R (s,a) = w1·ΔE + w2·ΔF - w3·ΔT Where ΔE is the improvement in emotional change, obtained based on the user's emotions before and after the accompanying behavior; ΔF is the improvement in the elderly's feedback on the accompanying measures, defined in three levels: -0.8 for poor, 0.1 for average, and 0.5 for good; ΔT is the change in the accompanying time cost, that is, the importance of the duration cost of the accompanying measures, expressed as the ratio between the duration of the accompanying measures and the improvement amount w1, w2, and w3 are weight factors, with values of 0.6, 0.3, and 0.1 respectively; Based on the state space S and behavior space A, calculate each state S t Take action A t The transition probability P(S t+1 |S t ,A t ) and the reward R(S) for each action t ,A t ), and obtain the optimal strategy through iteration.

8. The emotional companionship method applied to elderly care according to claim 7, characterized in that, It also includes the step: Construct a database, and update the database according to each emotion detection result, including emotion records, behavior selection and result feedback. The database is expressed as: Among them, E i : The emotional state at the i-th time; T i : Timestamp; B i : The behavior plan taken for E i ; R i : The effect score of the behavior plan.

9. A nursing care system for the elderly, characterized in that, It includes a processor and a memory. The memory stores a computer program. When the computer program is called and executed by the processor, it realizes the emotion recognition method described in any one of claims 1-5 or the emotional companion method described in any one of claims 6-8.

10. A computer-readable medium, characterized in that, The computer-readable medium stores a computer program. When the computer program is called and executed by the computer, it realizes the emotion recognition method described in any one of claims 1-5 or the emotional companion method described in any one of claims 6-8.

Citation Information

Patent Citations

  • Emotion recognition method based on multi-source information fusion and man-machine interaction system

    CN113139525A

Cited By

  • Intelligent analysis accompanying system for pain symptoms of middle-aged and elderly people

    CN120853832A