Intelligent old-age nursing emotion accompanying robot system based on multi-mode perception

Abstract

The invention discloses an intelligent old-age nursing emotion accompanying robot system based on multi-mode perception, and the system comprises a collection module which is used for passing through a visual sensor, a microphone array and an environment induction probe. According to the invention, multi-modal data such as facial expressions, voice contents, limb actions and home environment information of old people are collected in real time by integrating a plurality of sensor devices such as a visual sensor, a microphone array and an environment sensing probe; afterwards, the system processes and analyzes the collected multi-modal data by using advanced technical means such as a deep learning algorithm, a natural language processing technology and a behavior recognition algorithm, and accurately recognizes the emotional state, the potential demand and the health condition of the elderly; on the basis, the system generates a personalized emotion accompanying strategy, presents a'digital relatives' image highly similar to relatives of the elderly in shape and sound through a 3D holographic projection technology, executes the emotion accompanying strategy and interacts with the elderly.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart elderly care, in particular to a smart elderly care emotional accompanying robot system based on multi-modal perception. BACKGROUND

[0002] With the accelerated development of global population aging, the proportion of the elderly population is rising, and the problem of elderly care has become the focus of social attention. The traditional mode of elderly care mainly relies on family care and institutional care, but both modes face many challenges in actual operation. In terms of family care, with the miniaturization of family structure and the increase of empty-nest families, family members are not able to provide adequate daily care and emotional companionship for the elderly. Institutional care has problems such as uneven resource allocation and insufficient personalized services, making it difficult to meet the diverse needs of the elderly.

[0003] In this context, smart elderly care emerges as a new mode of elderly care. Smart elderly care uses modern information technology such as the Internet of Things, big data, cloud computing, and artificial intelligence to provide more convenient, efficient, and personalized elderly care services. Among them, emotional companionship is an indispensable part of smart elderly care services. The elderly often face emotional problems such as loneliness and loneliness in their later years, which not only affects their mental health but also may have adverse effects on their physical health. Therefore, how to provide effective emotional companionship services for the elderly has become a problem that needs to be solved in the field of smart elderly care.

[0004] Currently, some intelligent accompanying devices have appeared on the market, such as smart speakers and intelligent robots, but these devices still have many shortcomings in emotional companionship. On the one hand, most of these devices can only achieve simple voice interaction functions and cannot accurately identify the emotional state and emotional needs of the elderly; on the other hand, these devices lack personalized accompanying strategies and are difficult to provide targeted emotional support according to the specific circumstances of the elderly. In addition, existing devices are relatively single in terms of interaction mode and cannot provide immersive accompanying experience for the elderly. SUMMARY

[0005] To solve the above technical problems, a smart elderly care emotional accompanying robot system based on multi-modal perception is provided, which solves the problems raised in the background art.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is: A smart elderly care emotional accompanying robot system based on multi-modal perception, comprising: A collection module, the collection module is used to collect the facial expressions, voice content, body movements and home environment information of the elderly through a visual sensor, a microphone array and an environmental sensing probe; The preprocessing module is used to preprocess the collected multimodal data, including noise reduction, enhancement and standardization, in order to improve data quality. The feature extraction module is used to extract features from preprocessed facial expression data using deep learning algorithms to identify the emotional state of the elderly. The recognition module is used to perform speech recognition processing on the speech content, convert it into text information, and further analyze the semantic information in the text through natural language processing technology to identify the potential needs and emotional tendencies of the elderly. The judgment module is used to combine limb movement data and analyze the behavioral patterns of the elderly through behavior recognition algorithms to determine their activity status and health status. A generation module is used to generate personalized emotional companionship strategies based on emotional state, potential needs, emotional tendencies, and activity status. The interactive module is used to present a "digital relative" who is highly similar in appearance and voice to the elderly person's relatives through 3D holographic projection technology, to implement an emotional companionship strategy and to interact and communicate with the elderly person.

[0007] Preferably, the preprocessing of the acquired multimodal data specifically includes: For image data acquired by a visual sensor, a background subtraction method based on a Gaussian mixture model is used to remove complex home background interference, a nonlocal mean denoising algorithm is used to smooth facial skin noise, and histogram equalization is used to enhance the contrast of facial details in low light environments. For audio data acquired by microphone array, firstly, spectral subtraction is used to eliminate steady-state environmental noise, then wavelet packet transform is used to perform thresholding on non-stationary burst noise, and finally, blind source separation algorithm is used to separate human voice from reverberation. For the time-series signals acquired by the environmental sensing probe, the Kalman filter algorithm is used for data smoothing and outlier removal. The state update equation is as follows: ; And the covariance update equation is: ; in, Indicates the first The optimal state estimate at time t. For the first The observed value at time, For Kalman gain, For the observation matrix, To estimate the error covariance matrix, It is the identity matrix; Subsequently, linear interpolation was used to fill in the temporary data loss caused by transmission packet loss, and Z-score standardization was performed on all modal data, with the following formula: ; in The original data, The mean, The standard deviation is uniformly quantified to the interval [0, 1].

[0008] Preferably, the step of using deep learning algorithms to extract features from preprocessed facial expression data and identify the emotional state of the elderly specifically includes: A facial expression recognition model based on a dual-stream convolutional neural network is constructed. The spatial stream network uses the ResNet-50 architecture to extract texture features from static facial images, while the temporal stream network uses the 3D-CNN architecture to extract optical flow features from continuous video frames to capture dynamic changes in micro-expressions. The two feature vectors are concatenated in a fully connected layer, and a channel attention mechanism is introduced to assign higher weights to features in key regions such as the eyes and mouth. The weight calculation formula is as follows: ; in, For channel weight vectors, It is the Sigmoid activation function. , This is the weight matrix of the fully connected layer. This is a global average pooling feature. It is the ReLU activation function; Finally, the features are mapped to a probability distribution containing seven basic emotions—joy, sadness, anger, fear, surprise, disgust, and neutral—using a Softmax classifier. The calculation formula is as follows: ; The emotion corresponding to the highest probability is selected as the current emotional state, where For the first Output score for emotion categories.

[0009] Preferably, the step of analyzing semantic information in the text using natural language processing technology specifically includes: The preprocessed audio signal is input into an end-to-end speech recognition model based on the Transformer architecture, and the output is a text sequence containing timestamps; By using a bidirectional long short-term memory network combined with a conditional random field, named entity recognition was performed on the text to extract key symptom words and sentiment words such as "headache", "taking medicine", and "loneliness". A pre-trained language model based on BERT is constructed, which converts the identified text into 768-dimensional sentence vectors. These vectors are then input into a sentiment analysis head to calculate the sentiment polarity score and intensity of the text. The calculation formula is as follows: ; in, The base score is based on the matching of the sentiment dictionary. This is a score for modification strength based on dependency parsing. and Normalized weight coefficients and ; Simultaneously, the dependency relationships between entities are established through graph neural networks, and the co-occurrence probability is calculated: ; If an entity related to "physical discomfort" is detected, accompanied by a negative sentiment polarity score, If an entity of the type "missing children" is detected and If so, it is determined to be a need for emotional comfort.

[0010] Preferably, the step of combining limb movement data and analyzing the behavioral patterns of the elderly through behavior recognition algorithms specifically includes: The OpenPose algorithm was used to extract the two-dimensional coordinates of 25 key points of the human body from visual images, and then combined with depth camera data to convert them into a three-dimensional spatial coordinate sequence. Calculate the Euclidean distance, angle, and velocity vector between joints to construct a spatiotemporal skeleton diagram. The formulas for calculating the Euclidean distance and angle are as follows: ; The skeleton graph is input into the spatiotemporal graph convolutional network, and short-term actions and long-term behaviors are captured through multi-scale temporal convolutional layers. Hidden Markov Models are introduced to decode continuous action sequences and calculate the observed sequence. With preset behavior template The likelihood probability is given by the formula: ; in, The first calculation for the forward algorithm Always in a state of readiness The probability, The number of states; When the probability of "falling risk" At that time, it was determined to be an abnormal health condition.

[0011] Preferably, the generation of personalized emotional companionship strategies specifically includes: A multimodal emotional feature database for the elderly was established, storing historical emotional fluctuation curves, high-frequency demand words, and daily behavioral habit parameters. A weighted fusion algorithm was used to calculate the current comprehensive emotional index, with the following formula: ; in, Facial emotion score, ranging from [-1, 1]. To score the emotion of the voice, The score represents the behavioral state, with 1 for normal and -1 for abnormal. For dynamic weighting coefficients that change over time, satisfying If the current time period is a historically high-incidence period for depression, then and Increased to 0.4, It dropped to 0.2; The policy value function is updated based on the reinforcement learning algorithm, and the formula is: ; in For immediate rewards, such as a smile response being +1. As a discount factor, The learning rate; If sadness is detected and historical data shows that listening to old songs can effectively alleviate it, then the strategy of "playing classic operas" will be generated first.

[0012] Preferably, the presentation of a "digital relative" image that closely resembles the elderly relative in appearance and voice using 3D holographic projection technology specifically includes: Multi-view RGB-D video and audio data of relatives were pre-collected. A high-fidelity 3D facial mesh model was reconstructed using neural radiation field technology. Facial muscle control parameters were bound using BlendShape basis functions to achieve precise driving of micro-expressions such as blinking and mouth opening. The vertex displacement calculation formula is as follows: ; in Based on the vertices of the mesh, For the first The weight of each expression base, This corresponds to the shape offset. On the speech synthesis end, an acoustic model based on Tacotron2 and WaveRNN is used to clone the timbre, pitch and prosodic features of relatives, and convert the text to be replied into a speech waveform that is highly similar to the voiceprint of relatives. During the interaction, facial muscle parameters and lip-sync animation are adjusted in real time based on semantic understanding results to ensure that visual lip movements and auditory speech are perfectly aligned at the millisecond-level timestamp, and the alignment error must meet the following requirements. .

[0013] Preferably, the implementation of the emotional companionship strategy, which involves interacting and communicating with the elderly, specifically includes: When the strategy generates the "chat to relieve boredom" instruction, it calls the dialogue engine based on retrieval-enhanced generation (RAG) technology to retrieve historical topics of interest to the elderly from the local knowledge base and generate a reply text that matches the spoken language style of relatives by combining the large language model. On the holographic projection platform, the "digital family member" avatar is driven to perform corresponding body language, such as leaning forward to indicate attention and using gestures to assist in expression; at the same time, the pupillary changes and head posture of the elderly are monitored in real time, and the angle of gaze deviation is calculated. ; If detected If the duration of continuous eye closure exceeds 2 seconds, the interaction rhythm will be dynamically adjusted by reducing the background volume and switching topic keywords to avoid information overload.

[0014] Preferably, it also includes the analysis of home environment information, specifically including: Collect data on indoor temperature and humidity, light intensity, air quality, and the usage status of home appliances; Construct an environmental comfort evaluation model and use fuzzy logic reasoning to calculate the "livability index" of the current environment; When environmental data shows that the temperature is too low or the light is too dim, combined with the results of the elderly's emotion recognition, if "irritability" or "anxiety" is also detected, the system will automatically generate an "environment adjustment suggestion" strategy. The "digital family member" will then suggest adjusting the air conditioner temperature or turning on the warm light in a caring tone, and directly control the smart home to perform the adjustment operation through the Internet of Things protocol, so as to achieve dual intervention of environment and emotion.

[0015] Preferably, it also includes emergency response to sudden health risks in the elderly during the interaction process, specifically including: During continuous monitoring of facial expressions, voice, and body movements, multiple risk thresholds are set; a comprehensive risk scoring model is constructed, with the formula as follows: ; in The probability of a "painful" expression detected by a visual algorithm. The confidence scores for keywords such as "help" and "feeling unwell" were extracted for speech recognition. To determine the probability of motion recognition capturing a "falling" or "curling" posture. The weighting coefficients and , The Sigmoid activation function maps the scores to the [0, 1] interval; When the visual algorithm detects "painful" facial features for 10 consecutive frames, speech recognition extracts specific keywords, and motion recognition captures abnormal postures, it leads to... When the value is greater than 0.9, the highest level of emergency response mechanism is triggered; Immediately suspend routine emotional support conversations and switch to a serious medical guidance mode, displaying first aid guidance animations via holographic projection and playing clear voice instructions; At the same time, the system uses an environmental sensing probe to call up millimeter-wave radar to confirm vital signs, and sends alarm information, including the current location coordinates, vital sign data and a 5-second video clip of the scene, to the guardian's terminal and the community medical platform through an encrypted channel.

[0016] Compared with existing technologies, this invention provides a smart elderly care emotional companionship robot system based on multimodal perception, which has the following beneficial effects: This invention integrates multiple sensor devices, including visual sensors, microphone arrays, and environmental sensing probes, to collect multimodal data in real time, such as facial expressions, voice content, body movements, and home environment information of the elderly. Subsequently, the system uses advanced technologies such as deep learning algorithms, natural language processing, and behavior recognition algorithms to process and analyze the collected multimodal data, accurately identifying the elderly's emotional state, potential needs, and health status. Based on this, the system generates personalized emotional companionship strategies and uses 3D holographic projection technology to present a "digital relative" image that is highly similar in appearance and voice to the elderly's relatives, executing the emotional companionship strategies and interacting with the elderly. The system can also automatically adjust the home environment based on information about the living environment and the elderly person's emotions, achieving dual intervention through both environmental and emotional means. Finally, the system has an emergency response mechanism that can respond immediately to sudden health risks in the elderly person, displaying first aid guidance via holographic projection and simultaneously sending alarm information to guardians and medical platforms, providing timely safety assurance. These functions collectively improve the quality of life and mental health of the elderly. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system framework of the present invention; Figure 2 This is a schematic diagram of the method flow for S101-S104 in this invention; Figure 3 This is a schematic diagram of the method flow for S201-S204 in this invention; Figure 4 This is a schematic diagram of the method flow for S301-S305 in this invention; Figure 5 This is a schematic diagram of the method flow for S401-S405 in this invention. Detailed Implementation

[0018] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0019] Example 1 Please refer to Figure 1 As shown, a smart elderly care emotional companion robot system based on multimodal perception includes: The data acquisition module is used to collect facial expressions, voice content, body movements and home environment information of the elderly through visual sensors, microphone arrays and environmental sensing probes. The preprocessing module is used to preprocess the collected multimodal data, including noise reduction, enhancement and standardization, in order to improve data quality. The feature extraction module is used to extract features from preprocessed facial expression data using deep learning algorithms to identify the emotional state of the elderly. The recognition module is used to perform speech recognition processing on the speech content, convert it into text information, and further analyze the semantic information in the text through natural language processing technology to identify the potential needs and emotional tendencies of the elderly. The judgment module is used to combine limb movement data and analyze the behavioral patterns of the elderly through behavior recognition algorithms to determine their activity status and health status. The generation module is used to generate personalized emotional companionship strategies based on emotional state, potential needs, emotional tendencies, and activity status. The interactive module uses 3D holographic projection technology to present a "digital relative" who is highly similar in appearance and voice to the elderly, implements an emotional companionship strategy, and interacts and communicates with the elderly.

[0020] Please refer to Figure 2 As shown, the preprocessing of the collected multimodal data includes: S101. For image data acquired by the visual sensor, a background subtraction method based on Gaussian mixture model is used to remove complex home background interference, a non-local mean denoising algorithm is used to smooth facial skin noise, and histogram equalization is used to enhance the contrast of facial details in low light environment. S102. For the audio data collected by the microphone array, firstly, the steady-state noise of the environment is eliminated by spectral subtraction, then the non-stationary burst noise is thresholded based on wavelet packet transform, and finally, the human voice and reverberation are separated by blind source separation algorithm. S103. For the time-series signals acquired by the environmental sensing probe, the Kalman filter algorithm is used for data smoothing and outlier removal. The state update equation is as follows: ; And the covariance update equation is: ; in, Indicates the first The optimal state estimate at time t. For the first The observed value at time, For Kalman gain, For the observation matrix, To estimate the error covariance matrix, It is the identity matrix; S104. Subsequently, linear interpolation is used to fill in the temporary data loss caused by transmission packet loss, and Z-score standardization is performed on all modal data. The formula is as follows: ; in The original data, The mean, The standard deviation is uniformly quantified to the interval [0, 1].

[0021] Please refer to Figure 3 As shown, deep learning algorithms are used to extract features from preprocessed facial expression data to identify the emotional state of elderly people, specifically including: S201. Construct an expression recognition model based on a dual-stream convolutional neural network, wherein the spatial stream network uses the ResNet-50 architecture to extract texture features from static facial images, and the temporal stream network uses the 3D-CNN architecture to extract optical flow features from continuous video frames to capture dynamic changes in micro-expressions. S202. The two feature vectors are concatenated in a fully connected layer, and a channel attention mechanism is introduced to assign higher weights to key region features such as the eyes and mouth. The weight calculation formula is as follows: ; in, For channel weight vectors, It is the Sigmoid activation function. , This is the weight matrix of the fully connected layer. This is a global average pooling feature. It is the ReLU activation function; S203. Finally, the features are mapped to a probability distribution containing seven basic emotions—joy, sadness, anger, fear, surprise, disgust, and neutral—using a Softmax classifier. The calculation formula is as follows: ; S204. Select the emotion corresponding to the highest probability as the current emotional state, where For the first Output score for emotion categories.

[0022] Please refer to Figure 4 As shown, natural language processing techniques are used to analyze semantic information in text, specifically including: S301. Input the preprocessed audio signal into an end-to-end speech recognition model based on the Transformer architecture, and output a text sequence containing timestamps; S302. Using a bidirectional long short-term memory network combined with a conditional random field, named entity recognition is performed on the text to extract key symptom words and sentiment words such as "headache", "taking medicine", and "loneliness". S303. Construct a BERT-based pre-trained language model, convert the identified text into 768-dimensional sentence vectors, and input them into the sentiment analysis head to calculate the sentiment polarity score and intensity of the text. The calculation formula is as follows: ; in, The base score is based on the matching of the sentiment dictionary. This is a score for modification strength based on dependency parsing. and Normalized weight coefficients and ; S304. Simultaneously, the dependency relationships between entities are established through a graph neural network, and the co-occurrence probability is calculated: ; S305. If an entity related to "physical discomfort" is detected and accompanied by a negative sentiment polarity score, If an entity of the type "missing children" is detected and If so, it is determined to be a need for emotional comfort.

[0023] Please refer to Figure 5 As shown, by combining limb movement data, behavioral recognition algorithms are used to analyze the behavioral patterns of the elderly, specifically including: S401. Use the OpenPose algorithm to extract the two-dimensional coordinates of 25 key points of the human body from the visual image, and combine them with depth camera data to convert them into a three-dimensional spatial coordinate sequence. S402. Calculate the Euclidean distance, angle, and motion velocity vector between joints to construct a spatiotemporal skeleton diagram. The formulas for calculating the Euclidean distance and angle are as follows: ; S403. Input the skeleton graph into the spatiotemporal graph convolutional network and capture short-term actions and long-term behaviors through multi-scale temporal convolutional layers. S404. Introduce a Hidden Markov Model to decode continuous action sequences and calculate the observed sequence. With preset behavior template The likelihood probability is given by the formula: ; in, The first calculation for the forward algorithm Always in a state of readiness The probability, The number of states; S405, When the probability of "falling risk" At that time, it was determined to be an abnormal health condition.

[0024] Generate personalized emotional support strategies, specifically including: A multimodal emotional feature database for the elderly was established, storing historical emotional fluctuation curves, high-frequency demand words, and daily behavioral habit parameters. A weighted fusion algorithm was used to calculate the current comprehensive emotional index, with the following formula: ; in, Facial emotion score, ranging from [-1, 1]. To score the emotion of the voice, The score represents the behavioral state, with 1 for normal and -1 for abnormal. For dynamic weighting coefficients that change over time, satisfying If the current time period is a historically high-incidence period for depression, then and Increased to 0.4, It dropped to 0.2; The policy value function is updated based on the reinforcement learning algorithm, and the formula is: ; in For immediate rewards, such as a smile response being +1. As a discount factor, The learning rate; If sadness is detected and historical data shows that listening to old songs can effectively alleviate it, then the strategy of "playing classic operas" will be generated first.

[0025] Using 3D holographic projection technology, a "digital relative" image that closely resembles the elderly person's appearance and voice is presented, specifically including: Multi-view RGB-D video and audio data of relatives were pre-collected. A high-fidelity 3D facial mesh model was reconstructed using neural radiation field technology. Facial muscle control parameters were bound using BlendShape basis functions to achieve precise driving of micro-expressions such as blinking and mouth opening. The vertex displacement calculation formula is as follows: ; in Based on the vertices of the mesh, For the first The weight of each expression base, This corresponds to the shape offset. On the speech synthesis end, an acoustic model based on Tacotron2 and WaveRNN is used to clone the timbre, pitch and prosodic features of relatives, and convert the text to be replied into a speech waveform that is highly similar to the voiceprint of relatives. During the interaction, facial muscle parameters and lip-sync animation are adjusted in real time based on semantic understanding results to ensure that visual lip movements and auditory speech are perfectly aligned at the millisecond-level timestamp, and the alignment error must meet the following requirements. .

[0026] Implement emotional support strategies and interact with older adults, specifically including: When the strategy generates the "chat to relieve boredom" instruction, it calls the dialogue engine based on retrieval-enhanced generation (RAG) technology to retrieve historical topics of interest to the elderly from the local knowledge base and generate a reply text that matches the spoken language style of relatives by combining the large language model. On the holographic projection platform, the "digital family member" avatar is driven to perform corresponding body language, such as leaning forward to indicate attention and using gestures to assist in expression; at the same time, the pupillary changes and head posture of the elderly are monitored in real time, and the angle of gaze deviation is calculated. ; If detected If the duration of continuous eye closure exceeds 2 seconds, the interaction rhythm will be dynamically adjusted by reducing the background volume and switching topic keywords to avoid information overload.

[0027] It also includes the analysis of home environment information, specifically including: Collect data on indoor temperature and humidity, light intensity, air quality, and the usage status of home appliances; Construct an environmental comfort evaluation model and use fuzzy logic reasoning to calculate the "livability index" of the current environment; When environmental data shows that the temperature is too low or the light is too dim, combined with the results of the elderly's emotion recognition, if "irritability" or "anxiety" is also detected, the system will automatically generate an "environment adjustment suggestion" strategy. The "digital family member" will then suggest adjusting the air conditioner temperature or turning on the warm light in a caring tone, and directly control the smart home to perform the adjustment operation through the Internet of Things protocol, so as to achieve dual intervention of environment and emotion.

[0028] This also includes emergency response to sudden health risks in the elderly during the interaction process, specifically including: During continuous monitoring of facial expressions, voice, and body movements, multiple risk thresholds are set; a comprehensive risk scoring model is constructed, with the formula as follows: ; in The probability of a "painful" expression detected by a visual algorithm. The confidence scores for keywords such as "help" and "feeling unwell" were extracted for speech recognition. To determine the probability of motion recognition capturing a "falling" or "curling" posture. The weighting coefficients and , The Sigmoid activation function maps the scores to the [0, 1] interval; When the visual algorithm detects "painful" facial features for 10 consecutive frames, speech recognition extracts specific keywords, and motion recognition captures abnormal postures, it leads to... When the value is greater than 0.9, the highest level of emergency response mechanism is triggered; Immediately suspend routine emotional support conversations and switch to a serious medical guidance mode, displaying first aid guidance animations via holographic projection and playing clear voice instructions; At the same time, the system uses an environmental sensing probe to call up millimeter-wave radar to confirm vital signs, and sends alarm information, including the current location coordinates, vital sign data and a 5-second video clip of the scene, to the guardian's terminal and the community medical platform through an encrypted channel.

[0029] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A smart elderly care emotional companion robot system based on multimodal perception, characterized in that, include: The data acquisition module is used to acquire facial expressions, voice content, body movements and home environment information of the elderly through a visual sensor, microphone array and environmental sensing probe; The preprocessing module is used to preprocess the collected multimodal data, including noise reduction, enhancement and standardization, in order to improve data quality. The feature extraction module is used to extract features from preprocessed facial expression data using deep learning algorithms to identify the emotional state of the elderly. The recognition module is used to perform speech recognition processing on the speech content, convert it into text information, and further analyze the semantic information in the text through natural language processing technology to identify the potential needs and emotional tendencies of the elderly. The judgment module is used to combine limb movement data and analyze the behavioral patterns of the elderly through behavior recognition algorithms to determine their activity status and health status. A generation module is used to generate personalized emotional companionship strategies based on emotional state, potential needs, emotional tendencies, and activity status. The interactive module is used to present a "digital relative" who is highly similar in appearance and voice to the elderly person's relatives through 3D holographic projection technology, to implement an emotional companionship strategy and to interact and communicate with the elderly person.

2. The intelligent elderly care emotional companion robot system based on multimodal perception according to claim 1, characterized in that, The preprocessing of the acquired multimodal data specifically includes: For image data acquired by a visual sensor, a background subtraction method based on a Gaussian mixture model is used to remove complex home background interference, a nonlocal mean denoising algorithm is used to smooth facial skin noise, and histogram equalization is used to enhance the contrast of facial details in low light environments. For audio data acquired by microphone array, firstly, spectral subtraction is used to eliminate steady-state environmental noise, then wavelet packet transform is used to perform thresholding on non-stationary burst noise, and finally, blind source separation algorithm is used to separate human voice from reverberation. For the time-series signals acquired by the environmental sensing probe, the Kalman filter algorithm is used for data smoothing and outlier removal. The state update equation is as follows: ； And the covariance update equation is: ； in, Indicates the first The optimal state estimate at time t. For the first The observed value at time, For Kalman gain, For the observation matrix, To estimate the error covariance matrix, It is the identity matrix; Subsequently, linear interpolation was used to fill in the temporary data loss caused by transmission packet loss, and Z-score standardization was performed on all modal data, with the following formula: ； in The original data, The mean, The standard deviation is uniformly quantified to the interval [0, 1].

3. The intelligent elderly care emotional companion robot system based on multimodal perception according to claim 2, characterized in that, The method of using deep learning algorithms to extract features from preprocessed facial expression data and identify the emotional state of the elderly specifically includes: A facial expression recognition model based on a dual-stream convolutional neural network is constructed. The spatial stream network uses the ResNet-50 architecture to extract texture features from static facial images, while the temporal stream network uses the 3D-CNN architecture to extract optical flow features from continuous video frames to capture dynamic changes in micro-expressions. The two feature vectors are concatenated in a fully connected layer, and a channel attention mechanism is introduced to assign higher weights to features in key regions such as the eyes and mouth. The weight calculation formula is as follows: ； in, This is the channel weight vector. It is the Sigmoid activation function. , This is the weight matrix of the fully connected layer. This is a global average pooling feature. It is the ReLU activation function; Finally, the features are mapped to a probability distribution containing seven basic emotions—joy, sadness, anger, fear, surprise, disgust, and neutral—using a Softmax classifier. The calculation formula is as follows: ； The emotion corresponding to the highest probability is selected as the current emotional state, where For the first Output score for emotion categories.

4. The intelligent elderly care emotional companion robot system based on multimodal perception according to claim 3, characterized in that, The analysis of semantic information in text using natural language processing technology specifically includes: The preprocessed audio signal is input into an end-to-end speech recognition model based on the Transformer architecture, and the output is a text sequence containing timestamps; By using a bidirectional long short-term memory network combined with a conditional random field, named entity recognition was performed on the text to extract key symptom words and sentiment words such as "headache", "taking medicine", and "loneliness". A pre-trained language model based on BERT is constructed, which converts the identified text into 768-dimensional sentence vectors. These vectors are then input into a sentiment analysis head to calculate the sentiment polarity score and intensity of the text. The calculation formula is as follows: ； in, The base score is based on the matching of the sentiment dictionary. This is a score for modification strength based on dependency parsing. and Normalized weight coefficients and ; Simultaneously, the dependency relationships between entities are established through graph neural networks, and the co-occurrence probability is calculated: ； If the entity "physical discomfort" is detected and accompanied by a negative sentiment polarity score, If an entity of the type "missing children" is detected and If so, it is determined to be a need for emotional comfort.

5. The intelligent elderly care emotional companion robot system based on multimodal perception according to claim 4, characterized in that, The analysis of elderly people's behavioral patterns using behavior recognition algorithms, combined with limb movement data, specifically includes: The OpenPose algorithm was used to extract the two-dimensional coordinates of 25 key points of the human body from visual images, and then combined with depth camera data to convert them into a three-dimensional spatial coordinate sequence. Calculate the Euclidean distance, angle, and velocity vector between joints to construct a spatiotemporal skeleton diagram. The formulas for calculating the Euclidean distance and angle are as follows: ； The skeleton graph is input into the spatiotemporal graph convolutional network, and short-term actions and long-term behaviors are captured through multi-scale temporal convolutional layers. Hidden Markov Models are introduced to decode continuous action sequences and calculate the observed sequence. With preset behavior template The likelihood probability is given by the formula: ； in, The first calculation for the forward algorithm Always in a state of readiness The probability, The number of states; When the probability of "falling risk" At that time, it was determined to be an abnormal health condition.

6. The intelligent elderly care emotional companion robot system based on multimodal perception according to claim 5, characterized in that, The strategy for generating personalized emotional companionship specifically includes: A multimodal emotional feature database for the elderly was established, storing historical emotional fluctuation curves, high-frequency demand words, and daily behavioral habit parameters. A weighted fusion algorithm was used to calculate the current comprehensive emotional index, with the following formula: ； in, Facial emotion score, ranging from [-1, 1]. To score the emotion of the voice, The score represents the behavioral state, with 1 for normal and -1 for abnormal. For dynamic weighting coefficients that change over time, satisfying If the current time period is a historically high-incidence period for depression, then and Increased to 0.4, It dropped to 0.2; The policy value function is updated based on the reinforcement learning algorithm, and the formula is: ； in For immediate rewards, such as a smile response being +1. As a discount factor, The learning rate; If sadness is detected and historical data shows that listening to old songs can effectively alleviate it, then the strategy of "playing classic operas" will be generated first.

7. A smart elderly care emotional companion robot system based on multimodal perception according to claim 6, characterized in that, The aforementioned method of using 3D holographic projection technology to present "digital relatives" whose appearance and voice are highly similar to those of elderly relatives specifically includes: Multi-view RGB-D video and audio data of relatives were pre-collected. A high-fidelity 3D facial mesh model was reconstructed using neural radiation field technology. Facial muscle control parameters were bound using BlendShape basis functions to achieve precise driving of micro-expressions such as blinking and mouth opening. The vertex displacement calculation formula is as follows: ； in Based on the vertices of the mesh, For the first The weight of each expression base, This corresponds to the shape offset. On the speech synthesis end, an acoustic model based on Tacotron2 and WaveRNN is used to clone the timbre, pitch and prosodic features of relatives, and convert the text to be replied into a speech waveform that is highly similar to the voiceprint of relatives. During the interaction, facial muscle parameters and lip-sync animation are adjusted in real time based on semantic understanding results to ensure that visual lip movements and auditory speech are perfectly aligned at the millisecond-level timestamp, and the alignment error must meet the following requirements. .

8. A smart elderly care emotional companion robot system based on multimodal perception according to claim 7, characterized in that, The aforementioned strategy of providing emotional support and interacting with the elderly specifically includes: When the strategy generates the "chat to relieve boredom" instruction, it calls the dialogue engine based on retrieval augmented generation (RAG) technology to retrieve historical topics of interest to the elderly from the local knowledge base and generate reply text that matches the spoken language style of relatives by combining the large language model. On the holographic projection end, the "digital family member" avatar is driven to make corresponding body language, such as leaning forward to show attention and using gestures to assist in expression; at the same time, the pupil changes and head posture of the elderly are monitored in real time, and the angle of gaze deviation is calculated: ； If detected If the duration of continuous eye closure exceeds 2 seconds, the interaction rhythm will be dynamically adjusted by reducing the background volume and switching topic keywords to avoid information overload.

9. A smart elderly care emotional companion robot system based on multimodal perception according to claim 8, characterized in that, It also includes the analysis of home environment information, specifically including: Collect data on indoor temperature and humidity, light intensity, air quality, and the usage status of home appliances; Construct an environmental comfort evaluation model and use fuzzy logic reasoning to calculate the "livability index" of the current environment; When environmental data shows that the temperature is too low or the light is too dim, combined with the results of the elderly's emotion recognition, if "irritability" or "anxiety" is also detected, the system will automatically generate an "environment adjustment suggestion" strategy. The "digital family member" will then suggest adjusting the air conditioner temperature or turning on the warm light in a caring tone, and directly control the smart home to perform the adjustment operation through the Internet of Things protocol, so as to achieve dual intervention of environment and emotion.

10. A smart elderly care emotional companion robot system based on multimodal perception according to claim 9, characterized in that, This also includes emergency response to sudden health risks in the elderly during the interaction process, specifically including: During continuous monitoring of facial expressions, voice, and body movements, multiple risk thresholds are set; a comprehensive risk scoring model is constructed, with the formula as follows: ； in This represents the probability of a "painful" expression detected by a visual algorithm. To extract confidence scores for keywords such as "help" and "feeling unwell" from speech recognition. To determine the probability of motion recognition capturing a "falling" or "curling" posture. The weighting coefficients and , The Sigmoid activation function maps the scores to the [0, 1] interval; When the visual algorithm detects "painful" facial features for 10 consecutive frames, speech recognition extracts specific keywords, and motion recognition captures abnormal postures, leading to... When the value is greater than 0.9, the highest level of emergency response mechanism is triggered; Immediately suspend routine emotional support conversations and switch to a serious medical guidance mode, displaying first aid guidance animations via holographic projection and playing clear voice instructions; At the same time, the system uses an environmental sensing probe to call up millimeter-wave radar to confirm vital signs, and sends alarm information, including the current location coordinates, vital sign data and a 5-second video clip of the scene, to the guardian's terminal and the community medical platform through an encrypted channel.

Images