A multi-modal emotion recognition and interaction system for dementia patients

Abstract

This invention discloses a multimodal emotion recognition and interaction system for elderly people with dementia, including a multimodal data acquisition module, a micro-expression analysis module, a speech prosody analysis module, a body movement analysis module, a personality profile module, a multimodal fusion module, an emotion state recognition module, an adaptive interaction decision module, and a profile update module. By acquiring facial images, speech signals, and body postures, an aging compensation mechanism is introduced to eliminate the interference of facial relaxation on the recognition of movement units. When semantics is unavailable, emotion perception is achieved solely based on prosodic features and paralinguistic events. The system outputs six types of emotion probability distributions and confidence levels. When the confidence level is insufficient, an active inquiry mode is automatically triggered. This invention comprehensively improves the intelligence level and intervention timeliness of emotional care for elderly people with dementia.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and smart elderly care technology, specifically relating to a multimodal emotion recognition and interaction system for elderly people with dementia. Background Technology

[0002] Elderly people with dementia are unable to effectively express their emotional needs due to cognitive decline, making them highly susceptible to negative emotions such as anxiety, agitation, and distress. If these emotions are not identified and intervened in a timely manner, they may further trigger aggressive behavior or self-harm. In elderly care facilities, existing emotional monitoring methods mostly rely on single-modal signals—collecting only facial expressions or analyzing only speech content. However, in elderly people with moderate to severe dementia, facial muscle relaxation and skin wrinkles lead to a false alarm rate of over 20% for general micro-expression recognition models. Their language expression abilities deteriorate, tending towards phrases or even complete aphasia. Semantic content-based emotion analysis largely fails in this group, and the usability of single-modal solutions in real-world care scenarios is severely insufficient.

[0003] To obtain richer emotional information, some studies have introduced wearable physiological sensors such as ECG, skin conductance, and accelerometer wristbands. However, elderly people with moderate to severe dementia have extremely poor compliance with these devices, often pulling them off themselves or showing strong resistance, which can even directly induce agitated behavior, leading to unstable data collection. Other multimodal fusion schemes attempt to combine facial, voice, and physiological signals, but existing fusion strategies mostly use fixed weights or simple feature splicing, which cannot cope with the problem of a sharp drop in modal quality caused by the elderly turning their backs, occlusion, and environmental noise. In the scenario of continuous 24 / 7 monitoring in elderly care institutions, frequent loss of signals in a certain modality is inevitable. Fixed-weight fusion strategies experience a significant decrease in recognition ability or even stop working in this situation, making it difficult to meet the stability requirements for engineering implementation.

[0004] Therefore, there is an urgent need for a multimodal emotion recognition and interaction system for elderly people with dementia. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal emotion recognition and interaction system for elderly people with dementia, which solves problems such as the inability of elderly people with dementia to cooperate with wearable sensors, baseline drift in micro-expression recognition due to facial aging, failure of speech content analysis due to semantic degradation, fusion system shutdown due to modality loss, and insufficient personalization and lack of online learning ability in existing intervention strategies.

[0006] To achieve the above objectives, the present invention provides a multimodal emotion recognition and interaction system for elderly people with dementia. The system includes a multimodal data acquisition module, a micro-expression analysis module, a speech prosody analysis module, a body movement analysis module, a personality profile module, a multimodal fusion module, an emotion state recognition module, an adaptive interactive decision-making module, and a profile update module.

[0007] Furthermore, the multimodal data acquisition module is used to simultaneously acquire facial image sequences, voice signals, and body posture sequences of elderly people with dementia through non-contact visual sensors and audio sensors.

[0008] Furthermore, the micro-expression analysis module, the speech prosody analysis module, and the body movement analysis module are respectively connected to the multimodal data acquisition module.

[0009] Furthermore, the micro-expression analysis module is used to extract the temporal features of facial action unit intensity from the facial image sequence to obtain a micro-expression emotion feature vector; the speech prosody analysis module is used to extract the fundamental frequency trajectory, energy envelope, speech rate, and speech quality parameters from the speech signal to obtain a speech prosody feature vector; the body movement analysis module is used to extract the kinematic features of joints and repetitive stereotyped movement features from the body posture sequence to obtain a body movement feature vector.

[0010] Furthermore, the personalized profile module stores personalized profiles of elderly people with dementia and generates contextual feature vectors based on the current time, scene, and recent interaction records.

[0011] Furthermore, the multimodal fusion module is used to perform modal quality weighted fusion of the micro-expression emotion feature vector, speech prosody feature vector, body movement feature vector and context feature vector to generate a fused feature vector.

[0012] Furthermore, the emotional state recognition module is used to identify the current emotional state and corresponding confidence level of the elderly with dementia based on the fused feature vector.

[0013] Furthermore, the adaptive interaction decision module is used to select and output an interaction response based on the current emotional state, confidence level, and personalized profile.

[0014] Furthermore, the profile update module is used to update the personalized profile online based on the feedback of the interaction effect after each interaction is completed.

[0015] Furthermore, the multimodal data acquisition module includes a visual acquisition submodule, comprising at least one RGB-D camera, for synchronously acquiring the facial image sequence and body pose sequence at a preset frame rate.

[0016] The audio acquisition submodule includes a microphone array for acquiring the speech signal and performing beamforming to enhance the target human voice.

[0017] The temporal alignment submodule is used to align the facial image sequence, voice signal and body posture sequence on the time axis using a unified timestamp to form a synchronized multimodal data stream.

[0018] The data quality assessment submodule is used to calculate quality indicators, including sharpness, signal-to-noise ratio, and key point visibility rate, for each modality, generate a modality quality score, and reduce or block the contribution of the corresponding modality during the fusion stage when the modality quality score is lower than a preset threshold.

[0019] Furthermore, the micro-expression analysis module includes a face detection and alignment submodule, used to detect face regions from the facial image sequence and perform three-dimensional pose normalization.

[0020] The action unit extraction submodule is used to extract facial action unit intensities, including at least AU1, AU4, AU6, AU12, AU15, AU17, AU20, and AU25, from normalized face images based on a convolutional neural network.

[0021] The temporal feature modeling submodule is used to model the temporal changes in the intensity of the facial action units through a temporal convolutional network or a recurrent neural network, and extract micro-expression emotion feature vectors. The aging compensation submodule is used to perform aging compensation on the intensity of the facial motion unit based on the age information and facial laxity parameters of the elderly with dementia in the personalized portrait, so as to reduce the interference of facial laxity and wrinkles on micro-expression recognition.

[0022] Furthermore, the speech prosody analysis module includes: The voice activity detection submodule is used to detect valid voice segments from the voice signal.

[0023] The prosodic feature extraction submodule is used to extract the fundamental frequency mean, fundamental frequency variance, fundamental frequency jitter, energy envelope, speech rate, pause rate, tremolo, and harmonic-to-noise ratio from the effective speech segments.

[0024] The paralinguistic feature recognition submodule is used to identify paralinguistic events such as crying, shouting, groaning, and repetitive vocalizations in the valid speech segments.

[0025] The semantically omittable submodule is used to output a speech prosodic feature vector based solely on prosodic features and paralinguistic events when elderly people with dementia are unable to form comprehensible sentences.

[0026] Furthermore, the body posture and movement analysis module includes: The skeleton point extraction submodule is used to extract the two-dimensional or three-dimensional coordinate sequence of human joints from the body posture sequence. The kinematic feature extraction submodule is used to calculate motion amplitude, motion speed, trunk forward tilt angle, limb tremor frequency and center of gravity displacement features based on the joint coordinate sequence.

[0027] The agitated behavior recognition submodule is used to identify characteristic agitated or stereotyped behaviors of elderly people with dementia, including pacing back and forth, rubbing clothes, repeatedly patting, and aimless wandering.

[0028] The disability compensation submodule is used to adaptively scale the range of motion and speed thresholds based on the activity level of the elderly with dementia in the personalized profile.

[0029] Furthermore, the personalized profile module includes: The static profile storage submodule is used to store the cognitive impairment level, disease stage, personality traits, cultural background, family relationships, and past events that triggered negative emotions in elderly people with dementia.

[0030] The emotional baseline modeling submodule is used to learn the individual baseline distribution of modal features of each elderly person with dementia in a neutral emotional state within the initial observation window, and to use the individual baseline distribution for deviation measurement in subsequent emotion recognition.

[0031] The context-aware submodule is used to obtain the current time, current scene, recent care events, and recent interaction history in real time, and encode them into context feature vectors.

[0032] The privacy protection submodule is used to encrypt and store the static profile and emotional baseline locally. The system supports completing the emotion recognition inference only locally without transmitting the original audio and video data to the cloud.

[0033] Furthermore, the multimodal fusion module is used to perform quality-weighted fusion of the features of each modality to obtain a fused feature vector. : ;in, The number of modes participating in the fusion. Choose 4, which corresponds to micro-expressions, speech rhythm, body posture and movement, and personality profile, respectively; For the first The feature vectors output by the feature encoder for each modality; This is the learnable projection matrix for this mode; The dynamic fusion weights for this modality: In the formula The first output of the data quality assessment submodule Quality score for each modality; The first one determined based on the personalized profile Each modal reliability coefficient; For temperature coefficient, The larger the value, the more significant the difference in weights between modes.

[0034] Furthermore, the emotion state recognition module outputs the state of dementia in elderly people. Probability distribution of sentiment categories: ;in, For sentiment category indexing, The emotional categories include at least calm, pleasure, anxiety, pain, anger, and fear; and The first Weight vectors and biases corresponding to sentiment categories; For the first The score for emotion categories.

[0035] Final emotion recognition results and corresponding confidence level They are respectively: When the confidence level is When the value is below a preset threshold, the emotion state recognition module outputs an "uncertain" flag and triggers the adaptive interactive decision-making module to enter the active inquiry mode.

[0036] Furthermore, the adaptive interaction decision module includes an action selection submodule, which selects interaction actions based on reinforcement learning. : ;in, This represents a state vector consisting of the current sentiment category, confidence level, and personalized profile. The set of candidate interactive actions includes at least gentle verbal reassurance, playing nostalgic music preferred by the elderly, displaying images of family members, guiding breathing relaxation, prompting caregivers to intervene, and maintaining silent observation. The state-action value function, the function: ;in For learning rate, As a discount factor, To perform the action The next state after transition; immediate reward R: In the formula, The amount of decrease in negative emotion intensity measured by the emotion state recognition module before and after the action is performed; the greater the decrease in negative emotion intensity, the higher the reward. The positive feedback reward coefficient, The penalty coefficient for the number of times caregivers intervene. The resource cost incurred by this action.

[0037] Furthermore, the profile update module includes: a baseline online update submodule, used to update the emotional baseline feature μ of the elderly with dementia by exponential moving average: ;in For the first The updated baseline features. These are the multimodal features observed in the neutral state. To update the smoothing coefficient; a trigger event mining submodule is used to mine scene factors and care events that frequently accompany negative emotions in elderly people with dementia from historical interaction records and add them to the trigger event set of the static portrait; an abnormal emotion alarm submodule is used to send an alarm to caregivers or guardians through a remote communication interface when it is detected that the elderly person with dementia has sustained high-intensity negative emotions for more than a preset time, or exhibits aggressive or self-harming behavioral patterns.

[0038] Compared with the prior art, the beneficial effects of this invention are: This invention achieves accurate recording of each elderly person's trigger events and preferred intervention methods through a fully non-contact audio-visual acquisition architecture, personalized profiles, and emotional baseline modeling; it reduces the frequency of ineffective interventions by caregivers, realizes a closed-loop system for emotional perception, recognition, and intervention, and improves the overall intelligence level and intervention timeliness of emotional care for elderly people with dementia. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the components of a multimodal emotion recognition and interaction system for elderly people with dementia according to the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0041] This embodiment uses the daily emotional monitoring of elderly people with dementia under the care of a nursing home as an application scenario. The monitored subject is designated as Patient P, a 75-year-old female, clinically diagnosed with moderate Alzheimer's disease (MMSE score of 14), with a disease duration of 3.5 years. Her typical behavioral problem is the "sunset syndrome" agitation that occurs in the afternoon, manifested as repeated pacing, scratching at her clothes, sobbing softly, and repetitive questioning. Her language expression has become succinct and fragmented, unable to form complete and comprehensible sentences. Patient P uses a walking aid for independent movement, has mild hearing loss, and shows significant emotional responses to images of her deceased spouse and nostalgic opera music from the 1960s.

[0042] The system described in this embodiment is deployed in the single room occupied by patient P. The specific hardware includes: a binocular camera with an RGB-D depth sensor, fixedly installed in the corner of the room ceiling, with a horizontal viewing angle of 120 degrees, a frame rate of 30fps, and an effective range of 0.5m to 4.5m; a ring array consisting of 6 MEMS microphones integrated at the bottom of the camera, with a sampling rate of 48kHz; a 7-inch touchscreen and a pair of full-range speakers for feedback output; an edge computing box equipped with an 8-core ARM processor and a neural network acceleration unit (5TOPS computing power, 8GB memory, 256GB local storage), where all emotion recognition and inference are completed locally; and interconnection with the care duty station and family mobile APP via an encrypted local area network link to transmit alarms and briefings, while the original audio and video data does not leave the local area.

[0043] The overall workflow of the system is as follows: the multimodal data acquisition module continuously collects video and audio data of patient P → four parallel analysis modules (micro-expression, speech prosody, body movement, and personality profile) extract their respective modal features → the multimodal fusion module obtains fused features by quality weighting → the emotion state recognition module outputs emotion category and confidence level → the adaptive interactive decision-making module selects and executes feedback actions → the profile update module updates the profile online based on the intervention effect.

[0044] like Figure 1 The diagram shown is a schematic representation of a multimodal emotion recognition and interaction system for elderly people with dementia according to the present invention. The system includes a multimodal data acquisition module, a micro-expression analysis module, a speech prosody analysis module, a body movement analysis module, a personality profile module, a multimodal fusion module, an emotion state recognition module, an adaptive interactive decision-making module, and a profile update module.

[0045] The multimodal data acquisition module is used to simultaneously acquire facial image sequences, voice signals, and body posture sequences of elderly people with dementia through non-contact visual and audio sensors. The system does not rely on any physiological sensors that need to be worn by the elderly people with dementia.

[0046] The micro-expression analysis module is used to extract the temporal features of facial action unit intensity from the facial image sequence to obtain a micro-expression emotion feature vector.

[0047] The speech prosody analysis module is used to extract fundamental frequency trajectory, energy envelope, speech rate and speech quality parameters from the speech signal to obtain speech prosody feature vector.

[0048] The body posture analysis module is used to extract joint kinematic features and repetitive stereotyped movement features from the body posture sequence to obtain a body posture feature vector.

[0049] The personalized profile module is used to maintain a personalized profile of elderly people with dementia, including the stage of the disease, level of cognitive impairment, individual emotional baseline, living habits and historical triggering events, and to generate a contextual feature vector based on the current time, scene and recent interaction records.

[0050] The multimodal fusion module is used to perform modal quality weighted fusion of the micro-expression emotion feature vector, speech prosody feature vector, body movement feature vector and context feature vector to generate a fused feature vector.

[0051] The emotional state recognition module is used to identify the current emotional state and corresponding confidence level of the elderly with dementia based on the fused feature vector.

[0052] The adaptive interactive decision-making module is used to select and output an interactive response, including at least one of three types of feedback methods: voice, vision, and body, based on the current emotional state, confidence level, and personalized profile.

[0053] The profile update module is used to update the personalized profile online based on the feedback of the interaction effect after each interaction.

[0054] The system processes a typical emotionally charged event that occurred in patient P within a 6-minute time window from 15:48 to 15:54 on a certain afternoon, illustrating the collaborative working method of the modules.

[0055] At 15:48:12, the RGB-D camera in the room detected that patient P sat up from the bedside and began to walk between the bed and the window. The microphone array simultaneously picked up the low murmur without clear meaning.

[0056] The multimodal data acquisition module continuously outputs aligned video frames, depth maps, human key point sequences, and 48kHz audio frames at a frequency of 30fps.

[0057] It is important to note that in this embodiment, patient P did not wear any physiological sensors that require close contact with the skin, such as ECG, skin conductance, breathing belts, or accelerometer watches. In clinical practice, elderly people with moderate to severe dementia have extremely poor compliance with such wearable devices, often tearing them off themselves, resisting application, or even exhibiting agitation. Therefore, this system excluded such hardware from the initial architecture design and only uses a non-contact audiovisual channel to acquire all raw signals.

[0058] During the 30-second sliding window from 15:48:13 to 15:48:43, the micro-expression analysis module continuously output an AU4 (frowning) intensity average of 2.1 from the facial image sequence. Quantization), AU17 (lower lip lift) intensity 1.8, AU20 (corner of mouth horizontal stretch) intensity 1.5, micro-expression emotion feature vector Simultaneously, the speech prosody analysis module extracted a fundamental frequency mean of 248Hz (higher than the patient P's baseline of 187Hz in a calm state), a fundamental frequency jitter of 6.2%, and an energy standard deviation 2.3 times higher than the baseline from the speech signal. It also identified "whimpering" type paralinguistic events and the speech prosody feature vector. The body movement analysis module calculates a pacing frequency of 12 times per minute and a trunk forward tilt angle deviating from the baseline by 8.4 degrees based on the joint sequence, triggering the recognition of the stereotyped action of "repeatedly scratching clothing," and generating a body movement feature vector. The personalized profile module outputs the current context feature vector. This includes local time (afternoon, 30 minutes before sunset, coinciding with historically high-incidence periods), setting (alone room, no caregiver present), recent events (no regular outdoor walk in the afternoon), patient P's moderate AD course, individual emotional baseline, and historical trigger event labels.

[0059] The multimodal fusion module uses a quality-weighted approach to... Fuse into a fused feature vector Among them, due to the decline in patient P's language ability, the reliability coefficient of speech prosody was... The profile was set to 0.6 (baseline value 1.0), thus in terms of weight. Automatic weighting was applied; the confidence coefficients for micro-expressions, body movements, and contextual profiles were 1.0, 1.0, and 1.1, respectively, with the latter slightly higher reflecting a high degree of attention to the patient's contextual priors.

[0060] The emotion state recognition module outputs the probability distribution of F on six basic emotions (calm, pleasure, anxiety, pain, anger, and fear) [0.04, 0.02, 0.71, 0.18, 0.02, 0.03], and determines the current emotion state as "anxiety" with a confidence level of [0.04, 0.02, 0.71, 0.18, 0.02, 0.03]. Significantly higher than the preset threshold Therefore, it does not enter the active inquiry mode, but directly enters the decision-making process.

[0061] The adaptive interactive decision-making module combines the above states with the prior information in the profile that "there is a significant positive response to nostalgic opera music and spouse images," and selects an action from the candidate action set A. It is implemented through speakers and displays; at the same time, it does not immediately trigger "prompt caregiver intervention" to avoid wasting care resources on events that are only of moderate anxiety.

[0062] At 15:50:00, the system continued to monitor the intervention effect. By 15:53:30, the probability distribution output by the emotion state recognition module changed to [0.62, 0.21, 0.11, 0.03, 0.02, 0.01], which was determined to be "calm," and the negative emotion intensity decreased from 0.89 to 0.14. The profile update module uses this information to adjust the state-action value function. A positive update was given, and the combination of "30 minutes before sunset + living alone + no outdoor walk" was added to the set of triggering events for the static portrait with a confidence level of +0.05; at the same time, the neutral emotional baseline characteristics of patient P were updated. Update once according to the exponential moving average rule.

[0063] Within this 6-minute window, the system did not introduce any wearable physiological sensors. It completed the closed loop from perception to intervention to learning solely through an indoor non-contact audiovisual channel, thus avoiding an agitated outburst that could have escalated into aggressive behavior and saving caregivers from having to enter the room late at night to handle the situation. In the traditional, simply scheduled rounds mode, this incident would have been discovered at the earliest during the scheduled rounds at 16:00, a delay of about 12 minutes, and it often required more than 30 minutes of caregiver accompaniment to calm the person down.

[0064] The multimodal data acquisition module includes a visual acquisition submodule, which includes at least one RGB-D camera for synchronously acquiring the facial image sequence and body pose sequence at a preset frame rate.

[0065] The audio acquisition submodule includes a microphone array for acquiring the speech signal and performing beamforming to enhance the target human voice.

[0066] The temporal alignment submodule is used to align the facial image sequence, voice signal and body posture sequence on the time axis using a unified timestamp to form a synchronized multimodal data stream.

[0067] The data quality assessment submodule is used to calculate quality indicators, including sharpness, signal-to-noise ratio, and key point visibility rate, for each modality, generate a modality quality score, and reduce or block the contribution of the corresponding modality during the fusion stage when the modality quality score is lower than a preset threshold.

[0068] In this embodiment, the visual acquisition submodule uses a single RGB-D camera with an RGB resolution of 1920×1080, a depth resolution of 640×480, a depth accuracy of ±2mm@1m, a frame rate of 30fps, and a horizontal viewing angle of 120 degrees, capable of covering the entire activity area of ​​a 16-square-meter single-person room. RGB frames are used for facial analysis and lip-syncing, while depth frames are used for 3D human pose reconstruction and to avoid perspective confusion in 2D poses.

[0069] The audio acquisition submodule uses a 6-microphone circular array with a microphone spacing of 25mm. It employs a beamforming algorithm based on MVDR (Minimum Variance Distortionless Response) to spatially enhance the location of patient P. This can improve the target speech signal-to-noise ratio by approximately 8dB when patient P is 1.5m away from the array and there is 50dB of fan noise in the environment.

[0070] The timing alignment submodule uses a local hardware clock (accuracy ±1ms) to timestamp video and audio frames separately, aligning them to the same 10ms grid; it outputs a synchronous multimodal data packet once per second, containing 30 consecutive video frames and corresponding audio segments.

[0071] In this embodiment, the data quality assessment submodule implements the following rules: Laplacian variance is used for video clarity (threshold <80 is considered blurry), facial key point detection rate is used for face visibility (threshold <70% is considered unusable), and energy ratio before and after VAD is used for audio signal-to-noise ratio (threshold <3dB is considered unusable).

[0072] In a real-world test case, patient P fell asleep with his back to the camera at 22:30, and the visibility of his face dropped to 0. The modal quality score q1=0.05. After softmax weighting, the micro-expression channel weight α1 automatically dropped to 0.02. The system can still complete the "sleep-calm" emotion determination based on three modalities: body movements (chest movement caused by breathing), speech rhythm (sleep talk), and contextual profile. This avoids the "modal loss leads to shutdown" problem common in traditional single-modal systems.

[0073] Compared to the equal-weighted fusion scheme without quality assessment, this embodiment improved the emotion recognition F1 score from 0.62 to 0.81 during 72 hours of continuous operation testing (including the front, side, and back cameras), and the availability rate during the period when the face was not visible increased from 34% to 92%.

[0074] The micro-expression analysis module includes a face detection and alignment submodule, which is used to detect face regions from the facial image sequence and perform three-dimensional pose normalization.

[0075] The action unit extraction submodule is used to extract facial action unit intensities, including at least AU1, AU4, AU6, AU12, AU15, AU17, AU20, and AU25, from normalized face images based on a convolutional neural network.

[0076] The temporal feature modeling submodule is used to model the temporal changes in the intensity of the facial action units through a temporal convolutional network or a recurrent neural network, and extract micro-expression emotion feature vectors.

[0077] The aging compensation submodule is used to perform aging compensation on the intensity of the facial motion unit based on the age information and facial laxity parameters of the elderly with dementia in the personalized portrait, so as to reduce the interference of facial laxity and wrinkles on micro-expression recognition.

[0078] The face detection and alignment submodule in this embodiment uses a lightweight detector based on RetinaFace (MobileNet-V3 backbone), combined with 5-point facial landmark regression and 3DMM (three-dimensional deformation model) fitting, to normalize the detected face images into a frontal, 112×112 pixel standardized face, which can effectively offset the AU intensity estimation bias caused by the patient P's head-down, side-view, and other postures.

[0079] The action unit extraction submodule uses a ResNet-18 backbone network pre-trained on the BP4D and DISFA datasets. Before deployment of this system, it was fine-tuned using a privately labeled dataset containing approximately 200 elderly individuals (average age 72) covering 12 emotional scenarios, enabling the model to better generalize to aging faces. This submodule outputs the intensity of 12 commonly used facial action units, of which 8 are key AUs: AU1 (inner eyebrow raised), AU4 (frowning), AU6 (cheek raised), AU12 (corner of mouth raised), AU15 (corner of mouth lowered), AU17 (lower lip raised), AU20 (corner of mouth stretched laterally), and AU25 (lips opened), covering typical basic emotional archetypes such as sadness, pain, joy, and anger.

[0080] In this embodiment, the temporal feature modeling submodule employs a one-dimensional temporal convolutional network (TCN) (7 layers, dilation rates 1 / 2 / 4 / 8 / 16 / 32 / 64, receptive field 128 frames ≈ 4.3 seconds) to convolve each AU intensity temporally, outputting a 128-dimensional micro-expression emotion feature vector. Compared to LSTM, TCN reduces the latency on the edge computing box from 27ms to 9ms, meeting the requirements of real-time interaction.

[0081] The specific implementation of the aging compensation submodule is as follows: When patient P is admitted, the "baseline modeling stage" is entered. The system collects about 200 neutral facial images of patient P in a static and calm state, and calculates the "baseline bias" of each AU. For example, due to structural features such as forehead wrinkles, nasolabial folds, and drooping corners of the mouth, the average baseline of elderly patients in AU4 is 1.2 (while it is usually 0.2 for young people), and the baseline of AU15 is 0.9 (while it is usually 0.1 for young people).

[0082] During online operation, the baseline bias is subtracted from the AU intensity of each frame before it is fed into the temporal network, thereby avoiding misjudging "skin laxity" as "persistent frowning / sadness".

[0083] Offline evaluation on patient P showed that after adding aging compensation, the false alarm rate (misjudged as sadness or pain) of the system for the "neutral" state decreased from 19.3% to 4.7%; the recall rate for real sad events increased from 68% to 89%, effectively solving the "baseline drift" problem of general emotion recognition models in the elderly with dementia.

[0084] The speech prosody analysis module includes: a speech activity detection submodule, used to detect valid speech segments from the speech signal; a prosodic feature extraction submodule, used to extract the fundamental frequency mean, fundamental frequency variance, fundamental frequency jitter, energy envelope, speech rate, pause rate, tremolo, and harmonic-to-noise ratio from the valid speech segments; a paralinguistic feature recognition submodule, used to identify paralinguistic events such as crying, shouting, groaning, and repetitive sounds in the valid speech segments; and a semantic omission submodule, used to output speech prosodic feature vectors based solely on prosodic features and paralinguistic events when elderly people with dementia are unable to form comprehensible sentences, thereby avoiding reliance on the elderly person's language expression ability.

[0085] The speech activity detection submodule employs a WebRTC-VAD-based energy + spectral flatness dual-threshold algorithm, operating on the output of the aforementioned microphone array beamforming, achieving a speech segment hit rate of over 94% under 50dB background noise. The detected valid speech segments are then corrected for endpoints before being fed into prosodic feature extraction.

[0086] The prosodic feature extraction submodule extracts the following eight prosodic parameters based on librosa and Praat-Parselmouth: fundamental frequency mean F0_mean, fundamental frequency variance F0_var, fundamental frequency jitter (relative change in continuous cycle length), energy envelope RMS, speech rate (syllables per second), pause rate (percentage of silent frames), shimmer (relative change in continuous cycle amplitude), and harmonic-to-noise ratio (HNR). Each parameter corresponds to a 1-second analysis window and a 0.5-second sliding step, and after normalization, they are concatenated into a 64-dimensional speech prosodic feature vector.

[0087] The sub-language feature recognition submodule uses a lightweight CNN (3 layers of convolution + global pooling) to work on the Mel spectrogram. It is pre-trained on the publicly available CryingDetection and AGE-NV sub-language datasets and can recognize 5 types of events: crying, shouting, groaning, coughing, and repetitive vocalizations (echolalia, a common symptom of dementia in the elderly), with an average F1 score of 0.83.

[0088] In clinical practice, the language expression of elderly people with moderate to severe dementia often degenerates into phrases, fragments, or even complete aphasia. The traditional ASR (Action Speech Recognition) + NLU (Non-linguistic Understanding) link fails extensively in this scenario. This embodiment does not forcibly invoke ASR or require semantic decoding. Instead, when the context profile indicates that the patient P's current "language ability level" is ≤2 (0~5 levels are quantified, with level 2 indicating only fragmented output), the speech prosody analysis module only outputs prosodic feature vectors and paralinguistic events, skipping semantic decoding. When the ability level is ≥3, short keywords obtained through ASR are added as an aid.

[0089] In patient P's case, whose language ability level was consistently 2, this mechanism ensured that the speech pathway maintained effective signal output at all times. Based on patient P's 30-day trial data, this module still contributed effective emotion discrimination evidence even in speech events with zero semantically recognizable words (accounting for 63% of total speech events), increasing the effective coverage of the speech pathway from 37% in the control group (semantic-first scheme) to 96%.

[0090] The body posture analysis module includes: a skeletal point extraction submodule, used to extract two-dimensional or three-dimensional coordinate sequences of human joints from the body posture sequence; a kinematic feature extraction submodule, used to calculate movement amplitude, movement speed, trunk forward tilt angle, limb tremor frequency, and center of gravity displacement features based on the joint coordinate sequence; an agitated behavior recognition submodule, used to identify characteristic agitated or stereotyped behaviors of elderly people with dementia, including pacing back and forth, rubbing clothes, repeatedly patting, and aimless wandering; and a disability compensation submodule, used to adaptively scale the movement amplitude and speed thresholds based on the activity level of the elderly person with dementia in the personalized profile, so that the body posture analysis module is still effective for elderly people with dementia in wheelchairs or bedridden positions.

[0091] In this embodiment, the skeleton point extraction submodule uses MediaPipePose's BlazePose-Heavy model for inference on the edge computing box. Combined with the depth channel of the RGB-D camera, it directly outputs the 3D coordinates (unit: meters) of 17 joint points, with a frame rate of 25fps and an average joint point positioning error of <3cm@2m distance.

[0092] The kinematic feature extraction submodule calculates the following features within a 2-second sliding window based on the joint sequence: limb movement amplitude (maximum displacement of the wrist and ankle joints relative to the trunk center), movement velocity (first-order difference of joint displacement), trunk forward tilt angle (angle between the shoulder-hip line and the vertical direction), and limb tremor frequency (high-frequency wrist tremors after FFT). Frequency band energy), center of gravity displacement (world coordinate displacement of the midpoint of the hip). All features are normalized and concatenated to obtain a 96-dimensional body movement feature vector.

[0093] The agitated behavior recognition submodule was trained on the NTU-RGBD-60 and the system's proprietary "Agitated Behavior Annotation Set for Elderly People with Dementia" (containing approximately 15,000 segments across 8 categories, including pacing back and forth, rubbing clothes, repeatedly patting, aimless wandering, and scratching). The average F1 score was 0.78, significantly better than the 0.45 performance of the general action recognition model on this group.

[0094] The specific implementation of the disability compensation submodule is as follows: Maintain the "Activity Level" field in the patient profile. Level 0 = completely bedridden, Level 5 = free walking). When patient P uses a walking aid to move independently, this field is Level 3, and the system lowers the speed threshold for recognizing "pacing back and forth" from the default 0.6m / s to 0.35m / s. If the profile is switched to wheelchair stage (Level 2) or bedridden stage (Level 1), then: (a) the "walking" behavior template of the body movement pathway is automatically blocked, and the bedridden specific agitated behavior templates such as "repeatedly grasping the bed rails", "repeatedly lifting the blanket", and "abnormal frequency of turning over" are matched instead; (b) the overall movement amplitude threshold is scaled down to 30%~50% of the original value so that small movements can still be effectively captured.

[0095] In this embodiment, the overall recognition F1 score for body and emotional cues of patient P was 0.81. In another bedridden patient (level 1), the F1 score after disability compensation was 0.74, while the F1 score of the uncompensated control model was only 0.31. This indicates that the robustness of this module to elderly people with dementia at different stages of disability is significantly better than that of existing general solutions.

[0096] The personalized profile module includes: a static profile storage submodule, used to store the cognitive impairment level, disease stage, personality traits, cultural background, family relationships, and past events that triggered negative emotions in elderly people with dementia; an emotional baseline modeling submodule, used to learn the individual baseline distribution of each modality feature in a neutral emotional state for each elderly person with dementia within an initial observation window, and to use the individual baseline distribution for deviation measurement in subsequent emotion recognition; a context awareness submodule, used to acquire the current time, current scene, recent care events, and recent interaction history in real time, and encode them into context feature vectors; and a privacy protection submodule, used to encrypt and store the static profile and emotional baseline locally, and the system supports completing emotion recognition inference only locally without transmitting the original audio and video data to the cloud.

[0097] The following is an example (excerpt) of the static profile created for patient P in this embodiment: Cognitive impairment level = moderate (MMSE=14), disease stage = 4th year of progression, personality traits = {cautious, introverted, former primary school teacher}, cultural background = {Han nationality, northerner, religion = none, education level = undergraduate}, family relationships = {eldest son - visits twice a month, second daughter - calls once a week, deceased spouse - married in 1985, widowed in 2018}; past triggering negative emotional events = {being left alone at night with the lights off, a strange male caregiver, a white uniform in a hospital clinic, mentioning the spouse's death}; positive reaction resources = {3 excerpts from nostalgic operas, 7 photos of the spouse during their lifetime, 1 video of the granddaughter's birthday, video of the rural scenery of the hometown in the 1980s}.

[0098] The emotional baseline modeling submodule collected approximately 420 minutes of "neutral" data during the first week after patient P's admission (observation time ≥6 hours per day, excluding periods of significant agitation). The submodule statistically analyzed the patient's multimodal baseline distribution (mean and covariance), for example, AU4 baseline 1.2±0.4, F0 mean 187±18Hz, and pacing frequency 1.8 times / hour ±0.5. These individualized baselines were explicitly invoked as bias metrics in subsequent emotion recognition.

[0099] The context-aware submodule generates a 32-dimensional context feature vector every second. The specific encoding includes: hourly one-hot (24-dimensional compressed into 4-dimensional Fourier features), sunset flag, scene one-hot code (living room / dining room / activity room / bathroom), whether outdoor activities were carried out in the past hour, whether family members visited in the past 24 hours, and a summary of all interaction event types and emotional consequences in the past 6 hours.

[0100] The privacy protection submodule uses AES-256 to locally encrypt and store static profiles and baseline parameters, with the key managed by the TPM hardware security module. All emotion recognition neural network inference and decision-making are completed locally on the edge computing box; the system only sends two types of data externally: (a) structured emotion event summaries (category, confidence level, timestamp, and action taken), and (b) alarm event metadata. Raw video frames, raw audio frames, and raw keypoint coordinates are automatically overwritten after 48 hours of local storage on the edge device and are never uploaded to the cloud under any circumstances.

[0101] This design directly addresses the dual pain points of "family authorization and privacy concerns" in the guardianship of elderly people with dementia. Trial operation surveys show that after informing the public that the system would not distribute original audio and video recordings, the acceptance rate among 15 out of 16 families with dementia increased from 50% at the initial consultation to 95% willingness to install it. Simultaneously, local inference makes the system immune to offline and weak network environments; actual testing showed that all emotion recognition functions still functioned normally after 72 hours of offline operation.

[0102] The multimodal fusion module is used to perform quality-weighted fusion of the features of each modality according to the following formula to obtain the fused feature vector F: Where M is the number of modalities involved in the fusion, and in this system M is 4, corresponding to micro-expressions, speech prosody, body movements and personality profiles respectively; For the first The feature vectors output by the feature encoder for each modality; This is the learnable projection matrix for this mode; The dynamic fusion weights for this mode are calculated using the following formula: In the formula The first output of the data quality assessment submodule Quality score for each modality; The first one determined based on the personalized profile Each modal reliability coefficient, when the language ability of elderly people with dementia is poor. Accordingly reduced; For temperature coefficient, The larger the value, the more significant the difference in weights between modes; Indicates all Summation of each modality; the When a modality is of low quality or unavailable, its weights smoothly tend to zero, thereby achieving robust fusion under modality loss.

[0103] Modal projection matrices in this embodiment , , , All were obtained through end-to-end backpropagation on the labeled dataset; temperature coefficient The value was determined to be 2.5 through grid search (within the range of [0.5, 5.0], and the F1 test on the validation set is optimal at 2.5).

[0104] Credibility coefficient The default value is 1.0, which can be adjusted by the profiling rules. In this embodiment, it is set on patient P. (micro-expressions) (Voice recording, due to language proficiency level = 2) (posture), (The context of the profile reflects a strong trust in the patient's prior knowledge of the context.)

[0105] Using the aforementioned case of the intense event at 15:48, the modal quality score output by the data quality assessment submodule at that moment is: (Clear face) (The background shows an air conditioner fan) (Visible all over the body) (The image is always available). Substitute Calculation formula: The values ​​are as follows: Mode 1 = 2.5 × 0.92 × 1.0 = 2.30; Mode 2 = 2.5 × 0.78 × 0.6 = 1.17; Mode 3 = 2.5 × 0.95 × 1.0 = 2.375; Mode 4 = 2.5 × 1.00 × 1.1 = 2.75.

[0106] Its exponents are: exp(2.30)≈9.97, exp(1.17)≈3.22, exp(2.375)≈10.75, exp(2.75)≈15.64, totaling ≈39.58. Therefore... It is evident that because the image has high reliability and the voice reliability is explicitly downweighted, the system tends to use the image and body posture as the main decision-making basis for this patient, which is highly consistent with clinical experience in interpreting elderly people with dementia in the aphasia stage.

[0107] In another typical scenario, when patient P is lying in bed with their face pressed against the pillow, their face is almost invisible, making... After substitution This means that the weights of the micro-expression channels are almost zero; at this point, the fused feature F is dominated by the other three modalities.

[0108] The emotion state recognition module outputs the status of elderly people with dementia. Probability distribution of sentiment categories: ;in, For sentiment category indexing, The emotional categories include at least calm, pleasure, anxiety, pain, anger, and fear; and The first Weight vectors and biases corresponding to sentiment categories; For the first The score for emotion categories.

[0109] Final emotion recognition results and corresponding confidence level They are respectively: When the confidence level is When the value is below a preset threshold, the emotion state recognition module outputs an "uncertain" flag and triggers the adaptive interactive decision-making module to enter the active inquiry mode.

[0110] The adaptive interaction decision module includes an action selection submodule, which selects interaction actions based on reinforcement learning. : ;in, This represents a state vector consisting of the current sentiment category, confidence level, and personalized profile. The set of candidate interactive actions includes at least gentle verbal reassurance, playing nostalgic music preferred by the elderly, displaying images of family members, guiding breathing relaxation, prompting caregivers to intervene, and maintaining silent observation.

[0111] The state-action value function, the function: ;in For learning rate, As a discount factor, To perform the action The state that is then transitioned to.

[0112] Instant rewards : In the formula, The amount of decrease in negative emotion intensity measured by the emotion state recognition module before and after the action is performed; the greater the decrease in negative emotion intensity, the higher the reward. The positive feedback reward coefficient, The penalty coefficient for the number of times caregivers intervene. The reward function represents the resource cost incurred by this action. It causes the adaptive interactive decision-making module to favor actions that can steadily reduce negative emotions in elderly people with dementia and decrease the burden on caregivers.

[0113] The profile update module includes: a baseline online update submodule, used to update the emotional baseline feature μ of the elderly with dementia by exponential moving average: ;in For the first The updated baseline features. These are the multimodal features observed in the neutral state. To update the smoothing coefficient; a trigger event mining submodule is used to mine scene factors and care events that frequently accompany negative emotions in elderly people with dementia from historical interaction records and add them to the trigger event set of the static portrait; an abnormal emotion alarm submodule is used to send an alarm to caregivers or guardians through a remote communication interface when it is detected that the elderly person with dementia has sustained high-intensity negative emotions for more than a preset time, or exhibits aggressive or self-harming behavioral patterns.

[0114] Number of emotion categories in this embodiment Specifically, the categories are {calm, joy, anxiety, pain, anger, fear}. Each category has a corresponding weight vector. With bias The data was trained end-to-end on public datasets (IEMOCAP sentiment database, RAVDESS, and approximately 1000 labeled segments from our team's self-built "DEMA-1k multimodal sentiment database for elderly people with dementia"). The loss function is class-weighted cross-entropy to balance the samples (positive sentiment samples from elderly people with dementia are relatively scarce).

[0115] Confidence threshold In this embodiment, a conservative strategy is adopted, setting the value to 0.55. This value is determined by scanning the validation set. A trade-off curve of "false intervention rate - false negative rate" was plotted, and the operating point that minimizes the false intervention rate to <5% was selected. Samples with a false intervention rate below 0.55 account for approximately 14% of the total sample, and the system classifies them as "uncertain".

[0116] Returning to the intense event at 15:48, substituting the values ​​into the softmax classifier, the scores Z_c(F) are (calm = -1.42, pleasure = -2.11, anxiety = 1.86, pain = 0.51, anger = -2.41, fear = -1.98), and substituting these values ​​into the softmax classifier yields the probability distribution [0.04, 0.02, 0.71, 0.18, 0.02, 0.03]. =Anxiety, conf=0.71> =0.55, proceed directly to decision.

[0117] In contrast, consider another low-confidence case: At 19:20 one day, patient P faces an unidentified female caregiver. The system obtains the distribution [0.18, 0.05, 0.31, 0.15, 0.06, 0.25], conf=0.31<τ. At this time, the emotion state recognition module outputs an uncertainty flag. Based on this, the adaptive interactive decision-making module enters the active inquiry mode, playing a familiar family member's voice on the screen: "Mom, how are you feeling today?" After 3 seconds, it re-identifies the uncertainty as "fear" (conf0.62), and then takes reassuring actions with a family member's image, avoiding potentially erroneous strong interventions in uncertain situations.

[0118] Compared to the control system that does not introduce a confidence threshold, the overall false intervention rate in this embodiment decreased from 11.2% to 3.4% during the 30-day trial run, while the underreporting rate only increased from 7.1% to 8.0%. That is, the cost of about 1 percentage point underreporting was exchanged for nearly 8 percentage points of false intervention reduction, which significantly improved caregivers' acceptance of the system.

[0119] Candidate action set A is specifically defined as 6 categories: = Gentle verbal reassurance (system-generated TTS family voice, content is preset soothing phrases); =Play nostalgic music preferred by the elderly (randomly selected from positive response resources of still portraits); =Displays images of family members (with priority given to photos of the deceased spouse); =Guided breathing relaxation (voice prompts + rhythmic halo on screen); =Prompt caregivers to intervene (send a notification to the duty desk); =Continue silent observation (without intervention, continue monitoring for 30 seconds). State vector It is composed of sentiment category one-hot (6 dimensions), confidence level (1 dimension), and profile key field encoding (33 dimensions).

[0120] The Q function uses a two-layer fully connected neural network. This means outputting 6 action values ​​for each state. Learning rate. Discount factor Reward coefficient , , ,reflect (Disrupting human care) has the highest cost. (No operation) No cost. Exploration strategy adopted. greedy, It decreases linearly from 0.3 to 0.05.

[0121] The training data comes from three parts: (a) the initial simulation environment, where the Q network is pre-trained based on the state transition model designed by domain experts; (b) offline batch Q learning using real interaction records of 10 elderly people with dementia over a period of 4 weeks (approximately 2400 interaction events); and (c) continuous online updates after on-site deployment at the patient P's individual site.

[0122] In the first week of deployment, the system selected the "anxiety + sunset + living alone" state with near-average probability. After more than 500 interactive observations, the system gradually learned the patient's preferences. After the third week, a stable system was prioritized. The combination replaced the initial multi-movement swaying.

[0123] Following the aforementioned event 15:48, the system executes... Before the combined action was performed, the probability of "anxiety" was measured at 0.71 and the intensity of negative emotion at 0.89 (obtained by weighted summation of the probabilities of [anxiety, pain, anger, fear]). After 5.5 minutes, the probability of "calm" was measured at 0.62 and the intensity of negative emotion at 0.14. ; Instant rewards .

[0124] This positive reward is backpropagated to the Q network through the Bellman update rule, strengthening the preference for this state-action combination.

[0125] Compared to fixed rule systems (which always try first) →If it does not work Data from a 6-week trial of this reinforcement learning decision-making module on patient P showed that the average calming time per agitation event decreased from 18.4 minutes to 7.2 minutes. The number of times (caregiver intervention) was called down from 3.2 times per day to 0.6 times per day, which means that the use of human care resources was significantly reduced while ensuring the effect of emotional calming.

[0126] The profile update module includes: a baseline online update submodule, used to update the emotional baseline features μ of elderly people with dementia by exponential moving average according to the following formula: ;in denoted as the baseline features after the t-th update, x_t represents the multimodal features observed in the neutral state, and δ∈(0,1) is the update smoothing coefficient; the trigger event mining submodule is used to mine scene factors and care events that frequently accompany negative emotions in elderly people with dementia from historical interaction records and add them to the trigger event set of the static portrait; the abnormal emotion alarm submodule is used to send an alarm to caregivers or guardians through a remote communication interface when it is detected that the elderly person with dementia has sustained high-intensity negative emotions for more than a preset time, or exhibits aggressive or self-harming behavioral patterns.

[0127] The system learns continuously, which improves the accuracy of emotion recognition for each elderly person with dementia as the usage time increases, thereby overcoming the technical problems of large individual differences, deterioration of expressive ability, and atypical emotional cues among elderly people with dementia.

[0128] In this embodiment, the baseline online update submodule sets a smoothing coefficient. This means that each neutral observation is weighted with only 2% and incorporated into the baseline, equivalent to an effective averaging window of approximately 50 observations. This approach can track baseline drift caused by the progressive deterioration of patient P's disease while avoiding drastic baseline disturbances from a single abnormal observation. The system operates daily during stable periods when the patient is confirmed to be "calm" and has a confidence level >0.8 (typically after lunch). Minutes and after waking up in the morning (Minutes) Sample several neutral multimodal features As an update input.

[0129] The trigger event mining submodule employs a frequent pattern mining approach based on FP-Growth: it rolls over every 30 days, mining frequent co-occurring itemsets from historical logs of "high negative emotional events + context vectors". Two months after deployment in patient P, this module mined three new trigger patterns and automatically added them to the static profile: (a) the combination of "showering immediately after dinner" caused anxiety attacks in 82% of the samples; (b) the combination of "a strange male caregiver approaching the bedside" caused fear in 75% of the samples; and (c) the combination of "neighbors talking loudly in the hallway + living alone" caused alertness and sleep disruption in 68% of the samples.

[0130] These findings are then used by the adaptive interactive decision-making module for early warning. When the system identifies that the aforementioned high-risk combinations are about to occur (e.g., 19:45, when the "shower immediately after dinner" routine is about to begin during the normal time period), it will schedule the process in advance. (Nostalgic music)+ (Breathing guidance) for preventative intervention.

[0131] The specific rules for the abnormal emotion alarm submodule are as follows: when the system continuously monitors negative emotions (anxiety / pain / anger / fear) with a total probability > 0.7 and a duration ≥ 5 minutes, a Level 1 alarm is pushed to the caregiver's mobile device; when aggressive behavior (punching, pushing, throwing objects) or self-harming tendencies (banging against the wall, scratching oneself) are identified, a Level 2 emergency alarm is immediately pushed to the care duty station and family member's device, regardless of the duration, and preset comforting audio is automatically played to buy time before rescue arrives.

[0132] In patient P's case, the system issued 9 Level 1 alarms in the first 6 weeks (all of which were handled by caregivers within 3 minutes), 1 Level 2 alarm (due to a fall and struggle at night), and 0 false Level 2 alarms. During the same period, in the control room (2 elderly people with dementia) using the traditional on-time inspection plan, 34% of the negative emotional events were not detected in time, and all Level 2 emergency events had a delay of more than 10 minutes in handling.

[0133] The overall technical effectiveness of the system's continuous learning: At weeks 1, 4, 8, and 12 after deployment in patient P, the F1 score for emotion recognition was measured to be 0.71, 0.79, 0.84, and 0.87, respectively; during the same period, the F1 score of a control group without the image update module remained stable at [missing value]. between.

[0134] It is important to note that all the aforementioned effects of this system are achieved without relying on any physiological sensors worn by elderly people with dementia. This architectural choice directly addresses the long-standing pain point of poor wearability in the practice of monitoring elderly people with dementia, making the technical solution described in this invention not only innovative at the algorithm level but also highly feasible in terms of engineering implementation.

[0135] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-modal emotion recognition and interaction system for dementia patients, characterized in that, The system includes a multimodal data acquisition module, a micro-expression analysis module, a speech prosody analysis module, a body movement analysis module, a personality profile module, a multimodal fusion module, an emotion state recognition module, an adaptive interactive decision-making module, and a profile update module; The multimodal data acquisition module is used to simultaneously acquire facial image sequences, voice signals, and body posture sequences of elderly people with dementia through non-contact visual sensors and audio sensors. The micro-expression analysis module, the speech prosody analysis module, and the body movement analysis module are respectively connected to the multimodal data acquisition module; The micro-expression analysis module is used to extract the temporal features of facial action unit intensity from the facial image sequence to obtain a micro-expression emotion feature vector; the speech prosody analysis module is used to extract the fundamental frequency trajectory, energy envelope, speech rate and speech quality parameters from the speech signal to obtain a speech prosody feature vector; the body movement analysis module is used to extract the kinematic features of joints and repetitive stereotyped movement features from the body posture sequence to obtain a body movement feature vector. The personalized profile module stores personalized profiles of elderly people with dementia and generates contextual feature vectors based on the current time, scene, and recent interaction records. The multimodal fusion module is used to perform modal quality weighted fusion of the micro-expression emotion feature vector, speech prosody feature vector, body movement feature vector and context feature vector to generate a fused feature vector; The emotional state recognition module is used to identify the current emotional state and corresponding confidence level of the elderly with dementia based on the fused feature vector; The adaptive interaction decision module is used to select and output an interaction response based on the current emotional state, confidence level, and personalized profile. The profile update module is used to update the personalized profile online based on the feedback of the interaction effect after each interaction.

2. The system according to claim 1, characterized in that, The multimodal data acquisition module includes: The visual acquisition submodule includes at least one RGB-D camera for synchronously acquiring the facial image sequence and body pose sequence at a preset frame rate; The audio acquisition submodule includes a microphone array for acquiring the speech signal and performing beamforming to enhance the target human voice; The temporal alignment submodule is used to align the facial image sequence, voice signal and body posture sequence on the time axis using a unified timestamp to form a synchronized multimodal data stream; The data quality assessment submodule is used to calculate quality indicators, including sharpness, signal-to-noise ratio, and key point visibility rate, for each modality, generate a modality quality score, and reduce or block the contribution of the corresponding modality during the fusion stage when the modality quality score is lower than a preset threshold.

3. The system of claim 2, wherein, The micro-expression analysis module includes: The face detection and alignment submodule is used to detect face regions from the facial image sequence and perform three-dimensional pose normalization; The action unit extraction submodule is used to extract facial action unit intensities, including at least AU1, AU4, AU6, AU12, AU15, AU17, AU20, and AU25, from normalized face images based on a convolutional neural network. The temporal feature modeling submodule is used to model the temporal changes in the intensity of the facial action units through a temporal convolutional network or a recurrent neural network, and extract micro-expression emotion feature vectors. The aging compensation submodule is used to perform aging compensation on the intensity of the facial motion unit based on the age information and facial laxity parameters of the elderly with dementia in the personalized portrait, so as to reduce the interference of facial laxity and wrinkles on micro-expression recognition.

4. The system of claim 3, wherein, The speech prosody analysis module includes: A voice activity detection submodule is used to detect valid voice segments from the voice signal; The prosodic feature extraction submodule is used to extract the fundamental frequency mean, fundamental frequency variance, fundamental frequency jitter, energy envelope, speech rate, pause rate, tremolo and harmonic noise ratio from the effective speech segment. The paralinguistic feature recognition submodule is used to identify paralinguistic events such as crying, shouting, groaning, and repetitive vocalizations in the effective speech segments; The semantically omittable submodule is used to output a speech prosodic feature vector based solely on prosodic features and paralinguistic events when elderly people with dementia are unable to form comprehensible sentences.

5. The system of claim 4, wherein, The body posture and movement analysis module includes: The skeleton point extraction submodule is used to extract the two-dimensional or three-dimensional coordinate sequence of human joints from the body posture sequence. The kinematic feature extraction submodule is used to calculate motion amplitude, motion speed, trunk forward tilt angle, limb tremor frequency and center of gravity displacement features based on the joint coordinate sequence; The agitated behavior recognition submodule is used to identify characteristic agitated or stereotyped behaviors of elderly people with dementia, including pacing back and forth, rubbing clothes, repeatedly patting, and aimless wandering. The disability compensation submodule is used to adaptively scale the range of motion and speed thresholds based on the activity level of the elderly with dementia in the personalized profile.

6. The system of claim 5, wherein, The personalized profile module includes: The static portrait storage submodule is used to store the cognitive impairment level, disease stage, personality traits, cultural background, family relationships, and past events that triggered negative emotions in elderly people with dementia. The emotional baseline modeling submodule is used to learn the individual baseline distribution of each modal feature in the neutral emotional state of each elderly person with dementia within the initial observation window, and to use the individual baseline distribution for deviation measurement in subsequent emotion recognition. The context-aware submodule is used to acquire the current time, current scene, recent care events and recent interaction history in real time, and encode them into context feature vectors. The privacy protection submodule is used to encrypt and store the static profile and emotional baseline locally. The system supports completing the emotion recognition inference only locally without transmitting the original audio and video data to the cloud.

7. The system according to claim 6, characterized in that, The multimodal fusion module is used to perform quality-weighted fusion of features from each modality to obtain a fused feature vector. : ;in, The number of modes participating in the fusion. Choose 4, which corresponds to micro-expressions, speech rhythm, body posture and movement, and personality profile, respectively; For the first The feature vectors output by the feature encoder for each modality; This is the learnable projection matrix for this mode; The dynamic fusion weights for this modality: In the formula The first output of the data quality assessment submodule Quality score for each modality; The first one determined based on the personalized profile Each modal reliability coefficient; For temperature coefficient, The larger the value, the more significant the difference in weights between modes.

8. The system according to claim 1 or 7, characterized in that, The emotion state recognition module outputs the status of elderly people with dementia. Probability distribution of each sentiment category: ;in, For sentiment category indexing, The emotional categories include at least calm, pleasure, anxiety, pain, anger, and fear; and The first Weight vectors and biases corresponding to sentiment categories; For the first Score for sentiment-related categories; Final emotion recognition results and corresponding confidence level They are respectively: When the confidence level is When the value is below a preset threshold, the emotion state recognition module outputs an "uncertain" flag and triggers the adaptive interactive decision-making module to enter the active inquiry mode.

9. The system according to claim 8, characterized in that, The adaptive interaction decision module includes an action selection submodule, which selects interaction actions based on reinforcement learning. : ;in, This represents a state vector consisting of the current sentiment category, confidence level, and personalized profile. The set of candidate interactive actions includes at least gentle verbal reassurance, playing nostalgic music preferred by the elderly, displaying images of family members, guiding breathing relaxation, prompting caregivers to intervene, and maintaining silent observation. The state-action value function, the function: ;in For learning rate, As a discount factor, To perform the action The next state after that; Instant Rewards R: In the formula, The amount of decrease in negative emotion intensity measured by the emotion state recognition module before and after the action is performed; the greater the decrease in negative emotion intensity, the higher the reward. The positive feedback reward coefficient, The penalty coefficient for the number of times caregivers intervene. The resource cost incurred by this action.

10. The system according to claim 9, characterized in that, The profile update module includes: a baseline online update submodule, used to update the emotional baseline feature μ of the elderly with dementia by exponential moving average: ;in For the first The updated baseline features. These are the multimodal features observed in the neutral state. To update the smoothing coefficient; a trigger event mining submodule is used to mine scene factors and care events that frequently accompany negative emotions in elderly people with dementia from historical interaction records and add them to the trigger event set of the static portrait; an abnormal emotion alarm submodule is used to send an alarm to caregivers or guardians through a remote communication interface when it is detected that the elderly person with dementia has sustained high-intensity negative emotions for more than a preset time, or exhibits aggressive or self-harming behavioral patterns.

Images