Old people emotion recognition method and device based on multi-modal perception

Abstract

The embodiment of the invention provides an elderly emotion recognition method and device based on multi-modal perception, and the method and device achieve the optimization and enhancement of the signal quality through innovatively constructing a multi-modal data preprocessing mechanism and integrating the facial expression, voice and posture features. And designing a personalized feature mapping model based on historical emotion expression data, and establishing an adaptive feature fusion strategy for intelligent matching in combination with a cross-modal attention network. A hierarchical time sequence classification mechanism is introduced, dynamic modeling of the emotional development trend is realized through a long-short term memory network, and accurate prediction of the emotional state is supported. According to the method, the defects of the traditional technology in the aspects of multi-modal processing, personalized modeling, time sequence analysis and the like are effectively overcome, and the accuracy and reliability of sentiment recognition of the old people are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to an emotion recognition method and device for the elderly based on multi-modal perception. BACKGROUND

[0002] The existing emotion recognition method for the elderly has obvious deficiencies. The traditional system lacks systematicness in multi-modal data processing, and it is difficult to effectively integrate facial expressions, speech and posture features, which affects the accuracy of emotion recognition.

[0003] In addition, the existing technology has bottlenecks in personalized feature modeling. Most systems fail to fully consider the individual differences of the elderly, lack adaptive feature mapping mechanisms based on historical expression data, resulting in less than ideal recognition results.

[0004] The existing system has technical short boards in emotion time series analysis. It lacks dynamic modeling capability for emotion development trend, and it is difficult to achieve accurate prediction of emotion state through hierarchical time series classification, which affects the recognition reliability. The solution to these problems is of great significance to improve the emotion recognition level of the elderly. SUMMARY

[0005] In view of the problems in the prior art, the present application provides an emotion recognition method and device for the elderly based on multi-modal perception, which can effectively solve the deficiencies of traditional technology in multi-modal processing, personalized modeling and time series analysis, and significantly improve the accuracy and reliability of emotion recognition for the elderly.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides an emotion recognition method for the elderly based on multi-modal perception, comprising:

[0008] Collecting multi-modal data of the elderly, collecting facial video sequences through a high-definition camera, collecting speech signals using a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequences to obtain clear facial images, eliminating background noise from the speech signals to obtain clean speech, converting the human posture data into joint coordinate sequences and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial images, acoustic features of the clean speech, and posture features of the joint coordinate sequences, and constructing facial expression vectors, speech feature vectors, and posture feature vectors, respectively.

[0009] The personalized information model of the elderly is constructed, historical emotional expression data of the target elderly person is obtained from a user portrait database, personalized feature parameters in the historical emotional expression data are extracted, a feature mapping matrix is constructed based on the personalized feature parameters, the facial expression vector, the speech feature vector and the posture feature vector are input into a cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse the multi-modal features to obtain a fusion feature vector;

[0010] The hierarchical time sequence classifier is adopted for emotion recognition, a time sequence feature extraction module is constructed based on a long short-term memory network, the fusion feature vector is input into the time sequence feature extraction module to obtain an emotional state sequence, the emotional state sequence is time sequence modeled to obtain an emotional development trend feature, the current emotional state is probabilistically predicted in combination with the emotional development trend feature, an emotion recognition result is generated, and the emotion recognition result is updated to the user portrait database.

[0011] Further, the high-definition camera collected facial video sequence is input into an image enhancement network, the image enhancement network adopts an adaptive histogram equalization algorithm to perform illumination compensation on the facial region, detects an inter-frame motion amount based on an optical flow field estimation, constructs a motion compensation matrix to perform sharpening processing on a blurred region to obtain a clear facial image, a microphone array is adopted to collect a speech signal, a sound source direction is determined based on a beamforming algorithm, an adaptive filter is adopted to eliminate background noise in a non-target direction, and the speech signal is frame-windowed to obtain clean speech.

[0012] Human posture data are collected based on a depth camera, a human contour region is obtained by segmenting a depth image, three-dimensional space coordinates of human skeleton key points are calculated, the three-dimensional space coordinates are arranged in time sequence to construct a joint node coordinate sequence, the joint node coordinate sequence is trajectory smoothed based on a Kalman filtering algorithm, a skeleton topology constraint model is constructed to detect coordinate abnormalities caused by occlusion, an abnormal coordinate point is corrected using a time sequence interpolation algorithm, and a continuous and stable joint node coordinate sequence is generated.

[0013] Further, the clear facial image is subjected to face detection and alignment, a facial region image is extracted, the facial region image is input into a convolutional neural network, facial contour features are extracted from a shallow convolutional layer, facial expression features are extracted from a deep convolutional layer, different levels of features are fused using a feature pyramid network, the fused features are mapped to a feature space through a fully connected layer to obtain a facial expression vector, the clean speech is input into a speech feature extraction model, mel-frequency cepstral coefficients and pitch contour features are calculated, and an acoustic feature vector is constructed in combination with a speech fundamental frequency, a short-time energy and a zero-crossing rate.

[0014] The relative position relationship and motion speed between the joints are calculated for the joint coordinate sequence, a pose feature extraction model is constructed based on a graph convolution network, the pose feature extraction model models the human body skeleton as a graph structure, the joint coordinates and edge information are subjected to spatio-temporal convolution to obtain local pose features, the local features are weighted and aggregated through an attention pooling layer to obtain global pose features, and the global pose features are mapped into a pose feature vector.

[0015] Further, it also includes: inputting the identity of the target elderly into a user portrait database retrieval module, the retrieval module obtains historical emotional expression data based on an identity matching algorithm, segmenting the historical emotional expression data according to a time window, calculating the emotional change frequency and amplitude in each time window, extracting emotional expression habit features, classifying the emotional expression habit features based on a clustering analysis method, generating personalized feature parameters, the personalized feature parameters include emotional baseline level, emotional fluctuation range, and emotional duration;

[0016] A feature mapping network is constructed as a personalized information model, the personalized feature parameters are input into the feature mapping network, the feature mapping network performs nonlinear transformation on the feature parameters based on a multilayer perceptron structure to generate a feature mapping weight matrix, the feature mapping weight matrix is standardized by using a batch normalization layer, and the standardized result is converted into a feature mapping matrix by an activation function.

[0017] Further, it also includes: constructing a cross-modal attention network, the cross-modal attention network includes a self-attention subnetwork and a mutual-attention subnetwork, the face expression vector, the speech feature vector, and the pose feature vector are respectively input into the self-attention subnetwork, the self-attention subnetwork calculates the correlation score of the features within the same modality based on a query-key-value pair mechanism, generates attention weights according to the correlation score, and combines the features within the same modality by using the attention weights to obtain an intra-modal feature representation, the mutual-attention subnetwork constructs a cross-modal mapping relationship based on the feature mapping matrix, and calculates the semantic correlation between different modal features.

[0018] The intra-modal feature representation is input into a feature fusion module, the feature fusion module selectively fuses different modal features by using a gated update unit, adjusts the feature fusion weight based on the semantic correlation, and performs dimension reduction processing on the fused features to obtain a feature embedding vector, and the feature embedding vector and the original feature vector are combined through a residual connection structure to obtain a fused feature vector.

[0019] Further, the method further comprises: constructing a hierarchical time sequence classifier, arranging the fusion feature vectors in time sequence to construct a feature sequence, segmenting the feature sequence by a sliding window to obtain feature segments, constructing a time sequence feature extraction module based on a bidirectional long short-term memory network, the time sequence feature extraction module comprising a memory unit and a forgetting gate, the memory unit storing historical feature state information, the forgetting gate updating the memory unit state based on the current input feature, generating a hidden state vector through an output gate, the hidden state vector encoding the time sequence dependency of the feature sequence;

[0020] inputting the hidden state vector into a multi-layer perception decoder, the multi-layer perception decoder performing nonlinear transformation on the hidden state vector to obtain an emotion state probability distribution, selecting the class with the maximum probability as the emotion state label of the current time based on the maximum likelihood criterion, and combining the emotion state labels in the continuous time window into an emotion state sequence.

[0021] Further, the method further comprises: constructing a time sequence analysis module, calculating a transition probability matrix of different emotion categories in the emotion state sequence, extracting the duration and conversion law of the emotion state, dynamically updating the transition probability matrix using a recurrent neural network, selecting key time sequence features based on an attention mechanism, inputting the key time sequence features into a trend prediction network, the trend prediction network predicting the emotion development trend of the next time based on the historical transition law, and performing confidence evaluation on the emotion development trend to obtain an emotion development trend feature.

[0022] constructing an emotion prediction model, inputting the emotion state of the current time and the emotion development trend feature into the emotion prediction model, calculating the conditional probability distribution of different emotion categories based on a conditional random field, performing normalization processing on the conditional probability distribution to obtain an emotion recognition result, combining the emotion recognition result and the corresponding timestamp to form a state record, and writing the state record into an emotion history table of a user portrait database based on an incremental update strategy.

[0023] In a second aspect, the present application provides an old person emotion recognition device based on multi-modal perception, comprising:

[0024] The feature processing module is configured to collect multi-modal data of the old person, collect a face video sequence through a high-definition camera, collect a voice signal using a microphone array, collect human posture data based on a depth camera, perform illumination compensation and motion blur correction on the face video sequence to obtain a clear face image, eliminate background noise from the voice signal to obtain clean voice, convert the human posture data into a joint point coordinate sequence and correct coordinate abnormalities caused by occlusion, extract expression features of the clear face image, acoustic features of the clean voice, and posture features of the joint point coordinate sequence, and construct a face expression vector, a voice feature vector, and a posture feature vector, respectively.

[0025] a feature fusion module configured to construct a personalized information model for the elderly, to obtain historical emotional expression data of a target elderly person from a user portrait database, to extract personalized feature parameters in the historical emotional expression data, to construct a feature mapping matrix based on the personalized feature parameters, to input the facial expression vector, the speech feature vector, and the posture feature vector into a cross-modal attention network, and to combine the feature mapping matrix to perform adaptive fusion on the multi-modal features to obtain a fused feature vector;

[0026] an emotion recognition module configured to perform emotion recognition using a hierarchical time series classifier, to construct a time series feature extraction module based on a long short-term memory network, to input the fused feature vector into the time series feature extraction module to obtain an emotional state sequence, to perform time series modeling on the emotional state sequence to obtain an emotional development trend feature, to combine the emotional development trend feature to probabilistically predict a current emotional state, and to generate an emotion recognition result and update the emotion recognition result to the user portrait database.

[0027] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the emotion recognition method for the elderly based on multi-modal perception when executing the program.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the emotion recognition method for the elderly based on multi-modal perception.

[0029] In a fifth aspect, the present application provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions are executable by a processor to implement the steps of the emotion recognition method for the elderly based on multi-modal perception.

[0030] As can be seen from the above technical solutions, the present application provides an emotion recognition method and device for the elderly based on multi-modal perception, which realizes optimization and enhancement of signal quality by innovatively constructing a multi-modal data preprocessing mechanism and by integrating facial expression, speech, and posture features. A personalized feature mapping model based on historical emotional expression data is designed, and an adaptive feature fusion strategy is established for intelligent matching in combination with a cross-modal attention network. A hierarchical time series classification mechanism is introduced, and long short-term memory network is used to realize dynamic modeling of emotional development trends to support accurate prediction of emotional states. The method effectively solves the deficiencies of traditional technologies in multi-modal processing, personalized modeling, and time series analysis, and significantly improves the accuracy and reliability of emotion recognition for the elderly. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0032] Figure 1 The flowchart of the emotion recognition method for the elderly based on multi-modal perception in the embodiments of the present application;

[0033] Figure 2 The structural diagram of the emotion recognition device for the elderly based on multi-modal perception in the embodiments of the present application;

[0034] Figure 3 The structural diagram of the electronic device in the embodiments of the present application.

[0035] Reference signs:

[0036] Electronic device 9600, central processor 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer storage 9141, application / function storage 9142, data storage 9143, driver program storage 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0038] The acquisition, storage, use, processing and the like of data in the technical solutions of the present application all conform to the relevant provisions of national laws and regulations.

[0039] In view of the problems in the prior art, the application provides an old person emotion recognition method and device based on multi-modal perception, which innovatively constructs a multi-modal data preprocessing mechanism, integrates facial expressions, speech and posture features, and realizes optimization and enhancement of signal quality. A personalized feature mapping model based on historical emotion expression data is designed, a cross-modal attention network is combined, an adaptive feature fusion strategy is established for intelligent matching. A hierarchical time sequence classification mechanism is introduced, and a long short-term memory network is used to realize dynamic modeling of emotion development trend and support accurate prediction of emotion state. The method effectively solves the deficiencies of traditional technology in multi-modal processing, personalized modeling and time sequence analysis, and significantly improves the accuracy and reliability of old person emotion recognition.

[0040] In order to effectively solve the deficiencies of traditional technology in multi-modal processing, personalized modeling and time sequence analysis, and significantly improve the accuracy and reliability of old person emotion recognition, an embodiment of an old person emotion recognition method based on multi-modal perception is provided, as shown in Figure 1 , which specifically includes the following contents:

[0041] Step S101: Collecting multi-modal data of the old person, collecting facial video sequences through a high-definition camera, collecting speech signals using a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequences to obtain clear facial images, eliminating background noise from the speech signals to obtain clean speech, converting the human posture data into joint coordinate sequences and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial images, acoustic features of the clean speech, and posture features of the joint coordinate sequences, and constructing facial expression vectors, speech feature vectors, and posture feature vectors, respectively;

[0042] Optionally, the embodiment innovatively designs a data collection and feature extraction scheme based on multi-modal perception to solve the problems of poor data collection quality and inaccurate feature extraction in the old person emotion recognition scene. First, a multi-source data collection network is constructed to collect facial video data in real time through a high-definition camera. The embodiment designs an adaptive image enhancement algorithm: Enhanced_Image = a x Original_Image + b x Compensation_Matrix, where Original_Image is the original image, Compensation_Matrix is the illumination compensation matrix, a and b are adaptive weight coefficients, which are dynamically adjusted according to the ambient light intensity to realize adaptive enhancement of facial images under different lighting conditions. In indoor environments such as nursing homes, the algorithm can effectively improve the image quality and ensure the accuracy of subsequent expression recognition.

[0043] The embodiment deeply optimizes the voice signal collection mechanism. In view of the characteristics that the voice of the elderly is weak and easily disturbed by environmental noise, a spatial filtering scheme based on a microphone array is designed. The beamforming technology is used to enhance the voice signal in the target direction while suppressing the interference noise in other directions. Special attention is paid to the position change of the elderly when speaking, and the system uses an adaptive beam tracking algorithm to adjust the beam direction in real time to obtain the best voice enhancement effect. In application, it is found that this adaptive enhancement mechanism can effectively improve the signal-to-noise ratio of the voice of the elderly, and can accurately capture the voice features of the target elderly even in a scene where multiple people are talking at the same time. For example, in the public activity area of a nursing home, even if there are complex background noises such as TV sound and other old people's conversation, the system can still accurately extract the voice signal of the target elderly.

[0044] The embodiment innovatively realizes the collection and processing strategy of human posture data. Based on the depth camera, three-dimensional human skeleton data is obtained, and the characteristics of limited activity range and small motion amplitude of the elderly are specially considered. The system uses an improved skeleton extraction algorithm to accurately locate the key points of the human body through depth image segmentation. In view of the problem of partial joint occlusion caused by sitting, lying and other postures of the elderly, a coordinate correction mechanism based on the topology of the skeleton is designed. By establishing the spatial constraint relationship between the human body joints, the system can accurately infer the position of the occluded joint, ensuring the continuity and accuracy of the posture data. This intelligent correction mechanism has strong adaptability in practical application, and can accurately capture the body posture changes even in complex postures such as sitting in a wheelchair or leaning on a sofa.

[0045] The embodiment realizes the extraction and vectorization of multi-modal features through deep learning technology. For facial expression features, the system uses a multi-layer convolutional neural network to extract local and global features of the face. Special attention is paid to the areas of the elderly's face where wrinkles, eye circles and corners of the mouth are obvious emotional expressions, and a feature pyramid network is used to realize the fusion of multi-scale features. In terms of voice feature extraction, the system not only analyzes basic acoustic parameters, but also pays special attention to the prosodic features of the voice, such as speech rate, pause, intonation change, etc., which are of great significance to understanding the emotional state of the elderly. For posture features, the system captures the spatio-temporal dependency between human body joints through a graph convolution network, and encodes the continuous skeleton sequence into a fixed-dimensional feature vector. This multi-modal feature extraction strategy ensures that the system can comprehensively capture the emotional expression features of the elderly in different modalities.

[0046] The innovative design of this embodiment not only solves the problems of data collection and feature extraction in traditional methods, but also establishes a multi-modal perception framework that adapts to the characteristics of the elderly population. Through the collaborative collection and deep feature extraction of multi-source data, the system can accurately capture the subtle emotional expressions of the elderly. This targeted design ensures that the system maintains efficient perception and reliable feature extraction when facing the unique behavior patterns and expression methods of the elderly. In practical applications, this multi-modal perception scheme significantly improves the accuracy of emotion recognition for the elderly, providing a reliable data foundation for subsequent emotion analysis.

[0047] This embodiment realizes the intelligent emotion perception upgrade for the elderly population by establishing a complete data collection, signal processing, and feature extraction link. The system can dynamically adjust the collection strategy based on real-time environmental conditions, avoiding the limitations of traditional fixed parameter schemes. Through multi-modal data fusion and deep learning, the robustness and expression ability of feature extraction are significantly improved, providing comprehensive and accurate feature support for emotion recognition of the elderly. This intelligent perception mechanism exhibits strong environmental adaptability and recognition stability in the elderly care service scenario.

[0048] Step S102: Constructing an individualized information model for the elderly, obtaining the historical emotional expression data of the target elderly person from the user portrait database, extracting individualized feature parameters from the historical emotional expression data, constructing a feature mapping matrix based on the individualized feature parameters, inputting the facial expression vector, the speech feature vector, and the posture feature vector into the cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse the multi-modal features to obtain a fusion feature vector.

[0049] Optionally, this embodiment innovatively designs a feature fusion scheme based on individualized modeling to address the problems of large individualized differences and insufficient multi-modal feature fusion in emotion recognition for the elderly. This embodiment first constructs a user portrait retrieval engine to accurately locate the historical data of the target elderly person through an identity recognition module. The system designs a personalized feature extraction mechanism: Personal_Feature = f(Emotion_Baseline, Emotion_Range, Duration_Pattern), where Emotion_Baseline represents the emotional baseline level, Emotion_Range represents the emotional fluctuation range, and Duration_Pattern represents the emotional duration pattern. These parameters comprehensively characterize the emotional expression characteristics of the elderly. Through long-term observation in nursing homes, it is found that each elderly person has a unique emotional expression pattern, and this individualized modeling can significantly improve the accuracy of emotion recognition.

[0050] This embodiment deeply optimizes the historical data analysis mechanism. According to the timing characteristics of the emotional expression of the elderly, a data analysis strategy based on a sliding time window is designed. The system segments the historical emotional expression data by time, and extracts the statistical features of emotional changes in each time window. Special attention is paid to the key moments of emotional transition, and the regularity of emotional changes is captured by establishing an emotional state transition graph. For example, some elderly people have larger emotional fluctuations in the morning, while in the afternoon they tend to be calm; some elderly people have strong emotional reactions to specific events but the duration is short. This personalized timing analysis enables the system to more accurately understand the emotional expression patterns of the elderly.

[0051] This embodiment innovatively realizes the construction strategy of feature mapping network. Based on the extracted personalized feature parameters, the system designs a feature mapping network with a multi-layer perceptron structure. Through nonlinear transformation, the personalized parameters are converted into a feature mapping matrix: Mapping_Matrix = Normalize(MLP(Personal_Feature)), where MLP represents the multi-layer perceptron operation, and Normalize represents the normalization processing. This mapping mechanism can convert the personalized features of the elderly into a weight matrix that can be used for feature fusion, providing personalized guidance for subsequent multi-modal feature fusion. In particular, for elderly people with more introverted emotional expression, the system will pay more attention to subtle expression changes and voice features; for elderly people with emotional externalization, more attention will be paid to changes in posture features.

[0052] This embodiment realizes adaptive fusion of features through cross-modal attention network. In view of the inconsistency of multi-modal expression of the elderly, a dual attention structure containing self-attention and mutual attention mechanism is designed. At the self-attention level, the system respectively analyzes the internal correlation of facial expression, voice and posture features, and captures the feature dependency within the same modality. Through the mutual attention mechanism, the system establishes the semantic mapping between different modal features, realizing cross-modal information interaction. For example, when the elderly show sad emotions, facial expressions may not be obvious, but the trembling of the voice and the slight forward inclination of the body may better reflect their emotional state. This multi-modal collaborative analysis mechanism can more comprehensively understand the emotional expression of the elderly.

[0053] The innovative design of this embodiment not only solves the problems of personalized modeling and feature fusion in traditional methods, but also establishes a sustainable optimization framework for emotion recognition. Through the combination of personalized feature mapping and attention mechanism, the system can accurately capture the emotional expression features of the elderly in different modalities. This fusion mechanism based on personalized modeling ensures that the system always maintains efficient recognition ability and reliable understanding effect when facing the emotional expression of different elderly people. In the practice of elderly care services, this intelligent emotion recognition scheme significantly improves the accuracy of understanding the emotional state of the elderly.

[0054] This embodiment realizes the upgrade of emotion recognition for the elderly population by establishing a complete personalized modeling and feature fusion link. The system can dynamically adjust the fusion strategy based on historical data, avoiding the limitations of traditional fixed weight schemes. Through personalized feature mapping and multi-modal fusion, the accuracy and adaptability of emotion recognition are significantly improved, providing reliable technical support for emotion monitoring of the elderly. This intelligent recognition mechanism shows strong personalized adaptability and recognition stability in the elderly service scene.

[0055] Step S103: Adopting a hierarchical time series classifier for emotion recognition, the hierarchical time series classifier constructs a time series feature extraction module based on a long short-term memory network, inputs the fusion feature vector into the time series feature extraction module to obtain an emotion state sequence, performs time series modeling on the emotion state sequence to obtain an emotion development trend feature, combines the emotion development trend feature to probabilistically predict the current emotion state, generates an emotion recognition result, and updates the emotion recognition result to a user portrait database.

[0056] Optionally, to solve the problems of complex time series dependence and difficult prediction of emotion changes in emotion recognition for the elderly, this embodiment innovatively designs an emotion recognition scheme based on hierarchical time series analysis. This embodiment first constructs a long short-term memory network as the core module of time series feature extraction, and realizes selective retention of historical information through a gating mechanism. The system designs a state updating mechanism: Cell State = f_t x Previous State + i_t x Current Input, where f_t is the output of the forgetting gate, i_t is the output of the input gate, Previous State is the historical state, and Current Input is the current input. Through this mechanism, the dynamic tracking of the emotion state of the elderly is realized. In the practical application of the nursing home, this time series modeling method can effectively capture the continuity features of the emotion changes of the elderly, providing time series support for emotion recognition.

[0057] This embodiment deeply optimizes the temporal feature extraction strategy. Considering the gradual nature of emotional expression in the elderly, a feature sequence construction method based on a sliding window is designed. The system segments the fused feature vector along the time dimension, with each time window containing a continuous feature sequence. Particular attention is paid to the transition process of emotional states, simultaneously considering forward and backward temporal dependencies through a bidirectional LSTM network. This bidirectional analysis mechanism enables a more comprehensive understanding of the contextual information of emotional changes. For example, when an elderly person gradually becomes anxious from a calm state, the system can accurately capture this emotional transition process by analyzing the feature changes before and after the transition. In practice, it has been found that emotional changes in the elderly often have a certain inertia; this temporal-based analysis method can effectively improve the accuracy of emotion recognition.

[0058] This embodiment innovatively implements a modeling strategy for emotion development trends. The system designs a recurrent neural network structure and constructs an emotion development trend prediction model by analyzing the transition patterns in the emotion state sequence: Trend_Score = RNN(State_Sequence, Transition_Matrix), where State_Sequence is the emotion state sequence, Transition_Matrix is ​​the state transition matrix, and RNN is the recurrent neural network operation. This trend analysis mechanism can predict the possible direction of emotion changes, providing predictive guidance for emotion recognition. In particular, for elderly people who are prone to mood swings, the system will focus more on short-term emotion change trends; for elderly people with relatively stable emotions, it will consider long-term emotion development patterns more.

[0059] This embodiment achieves probabilistic prediction of emotional states using conditional random fields. Addressing the uncertainty of emotional expression in the elderly, a state prediction mechanism based on a probabilistic graphical model is designed. The system comprehensively considers the current emotional state and the predicted development trend, calculating the probability of different emotion categories using conditional probability. This probabilistic prediction method can better handle the uncertainty in emotion recognition and provide highly reliable recognition results. For example, when the system detects that an elderly person exhibits fatigue characteristics, it will analyze historical data to determine whether this state might evolve into a negative emotion, thus providing an early warning.

[0060] This innovative design not only solves the problems of temporal modeling and trend prediction in traditional methods, but also establishes a continuously optimizing emotion recognition framework. Through a combination of hierarchical temporal analysis and probabilistic prediction, the system can accurately understand the emotional change patterns of the elderly. This deep learning-based recognition mechanism ensures that the system maintains high recognition efficiency and reliable prediction results when faced with complex emotional expressions. In elderly care service practice, this intelligent emotion recognition solution significantly improves the understanding and prediction of the emotional states of the elderly.

[0061] The embodiment realizes the emotion recognition upgrade for the elderly population by establishing a complete time sequence analysis and state prediction link. The system can dynamically adjust the prediction strategy based on historical data, avoiding the limitations of traditional fixed models. Through multi-level time sequence analysis and probability prediction, the accuracy and predictability of emotion recognition are significantly improved, providing reliable technical support for emotion monitoring of the elderly. This intelligent recognition mechanism shows strong prediction ability and service value in the elderly service scene, and can timely discover the emotional changes of the elderly and provide effective decision support for related caregivers.

[0062] From the above description, the emotion recognition method for the elderly based on multi-modal perception provided by the embodiments of the application can innovatively construct a multi-modal data preprocessing mechanism, optimize and enhance the signal quality by integrating facial expressions, speech, and posture features. A personalized feature mapping model based on historical emotional expression data is designed, combined with a cross-modal attention network, to establish an adaptive feature fusion strategy for intelligent matching. A hierarchical time sequence classification mechanism is introduced, and a long short-term memory network is used to realize dynamic modeling of emotional development trends, supporting accurate prediction of emotional states. This method effectively solves the shortcomings of traditional technologies in multi-modal processing, personalized modeling, and time sequence analysis, significantly improving the accuracy and reliability of emotion recognition for the elderly.

[0063] In an embodiment of the emotion recognition method for the elderly based on multi-modal perception of the application, the following content can also be included:

[0064] Step S201: input the facial video sequence collected by the high-definition camera into the image enhancement network, the image enhancement network uses an adaptive histogram equalization algorithm to perform illumination compensation on the facial region, detects the inter-frame motion based on optical flow field estimation, constructs a motion compensation matrix to sharpen the blurred area to obtain a clear facial image, uses a microphone array to collect speech signals, determines the sound source direction based on a beamforming algorithm, uses an adaptive filter to eliminate background noise in non-target directions, and performs frame and window processing on the speech signals to obtain clean speech.

[0065] Step S202: collect human posture data based on a depth camera, segment the depth image to obtain a human contour region, calculate the three-dimensional space coordinates of the human skeleton key points, arrange the three-dimensional space coordinates in time sequence to construct a joint node coordinate sequence, perform trajectory smoothing on the joint node coordinate sequence based on a Kalman filter algorithm, construct a skeleton topology constraint model to detect coordinate abnormalities caused by occlusion, use a time sequence interpolation algorithm to correct abnormal coordinate points, and generate a continuous and stable joint node coordinate sequence.

[0066] Optionally, the embodiment is aimed at the problems of unstable data collection quality and inaccurate posture tracking in elderly emotion recognition, and innovatively designs a multi-modal data preprocessing scheme. The embodiment first constructs an image enhancement network, and performs illumination compensation on the face region through an adaptive histogram equalization algorithm. The system designs an illumination compensation mechanism: Enhanced_Image = AHE(ROI, Window_Size, Clip_Limit), where ROI is the face region of interest, Window_Size is the local window size, and Clip_Limit is the histogram clipping threshold. Through this mechanism, adaptive enhancement of face images under different illumination conditions is realized. In the actual application in the nursing home, due to the complex and variable indoor lighting conditions, this adaptive enhancement method can effectively improve the quality of face images, providing clear visual input for subsequent expression recognition.

[0067] The embodiment deeply optimizes the motion blur correction mechanism. For the image blur problem caused by the head movement of the elderly, a motion compensation strategy based on optical flow field estimation is designed. The system accurately estimates the motion amplitude and direction by calculating the pixel displacement field between adjacent frames. Special attention is paid to the blur caused by sudden head turning or nodding of the elderly, and the blur area is accurately sharpened by constructing a motion compensation matrix. This correction method based on motion estimation can effectively handle the image quality problem in dynamic scenes, ensuring that the system can continuously obtain clear face images. For example, when the elderly frequently turn their heads while watching TV or talking with others, the system can still maintain the clarity of the face images.

[0068] The embodiment innovatively realizes the spatial filtering strategy of the voice signal. Based on the microphone array, a sound source positioning system is constructed, and the accurate tracking of the target sound source is realized through the beamforming algorithm. The system uses the delay accumulation method to calculate the direction of the sound source, and determines the spatial position of the sound source according to the time delay difference of the signals received by each microphone. Special attention is paid to the position change of the elderly when speaking, and the gain direction is adjusted in real time through the adaptive beam tracking algorithm. This spatial filtering mechanism can effectively suppress the interference noise in non-target directions and extract clear target voice signals. In the noisy environment of the nursing home, even if there are multiple people speaking, TV sound, and other background noise, the system can accurately capture the voice of the target elderly person.

[0069] The embodiment realizes the accurate capture of human posture through depth image processing technology. For the point cloud data collected by the depth camera, the system designs an adaptive segmentation algorithm to extract the human contour. Special consideration is given to the case where the elderly may use wheelchairs or crutches and other assistive devices, and the human region is accurately separated through background modeling and foreground extraction. The system uses an improved skeleton extraction algorithm to calculate the three-dimensional spatial coordinates of the human key points. This posture capture method based on depth information overcomes the limitations of traditional RGB images under occlusion and illumination changes.

[0070] The embodiment deeply optimizes the smoothing mechanism of posture data. The system uses Kalman filter algorithm to smooth the trajectory of the joint coordinate sequence: State_t = AState_t-1 + BControl_t + W_t, where State_t represents the current state, A is the state transition matrix, Control_t is the control input, and W_t is the system noise. This state estimation method can effectively suppress coordinate jumps and generate smooth motion trajectories. In particular, for the slow motion characteristics of the elderly, the system adjusts the filter parameters to achieve better trajectory tracking effect.

[0071] The innovative design of the embodiment not only solves the data quality and trajectory tracking problems in traditional methods, but also establishes a reliable multi-modal data preprocessing framework. Through multi-level signal processing and feature enhancement, the system can accurately capture the facial expressions, speech features and body postures of the elderly. This deep learning-based preprocessing mechanism ensures that the system always maintains high-quality data acquisition capability in the face of complex real-world environments. In the practice of elderly care services, this intelligent data preprocessing scheme significantly improves the accuracy of subsequent emotion recognition.

[0072] The embodiment realizes multi-modal perception upgrade for the elderly population by establishing a complete data acquisition and preprocessing link. The system can dynamically adjust the processing strategy according to real-time environmental conditions, avoiding the limitations of traditional fixed parameter schemes. Through multi-dimensional signal enhancement and trajectory smoothing, the data quality and feature extraction reliability are significantly improved, providing a stable data foundation for emotion recognition of the elderly. This intelligent preprocessing mechanism exhibits strong environmental adaptability and data processing effect in the context of elderly care services.

[0073] In an embodiment of the multi-modal perception-based emotion recognition method for the elderly in the present application, the following content can be specifically included:

[0074] Step S301: Perform face detection and alignment on the clear face image, extract the face region image, input the face region image into a convolutional neural network, extract face contour features from shallow convolutional layers, extract face expression features from deep convolutional layers, use a feature pyramid network to fuse features at different levels, map the fused features to a feature space through a fully connected layer to obtain a face expression vector, input the clean speech into a speech feature extraction model, calculate the Mel frequency cepstral coefficient and pitch contour feature, and construct an acoustic feature vector combining the speech fundamental frequency, short-time energy and zero-crossing rate.

[0075] Step S302: Calculate the relative position relationship and motion speed between the nodes based on the node coordinate sequence, construct a pose feature extraction model based on a graph convolution network, model the human skeleton as a graph structure, perform spatio-temporal convolution on the node coordinates and edge information to obtain local pose features, aggregate the local features through an attention pooling layer to obtain global pose features, and map the global pose features to a pose feature vector.

[0076] Optionally, the embodiment innovatively designs a multi-modal feature extraction scheme to address the problems of incomplete feature extraction and insufficient modal feature expression in emotion recognition for the elderly. The embodiment first constructs a face detection and alignment module, and uses an improved multi-scale face detection algorithm to locate the facial region. The system designs a feature extraction mechanism: Feature_Map = CNN(Face_ROI, Layer_depth), where Face_ROI is the facial region of interest, and Layer_depth is the network layer depth. Multi-level features of the face are extracted through different depth convolution layers. In the practical application of the nursing home, this multi-level feature extraction method can fully capture the subtle expression changes of the elderly, especially for the characteristics of the elderly with many facial wrinkles and non-obvious expressions.

[0077] The embodiment optimizes the facial feature fusion mechanism. A multi-scale feature fusion strategy based on a feature pyramid network is designed to address the complexity of the elderly's expression features. The system extracts local features such as texture and edge in the shallow network, which can reflect the subtle changes of facial muscles; and extracts high-level semantic features in the deep network to capture the overall expression pattern. Through the feature pyramid structure, adaptive fusion of features at different levels is achieved, ensuring that both subtle expression changes and overall emotional expression can be captured. For example, when the elderly show a smile, the system can not only capture the upward movement of the corners of the mouth, but also identify the subtle changes around the eyes, so as to more accurately judge whether it is a real happy expression.

[0078] The embodiment innovatively realizes a multi-dimensional extraction strategy for speech features. Based on clean speech signals, the system calculates the Mel-frequency cepstral coefficients (MFCC), which can effectively represent the acoustic features of speech. Special attention is paid to the changes in pitch contour, and the fundamental frequency features are extracted through short-time Fourier transform and autocorrelation analysis. The system also monitors the short-time energy and zero-crossing rate of speech, which directly reflects the emotional state of the speaker. For example, when the elderly are in a low mood, the speech energy is usually low, the speech rate is slow, and the fundamental frequency changes gently; while in an excited state, the speech energy increases, the speech rate accelerates, and the fundamental frequency changes more dramatically. This multi-dimensional acoustic feature analysis can accurately capture the emotional information contained in the speech of the elderly.

[0079] The embodiment realizes the extraction of human posture features through a graph convolution network. For the sequence of joint coordinates, the system first calculates the relative position relationship between the joints to construct the topology of the human skeleton. Special attention is paid to common posture features of the elderly, such as forward leaning and rickets, which are often related to specific emotional states. The system extracts features from the skeleton structure through graph convolution operation: GCN_Feature = GraphConv(Node_Coord, Edge_Weight), where Node_Coord is the joint coordinate and Edge_Weight is the edge weight, which captures the spatiotemporal features of human posture in this way.

[0080] The embodiment deeply optimizes the aggregation mechanism of posture features. The system designs a feature pooling strategy based on attention mechanism, which adaptively weights the features of different joints. Special attention is paid to the movement features of body parts with obvious emotional expression, such as arms and head. Through dynamic adjustment of attention weights, the system can highlight the contribution of important joints while suppressing noise. This intelligent feature aggregation method can accurately capture emotional cues in the body language of the elderly, providing important auxiliary information for emotion recognition.

[0081] The innovative design of the embodiment not only solves the problem of incomplete feature extraction in traditional methods, but also establishes a multi-modal feature expression framework. Through the combination of deep learning and attention mechanism, the system can accurately capture the emotional performance of the elderly in facial expressions, speech and posture. This multi-modal based feature extraction mechanism ensures that the system always maintains efficient feature extraction ability and reliable expression effect when facing the complex emotional expression of the elderly. In the practice of elderly care services, this intelligent feature extraction scheme significantly improves the accuracy of emotion recognition.

[0082] The embodiment realizes the upgrade of multi-modal feature analysis for the elderly population by establishing a complete feature extraction and expression link. The system can dynamically adjust the extraction strategy according to the characteristics of different modalities, avoiding the limitations of traditional single feature extraction methods. Through multi-dimensional feature extraction and fusion, the comprehensiveness and accuracy of feature expression are significantly improved, providing reliable feature support for subsequent emotion recognition. This intelligent feature extraction mechanism shows strong adaptability and analysis effect in the scene of elderly care services.

[0083] In an embodiment of the multi-modal perception based emotion recognition method for the elderly in the present application, the following content can also be specifically included:

[0084] Step S401: input the identity of the target elderly into the user portrait database retrieval module, the retrieval module obtains historical emotional expression data based on an identity matching algorithm, segments the historical emotional expression data according to a time window, calculates the emotional change frequency and amplitude in each time window, extracts emotional expression habit features, classifies the emotional expression habit features based on a clustering analysis method, and generates personalized feature parameters, the personalized feature parameters include an emotional baseline level, an emotional fluctuation range, and an emotional duration;

[0085] Step S402: construct a feature mapping network as a personalized information model, input the personalized feature parameters into the feature mapping network, the feature mapping network performs nonlinear transformation on the feature parameters based on a multilayer perceptron structure, generates a feature mapping weight matrix, standardizes the feature mapping weight matrix using a batch normalization layer, and converts the standardized result into a feature mapping matrix through an activation function.

[0086] Optionally, the embodiment innovatively designs a set of personalized modeling scheme based on user portrait to solve the problems of inaccurate personalized feature extraction and large emotional expression mode difference in elderly emotion recognition. The embodiment first constructs an identity recognition and retrieval module to accurately locate the historical data of the target elderly through biological feature matching. The system designs a time sequence feature extraction mechanism: Emotion_Pattern = Analysis(Time_Window, Frequency, Amplitude), where Time_Window is the size of the time window, Frequency is the emotional change frequency, and Amplitude is the emotional fluctuation amplitude. This mechanism accurately describes the personalized emotional expression pattern of the elderly. In the practical application of the nursing home, this personalized analysis method can effectively distinguish the emotional expression characteristics of different elderly people, providing personalized reference for subsequent emotion recognition.

[0087] The embodiment deeply optimizes the historical data analysis mechanism. According to the time sequence characteristics of elderly emotional expression, a data segmentation strategy based on sliding time window is designed. The system divides the historical emotional expression data in the time dimension, extracts key statistical features in each time window. Special attention is paid to the time pattern of emotional changes, and personalized emotional expression features are constructed by calculating the emotional change frequency and amplitude. For example, some elderly people have frequent emotional changes but small amplitude, showing subtle emotional fluctuations; while some elderly people have less emotional changes but large fluctuation amplitude, which may produce obvious emotional changes under certain event triggers. This time-based analysis method can accurately capture the emotional expression habits of different elderly people.

[0088] This embodiment innovatively implements cluster analysis of emotion expression habits. The system employs an improved K-means clustering algorithm to classify emotion expression features, determining the category of a sample by calculating the Euclidean distance between feature vectors. Particular attention is paid to key parameters such as baseline level, fluctuation range, and duration of emotion expression, as these parameters directly reflect the characteristics of emotional expression in the elderly. For example, the baseline level reflects the elderly's normal emotional tendency, the fluctuation range indicates the sensitivity to emotional changes, and the duration reflects the stability of the emotional state. This multi-dimensional feature analysis provides an important basis for constructing a personalized emotion recognition model.

[0089] This embodiment achieves dynamic transformation of personalized parameters through a feature mapping network. A feature mapping model is designed based on a multilayer perceptron structure, transforming personalized feature parameters into feature mapping weights through nonlinear transformation: Weight_Matrix = MLP(Personal_Features), where Personal_Features are the personalized feature parameters and MLP is the multilayer perceptron operation. This mapping mechanism can transform the personalized features of elderly individuals into a weight matrix that can be used for feature fusion, providing personalized guidance for subsequent multimodal feature fusion.

[0090] This embodiment deeply optimizes the standardization process of feature mapping. The system employs batch normalization to standardize the feature mapping weights, adjusting the weight distribution by calculating the mean and variance within each batch. Particular attention is paid to scale differences between different feature dimensions, ensuring that features across all dimensions are within the same numerical range through standardization. This standardization mechanism not only improves the model's training efficiency but also enhances the generalization ability of the feature mapping.

[0091] This embodiment's innovative design not only solves the personalized modeling problem in traditional methods but also establishes a continuously optimizing feature mapping framework. By combining personalized parameter extraction and feature mapping, the system can accurately capture the unique emotional expression patterns of the elderly. This user profile-based personalized modeling mechanism ensures that the system maintains high adaptability and reliable recognition results when facing different elderly individuals. In elderly care service practice, this intelligent personalized modeling scheme significantly improves the accuracy of emotion recognition.

[0092] This embodiment achieves an upgrade in personalized emotion recognition for the elderly by establishing a complete data analysis and feature mapping chain. The system can dynamically adjust the mapping strategy based on historical data, avoiding the limitations of traditional fixed models. Through multi-dimensional feature analysis and mapping transformation, the accuracy and adaptability of personalized modeling are significantly improved, providing reliable personalized support for emotion recognition in the elderly. This intelligent modeling mechanism demonstrates strong personalized adaptability and recognition effectiveness in elderly care service scenarios.

[0093] In an embodiment of the multi-modal perception-based emotion recognition method for the elderly person in the present application, the following content can also be specifically included:

[0094] Step S501: A cross-modal attention network is constructed, which includes a self-attention subnetwork and an inter-attention subnetwork. The facial expression vector, the speech feature vector, and the posture feature vector are respectively input into the self-attention subnetwork. The self-attention subnetwork calculates a correlation score of the features within the same modality based on a query-key-value pair mechanism, generates an attention weight according to the correlation score, and combines the feature vectors by weighting to obtain an intra-modal feature representation. The inter-attention subnetwork constructs a cross-modal mapping relationship based on the feature mapping matrix, and calculates the semantic correlation between different modal features.

[0095] Step S502: The intra-modal feature representation is input into a feature fusion module. The feature fusion module selectively fuses different modal features by using a gated update unit, adjusts the feature fusion weight based on the semantic correlation, performs dimension reduction processing on the fused features to obtain a feature embedding vector, and combines the feature embedding vector with the original feature vector through a residual connection structure to obtain a fused feature vector.

[0096] Optionally, the present embodiment innovatively designs a multi-modal fusion scheme based on personalized feature mapping to solve the problems of insufficient personalized modeling and insufficient cross-modal feature fusion in emotion recognition for the elderly person. The present embodiment first constructs a user portrait retrieval engine to accurately locate the historical data of the target elderly person through an identity matching algorithm. The system designs an emotion feature extraction mechanism: Emotion_Feature=f(Base_Level,Wave_Range,Duration), wherein Base_Level is the baseline level of emotion, Wave_Range is the emotion fluctuation range, and Duration is the duration. These parameters comprehensively characterize the emotional expression pattern of the elderly person. In the long-term observation of the nursing home, each elderly person shows unique emotional expression habits. This personalized feature extraction is crucial to improve the accuracy of emotion recognition.

[0097] This embodiment deeply optimizes the historical data analysis strategy. According to the timing characteristics of the emotional expression of the elderly, a sliding time window is used to segment the historical data. The system calculates the statistical characteristics of the emotional changes in each time window, including the change frequency, fluctuation amplitude and other key indicators. Special attention is paid to the persistence and conversion characteristics of the emotional state, and a clustering analysis method is used to identify typical emotional expression patterns. For example, some elderly people have frequent emotional changes but small amplitude, while some elderly people have infrequent emotional changes but large amplitude once changed. This time window-based analysis method can accurately capture the emotional expression characteristics of individuals.

[0098] This embodiment innovatively realizes the construction of cross-modal attention network. A double attention mechanism is designed for the heterogeneity of facial expressions, speech and posture features. At the self-attention level, the system uses a query-key-value mechanism to calculate feature correlation: Attention_Score = Softmax(Q x K^T / sqrt(d)), where Q is the query matrix, K is the key matrix, and d is the feature dimension. This self-attention mechanism can capture the feature dependency within the same modality and enhance the consistency of feature expression. At the mutual attention level, the system establishes semantic relationships between different modalities based on feature mapping matrices to realize information exchange between modalities.

[0099] This embodiment deeply optimizes the feature fusion strategy. A selective fusion mechanism based on a gating update unit is designed for the complementarity of different modal features. The system learns the importance weight of the feature to dynamically adjust the contribution of each modality. Special attention is paid to the semantic consistency between modalities. When some modal features show strong correlation, the system will increase the weight of these features in the fusion process. Through the residual connection structure, the system can retain important information of the original features while incorporating new feature expressions, thereby generating a more comprehensive fused feature vector.

[0100] The innovative design of this embodiment not only solves the feature fusion problem in traditional methods, but also establishes a sustainable and optimized personalized modeling framework. Through the combination of personalized parameter extraction and cross-modal attention mechanism, the system can accurately capture the emotional expression characteristics of the elderly. This attention-based fusion mechanism ensures that the system always maintains efficient feature extraction capability and reliable fusion effect when facing complex emotional expressions. In the practice of elderly care services, this intelligent feature fusion scheme significantly improves the accuracy of emotion recognition.

[0101] This embodiment realizes the upgrade of emotion recognition for the elderly population by establishing a complete personalized modeling and feature fusion link. The system can dynamically adjust the fusion strategy based on historical data, avoiding the limitations of traditional fixed weight schemes. Through personalized feature mapping and multi-modal fusion, the accuracy and adaptability of emotion recognition are significantly improved, providing reliable technical support for emotion monitoring of the elderly. This intelligent fusion mechanism exhibits strong personalized adaptability and recognition stability in the context of elderly care services.

[0102] This embodiment not only improves the accuracy of emotion recognition, but also establishes an evolving intelligent system through continuous data accumulation and model optimization. This personalized modeling-based fusion mechanism ensures that the system can continuously improve over time, providing increasingly accurate emotion recognition services for the elderly. In practical applications, this self-optimization mechanism significantly improves the long-term service quality and user satisfaction of the system.

[0103] In an embodiment of the multi-modal perception-based emotion recognition method for the elderly according to the present application, the following content can be specifically included:

[0104] Step S601: Construct a hierarchical time series classifier, arrange the fusion feature vectors in time sequence to construct a feature sequence, segment the feature sequence by sliding window to obtain feature segments, construct a time series feature extraction module based on a bidirectional long short-term memory network, the time series feature extraction module includes a memory unit and a forgetting gate, the memory unit stores historical feature state information, the forgetting gate updates the memory unit state based on the current input feature, generates a hidden state vector through an output gate, and the hidden state vector encodes the time series dependency of the feature sequence;

[0105] Step S602: Input the hidden state vector into a multi-layer perceptron decoder, the multi-layer perceptron decoder performs non-linear transformation on the hidden state vector to obtain an emotion state probability distribution, selects the class with the maximum probability as the emotion state label at the current time based on the maximum likelihood criterion, and combines the emotion state labels within the continuous time window into an emotion state sequence.

[0106] Optionally, the embodiment is directed to the problems of insufficient time sequence modeling and inaccurate state prediction in the emotion recognition of the elderly, and a set of emotion recognition scheme based on hierarchical time sequence analysis is innovatively designed. The embodiment first constructs a time sequence feature processing engine, and segments the fusion feature vector through a sliding window mechanism. The system designs a state updating mechanism: State_t = f_t x State_t-1 + i_t x Input_t, where State_t is the current state, f_t is the forgetting gate output, i_t is the input gate output, and Input_t is the current input. Through this mechanism, the dynamic tracking of the emotional state of the elderly is realized. In the practical application of the nursing home, due to the continuity and gradualness of the emotional changes of the elderly, this time sequence-based analysis method can effectively capture the process characteristics of emotional changes.

[0107] The embodiment deeply optimizes the time sequence feature extraction strategy. In view of the continuity of the emotional expression of the elderly, a feature extraction module based on a bidirectional long short-term memory network is designed. The system realizes comprehensive analysis of the emotional state sequence through information transmission in the forward and backward directions. Special attention is paid to the key moment of emotional conversion, and important historical state information is stored through the memory unit. For example, when the elderly gradually become anxious from a calm state, the system can accurately capture the process of this emotional change by analyzing the feature changes before and after the moment. In the nursing service, this bidirectional analysis mechanism can discover the trend of emotional changes in advance and provide a basis for timely intervention.

[0108] The embodiment innovatively realizes the encoding mechanism of the emotional state. Based on the feature extraction result of the bidirectional LSTM, the system generates a hidden state vector containing rich time sequence information: Hidden_State = BiLSTM(Feature_Sequence), where Feature_Sequence is the time sequence feature sequence, and BiLSTM is the bidirectional long short-term memory network operation. This time sequence encoding mechanism can capture the long-term dependence of the emotional state and provide reliable feature support for subsequent state prediction. In particular, for the elderly with relatively slow emotional changes, the system will pay more attention to long-term state changes; for the elderly with large emotional fluctuations, the system will pay more attention to short-term feature changes.

[0109] The embodiment deeply optimizes the state decoding strategy. The system uses a multi-layer perceptron structure to perform nonlinear transformation on the hidden state vector to generate the probability distribution of the emotional state. The most likely emotional state is selected through the maximum likelihood criterion to realize accurate recognition of the current emotion. Special attention is paid to the continuity constraint of the emotional state, and the system generates a continuous emotional state sequence through the state combination within the time window. This probability-based decision mechanism can effectively handle the uncertainty in emotion recognition and provide reliable recognition results.

[0110] The innovative design of this embodiment not only solves the timing modeling problem in traditional methods, but also establishes a sustainable optimization framework for emotion recognition. Through hierarchical timing analysis and state prediction, the system can accurately understand the emotional change patterns of the elderly. This deep learning-based recognition mechanism ensures that the system always maintains high recognition ability and reliable prediction effect when facing complex emotional expressions. In the practice of elderly care services, this intelligent emotion recognition scheme significantly improves the understanding and prediction of the emotional state of the elderly.

[0111] This embodiment realizes the upgrade of emotion recognition for the elderly population by establishing a complete timing analysis and state prediction link. The system can dynamically adjust the prediction strategy based on historical data, avoiding the limitations of traditional fixed models. Through multi-level timing analysis and state prediction, the accuracy and predictability of emotion recognition are significantly improved, providing reliable technical support for emotional monitoring of the elderly. This intelligent recognition mechanism exhibits strong prediction ability and service value in the context of elderly care services.

[0112] This embodiment not only improves the accuracy of emotion recognition, but also establishes an evolving intelligent system through continuous data accumulation and model optimization. This timing learning-based recognition mechanism ensures that the system can continuously improve over time, providing increasingly accurate emotion recognition services for the elderly. In practical applications, this self-optimization mechanism significantly improves the long-term service quality and user satisfaction of the system, providing strong technical support for intelligent elderly care services.

[0113] In an embodiment of the multi-modal perception-based emotion recognition method for the elderly in the present application, the following content can be specifically included:

[0114] Step S701: Construct a timing analysis module, calculate the transition probability matrix of different emotion categories in the emotion state sequence, extract the duration and transition rules of the emotion state, use a recurrent neural network to dynamically update the transition probability matrix, select key timing features based on the attention mechanism, input the key timing features into a trend prediction network, the trend prediction network predicts the emotional development trend at the next time based on historical transition rules, and performs confidence evaluation on the emotional development trend to obtain emotion development trend features;

[0115] Step S702: Construct an emotion prediction model, input the current time emotion state and the emotion development trend features into the emotion prediction model, calculate the conditional probability distribution of different emotion categories based on conditional random fields, normalize the conditional probability distribution to obtain emotion recognition results, combine the emotion recognition results with the corresponding time stamp to form a state record, and write the state record into the emotion history table of the user portrait database based on an incremental update strategy.

[0116] Optionally, the embodiment is directed to the problems of inaccurate timing prediction and unclear emotion transition rules in the emotion recognition of the elderly. A set of emotion prediction scheme based on timing analysis is innovatively designed. The embodiment first constructs a timing analysis module to calculate the emotion state transition probability through a Markov chain model. The system designs a state transition mechanism: Transition_Prob = P(St+1|St) = Count(St→St+1) / Count(St), where St represents the emotion state at time t, and Count represents the count of state transition. In the long-term observation of the nursing home, it is found that the emotion changes of the elderly often follow certain rules, for example, the transition from calm to anxiety may pass through a slight state of agitation. The calculation of this transition probability provides an important basis for subsequent trend prediction.

[0117] The embodiment deeply optimizes the timing feature extraction strategy of the emotion state. For the continuity feature of the emotion changes of the elderly, a dynamic updating mechanism based on recurrent neural network is designed. The system realizes the modeling of long-term dependence through LSTM unit, and pays special attention to the duration and transition time of the emotion state. For example, some elderly people are prone to emotional fluctuations in the morning. This time-related pattern can be captured through a recurrent network. Through the attention mechanism, the system can automatically identify important timing features, which are often related to the key moments of emotion transition. In practical applications, this dynamic feature extraction method can accurately grasp the key nodes of the emotion changes of the elderly.

[0118] The embodiment innovatively realizes the prediction mechanism of the emotion trend. Based on the historical transition rules, the system uses a trend prediction network to predict the future emotion state: Future_State = Network(History_States, Attention_Weights), where History_States is the historical state sequence, and Attention_Weights is the attention weight. This prediction mechanism not only considers the transition rules of the historical state, but also highlights the influence of key timing features through the attention mechanism. In particular, the system will perform confidence evaluation on the prediction results, and when the prediction uncertainty is high, the prediction weight will be adjusted accordingly to ensure the reliability of the prediction results.

[0119] The embodiment deeply optimizes the construction strategy of the emotion prediction model. The system uses a conditional random field model to realize the sequence labeling of the emotional state, which can fully consider the dependency between states. In view of the characteristics of the emotional expression of the elderly, the system pays special attention to the influence of environmental factors and physiological state on the change of emotion. For example, before and after meal time, medication time and other specific periods, the emotional state of the elderly may have regular changes. Through the modeling of conditional random field, the system can integrate these context information into the prediction process.

[0120] The innovative design of the embodiment not only solves the timing prediction problem in the traditional method, but also establishes a sustainable optimization emotion monitoring framework. Through the combination of state transition analysis and trend prediction, the system can accurately grasp the emotional change rule of the elderly. This time-based prediction mechanism ensures that the system always maintains efficient prediction ability and reliable monitoring effect when facing complex emotional changes. In the practice of pension services, this intelligent prediction scheme significantly improves the forward-looking and accuracy of emotional care.

[0121] The embodiment realizes the upgrade of emotion monitoring for the elderly population by establishing a complete data update and storage link. The system uses an incremental update strategy to write new emotional state records into the user portrait database in a timely manner. This dynamic updating mechanism ensures that the system can continuously accumulate personalized emotional expression data, providing more rich historical basis for subsequent prediction analysis. In practical application, this continuous optimization mechanism significantly improves the personalized service ability and prediction accuracy of the system.

[0122] The embodiment not only improves the accuracy of emotion prediction, but also establishes an evolving intelligent system through continuous data accumulation and model optimization. This prediction mechanism based on time series analysis ensures that the system can continuously improve with use, providing more and more accurate emotional care services for the elderly. In the practice of pension services, this self-optimization mechanism significantly improves the long-term service quality and early warning effect of the system, providing reliable technical support for the mental health management of the elderly.

[0123] In order to effectively solve the deficiencies of traditional technology in multi-modal processing, personalized modeling and time series analysis, and significantly improve the accuracy and reliability of emotion recognition of the elderly, the present application provides an embodiment of a multi-modal perception-based emotion recognition device for implementing all or part of the content of the multi-modal perception-based emotion recognition method for the elderly, see Figure 2 , the multi-modal perception-based emotion recognition device specifically includes the following content:

[0124] The feature processing module 10 is used for collecting multi-modal data of the elderly, collecting a facial video sequence through a high-definition camera, collecting a voice signal through a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequence to obtain a clear facial image, eliminating background noise from the voice signal to obtain clean voice, converting the human posture data into a joint point coordinate sequence and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial image, acoustic features of the clean voice, and posture features of the joint point coordinate sequence, and constructing a facial expression vector, a voice feature vector, and a posture feature vector respectively.

[0125] The feature fusion module 20 is used for constructing an individualized information model of the elderly, obtaining historical emotional expression data of a target elderly person from a user portrait database, extracting individualized feature parameters in the historical emotional expression data, constructing a feature mapping matrix based on the individualized feature parameters, inputting the facial expression vector, the voice feature vector, and the posture feature vector into a cross-modal attention network, and performing adaptive fusion on multi-modal features by the cross-modal attention network in combination with the feature mapping matrix to obtain a fusion feature vector.

[0126] The emotion recognition module 30 is used for performing emotion recognition by using a hierarchical time sequence classifier, constructing a time sequence feature extraction module based on a long short-term memory network, inputting the fusion feature vector into the time sequence feature extraction module to obtain an emotional state sequence, performing time sequence modeling on the emotional state sequence to obtain an emotional development trend feature, performing probability prediction on a current emotional state in combination with the emotional development trend feature, generating an emotion recognition result, and updating the emotion recognition result to the user portrait database.

[0127] As can be known from the above description, the emotion recognition device for the elderly based on multi-modal perception provided by the embodiment of the application can realize optimization and enhancement of signal quality by innovatively constructing a multi-modal data preprocessing mechanism and by integrating facial expression, voice, and posture features. An individualized feature mapping model based on historical emotional expression data is designed, a cross-modal attention network is combined, an adaptive feature fusion strategy is established, a hierarchical time sequence classification mechanism is introduced, dynamic modeling of an emotional development trend is realized through a long short-term memory network, and accurate prediction of an emotional state is supported. The method effectively solves the deficiencies of traditional technologies in multi-modal processing, individualized modeling, and time sequence analysis, and significantly improves the accuracy and reliability of emotion recognition of the elderly.

[0128] From the hardware level, in order to effectively solve the deficiencies of traditional technology in multi-modal processing, personalized modeling and time series analysis, and significantly improve the accuracy and reliability of elderly emotion recognition, the present application provides an embodiment of an electronic device for implementing all or part of the content of the multi-modal perception-based elderly emotion recognition method. The electronic device specifically includes the following content:

[0129] A processor, a memory, a communications interface, and a bus; wherein the processor, the memory, the communications interface, and the bus complete mutual communication through the bus; the communications interface is used to realize information transmission between the multi-modal perception-based elderly emotion recognition device and the core business system, the user terminal, and the related database and other related devices; the logic controller can be a desktop computer, a tablet computer, and a mobile terminal, etc., and the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiments of the multi-modal perception-based elderly emotion recognition method and the embodiments of the multi-modal perception-based elderly emotion recognition device, the contents of which are incorporated herein, and the repeated parts will not be described again.

[0130] It can be understood that the user terminal can include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. The smart wearable device can include smart glasses, a smart watch, a smart bracelet, etc.

[0131] In actual application, part of the multi-modal perception-based elderly emotion recognition method can be executed on the electronic device as described above, or all operations can be completed in the client device. Specifically, the selection can be made according to the processing capacity of the client device and the limitation of the user's use scenario, etc. The present application does not limit this. If all operations are completed in the client device, the client device can also include a processor.

[0132] The above-mentioned client device can have a communication module (i.e. a communication unit) and can be communicatively connected with a remote server to realize data transmission with the server. The server can include a server of the task scheduling center side, and the server of the intermediate platform can also be included in other implementation scenarios, such as the server of the third-party server platform communicatively linked with the task scheduling center server. The server can include a single computer device, a server cluster composed of multiple servers, or a distributed server structure.

[0133] Figure 3A schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is noted that the configuration shown in FIG. 9 is exemplary; other types of configurations can also be used to supplement or replace the configuration shown in FIG. 9 to implement telecommunication functions or other functions. Figure 3 Figure 3 It is noted that the configuration shown in FIG. 9 is exemplary; other types of configurations can also be used to supplement or replace the configuration shown in FIG. 9 to implement telecommunication functions or other functions.

[0134] In an embodiment, the emotion recognition method based on multi-modal perception for the elderly can be integrated into the central processor 9100. The central processor 9100 can be configured to perform the following controls:

[0135] Step S101: Collecting multi-modal data of the elderly, collecting facial video sequences through a high-definition camera, collecting voice signals through a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequences to obtain clear facial images, eliminating background noise from the voice signals to obtain clean voice, converting the human posture data into a joint coordinate sequence and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial images, acoustic features of the clean voice, and posture features of the joint coordinate sequence, and constructing facial expression vectors, voice feature vectors, and posture feature vectors, respectively.

[0136] Step S102: Constructing an individualized information model for the elderly, obtaining historical emotional expression data of a target elderly person from a user portrait database, extracting individualized feature parameters from the historical emotional expression data, constructing a feature mapping matrix based on the individualized feature parameters, inputting the facial expression vectors, voice feature vectors, and posture feature vectors into a cross-modal attention network, and the cross-modal attention network combining the feature mapping matrix to adaptively fuse multi-modal features to obtain a fusion feature vector.

[0137] Step S103: Performing emotion recognition using a hierarchical time sequence classifier, constructing a time sequence feature extraction module based on a long short-term memory network, inputting the fusion feature vector into the time sequence feature extraction module to obtain an emotional state sequence, performing time sequence modeling on the emotional state sequence to obtain emotional development trend features, combining the emotional development trend features to probabilistically predict the current emotional state, generating an emotion recognition result, and updating the emotion recognition result to the user portrait database.

[0138] ​As can be seen from the above description, the electronic device provided by the embodiment of the application innovatively constructs a multi-modal data preprocessing mechanism, optimizes and enhances the signal quality by integrating facial expressions, voices and posture features. A personalized feature mapping model based on historical emotional expression data is designed, a cross-modal attention network is combined, an adaptive feature fusion strategy is established for intelligent matching. A hierarchical time sequence classification mechanism is introduced, a long short-term memory network is used to realize dynamic modeling of emotional development trends, and accurate prediction of emotional states is supported. The method effectively solves the deficiencies of traditional technologies in multi-modal processing, personalized modeling and time sequence analysis, and significantly improves the accuracy and reliability of emotion recognition of the elderly.

[0139] In another embodiment, the multi-modal perception based emotion recognition device for the elderly can be configured separately from the central processor 9100, for example, the multi-modal perception based emotion recognition device for the elderly can be configured as a chip connected with the central processor 9100, and the multi-modal perception based emotion recognition method function is realized through the control of the central processor.

[0140] As shown in Figure 3 , the electronic device 9600 can also include a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily include all the components shown in Figure 3 ; in addition, the electronic device 9600 can also include components not shown in Figure 3 , which can refer to prior art.

[0141] As shown in Figure 3 , the central processor 9100, also known as a controller or operation control, can include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 9600.

[0142] The memory 9140, for example, can be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. Information related to failure can be stored, and programs for executing related information can also be stored. The central processor 9100 can execute the programs stored in the memory 9140 to realize information storage or processing, etc.

[0143] The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0144] The memory 9140 can be a solid state memory such as a read only memory (ROM), a random access memory (RAM), a SIM card, or the like. It can also be a memory that retains information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes referred to as an EPROM or the like. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 can include an application / function storage 9142 for storing application programs and function programs or for storing a flow for executing an operation of the electronic device 9600 by the central processing unit 9100.

[0145] The memory 9140 can also include a data storage 9143 for storing data such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. A driver storage 9144 of the memory 9140 can include various drivers of the electronic device for a communication function and / or for performing other functions of the electronic device such as a messaging application, an address book application, and the like.

[0146] The communication module 9110 is a transmitter / receiver that transmits and receives signals via an antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0147] Based on different communication technologies, a plurality of communication modules 9110 such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, and the like can be provided in the same electronic device. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby enabling a conventional telecommunication function. The audio processor 9130 can include any suitable buffer, decoder, amplifier, and the like. In addition, the audio processor 9130 is also coupled to the central processing unit 9100, thereby enabling recording on a local machine through the microphone 9132 and enabling playing of a sound stored on the local machine through the speaker 9131.

[0148] The embodiments of the present application also provide a computer readable storage medium capable of realizing all steps of the multi-modal perception based emotion recognition method for the elderly with the execution subject being a server or a client in the above-mentioned embodiments, and the computer program is stored on the computer readable storage medium, and when the processor executes the computer program, all steps of the multi-modal perception based emotion recognition method for the elderly with the execution subject being a server or a client in the above-mentioned embodiments are realized, for example, the following steps are realized when the processor executes the computer program:

[0149] Step S101: Collecting multi-modal data of the elderly, collecting a facial video sequence through a high-definition camera, collecting a voice signal through a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequence to obtain a clear facial image, eliminating background noise from the voice signal to obtain clean voice, converting the human posture data into a joint coordinate sequence and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial image, acoustic features of the clean voice, and posture features of the joint coordinate sequence, and constructing a facial expression vector, a voice feature vector, and a posture feature vector respectively;

[0150] Step S102: Constructing an individual information model for the elderly, obtaining historical emotional expression data of a target elderly person from a user portrait database, extracting individual feature parameters in the historical emotional expression data, constructing a feature mapping matrix based on the individual feature parameters, inputting the facial expression vector, the voice feature vector, and the posture feature vector into a cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse multi-modal features to obtain a fusion feature vector;

[0151] Step S103: Performing emotion recognition by using a hierarchical time sequence classifier, constructing a time sequence feature extraction module based on a long short-term memory network, inputting the fusion feature vector into the time sequence feature extraction module to obtain an emotional state sequence, performing time sequence modeling on the emotional state sequence to obtain an emotional development trend feature, combining the emotional development trend feature to probabilistically predict the current emotional state, generating an emotion recognition result, and updating the emotion recognition result to the user portrait database.

[0152] From the above description, the computer readable storage medium provided by the embodiments of the application can realize the optimization and enhancement of signal quality by innovatively constructing a multi-modal data preprocessing mechanism and integrating facial expression, voice and posture features. A personalized feature mapping model based on historical emotional expression data is designed, a cross-modal attention network is combined, an adaptive feature fusion strategy is established for intelligent matching. A hierarchical time series classification mechanism is introduced, and a long short-term memory network is used to realize dynamic modeling of emotional development trends, supporting accurate prediction of emotional states. This method effectively solves the shortcomings of traditional technologies in multi-modal processing, personalized modeling and time series analysis, and significantly improves the accuracy and reliability of emotion recognition for the elderly.

[0153] The embodiments of the application also provide a computer program product capable of realizing all steps of the multi-modal perception based emotion recognition method for the elderly in the above-mentioned embodiments, wherein the execution subject is a server or a client. The computer program / instructions are executed by a processor to realize the steps of the multi-modal perception based emotion recognition method for the elderly, for example, the computer program / instructions realize the following steps:

[0154] Step S101: Collecting multi-modal data of the elderly, collecting facial video sequences through a high-definition camera, collecting voice signals using a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequences to obtain clear facial images, eliminating background noise from the voice signals to obtain clean voice, converting the human posture data into joint coordinate sequences and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial images, acoustic features of the clean voice, and posture features of the joint coordinate sequences, and constructing facial expression vectors, voice feature vectors, and posture feature vectors, respectively;

[0155] Step S102: Constructing an elderly personalized information model, obtaining historical emotional expression data of a target elderly person from a user portrait database, extracting personalized feature parameters in the historical emotional expression data, constructing a feature mapping matrix based on the personalized feature parameters, inputting the facial expression vectors, the voice feature vectors, and the posture feature vectors into a cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse multi-modal features to obtain a fusion feature vector;

[0156] Step S103: performing emotion recognition by using a hierarchical time sequence classifier, wherein the hierarchical time sequence classifier is configured to construct a time sequence feature extraction module based on a long short-term memory network, input the fusion feature vector into the time sequence feature extraction module to obtain an emotion state sequence, perform time sequence modeling on the emotion state sequence to obtain an emotion development trend feature, perform probability prediction on a current emotion state in combination with the emotion development trend feature, generate an emotion recognition result, and update the emotion recognition result to a user portrait database.

[0157] As can be seen from the above description, the computer program product provided by the embodiment of the present application innovatively constructs a multi-modal data preprocessing mechanism, optimizes and enhances the signal quality by integrating facial expression, voice and posture features. A personalized feature mapping model based on historical emotion expression data is designed, a cross-modal attention network is combined, an adaptive feature fusion strategy is established for intelligent matching. A hierarchical time sequence classification mechanism is introduced, dynamic modeling of the emotion development trend is realized through a long short-term memory network, and accurate prediction of the emotion state is supported. The method effectively solves the deficiencies of the traditional technology in multi-modal processing, personalized modeling and time sequence analysis, and significantly improves the accuracy and reliability of the emotion recognition of the elderly.

[0158] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0159] The present application is described with reference to flowcharts and / or block diagrams of the method, device (apparatus), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks Figure 1 The device that implements the functions specified in one or more flows and / or blocks

[0160] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in conjunction with the disclosed methods can be implemented on practitioners' computers in computer software, firmware, hardware, or combinations of them. Figure 1

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in conjunction with the disclosed methods can be implemented on practitioners' computers in computer software, firmware, hardware, or combinations of them. Figure 1

[0162] The principles and implementations of the present application are described in the specific embodiments, the above examples are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed; in view of the above, the content of the specification should not be understood as the limitation of the present application.​​

Claims

1. A method for emotion recognition of the elderly based on multi-modal perception, characterized in that, The method comprises: Collecting multi-modal data of the elderly, collecting facial video sequences through a high-definition camera, collecting voice signals using a microphone array, collecting human posture data based on a depth camera, performing illumination compensation and motion blur correction on the facial video sequences to obtain clear facial images, eliminating background noise from the voice signals to obtain clean voice, converting the human posture data into a joint coordinate sequence and correcting coordinate abnormalities caused by occlusion, extracting expression features of the clear facial images, acoustic features of the clean voice, and posture features of the joint coordinate sequence, and constructing facial expression vectors, voice feature vectors, and posture feature vectors respectively; Building an individual information model for the elderly, obtaining historical emotional expression data of the target elderly person from a user portrait database, extracting individual feature parameters from the historical emotional expression data, constructing a feature mapping matrix based on the individual feature parameters, inputting the facial expression vectors, voice feature vectors, and posture feature vectors into a cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse multi-modal features to obtain a fusion feature vector; Using a hierarchical time series classifier for emotion recognition, the hierarchical time series classifier constructs a time series feature extraction module based on a long short-term memory network, inputs the fusion feature vector into the time series feature extraction module to obtain an emotional state sequence, performs time series modeling on the emotional state sequence to obtain emotional development trend features, combines the emotional development trend features to probabilistically predict the current emotional state, generates an emotion recognition result, and updates the emotion recognition result to the user portrait database. 2.The multi-modal perception based emotion recognition method for the elderly according to claim 1, wherein, The collection of multi-modal data of the elderly, the collection of facial video sequences through a high-definition camera, the collection of voice signals using a microphone array, the collection of human posture data based on a depth camera, the illumination compensation and motion blur correction on the facial video sequences to obtain clear facial images, the elimination of background noise from the voice signals to obtain clean voice, and the conversion of the human posture data into a joint coordinate sequence and the correction of coordinate abnormalities caused by occlusion, comprise: Inputting the facial video sequences collected by the high-definition camera into an image enhancement network, the image enhancement network uses an adaptive histogram equalization algorithm to perform illumination compensation on the facial region, estimates the inter-frame motion based on the optical flow field, constructs a motion compensation matrix to sharpen the blurred area to obtain a clear facial image, collects voice signals using a microphone array, determines the sound source direction based on the beamforming algorithm, uses an adaptive filter to eliminate background noise in non-target directions, and performs frame windowing processing on the voice signals to obtain clean voice; The human body posture data is collected based on a depth camera, a human body contour region is obtained by segmenting a depth image, three-dimensional space coordinates of human body skeleton key points are calculated, the three-dimensional space coordinates are arranged in time sequence to construct a joint node coordinate sequence, a Kalman filtering algorithm is used to smooth the joint node coordinate sequence, a skeleton topology constraint model is constructed to detect abnormal coordinates caused by occlusion, a time sequence interpolation algorithm is used to correct abnormal coordinate points, and a continuous and stable joint node coordinate sequence is generated. 3.The multi-modal perception based emotion recognition method for the elderly according to claim 1, wherein, The expression features of the clear face image, the acoustic features of the clean speech, and the posture features of the joint node coordinate sequence are extracted, and a facial expression vector, a speech feature vector, and a posture feature vector are constructed, including: Face detection and alignment are performed on the clear face image, a face region image is extracted, the face region image is input into a convolutional neural network, face contour features are extracted from a shallow convolutional layer, face expression features are extracted from a deep convolutional layer, feature pyramid networks are used to fuse features at different levels, and a fully connected layer is used to map the fused features to a feature space to obtain a facial expression vector; the clean speech is input into a speech feature extraction model, mel-frequency cepstrum coefficients and pitch contour features are calculated, and an acoustic feature vector is constructed in combination with a speech fundamental frequency, short-time energy, and a zero-crossing rate; The relative position relationship and motion speed between the joint nodes are calculated for the joint node coordinate sequence, a posture feature extraction model is constructed based on a graph convolution network, the posture feature extraction model models the human body skeleton as a graph structure, spatial and temporal convolution is performed on the joint node coordinates and edge information to obtain local posture features, the local features are weighted and aggregated through an attention pooling layer to obtain global posture features, and the global posture features are mapped to a posture feature vector. 4.The multi-modal perception based emotion recognition method for the elderly according to claim 1, wherein, The personalized information model of the elderly is constructed, historical emotional expression data of a target elderly person is obtained from a user portrait database, personalized feature parameters in the historical emotional expression data are extracted, a feature mapping matrix is constructed based on the personalized feature parameters, including: An identity of the target elderly person is input into a user portrait database retrieval module, the retrieval module obtains historical emotional expression data based on an identity matching algorithm, the historical emotional expression data is segmented according to a time window, the emotional change frequency and amplitude in each time window are calculated, emotional expression habit features are extracted, the emotional expression habit features are classified based on a clustering analysis method, and personalized feature parameters are generated, the personalized feature parameters include an emotional baseline level, an emotional fluctuation range, and an emotional duration; A feature mapping network is constructed as the personalized information model, the personalized feature parameters are input into the feature mapping network, the feature mapping network performs nonlinear transformation on the feature parameters based on a multilayer perceptron structure to generate a feature mapping weight matrix, the feature mapping weight matrix is standardized by a batch normalization layer, and the standardized result is converted into a feature mapping matrix by an activation function. 5.The multi-modal perception based emotion recognition method for the elderly according to claim 1, wherein, The face expression vector, the speech feature vector, and the posture feature vector are input into a cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse the multi-modal features to obtain a fused feature vector, including: A cross-modal attention network is constructed, and the cross-modal attention network includes a self-attention subnetwork and a mutual-attention subnetwork. The face expression vector, the speech feature vector, and the posture feature vector are input into the self-attention subnetwork, respectively. The self-attention subnetwork calculates a correlation score of features in the same modality based on a query-key-value pair mechanism, generates an attention weight according to the correlation score, and combines the features in the same modality by weighting based on the attention weight to obtain an intra-modal feature representation. The mutual-attention subnetwork constructs a cross-modal mapping relationship based on the feature mapping matrix, and calculates a semantic correlation between different modal features. The intra-modal feature representation is input into a feature fusion module. The feature fusion module selectively fuses different modal features by using a gated update unit, adjusts a feature fusion weight based on the semantic correlation, performs dimension reduction processing on the fused features to obtain a feature embedding vector, and combines the feature embedding vector with the original feature vector through a residual connection structure to obtain a fused feature vector. 6.The multi-modal perception based emotion recognition method for the elderly according to claim 1, wherein, The hierarchical temporal classifier is used for emotion recognition. The hierarchical temporal classifier constructs a temporal feature extraction module based on a long short-term memory network, inputs the fused feature vector into the temporal feature extraction module to obtain an emotion state sequence, including: A hierarchical temporal classifier is constructed. The fused feature vector is arranged in time sequence to construct a feature sequence, the feature sequence is segmented by a sliding window to obtain a feature segment, a temporal feature extraction module is constructed based on a bidirectional long short-term memory network, the temporal feature extraction module includes a memory unit and a forgetting gate, the memory unit stores historical feature state information, the forgetting gate updates the memory unit state based on the current input feature, and an implicit state vector is generated through an output gate, the implicit state vector encodes the temporal dependence of the feature sequence; The implicit state vector is input into a multi-layer perceptron decoder, the multi-layer perceptron decoder performs nonlinear transformation on the implicit state vector to obtain an emotion state probability distribution, selects the class with the maximum probability as the emotion state label at the current time based on the maximum likelihood criterion, and combines the emotion state labels in the continuous time window into an emotion state sequence. 7.The multi-modal perception based emotion recognition method for the elderly according to claim 1, wherein, The emotion state sequence is modeled in time sequence to obtain an emotion development trend feature, the current emotion state is probabilistically predicted based on the emotion development trend feature, an emotion recognition result is generated, and the emotion recognition result is updated to a user portrait database, including: The time sequence analysis module is constructed to calculate a transition probability matrix of different emotional categories in the emotional state sequence, extract a duration and a conversion law of the emotional state, dynamically update the transition probability matrix by using a recurrent neural network, select key time sequence features based on an attention mechanism, input the key time sequence features into a trend prediction network, the trend prediction network predicts an emotional development trend at a next moment based on a historical transition law, and performs confidence evaluation on the emotional development trend to obtain an emotional development trend feature; The emotion prediction model is constructed, the emotional state at a current moment and the emotional development trend feature are input into the emotion prediction model, a conditional probability distribution of different emotional categories is calculated based on a conditional random field, the conditional probability distribution is normalized to obtain an emotion recognition result, the emotion recognition result is combined with a corresponding timestamp to form a state record, and the state record is written into an emotional history table of a user portrait database based on an incremental update strategy.

8. A multi-modal perception based emotion recognition device for elderly, characterized in that, The device comprises: The feature processing module is configured to collect multi-modal data of the elderly, collect a facial video sequence by using a high-definition camera, collect a voice signal by using a microphone array, collect human posture data based on a depth camera, perform illumination compensation and motion blur correction on the facial video sequence to obtain a clear facial image, eliminate background noise from the voice signal to obtain clean voice, convert the human posture data into a joint coordinate sequence and correct coordinate abnormalities caused by occlusion, extract expression features of the clear facial image, acoustic features of the clean voice, and posture features of the joint coordinate sequence, and construct a facial expression vector, a voice feature vector, and a posture feature vector, respectively. The feature fusion module is configured to construct an elderly individual information model, obtain historical emotional expression data of a target elderly person from a user portrait database, extract individual feature parameters in the historical emotional expression data, construct a feature mapping matrix based on the individual feature parameters, input the facial expression vector, the voice feature vector, and the posture feature vector into a cross-modal attention network, and the cross-modal attention network combines the feature mapping matrix to adaptively fuse the multi-modal features to obtain a fusion feature vector. The emotion recognition module is configured to perform emotion recognition by using a hierarchical time sequence classifier, construct a time sequence feature extraction module based on a long short-term memory network, input the fusion feature vector into the time sequence feature extraction module to obtain an emotional state sequence, perform time sequence modeling on the emotional state sequence to obtain an emotional development trend feature, combine the emotional development trend feature to probabilistically predict a current emotional state, generate an emotion recognition result, and update the emotion recognition result to the user portrait database.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the multi-modal perception-based emotion recognition method for the elderly according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the multi-modal perception-based emotion recognition method for the elderly according to any one of claims 1 to 7 when executed by the processor.

Images