Elderly accompanying implementation method and device based on multi-modal emotion recognition

Abstract

The invention provides an elderly accompanying implementation method and device based on multi-modal emotion recognition, and belongs to the technical field of information management. The method comprises the following steps: collecting multi-modal emotion data of an elderly user; performing multi-modal feature extraction and cross-modal feature fusion on the multi-modal emotion data by using a multi-modal analysis model and a cross-modal distillation pipeline provided by a multi-modal fusion framework to obtain an emotion feature recognition vector; inputting the emotion feature recognition vector into an emotion recognition deep learning model so as to recognize the current emotion state of the elderly user in combination with an emotion state attention mechanism; sampling behavior log modal data, constructing a dynamic space-time atlas, and performing root cause analysis on the current emotional state in combination with the dynamic space-time atlas to obtain a root cause analysis result; and pushing accompanying interaction service and / or medical care auxiliary early warning suggestions according to the current emotional state and the root cause analysis result. The problem that accurate nursing services and accompanying strategies are difficult to provide in the prior art can be solved.

Description

Technical Field

[0001] This application belongs to the field of information management technology, specifically relating to a method and device for providing companionship to the elderly based on multimodal emotion recognition. Background Technology

[0002] The elderly companionship system is a comprehensive optimization system that integrates advanced artificial intelligence technologies (such as machine learning, deep learning, data analysis, and natural language processing) to improve the efficiency and quality of elderly care services. By monitoring and intelligently analyzing multi-dimensional data on the elderly's physical and mental health, care needs, and lifestyle habits in real time, it can provide more precise, personalized, and efficient care services.

[0003] Most existing technologies utilize multimodal feature extraction and fusion to dynamically model the emotions of elderly users, thereby providing psychological health guidance. Specifically, multimodal feature extraction and fusion yields psychological health fusion feature data; and by introducing a dynamic emotion modeling module, emotional state sequence modeling is optimized to obtain dynamic emotion modeling classification data, which is then used to generate adaptive psychological health guidance strategies, resulting in psychological health guidance strategies.

[0004] However, existing elderly companionship systems based on multimodal feature extraction and fusion methods have a disconnect between the feature vectors obtained after multimodal feature fusion and the user's emotional state. This makes it difficult to deeply analyze the reasons for changes in the user's emotional state, which in turn makes it difficult to provide users with accurate care services and companionship strategies. Summary of the Invention

[0005] This application provides a solution for elderly companionship based on multimodal emotion recognition, which can explore the relationship between the feature vector after multimodal feature fusion and the user's emotional state, deeply analyze the reasons for changes in the user's emotional state, and provide users with accurate care services and companionship strategies.

[0006] According to a first aspect of this application, embodiments of this application provide a method for providing companionship to the elderly based on multimodal emotion recognition, including: Collect multimodal emotional data from elderly users, including visual modal data, auditory modal data, physiological modal data, and behavioral log modal data; Using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework, multimodal feature extraction and cross-modal feature fusion are performed on visual modal data, auditory modal data, physiological modal data and behavioral log modal data to obtain emotional feature recognition vectors for elderly users; The emotion feature recognition vector is input into the emotion recognition deep learning model, and the emotion recognition deep learning model is used to identify the current emotional state of elderly users based on the emotion state attention mechanism. By sampling behavioral log modal data, constructing a dynamic spatiotemporal graph, and combining the emotional feature recognition vector and the dynamic spatiotemporal graph, root cause analysis of the current emotional state is performed to obtain the root cause analysis results. Based on the current emotional state and root cause analysis results, elderly users will be given companionship and interaction services and / or medical care assistance warning suggestions.

[0007] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of collecting multimodal emotional data of elderly users, wherein the multimodal emotional data includes visual modality, auditory modality, physiological signal modality, and behavioral log modality, includes: A series of consecutive multi-frame image sequences containing facial expressions, body movements, and gestures of elderly users were collected as visual modality data. Collect auditory modal data including the speech characteristics, speech content, and ambient sounds of elderly users; Collect physiological modal data of elderly users through wearable devices or physiological signal sensors; Record the interactive behavior data of elderly users to obtain behavioral log modal data.

[0008] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework to extract multimodal features and fuse cross-modal features from visual modal data, auditory modal data, physiological modal data, and behavioral log modal data to obtain the emotional feature recognition vector of the elderly user includes: The improved MViT v2 model provided by the multimodal fusion framework is used to identify the behavioral characteristics and motion cues of consecutive multi-frame image sequences in visual modal data by combining an attention mechanism. Based on the behavioral characteristics and motion cues, the visual feature vector corresponding to the visual modal data is output. Using the ECAPA model provided by the multimodal fusion framework, frame-level features of the corresponding feature spectra of auditory modal data are extracted. Based on the attention statistical pooling mechanism, speech feature vectors containing speech emotion and speech content are extracted from the frame-level features. Using the large text processing model provided by the multimodal fusion framework, combined with the pre-built text semantic vector space, physiological modality data and behavioral log modality data are mapped to the text semantic vector space respectively to obtain text feature vectors; Using a cross-modal distillation pipeline, and based on a cross-attention fusion mechanism with emotion state weights, adversarially learn the joint distribution of visual feature vectors, speech feature vectors, and text feature vectors, and fuse them to obtain an emotion feature recognition vector that includes emotional dependencies.

[0009] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of using the improved MViT v2 model provided by the multimodal fusion framework, combined with an attention mechanism to identify behavioral characteristics and motion cues in a series of consecutive image frames in the visual modality data, and outputting the visual feature vector corresponding to the visual modality data based on the behavioral characteristics and motion cues, includes: A sequence of consecutive multi-frame images is input into the improved MViT v2 model. Based on the built-in time component generation mechanism, the behavioral characteristics in the sequence of consecutive multi-frame images are identified, and the time components corresponding to the behavioral characteristics are obtained. Using the fixed stride pooling strategy built into the improved MViT v2 model, a fixed stride pooling operation is performed on a series of consecutive frames of images in the time dimension to obtain motion cues of human behavior. The attention mechanism of the improved MViT v2 model is used to process a series of consecutive multi-frame image sequences according to temporal components and motion cues, and output the behavioral feature vector corresponding to the visual modality data. Multiple classification attention heads are used to predict the behavior category labels corresponding to the behavior feature vectors; where the behavior category labels include facial expressions, body postures, and gestures.

[0010] Preferably, in the above-mentioned method for providing companionship to the elderly, the steps of using the ECAPA model provided by the multimodal fusion framework to extract frame-level features from the feature spectrograms corresponding to the auditory modality data, and extracting speech feature vectors containing speech emotion and speech content from the frame-level features according to the attention statistical pooling mechanism, include: Auditory modality data is converted into feature spectrograms, and a time-delay neural network is used to perform feature detection on the feature spectrograms in the time dimension. The features are then fused to obtain speech convolutional features at different time steps. The speech convolutional features are input into the pre-designed speech emotion channel and speech content channel respectively. In the speech emotion channel, a multi-scale feature fusion mechanism is used to fuse the speech convolutional features to obtain frame-level features containing emotional states. In the speech content channel, an emotional keyword matching mechanism is used to extract emotional keywords from the speech convolutional features to obtain frame-level features containing emotional content. Using an attention-based statistical pooling mechanism that integrates emotion weights and environmental anomaly weights, the weighted mean and weighted standard deviation of frame-level features are calculated and concatenated to obtain a fixed-dimensional speech feature vector. By using a nonlinear transformation method, the identity vector of elderly users is embedded into the language feature vector, and the speech feature vector after embedding the identity vector is output.

[0011] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of using a cross-modal distillation pipeline and, based on a cross-attention fusion mechanism with emotion state weights, adversarially learning the joint distribution of visual feature vectors, speech feature vectors, and text feature vectors to fuse them to obtain an emotion feature recognition vector containing emotional dependencies includes: Based on the distribution of emotion features corresponding to visual feature vectors, speech feature vectors, and text feature vectors in the same time sequence, design emotion state weights. Using a cross-modal distillation pipeline, adversarial learning is performed on the joint distribution of visual, speech, and text feature vectors based on feature similarity among them; whereby... If the feature similarity is greater than or equal to the predetermined similarity threshold, then the emotion feature recognition vector is obtained by adversarially learning the joint distribution of visual feature vector, speech feature vector and text feature vector according to the alignment loss function. If the feature similarity is less than a predetermined similarity threshold, a cross-attention fusion mechanism is used, which combines emotional state weights and fuses the linear layer projections corresponding to the visual feature vector, speech feature vector, and text feature vector respectively, to obtain an emotional feature recognition vector that includes emotional dependencies.

[0012] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of inputting the emotion feature recognition vector into an emotion recognition deep learning model and using the emotion recognition deep learning model to identify the current emotional state of the elderly user based on the emotion state attention mechanism includes: A deep learning model for emotion recognition is constructed using a bidirectional LSTM network combined with an emotion state attention mechanism. The emotion feature recognition vector is input into the emotion recognition deep learning model for feature learning, and the emotion state attention mechanism is combined to capture the emotion dependency relationship in the emotion feature recognition vector; Based on emotional dependence, the current emotional state is output through multiple emotional classification attention heads.

[0013] Preferably, in the above-mentioned method for providing companionship to the elderly, the steps of sampling behavioral log modal data, constructing a dynamic spatiotemporal graph, and combining emotional feature recognition vectors and the dynamic spatiotemporal graph to perform root cause analysis on the current emotional state and obtain the root cause analysis results include: The sampling rate is dynamically adjusted based on the time decay factor. The dynamically adjusted sampling rate is used to sample behavioral log modal data to obtain multiple text feature sequences. Multiple text feature sequences are mapped to a semantic vector space, and the semantic vector corresponding to each text feature sequence is extracted. Based on the semantic vectors and context positions of multiple text feature sequences, the semantic association and text type of multiple text feature sequences are extracted. By combining temporal order, semantic vectors, semantic associations, and text types, dynamic spatiotemporal maps corresponding to multiple text feature sequences are constructed. Align the sentiment feature recognition vector with the text feature sequence of the dynamic spatiotemporal graph in time; Calculate the contribution rate of each emotion feature recognition vector to the current emotion state and match it with time-aligned text feature sequences; Based on the dynamic path weighting algorithm combined with semantic association, text feature sequences with emotional contribution in the dynamic spatiotemporal graph are calculated and searched. By using text feature sequences and sentiment contribution, the root cause analysis results in the root cause database are obtained through matching.

[0014] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of pushing companionship and interaction services and / or medical assistance and early warning suggestions to elderly users based on their current emotional state and root cause analysis results includes: Design a virtual companion assistant to deliver emotional companionship services tailored to elderly users' current emotional state and root cause analysis results via multimedia. These services include: The virtual companion assistant engages in natural language conversations with elderly users via voice and text, and adjusts its conversation style and content based on the user's current emotional state in real time. Based on the current emotional state of elderly users, interest-based push cases are matched with the user profiles of elderly users and the interest database of the profiles. Through multimedia methods, combined with interest-based push cases, interest-recommended content and emotion-regulating content are pushed to elderly users. Based on the current emotional state, match the elderly users' usual entertainment methods and push interactive entertainment activities and interpersonal contact methods to them.

[0015] Preferably, in the above-mentioned method for providing companionship to the elderly, the step of pushing companionship and interaction services and / or medical and nursing assistance warning suggestions to the elderly user based on the current emotional state and root cause analysis results includes: generating and pushing the elderly user's emotional state dashboard to relevant medical and nursing staff according to the current emotional state, so as to provide medical and nursing staff with warnings and care for the elderly user's emotional state; Real-time monitoring of the fluctuations and duration of the current emotional state; When the fluctuation amplitude is greater than or equal to the predetermined amplitude threshold, and any one of the visual modal data, auditory modal data, and physiological modal data triggers an abnormal warning signal, an abnormal alarm signal and the location status information of the elderly user are pushed to medical staff. When the duration of the current emotional state within the abnormal state range is greater than or equal to a predetermined time threshold, the current emotional state and the root cause analysis results are used to match early warning cases in the early warning case library. Based on the expert experience decision tree constructed using fuzzy logic and combined with early warning cases, risk suggestion reports and abnormal early warning signals are generated and pushed to medical staff.

[0016] Preferably, the above-mentioned method for providing companionship to the elderly, after the step of pushing companionship and interaction services and / or medical assistance and early warning suggestions to the elderly user based on the current emotional state and root cause analysis results, further includes: By combining the companionship and interaction services selected by elderly users during the scheduled time period with their user profiles, a user profile of the elderly user is generated. User profile feature vectors are extracted from user profiles and cross-fused with emotion feature recognition vectors to obtain user emotion feature vectors; The fused user sentiment feature vector is input into the long-term risk prediction model, and the positive and negative time-series sentiment features of the user sentiment feature vector are captured through forward and backward propagation. By fusing positive and negative temporal sentiment features according to temporal relationships, a fused temporal dependency relationship is obtained. Risk factor features are then extracted from the user's sentiment feature vector based on the temporal dependency relationship. The risk factors are classified and predicted, and risk prediction labels are output.

[0017] Preferably, the above-mentioned method for providing companionship to the elderly further includes, after the step of collecting multimodal emotional data from elderly users: By fusing multimodal emotional data according to multidimensional evaluation criteria, a multidimensional flow state vector is obtained; The multidimensional flow state vector is input into a pre-built flow state assessment model. The flow state assessment model is combined with the current user behavior of elderly users to predict the location and trend of the elderly user's flow channel. Based on the loss function of the flow state assessment model, combined with the location and trend of the flow channel, the flow state of elderly users is maintained or adjusted. Maintaining or adjusting the flow state of elderly users includes: changing the difficulty of the current user behavior, decomposing or adding behavioral goals and / or providing behavioral reference suggestions. The loss function is designed based on the target flow state and the location and trend of the flow channel.

[0018] According to a second aspect of this application, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the elderly companionship method based on multimodal emotion recognition provided by any of the above technical solutions.

[0019] The technical solution of this application has at least the following technical effects: The technical solution for elderly companionship based on multimodal emotion recognition provided in this application first collects multimodal emotion data of elderly users, including visual modality data, auditory modality data, physiological modality data, and behavioral log modality data. This allows for the acquisition of emotional states from different modalities as much as possible, more accurately reflecting the emotional state of elderly users. Then, using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework, feature extraction and cross-modal feature fusion are performed on the above-mentioned multimodal emotion data. This yields a comprehensive emotional feature recognition vector for elderly users, which accurately reflects their emotional characteristics. After inputting this emotional feature recognition vector into the emotion recognition deep learning model, ... By using an emotion recognition deep learning model based on the emotion state attention mechanism, the current emotional state of elderly users can be accurately identified. Then, by sampling behavioral log modal data to construct a dynamic spatiotemporal graph, and combining this graph with emotion feature recognition vectors to perform root cause analysis of the current emotional state, the root cause analysis results can be obtained, accurately determining the reasons affecting the elderly user's current emotional state. Finally, based on the current emotional state and root cause analysis results, companionship and interaction services and / or medical care assistance warning suggestions are pushed to elderly users. This provides personalized companionship services for elderly users with different emotional types and states, effectively solving the emotional companionship challenges of the elderly and effectively improving their mental health and subjective well-being. This approach addresses the problem in existing elderly companionship solutions based on multimodal feature extraction and fusion methods where the feature vector after multimodal feature fusion is disconnected from the user's emotional state, making it difficult to deeply analyze the reasons for changes in the user's emotional state and provide accurate care services and companionship strategies. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for providing companionship to the elderly based on multimodal emotion recognition, provided in an embodiment of this application; Figure 2 for Figure 1 The illustrated embodiment provides a flowchart of a method for multimodal feature extraction and cross-modal feature fusion of multimodal sentiment data; Figure 3 for Figure 1 The illustrated embodiment provides a flowchart of a method for predicting risk factors based on user profiles; Figure 4 for Figure 1 The illustrated embodiment provides a flowchart of a method for maintaining or adjusting a state of cardiac flow. Figure 5 A schematic diagram of a device for providing companionship to the elderly based on multimodal emotion recognition, provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0023] In this application, unless otherwise expressly specified and limited, the terms "above" and "below" the second feature can refer to direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0024] The existing technology has the following drawbacks: Existing elderly companionship systems based on multimodal feature extraction and fusion methods have a disconnect between the feature vectors obtained after multimodal feature fusion and the user's emotional state. This makes it difficult to deeply analyze the reasons for changes in the user's emotional state, which in turn makes it difficult to provide users with accurate care services and companionship strategies.

[0025] To address the aforementioned issues, the following embodiments of this application provide a solution for elderly companionship based on multimodal emotion recognition. This solution can be applied in scenarios such as hospital wards, nursing homes, and home care, enabling personalized companionship and emotional comfort for elderly users. Based on the emotional communication needs of empty-nest elderly in today's aging society, it can non-invasively perceive and classify the emotional states of the elderly, thereby providing personalized companionship services for elderly people with different emotional types, effectively solving the problem of emotional companionship for the elderly, and improving their mental health and subjective well-being.

[0026] To achieve the above objectives, see [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating a method for providing companionship to the elderly based on multimodal emotion recognition, provided as an embodiment of this application. Figure 1 As shown, this method for providing companionship to the elderly based on multimodal emotion recognition includes: S110: Collect multimodal emotional data from elderly users, including visual modal data, auditory modal data, physiological modal data, and behavioral log modal data.

[0027] The technical solution provided in this application embodiment can accurately identify the emotional state of elderly users by collecting multimodal emotional data, and then push corresponding emotional services and / or medical advice to meet the emotional needs and medical care services of elderly users.

[0028] Specifically, as a preferred embodiment, in the above-mentioned method for providing companionship to the elderly, step S110: collecting multimodal emotional data of elderly users, wherein the multimodal emotional data includes visual modality, auditory modality, physiological signal modality, and behavioral log modality, includes the following steps: S111: Collect consecutive multi-frame image sequences containing facial expressions, body movements, and gestures of elderly users as visual modal data.

[0029] The visual modal data includes a series of consecutive multi-frame images of facial expressions, body movements, and gestures. Specifically, for facial expressions: it requires identifying basic expressions and their intensity, including happiness, sadness, anger, surprise, fear, disgust, and neutrality; capturing micro-expressions; and identifying eye contact, including alertness / lack of focus, fixation time, and fixation direction. For body movements, it includes recognizing sitting, standing, lying, activity levels (active / still), and specific actions (such as repeatedly rubbing hands, pacing, and curling up). For gesture recognition, it mainly includes understanding simple interactive gestures (such as waving, shaking hands, and thumbs-up).

[0030] S112: Collect auditory modal data including the speech characteristics, speech content, and environmental sounds of elderly users. Specifically, the auditory modal data includes speech feature recognition, speech content recognition, and environmental sound analysis. This includes analyzing features such as tone, speech rate, volume, and pauses to identify specific emotional states, including positive, negative, neutral, excited, and / or depressed. Speech content keyword recognition includes identifying specific keywords (such as "pain," "unhappy," "homesick," and / or "call a nurse") and combining this with emotion to determine the level of urgency. Environmental sound analysis includes identifying abnormal sounds (such as coughing, groaning, falling sounds, and / or the sound of objects dropping).

[0031] S113: Collect physiological modal data of elderly users through wearable devices or physiological signal sensors. The physiological modal data mainly includes heart rate / heart rate variability and skin conductance response. Heart rate / heart rate variability is identified through wearable wristbands or mattress sensors (low-invasive) to assist in determining stress, anxiety, and relaxation states. Skin conductance response is identified through wearable devices to assist in determining emotional arousal (excitement and tension, etc.). Additionally, this application can also collect signals such as electrocardiogram, finger pulse, respiratory rate, and pulse to monitor emotion-related physiological indicators. S114: Record the interactive behavior data of elderly users to obtain behavioral log modal data.

[0032] Behavioral log modal data: This data is obtained by integrating and analyzing behavioral data such as the frequency and duration of interaction between elderly users and the companionship system, the types of content they prefer, and the number of times they refuse to interact. It can serve as supplementary evidence of their current emotional state.

[0033] Figure 1 The technical solution provided in the illustrated embodiment, after step S110: collecting multimodal emotional data of elderly users, further includes: S120: Using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework, multimodal feature extraction and cross-modal feature fusion are performed on visual modal data, auditory modal data, physiological modal data, and behavioral log modal data to obtain the emotional feature recognition vector of elderly users. The emotional labels corresponding to the emotional feature recognition vector here include: basic emotions (happiness, sadness, anger, surprise, fear, disgust, and neutrality) + complex / medical-related states (calm, anxiety, depression, pain, confusion, and pleasure).

[0034] The technical solution provided in this application extracts multimodal features from the multimodal emotional data by using a multimodal analysis model provided by a multimodal fusion framework. This extracts multimodal features that reflect the emotional state of elderly users. Furthermore, by performing cross-modal feature fusion on the multimodal features through a cross-modal distillation pipeline, an emotional feature recognition vector for the elderly user can be obtained. This emotional feature recognition vector is obtained by fusing multiple modal features, thereby accurately reflecting the current emotional state of the elderly user.

[0035] To address the diverse multimodal emotion data, including visual, auditory, physiological, and behavioral log data, this application designs targeted multimodal analysis models, including an improved MViT v2 model, an ECAPA model, and other large-scale text processing models. Furthermore, a cross-media distillation pipeline is used to fuse the feature vectors extracted by these models across modalities, resulting in a fused emotion feature recognition vector. Specific testing on a dataset of elderly individuals demonstrates an accuracy exceeding 85%.

[0036] Specifically, as a preferred embodiment, such as Figure 2 As shown, in the above-mentioned method for providing companionship to the elderly, step S120, which involves using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework to extract multimodal features and fuse cross-modal features from visual modal data, auditory modal data, physiological modal data, and behavioral log modal data to obtain the emotional feature recognition vector of the elderly user, includes: S121: Using the improved MViT v2 model provided by the multimodal fusion framework, combined with the attention mechanism, the behavioral characteristics and motion cues of consecutive multi-frame image sequences in the visual modality data are identified, and the visual feature vector corresponding to the visual modality data is output based on the behavioral characteristics and motion cues.

[0037] The improved MViT2 v2 model provided in this application primarily improves upon the temporal component generation mechanism to dynamically adjust the temporal components corresponding to different visual modalities; employs a fixed stride pooling strategy to perform fixed-stride pooling operations on consecutive multi-frame image sequences in the temporal dimension; and utilizes multiple classification attention heads to obtain various types of visual feature vectors. By using the improved MViT v2 model, the behavioral characteristics and motion cues of consecutive multi-frame image sequences in the aforementioned visual modal data can be identified using an attention mechanism. These cues include subtle human behavior-related motion cues, such as slight head movements and finger gestures. Furthermore, based on these behavioral characteristics and motion cues, a visual feature vector corresponding to the visual modal data is output. This visual feature vector accurately reflects the behavioral actions of elderly users in the visual modality.

[0038] Specifically, as a preferred embodiment, in the elderly companionship method provided in this application, step S121 above: using the improved MViT v2 model provided by the multimodal fusion framework, combined with an attention mechanism to identify behavioral characteristics and motion cues in a continuous multi-frame image sequence in visual modal data, and outputting the visual feature vector corresponding to the visual modal data based on the behavioral characteristics and motion cues, specifically includes: S1211: Input a series of consecutive frames of images into the improved MViT v2 model. Based on the built-in time component generation mechanism, identify the behavioral characteristics in the series of consecutive frames of images and obtain the time components corresponding to the behavioral characteristics.

[0039] S1212: Using the fixed stride pooling strategy built into the improved MViT v2 model, fixed stride pooling is performed on a series of consecutive frames of images in the time dimension to obtain motion cues of human behavior.

[0040] S1213: Using the attention mechanism of the improved MViT v2 model, the sequence of consecutive multi-frame images is processed according to the temporal component and motion cues, and the behavioral feature vector corresponding to the visual modality data is output.

[0041] S1214: Use multiple classification attention heads to classify and predict the behavior category label corresponding to the behavior feature vector; where the behavior category label includes facial expression, body posture and gesture.

[0042] The improvements to this modified MViT v2 model include a temporal component generation mechanism, a fixed-stride pooling strategy, and multiple classification attention heads. Specifically, after acquiring a series of consecutive image frames, the model inputs these frames into the MViT v2 model. The temporal component generation mechanism identifies behavioral characteristics within the sequence for a predetermined duration, obtaining the temporal components corresponding to these characteristics. Then, a fixed-stride pooling strategy is applied to the sequence over the temporal dimension to obtain key and value tensors that preserve subtle motion cues related to user behavior. After processing the sequence with these temporal components, key tensors, and value tensors, a behavioral feature vector containing specific user behaviors is obtained. Finally, multiple pre-trained classification attention heads are used to classify these behavioral feature vectors, outputting corresponding behavior category labels. Each specialized attention head corresponds to a predefined behavior category. By using the improved MViT v2 model and combining it with the attention mechanism to identify the visual feature vectors corresponding to the visual modal data, we can accurately identify the behavioral characteristics of elderly users and analyze the psychological state corresponding to their behavior.

[0043] Figure 2 The technical solution provided in the illustrated embodiment, after step S121: outputting the visual feature vector corresponding to the visual modality data based on behavioral characteristics and motion cues, further includes: S122: Using the ECAPA model provided by the multimodal fusion framework, frame-level features of the corresponding feature spectra of auditory modal data are extracted. Based on the attention statistical pooling mechanism, speech feature vectors containing speech emotion and speech content are extracted from the frame-level features.

[0044] The technical solution provided in this application embodiment is that the voiceprint recognition model ECAPA model is a pre-trained audio classification network model. The ECAPA model can accurately extract the frame-level features of the feature spectrograms corresponding to the auditory modality data (such as Fbank / MFCC feature sequences), and further perform multi-layer feature fusion. According to the attention statistical pooling mechanism provided by the ECAPA model, the speech feature vector containing speech emotion and speech content is extracted from the frame-level features.

[0045] Specifically, as a preferred embodiment, step S122 above: using the ECAPA model provided by the multimodal fusion framework, extracting frame-level features of the feature spectrogram corresponding to the auditory modality data, and extracting speech feature vectors containing speech emotion and speech content from the frame-level features according to the attention statistical pooling mechanism, specifically including: S1221: Convert auditory modality data into feature spectrograms, use a time-delay neural network to perform feature detection on the feature spectrograms in the time dimension, and fuse them to obtain speech convolutional features at different time steps.

[0046] S1222: The speech convolutional features are input into the pre-designed speech emotion channel and speech content channel, respectively. In the speech emotion channel, a multi-scale feature fusion mechanism is used to fuse the speech convolutional features to obtain frame-level features containing emotional states. In the speech content channel, an emotion keyword matching mechanism is used to extract emotion keywords from the speech convolutional features to obtain frame-level features containing emotion content. This speech content channel can recognize features such as speech content and environmental noise.

[0047] S1223: Using an attention-based statistical pooling mechanism that integrates emotion weights and environmental anomaly weights, the weighted mean and weighted standard deviation of frame-level features are calculated and concatenated to obtain a fixed-dimensional speech feature vector.

[0048] The emotion weight and environmental anomaly weight can be obtained through the aforementioned speech emotion channel and speech content channel. The ECAPA model uses an attention mechanism with an attention statistical pooling layer to calculate a weight for frame-level features. Specifically, the ECAPA model automatically learns which time frames contain richer language emotion and speech content, and assigns higher weights to these frames; finally, it uses a weighted mean and weighted standard deviation to represent the statistical information of the entire sequence. The weighted mean and weighted standard deviation are concatenated to form a fixed-length speech feature vector.

[0049] S1224: An elderly user's identity vector is embedded into the language feature vector using a non-linear transformation, outputting a speech feature vector with the embedded identity vector. A fully connected layer can be added to the ECAPA model to map to the final identity embedding vector. A loss function is then used to separate different speakers, improving the model's overall discriminative ability, ultimately outputting the speech feature vector with the embedded identity vector.

[0050] The technical solution provided in this application converts auditory modal data (such as audio waveforms) into feature spectrograms, which are essentially feature sequences. Then, a time-delay neural network is used to detect features in the feature spectrogram in the time dimension, resulting in a multi-dimensional feature sequence. By fusing this feature sequence, speech convolutional features at different time steps can be obtained. Specifically, the input audio waveform undergoes pre-emphasis, framing, and windowing operations to segment the continuous audio signal into short-time signals. A time-delay neural network is used to detect speech convolutional features in the time dimension, such as MFCCs or Fbank features. These features can simulate the human ear's frequency perception characteristics, and are fused to obtain speech convolutional features at different time steps. Then, the speech convolutional features are input into the speech emotion channel and speech content channel respectively, and frame-level features are extracted to obtain frame-level features containing emotional state and emotional content. Specifically, a multi-scale feature fusion mechanism is used in the speech emotion channel to fuse the speech convolutional features, resulting in frame-level features containing emotional state. In the speech content channel, an emotional keyword matching mechanism is used to extract emotional keywords from the speech convolutional features, resulting in frame-level features containing emotional content. The multi-scale feature fusion mechanism here automatically learns and emphasizes important channels through the following steps: A feature sequence channel is used to perform global average pooling on features across the entire time dimension, generating a channel-level statistical vector that summarizes the global information of each channel. An activation channel is used to input this vector into a small two-layer fully connected network (bottleneck structure) to learn the weight of each channel (a scalar between 0 and 1). This weight represents the "importance" of the channel. Through the activation channel, sentiment weights and environmental anomaly weights are obtained. Then, an attention-based statistical pooling mechanism that integrates sentiment and environmental anomaly weights is used to calculate and concatenate the weighted mean and weighted standard deviation of frame-level features, resulting in a fixed-dimensional speech feature vector. Finally, through a non-linear transformation, the identity vector of the elderly user is embedded into the above speech feature vector, outputting the speech feature vector with the embedded identity vector.

[0051] in addition, Figure 2 The technical solution provided in the illustrated embodiment, in step S122: using the ECAPA model provided by the multimodal fusion framework to extract frame-level features of the feature spectrogram corresponding to the auditory modality data, and after extracting speech feature vectors containing speech emotion and speech content from the frame-level features according to the attention statistical pooling mechanism, further includes: S123: Using the large-scale text processing model provided by the multimodal fusion framework, combined with a pre-constructed text semantic vector space, physiological modality data and behavioral log modality data are mapped to the text semantic vector space respectively to obtain text feature vectors. This application embodiment pre-constructs a text semantic vector space. Since physiological modality data and behavioral log modality data are essentially text types, after vectorizing the keywords of the aforementioned physiological modality data and behavioral log modality data, they can be mapped to the aforementioned text semantic vector space to obtain text feature vectors, thereby accurately identifying the physiological modality and historical behavior of elderly users.

[0052] S124: Using a cross-modal distillation pipeline, adversarially learn the joint distribution of visual feature vectors, speech feature vectors, and text feature vectors based on a cross-attention fusion mechanism with emotion state weights, and fuse them to obtain an emotion feature recognition vector that includes emotion dependencies.

[0053] By using a cross-modal distillation pipeline to distill the key content of the aforementioned visual, speech, and text feature vectors, and adversarially learning the joint distribution of these feature vectors based on a cross-attention fusion mechanism corresponding to emotional state weights, an emotional feature recognition vector containing emotional dependencies can be obtained. This emotional feature recognition vector can then accurately extract the emotional dependencies of elderly users, such as their dependencies on relatives or specific objects.

[0054] This application primarily obtains the sentiment feature recognition vector containing emotional dependencies through two methods: when the feature similarity is high, the joint distribution of the various feature vectors is learned according to the alignment loss function to obtain the sentiment feature recognition vector; when the feature similarity is low, the feature vectors are fused through a cross-attention fusion mechanism to obtain the sentiment feature recognition vector containing emotional dependencies. These methods enable refined extraction of the sentiment feature recognition vector.

[0055] Specifically, as a preferred embodiment, Figure 2 In the elderly companionship implementation method shown, step S124: using a cross-modal distillation pipeline, based on a cross-attention fusion mechanism based on emotional state weights, adversarially learns the joint distribution of visual feature vectors, speech feature vectors, and text feature vectors, and fuses them to obtain an emotional feature recognition vector containing emotional dependencies, specifically including: S1241: Design emotional state weights according to the emotional feature distributions corresponding to visual feature vectors, speech feature vectors, and text feature vectors in the same time sequence. Since the visual feature vectors, speech feature vectors, and text feature vectors have a temporal correspondence, extracting the emotional feature distributions in the same time sequence can determine the emotional features of a specific time period, thereby designing emotional state weights for the above feature vectors in different time sequences.

[0056] S1242: Using a cross-modal distillation pipeline, adversarially learn the joint distribution of visual, speech, and text feature vectors based on feature similarity among visual, speech, and text feature vectors.

[0057] S1243: If the feature similarity is greater than or equal to the predetermined similarity threshold, then according to the alignment loss function, the joint distribution of the visual feature vector, speech feature vector and text feature vector is learned to obtain the emotion feature recognition vector.

[0058] S1244: If the feature similarity is less than the predetermined similarity threshold, the cross-attention fusion mechanism is used to combine the emotional state weights and fuse the linear layer projections corresponding to the visual feature vector, speech feature vector and text feature vector respectively to obtain the emotional feature recognition vector containing emotional dependence.

[0059] In this design, the feature similarity of the multimodal features is greater than or equal to a predetermined similarity threshold. An alignment loss function is used to learn the joint feature distribution, resulting in a more refined emotion feature recognition vector. Specifically, the cross-modal adversarial distillation pipeline includes a generator network, a discriminator network, and a feature alignment loss function. The generator network primarily performs two functions: learning the joint distribution of cross-modal features and generating fused features. The generator network can acquire and learn the joint distribution of the aforementioned visual, speech, and text feature vectors through adversarial learning, thereby solving the problem of fragmented multimodal data and difficulty in feature-level fusion. Specifically, if the visual, speech, and text feature vectors are independently encoded to generate 32-dimensional semantic vectors, a cross-modal adversarial distillation pipeline is designed. A generator network (containing 4 fully connected layers) is constructed within this pipeline to learn the joint distribution of visual-speech-text features, ultimately obtaining the final emotion feature recognition vector. The adversarial distillation pipeline is 32-dimensional. Modal feature vectors with dimensions higher than or equal to 32 will compete with each other when input into the pipeline, requiring adversarial competition for the pipeline's dimensions to achieve a joint distribution of visual and textual features. In this way, the generator network can learn this joint distribution of visual and textual features based on the loss function, thus obtaining the defect feature vector.

[0060] Furthermore, if the feature similarity is less than a predetermined similarity threshold, a cross-attention fusion mechanism is used. This mechanism combines emotional state weights with the linear layer projections corresponding to the visual, speech, and text feature vectors to obtain an emotional feature recognition vector that includes emotional dependencies. By using the cross-attention fusion mechanism combined with emotional state weights to fuse the aforementioned multimodal feature vectors, the similarity between different feature vectors can be mined to obtain specific emotional dependencies.

[0061] Specifically, cross-attention is a mechanism that allows information to be exchanged between different modalities or different feature representations. This application introduces an emotional state weight W, and uses this emotional state weight to fuse the linear layer projection of the above feature vectors, resulting in the following core formula:

[0062] Where Q is the projection from the visual feature vector, and K and V are the linear layer projections from the speech feature vector and text feature vector, respectively. Let W be a constant and W be the emotional state weight. Therefore, this emotional state weight can reflect the emotional dependency relationship. Thus, when the feature similarity is less than a predetermined similarity threshold, a cross-attention fusion mechanism is used to combine the emotional state weight with the linear layer projections corresponding to the above feature vectors to obtain an emotional feature recognition vector that includes the emotional dependency relationship.

[0063] Figure 1 The elderly companionship method based on multimodal emotion recognition provided in the illustrated embodiment, in step S120: after using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework to perform multimodal feature extraction and cross-modal feature fusion on the visual modality data, auditory modality data, physiological modality data, and behavioral log modality data to obtain the emotional feature recognition vector of the elderly user, further includes: S130: Input the emotion feature recognition vector into the emotion recognition deep learning model, and use the emotion recognition deep learning model to identify the current emotional state of elderly users based on the emotion state attention mechanism.

[0064] The technical solution provided in this application embodiment inputs the emotion feature recognition vector into the emotion recognition deep learning model. By using the emotion recognition deep learning model based on the emotion state attention mechanism, the current emotional state of elderly users can be accurately identified.

[0065] Specifically, as a preferred embodiment, in the above-mentioned method for providing companionship to the elderly, step S130: inputting the emotional feature recognition vector into the emotional recognition deep learning model, and using the emotional recognition deep learning model to identify the current emotional state of the elderly user based on the emotional state attention mechanism, specifically includes: S131: A deep learning model for emotion recognition is constructed using a bidirectional LSTM network combined with an emotion state attention mechanism.

[0066] S132: Input the emotion feature recognition vector into the emotion recognition deep learning model for feature learning, and combine the emotion state attention mechanism to capture the emotion dependency relationship in the emotion feature recognition vector.

[0067] S133: Based on the emotional dependency relationship, the current emotional state is output through multiple emotional classification attention heads.

[0068] The technical solution provided in this application embodiment includes an emotion recognition deep learning model comprising the aforementioned bidirectional LSTM network and an emotion state attention mechanism. After inputting the emotion feature recognition vector into the emotion recognition deep learning model, the model uses the left and right subnetworks of the bidirectional LSTM network to capture the forward and backward temporal features of the emotion feature recognition vector through forward and backward propagation, respectively. The forward and backward temporal features are fused according to temporal relationships to obtain fused dependency temporal information. Then, the emotion state attention mechanism is used to capture the emotional dependencies in the emotion feature recognition vector from the forward and backward temporal information. Finally, the emotion feature recognition vector is classified according to the aforementioned multiple emotion classification attention heads to output the current emotion state. Specifically, the bidirectional LSTM model combined with the emotion state attention mechanism takes a sequence of emotion feature recognition vectors as input, processes it using the emotion state attention mechanism to obtain the current emotion state, and finally, the predicted label of the current emotion state is obtained through classification by multiple emotion classification attention heads in the softmax layer.

[0069] Figure 1 The elderly companionship method based on multimodal emotion recognition provided in the illustrated embodiment, after step S130: inputting the emotion feature recognition vector into the emotion recognition deep learning model, and using the emotion recognition deep learning model to identify the current emotional state of the elderly user according to the emotion state attention mechanism, further includes: S140: Sample behavioral log modal data, construct a dynamic spatiotemporal graph, combine emotional feature recognition vectors and dynamic spatiotemporal graph to perform root cause analysis on the current emotional state, and obtain the root cause analysis results.

[0070] The technical solution provided in this application collects behavioral log modal data and statistically analyzes users' historical behavior, thereby constructing a dynamic spatiotemporal graph that reflects the user's activity time and location. By combining the aforementioned emotional feature recognition vector and the dynamic spatiotemporal graph, a root cause analysis of the current emotional state can be performed, thus obtaining the aforementioned root cause analysis results.

[0071] Specifically, as a preferred embodiment, in the above-mentioned method for providing companionship to the elderly, step S140: sampling behavioral log modal data, constructing a dynamic spatiotemporal graph, and combining emotional feature recognition vectors and the dynamic spatiotemporal graph to perform root cause analysis on the current emotional state to obtain the root cause analysis results, includes: S141: The sampling rate is dynamically adjusted according to the time decay factor, and the behavioral log modal data is sampled using the dynamically adjusted sampling rate to obtain multiple text feature sequences. The time decay factor is used to adjust the sampling rate because the further away from the time corresponding to the sentiment feature recognition vector, the lower the correlation with the current sentiment state. Therefore, this application designs a time decay factor to dynamically adjust the sampling rate, and samples the above-mentioned behavioral log modal data according to this sampling rate.

[0072] S142: Map multiple text feature sequences to a semantic vector space, extract the semantic vector corresponding to each text feature sequence, and extract the semantic association and text type of the multiple text feature sequences based on the semantic vectors and contextual positions. By mapping the multiple text feature sequences to the semantic vector space, extracting the semantic vector corresponding to each text feature sequence, and combining it with the contextual position, the semantic association and text type related to the current emotional state can be obtained. This allows for the specific determination of user activities, physical condition, and social behavior related to the current emotional state of elderly users.

[0073] S143: Combining temporal order, semantic vectors, semantic associations, and text types, a dynamic spatiotemporal graph corresponding to multiple text feature sequences is constructed. This dynamic spatiotemporal graph can reflect the correlation between different user behaviors over time, as well as the correlation between different user behaviors at the spatial level and in terms of behavioral logic.

[0074] S144: Align the sentiment feature recognition vector with the text feature sequence of the dynamic spatiotemporal graph in time. This embodiment of the application, by aligning the aforementioned sentiment feature recognition vector with the text feature sequence of the dynamic spatiotemporal graph in time, can specifically determine user behaviors that are strongly correlated with the sentiment feature recognition vector in time.

[0075] S145: Calculate the contribution rate of each emotion feature recognition vector to the current emotional state and match it with time-aligned text feature sequences. Because there are multiple collected emotion feature recognition vectors, this embodiment of the application can determine one or more text feature sequences that are strongly correlated with the current emotional state in time and have the largest contribution rate by calculating the contribution rate of each emotion feature recognition vector to the current emotional state and matching them with time-aligned text feature sequences, thereby determining the specific reasons affecting the current emotional state of elderly users.

[0076] S146: Using a dynamic path weighting algorithm combined with semantic association, calculate and locate text feature sequences with emotional contribution in the dynamic spatiotemporal graph. By combining the dynamic path weighting algorithm with semantic association, it is possible to identify text feature sequences with emotional contribution and logical association that are aligned with the time in the aforementioned dynamic spatiotemporal graph, thereby further identifying user behaviors that contribute to the current emotional state.

[0077] S147: Using text feature sequences and sentiment contribution, the root cause analysis results are obtained from the root cause database. This root cause database includes the correspondence between sentiment states and root causes, as well as information such as the user behavior corresponding to the root cause and its sentiment contribution to the sentiment state. The root cause analysis results include the root causes affecting the current sentiment state and the probability of the root causes influencing the behavior.

[0078] The technical solution provided in this application, by mapping the multiple text feature sequences to a semantic vector space, extracts the meaning vector corresponding to each text feature sequence, and combines it with the context position, can obtain the semantic association and text type related to the current emotional state, thereby specifically determining information such as user activities, physical condition, and social behavior associated with the current emotional state of elderly users; then, by aligning the above-mentioned emotional feature recognition vectors and text feature sequences in time with the dynamic spatiotemporal graph, it is possible to specifically determine user behaviors that are strongly correlated with the emotional feature recognition vectors in time, thereby determining one or more text feature sequences that are strongly correlated with the current emotional state in time and have the largest contribution rate; then, by using a dynamic path weight algorithm combined with semantic association, the above-mentioned dynamic spatiotemporal graph is used to determine text feature sequences that have emotional contribution and logical association with the above-mentioned time-aligned text feature sequences, thereby further determining user behaviors that contribute to the current emotional state, and thus accurately matching the causes affecting the current emotional state from the root cause database.

[0079] Figure 1 The technical solution provided in the illustrated embodiment, in step S140: sampling behavioral log modal data, constructing a dynamic spatiotemporal graph, combining the emotional feature recognition vector and the dynamic spatiotemporal graph to perform root cause analysis on the current emotional state, and obtaining the root cause analysis results, further includes: S150: Based on the current emotional state and root cause analysis results, push companionship and interaction services and / or medical care assistance warning suggestions to elderly users.

[0080] The technical solution provided in this application, after obtaining the current emotional state and root cause analysis results, can push personalized, age-appropriate emotional companionship service content and / or medical care assistance warning suggestions to elderly users. Specifically, based on different emotional states or emotional communication needs of the elderly, a corresponding companionship service content resource library is created and classified and sorted hierarchically. Companionship service content suitable for the needs of elderly people with different emotional types is pushed to them. The emotional recognition results are dynamically matched with the personalized service strategy library to achieve a closed loop from emotional perception to accurate response.

[0081] Specifically, as a preferred embodiment, in the above-described method for providing companionship to the elderly, step S150: the step of pushing companionship interaction services and / or medical care assistance warning suggestions to the elderly user based on the current emotional state and root cause analysis results, includes: S151: Design a virtual companion assistant to deliver emotional companionship services tailored to elderly users' current emotional state and root cause analysis results via multimedia. These emotional companionship services include: S152: Engages in natural language conversations with elderly users via voice and text, and adjusts the dialogue style and content of the virtual companion assistant based on the user's current emotional state in real time.

[0082] S153: Based on the current emotional state, match the elderly user's profile with interest-based recommendation cases from the profile's interest library, and use multimedia methods combined with these cases to push interest-based recommended content and mood-regulating content to the elderly user. The elderly user's profile includes basic information such as name, age, medical history, hobbies, contraindications, frequently contacted persons, and cognitive ability assessments, as well as behavioral log modal data.

[0083] S154: Based on the elderly user's current emotional state and their common entertainment habits, the system pushes interactive entertainment activities and interpersonal connection methods to the elderly user. The pushed interactive entertainment activities include light games, such as simple memory matching, board games (chess, checkers), or brain teasers. Digital memoirs, such as guiding the elderly to upload videos, old photos, or tell stories to create simple electronic photo albums or story collections. Virtual pets / plants, such as providing low-maintenance virtual life for care and interaction. Simple fitness / activity guidance activities, such as animated / video guidance for seated stretching exercises or finger exercises. Interpersonal connection methods mainly involve family members connecting and participating; these include authorized family video calls: the system can initiate or accept video call requests from relatives and friends (requiring the elderly user's consent / nurse's assistance). Emotional status briefings: family members can view an overall emotional status briefing for the elderly (with authorization) through the app (not detailed data, such as "today's mood is stable, with more interaction"). Messages and care delivery: family members can send text, voice, or video messages, which the system will play for the elderly at an appropriate time. Digital Memory Sharing: Family members can upload photos and videos to the elderly person's digital memory.

[0084] This application embodiment designs a virtual companion assistant (which can be software or a physical structure) to achieve intelligent companionship and interaction. Specific intelligent companionship and interaction methods are as follows: Natural language dialogue: Supports voice and text chat, enabling daily greetings, casual conversation, guided recollections (nostalgia therapy), and answering simple questions (such as time and weather). Emotional response: Adjusts the tone, content, and expression (virtual avatar) based on the identified emotions of the elderly person. For example, offering comfort when the elderly person is sad and sharing joy when they are happy. Personalized settings: The virtual assistant's image, name, and voice can be set. Furthermore, the virtual companion assistant provided in this application embodiment can adjust its dialogue method and content according to the real-time current emotional state. The technical solution provided in this application embodiment can also match the current emotional state with the elderly user's user profile and interest-based recommendation cases from the interest database, and push interest-recommended content and emotion-regulating content to the elderly user through multimedia methods combined with these interest-based recommendation cases. The specific multimedia content pushed includes: based on identified emotions, personal profiles (interests, cultural background, and cognitive abilities, etc.), and interaction history, music (soothing, nostalgic, or inspiring), opera, crosstalk, audiobooks, clips from old movies / documentaries, scenic videos, and photo collections, etc. Additionally, the emotion-regulating content pushed to elderly users includes: proactively pushing content targeting specific emotions (such as relaxation-guided meditation / breathing exercises for anxiety, and heartwarming stories for sadness, etc.).

[0085] Furthermore, this application can also push medical and nursing assistance warning suggestions to elderly users based on their current emotional state and root cause analysis results. Specifically, as a preferred embodiment, in the above-mentioned elderly companionship implementation method, step S150: pushing companionship interaction services and / or medical and nursing assistance warning suggestions to elderly users based on their current emotional state and root cause analysis results includes: S155: generating and pushing an emotional state dashboard of the elderly user to relevant medical and nursing personnel according to their current emotional state, so as to provide medical and nursing personnel with emotional state warnings and care for the elderly user.

[0086] S156: Real-time monitoring of the state fluctuations and duration corresponding to the current emotional state.

[0087] S157: When the amplitude of state fluctuation is greater than or equal to a predetermined amplitude threshold, and any one of the visual modal data, auditory modal data, and physiological modal data triggers an abnormal warning signal, an abnormal alarm signal and the location status information of the elderly user are pushed to medical staff. The predetermined amplitude threshold is the threshold corresponding to the amplitude of emotional state fluctuation, and this predetermined amplitude threshold can be set to a specific value through actual measurement. Relevant indicators reflecting state fluctuation include physiological indicators such as heart rate, blood pressure, and skin conductance signals. This application can set corresponding amplitude thresholds by actually measuring the fluctuation amplitude of the above-mentioned physiological indicators corresponding to different emotions.

[0088] S158: When the duration of the current emotional state within the abnormal state range is greater than or equal to a predetermined time threshold, the current emotional state and root cause analysis results are matched against warning cases in the warning case library. The predetermined time threshold is the threshold corresponding to the duration of the emotional state within the abnormal state range. This predetermined time threshold can be set to a specific value through actual measurement, such as 10 minutes, 12 minutes, or 15 minutes.

[0089] S159: Based on the expert experience decision tree constructed using fuzzy logic and combined with early warning cases, risk suggestion reports and abnormal early warning signals are generated and pushed to medical staff.

[0090] Specifically, the emotional state dashboard includes: providing medical staff with an overview of the emotional state of the elderly at the ward / department level (heat map and / or emotion distribution map). Detailed emotional trend charts for individual elderly individuals (by hour / day / week). The abnormal warning signals here include: (1) Real-time abnormal warning: When strong negative emotions (such as violent crying, extreme anger and / or suicidal tendencies), suspected pain expressions, keyword triggers (such as "help" or "I don't want to live anymore"), abnormal sounds (such as continuous groaning or falling sounds) are detected, a pop-up window, sound alarm or SMS notification is immediately sent to the nurse station / duty mobile phone, including the elderly's location and preliminary judgment. (2) Potential risk warning: Based on continuous monitoring, patterns that may indicate depression, cognitive decline, or physical discomfort, such as persistent low mood, social withdrawal, and significant reduction in activity, are identified, and risk reports are generated for medical staff to refer to.

[0091] As an example, the emotional distribution map provided by the aforementioned emotional state dashboard is used to monitor the current emotional state of elderly users in real time. If the emotional distribution map reflects a current emotional state whose short-term fluctuation amplitude is greater than or equal to a predetermined amplitude threshold, and whose duration is greater than or equal to a predetermined time threshold (e.g., 1 minute), the current emotional state and root cause analysis results are used to match early warning cases in the early warning case library. An expert experience decision tree is used to search for similar cases from the early warning cases to determine if the elderly user's mental state is in a severely negative emotional state, such as suicidal tendencies, and if keywords triggering strong negative situations (e.g., "I don't want to live anymore") are present. At this point, a risk suggestion report and an abnormal early warning signal containing the elderly user's location are generated and promptly pushed to the medical staff at the nurses' station, who then provide the elderly user with assistance services as quickly as possible.

[0092] The technical solution provided in this application provides a platform for interaction and sharing among the elderly. This platform allows elderly individuals with matched emotional types and mutual support to engage in interactive activities such as communication, video sharing, and photo sharing, fully leveraging the positive effects of mutual companionship and peer support. Furthermore, based on the emotional needs of elderly people living alone, the platform categorizes their emotional states and designs companionship service strategies for different emotional states, achieving personalized and targeted emotional companionship services to accurately meet their psychological needs. It utilizes social incentive service strategies such as automatically accessing relatives and friends for video calls, and dynamically identifies and updates the emotional states of the elderly through continuous multimodal emotional data collection and analysis. Additionally, as a preferred embodiment, such as... Figure 3 As shown, the above-mentioned method for providing companionship to the elderly, after step S150: pushing companionship interaction services and / or medical assistance warning suggestions to the elderly user based on the current emotional state and root cause analysis results, further includes: S210: Combine the companionship and interaction services selected by the elderly user within the predetermined time period with the user profile to generate a user profile for the elderly user. Companionship and interaction services can be obtained from behavioral log modal data, such as video calls with family members, chess games, or chats with others. The user profile here can include information such as the elderly user's physical health status, behavioral habits, and relatives and friends.

[0093] S220: Extract user profile feature vectors from user profiles and cross-fuse them with sentiment feature recognition vectors to obtain user sentiment feature vectors. Specifically, the cross-entropy function can be used to fuse the aforementioned sentiment feature recognition vectors and user profile feature vectors to obtain the user sentiment feature vectors.

[0094] S230: The fused user sentiment feature vector is input into the long-term risk prediction model. Through forward and backward propagation, the model captures the positive and negative temporal sentiment features of the user sentiment feature vector. This long-term risk prediction model can use a bidirectional LSTM model with a hybrid attention mechanism to capture the positive and negative temporal sentiment features of the user sentiment feature vector.

[0095] S240: Fuse positive and negative temporal sentiment features according to temporal relationships to obtain fused temporal dependencies, and extract risk factor features from user sentiment feature vectors based on temporal dependencies.

[0096] S250: Classifies and predicts risk factor characteristics, outputting risk prediction labels. Based on the above risk factor characteristics, it can combine emotion recognition results and the elderly's health records to pop up personalized nursing suggestions on the nurse's workstation interface (e.g., "Grandma Wang is in low spirits, it is recommended to arrange a video call with her family or a 10-minute nostalgic chat," "Grandpa Li seems to be in pain, it is recommended to have a physical examination").

[0097] The technical solution provided in this application combines companionship and interaction services for elderly users with user profiles to generate user portraits of elderly users. User portrait feature vectors are extracted from these portraits and cross-fused with the aforementioned emotional feature recognition vectors to obtain user emotional feature vectors. These vectors reflect information such as user behavior habits, social relationships, and long-term physical condition. These emotional feature vectors are then input into a long-term risk prediction model to capture both positive and negative temporal emotional features. Furthermore, the captured positive and negative temporal emotional features are used to obtain a fused temporal dependency relationship. Based on this temporal relationship, risk factor features are extracted. These risk factor features include negative psychological states (such as depression, anxiety, and irritability) and sub-health conditions (such as chronic diseases and disease progression trends).

[0098] Furthermore, this application embodiment can also monitor and maintain the emotional state of elderly users at all times. Specifically, to promote the health of elderly users, it can help them maintain a state of "flow." When in a state of flow, a person exhibits the following characteristics: (1) Complete immersion: Focusing wholeheartedly on what you are doing and forgetting yourself.

[0099] (2) High concentration: Attention is highly focused, and external distractions (such as sound and time) seem to disappear.

[0100] (3) Integration of action and consciousness: You no longer need to think about "what to do", and your actions and behaviors are natural and smooth, as if they are instinctive.

[0101] (4) Sense of control: Feeling that you have full control over the current activity and are not worried about failure.

[0102] (5) Distorted sense of time: feeling that time passes very quickly ("a few hours pass in the blink of an eye") or very slowly (being able to clearly perceive every detail).

[0103] (6) Intrinsic motivation: The activity itself is the reward; doing it makes you feel happy and satisfied, rather than for external rewards.

[0104] This application is able to design multidimensional evaluation criteria based on the features corresponding to the above-mentioned flow state, and combine the above-mentioned multimodal emotional data to determine whether a user is in a flow state.

[0105] Specifically, as a preferred embodiment, such as Figure 4 As shown, the above-mentioned method for providing companionship to the elderly, after step S110: collecting multimodal emotional data from elderly users, further includes: S310: Multimodal emotional data is fused according to multidimensional evaluation criteria to obtain a multidimensional flow state vector. The multidimensional evaluation criteria are designed based on flow-related characteristics, such as focus, amplitude of behavioral movements, blood oxygen levels, and whether the current health status is pleasant and satisfying. These evaluation criteria can combine the collected multimodal emotional data to output a multidimensional flow state vector. Here, corresponding weights can be designed for each multidimensional evaluation criterion, and the weights are combined with the multimodal emotional data corresponding to each criterion to obtain the aforementioned multidimensional flow state vector.

[0106] S320: Input the multidimensional flow state vector into a pre-built flow state assessment model. Use the flow state assessment model, combined with the elderly user's current user behavior, to predict the location and trend of the elderly user's flow channels. Specific flow channel locations include: Anxiety zone: Challenge far exceeds skill level. Arousal zone: Challenge slightly exceeds skill level, with learning potential. Flow zone: Challenge and skill level perfectly matched. The system objective of this embodiment is to maintain the state of the flow zone as much as possible. Control zone: Skill level exceeds challenge level, possibly feeling relaxed but bored. Boredom zone: Skill level far exceeds challenge level.

[0107] S330: Based on the loss function of the flow state assessment model, combined with the location and trend of the flow channel, maintain or adjust the flow state of elderly users; wherein, maintaining or adjusting the flow state of elderly users includes: changing the difficulty of the current user behavior, decomposing or adding behavioral goals and / or providing behavioral reference suggestions; wherein, the loss function is designed based on the target flow state and the location and trend of the flow channel.

[0108] "Flow" describes an extreme psychological state of complete immersion in an activity. Entering and maintaining a flow state requires a balance between challenge and skill level—this is the core principle. Flow occurs when the challenge of the activity matches the individual's skill level, and both are at a high level. Specifically: high challenge, low skill → leads to anxiety; low challenge, high skill → leads to boredom. Flow is most likely to occur only when both challenge and skill are elevated to a certain level and balanced. This is like a continuously upward-extending channel.

[0109] This application uses a flow state assessment model (e.g., a deep learning fusion network) to fuse heterogeneous data from different modalities, generating a unified and quantifiable "flow index." This index is not a simple numerical value, but a multi-dimensional vector that includes: focus level, challenge-skill balance, intrinsic motivation level, and emotional valence (positive / negative). Then, based on this "flow index" and historical data, the user's current flow state is determined.

[0110] In addition, the embodiments of this application can also predict the user's flow channel trend, that is, predict which area the flow channel will enter (for example: after skill improvement, the current task may soon become boring).

[0111] In the technical solutions provided in this application, if the user tends towards "anxiety," the strategy adopted is not direct comfort, but rather implicit prompts: providing minimalist prompts or the first step of breaking down the goal at the edge of the interface. Dynamically reducing difficulty: subtly lowering a parameter of the task without being noticed. Providing "scaffolding": offering a small tool or reference case. If the user tends towards "boredom," the strategy adopted includes dynamically increasing the challenge and offering exploratory suggestions; dynamically increasing adjustments includes introducing new goals, time limits, or more complex elements. Offering exploratory suggestions includes: "Want to try another approach?" The core principle: all adjustments should be gradual and contextualized, as if everything happens naturally, rather than being imposed by the system.

[0112] The technical solution provided in this application, by fusing the aforementioned multimodal emotional data according to multidimensional evaluation criteria, can obtain a multidimensional flow state vector. This flow state vector is then input into a pre-constructed flow state assessment model. This model, combined with the elderly user's current behavior, predicts the location and trend of the elderly user's flow channels. Finally, based on the loss function of the flow state assessment model, and considering the location and trend of the flow channels, the elderly user's flow state is maintained or adjusted. This promotes the elderly user's entry into and maintenance of a flow state, enabling them to release pleasure-related neurotransmitters. This not only enhances their focus and creativity but also brings a strong sense of pleasure, thereby strengthening the flow experience. For example, when an elderly user is playing chess with a virtual companion assistant, adjusting the difficulty of the game or offering exploratory suggestions can help the elderly user enter and maintain a flow state.

[0113] In summary, the elderly companionship method based on multimodal emotion recognition provided in this application first collects multimodal emotion data of elderly users, including visual modality data, auditory modality data, physiological modality data, and behavioral log modality data. This allows for the acquisition of emotional states from different modalities as much as possible, more accurately reflecting the emotional state of elderly users. Then, using the multimodal analysis model and cross-modal distillation pipeline provided by the multimodal fusion framework, feature extraction and cross-modal feature fusion are performed on the aforementioned multimodal emotion data. This yields a comprehensive emotional feature recognition vector for the elderly user, which accurately reflects the emotional characteristics of the elderly user. This emotional feature recognition vector is then input into an emotion recognition deep learning model. Subsequently, using the emotion recognition deep learning model based on the emotion state attention mechanism, the current emotional state of elderly users can be accurately identified. Then, a dynamic spatiotemporal graph is constructed by sampling behavioral log modal data. Combining this dynamic spatiotemporal graph with the emotion feature recognition vector, a root cause analysis of the current emotional state is performed, yielding root cause analysis results and accurately determining the causes affecting the elderly user's current emotional state. Finally, based on the current emotional state and root cause analysis results, companionship and interaction services and / or medical assistance warning suggestions are pushed to the elderly user. This provides personalized companionship services for elderly users with different emotional types and states, effectively solving the emotional companionship problem for the elderly and effectively improving their mental health and subjective well-being. This approach addresses the problem in existing elderly companionship solutions based on multimodal feature extraction and fusion methods where the feature vector after multimodal feature fusion is disconnected from the user's emotional state, making it difficult to deeply analyze the causes of changes in the user's emotional state and provide accurate care services and companionship strategies. Furthermore, the above embodiments of this application can iteratively reserve companionship service content resources, including dynamically adjusting service strategies based on multimodal data feedback and elderly behavior analysis and evaluation. Based on factors such as seniors' self-evaluation, service usage time, interaction interruption rate, and special needs during holidays, the service strategy is optimized, and the resource library of companionship services is dynamically adjusted and improved. Through continuous iteration, the system can evolve from "ordinary companionship" to "an elderly partner who understands you."

[0114] In addition, it should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0115] Additionally, see Figure 5 , Figure 5 This is a schematic diagram of a device for providing companionship to the elderly based on multimodal emotion recognition, provided as an embodiment of this application. Figure 5 As shown, the elderly companionship device includes: The system consists of a perception layer 100, an edge / local server 200, a cloud server 300, and an application layer 400; among which, The perception layer 100 includes a camera, a microphone array, sensors, and a terminal interaction device; The edge / local server 200 includes a data receiving module, a multimodal feature extraction module, an emotion recognition engine, a decision and response generation module, a local database, and an early warning triggering module. Specific functions are as described in the above method embodiments and will not be repeated here.

[0116] The following management procedures are performed on cloud server 300: Elderly records management: creating and maintaining information about the elderly (name, age, medical history, hobbies, contraindications, frequently contacted persons, and cognitive ability assessment).

[0117] Centralized configuration management: Manage the status and configuration of cameras, microphones, sensors, tablets / robot terminals in the ward.

[0118] Access control: Hierarchical access settings (administrators, doctors, nurses, caregivers, and family members).

[0119] Content Management: Uploading, reviewing, and categorizing multimedia content libraries.

[0120] Model training and update: Provide an interface to update the sentiment recognition model (strict data privacy protection is required).

[0121] Big data analytics and logs and auditing: Record all system operations, alerts, and interaction history.

[0122] The corresponding software parameters include: 1. Emotion recognition performance: Accuracy: >85% (tested and validated on a specific dataset of elderly populations).

[0123] Modal fusion strategies: weighted fusion / decision-level fusion / attention-based feature fusion.

[0124] Recognition latency: <3 seconds (from data acquisition to output result).

[0125] Supported emotional labels: basic emotions (happiness, sadness, anger, surprise, fear, disgust, neutral) + complex / medical-related states (calm, anxiety, depression, pain, confusion, pleasure).

[0126] 2. Data Processing and Privacy: Data anonymization: All audio and video data undergoes feature extraction on edge devices or local servers, and the original data is not stored for long periods or is only encrypted and stored for a very short time.

[0127] Storage strategy: The identified structured data (emotion tags, behavior logs, metadata) is encrypted and stored; the raw audio and video data is cached shortly as needed or processed and discarded in real time.

[0128] Compliance: Strictly comply with medical data privacy regulations such as HIPAA, GDPR, and China's Personal Information Protection Law.

[0129] Edge computing capabilities: Supports real-time processing on terminal devices or local servers, reducing network dependence and the risk of privacy leaks.

[0130] 3. Hardware Interface and Compatibility: Camera: Supports mainstream RTSP / IP cameras; 1080P+ resolution recommended; supports infrared night vision.

[0131] Microphone: Supports array microphones with noise reduction and directional sound pickup capabilities.

[0132] Physiological sensor interface: Supports Bluetooth / BLE protocol access to common medical / consumer wristband and mattress sensors (such as heart rate, heart rate variability HRV and / or skin conductance).

[0133] Terminal devices: Compatible with Android / iOS tablets; supports specific companion robot hardware platforms.

[0134] Network: Supports hospital intranet deployment; requires stable bandwidth to ensure audio and video streams.

[0135] 4. Interaction performance: Automatic Speech Recognition (ASR): Chinese Mandarin recognition accuracy >95% (quiet environment); supports certain dialects; response time <2 seconds.

[0136] Text-to-Speech (TTS): Naturalness MOS > 4.0; offers a variety of gentle and friendly elderly voice options.

[0137] Virtual assistant response: Conversation response time < 1.5 seconds.

[0138] 5. Reliability: System availability: >99.5% (annual).

[0139] Fault recovery: Key modules support redundancy; data backup mechanism; rapid failover.

[0140] False positive rate: <5% (model and rules need continuous optimization).

[0141] 6. Deployment and Maintenance: Deployment method: Local deployment is supported (recommended to ensure data security).

[0142] Update mechanism: Supports remote secure updates (models, software).

[0143] Log monitoring: Provides system operation status monitoring and alarm functions.

[0144] In summary, the technical solution provided in this application can realize the delivery of personalized and age-appropriate emotional companionship services. Specifically, based on different emotional states or emotional communication needs of the elderly, a corresponding companionship service content resource library is created and classified and sorted in a gradient. Companionship service content suitable for the needs of elderly people with different emotional types is pushed to them. The emotion recognition results are dynamically matched with the personalized service strategy library to achieve a closed loop from emotion perception to accurate response.

[0145] In addition, see Figure 6 , Figure 6 The provided electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the elderly companionship implementation method based on multimodal emotion recognition of any of the above embodiments.

[0146] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments can include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), or in-vehicle terminals, as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0147] like Figure 6As shown, the electronic device can include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 1002 or a program loaded from a storage device 1003 into a random access memory RAM 1004. The RAM 1004 also stores various programs and data required for the operation of the aforementioned electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 can operate a defect autonomous decision-making device for industrial inspection and exchange data with other devices wirelessly or via wired communication. Although the diagram shows a model building device with various systems, it should be understood that it is not required to implement or have all of the systems shown. It is possible to implement or have more or fewer systems alternatively.

[0148] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram can represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0151] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0152] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for providing companionship to the elderly based on multimodal emotion recognition, characterized in that, include: Collect multimodal emotional data from elderly users; A multimodal fusion framework is used to extract multimodal features and fuse cross-modal features from the multimodal emotional data to obtain the emotional feature recognition vector of the elderly user. The emotional feature recognition vector is input into the emotional recognition deep learning model, and the current emotional state of the elderly user is identified according to the emotional state attention mechanism. A dynamic spatiotemporal graph corresponding to the multimodal emotion data is constructed, and the root cause analysis of the current emotion state is performed by combining the emotion feature recognition vector and the dynamic spatiotemporal graph to obtain the root cause analysis results; Based on the current emotional state and the root cause analysis results, companionship and interaction services and / or medical assistance and early warning suggestions are pushed to the elderly user.

2. The method as described in claim 1, characterized in that, The multimodal emotion data includes visual modal data, auditory modal data, physiological modal data, and behavioral log modal data; the step of using a multimodal fusion framework to extract multimodal features and fuse cross-modal features from the multimodal emotion data to obtain the emotion feature recognition vector of the elderly user includes: Using the improved MViT v2 model provided by the multimodal fusion framework, visual feature vectors corresponding to the visual modal data are identified; Using the ECAPA model provided by the multimodal fusion framework, extract speech feature vectors corresponding to the auditory modality data; Using the large text processing model provided by the multimodal fusion framework, the physiological modality data and behavioral log modality data are mapped to the text semantic vector space respectively to obtain text feature vectors; The joint distribution of the visual feature vector, speech feature vector, and text feature vector is learned using a cross-modal distillation pipeline, and then fused to obtain an emotion feature recognition vector that includes emotion dependencies.

3. The method as described in claim 2, characterized in that, The step of identifying the visual feature vectors corresponding to the visual modality data using the improved MViT v2 model provided by the multimodal fusion framework includes: The visual modality data is input into the improved MViT v2 model, and the behavioral characteristics in the continuous multi-frame image sequence are identified according to the built-in time component generation mechanism to obtain the time component corresponding to the behavioral characteristics. Using the fixed stride pooling strategy built into the improved MViT v2 model, a fixed stride pooling operation is performed on the continuous multi-frame image sequence in the time dimension to obtain motion cues of human behavior. The continuous multi-frame image sequence is processed according to the time component and motion cues to output the behavioral feature vector corresponding to the visual modality data; Multiple classification attention heads are used to classify and predict the behavior category label corresponding to the behavior feature vector.

4. The method as described in claim 2, characterized in that, The step of extracting the speech feature vectors corresponding to the auditory modality data using the ECAPA model provided by the multimodal fusion framework includes: A time-delay neural network is used to perform feature detection on the feature spectrogram corresponding to the auditory modality data in the time dimension, and the detected speech convolutional features are input into the speech emotion channel and the speech content channel respectively. In the speech emotion channel, a multi-scale feature fusion mechanism is used to fuse the speech convolutional features to obtain frame-level features containing emotional states; in the speech content channel, an emotion keyword matching mechanism is used to extract emotion keywords from the speech convolutional features to obtain frame-level features containing emotion content. Using an attention-based statistical pooling mechanism that integrates emotion weights and environmental anomaly weights, the weighted mean and weighted standard deviation of the two types of frame-level features are calculated and concatenated to obtain a fixed-dimensional speech feature vector. By using a nonlinear transformation method, the identity vector of the elderly user is embedded into the speech feature vector, and the speech feature vector after embedding the identity vector is output.

5. The method as described in claim 2, characterized in that, The step of learning the joint distribution of the visual feature vector, speech feature vector, and text feature vector using a cross-modal distillation pipeline, and fusing them to obtain an emotion feature recognition vector containing emotion dependencies, includes: Align and extract the emotional feature distributions corresponding to the visual feature vectors, speech feature vectors and text feature vectors of the same temporal sequence to obtain the emotional state weights; Using a cross-modal distillation pipeline, adversarial learning is performed on the joint distribution of the visual feature vectors, speech feature vectors, and text feature vectors based on the feature similarity between different feature vectors; wherein, If the feature similarity is greater than or equal to a predetermined similarity threshold, the joint distribution of the visual feature vector, speech feature vector, and text feature vector is learned adversarially according to the alignment loss function to obtain the emotion feature recognition vector. If the feature similarity is less than a predetermined similarity threshold, a cross-attention fusion mechanism is used to combine the emotional state weights and fuse the linear layer projections corresponding to the visual feature vector, speech feature vector, and text feature vector respectively to obtain the emotional feature recognition vector containing emotional dependencies.

6. The method as described in claim 1, characterized in that, The steps of constructing the dynamic spatiotemporal graph corresponding to the multimodal sentiment data, and combining the sentiment feature recognition vector and the dynamic spatiotemporal graph to perform root cause analysis on the current sentiment state to obtain the root cause analysis results include: The sampling rate is dynamically adjusted based on the time decay factor. The dynamically adjusted sampling rate is used to sample behavioral log modal data to obtain multiple text feature sequences. The multiple text feature sequences are mapped to the semantic vector space to obtain the semantic vector corresponding to each text feature sequence. Based on the semantic vectors and context positions of the multiple text feature sequences, the semantic associations and text types of the multiple text feature sequences are extracted. In chronological order, using the semantic vectors, semantic associations, and text types, a dynamic spatiotemporal map corresponding to the multiple text feature sequences is constructed. Align the emotion feature recognition vector with the text feature sequence of the dynamic spatiotemporal graph in time; Calculate the contribution rate of each emotion feature recognition vector to the current emotion state, and match the time-aligned text feature sequence; Based on the dynamic path weight algorithm and the semantic association, text feature sequences with emotional contribution in the dynamic spatiotemporal graph are calculated and searched. Using the text feature sequence and the sentiment contribution, the root cause analysis results in the root cause database are obtained through matching.

7. The method as described in claim 1, characterized in that, The step of pushing companionship and interaction services and / or medical assistance warning suggestions to the elderly user based on the current emotional state and the root cause analysis results includes: Design a virtual companion assistant to push emotional companionship services to the elderly user through multimedia methods, tailored to their current emotional state and the root cause analysis results; wherein, the emotional companionship services include: The virtual companion assistant engages in natural language dialogue with the elderly user via voice and text, and adjusts the dialogue method and content based on the user's current emotional state in real time. Based on the current emotional state and the interest-based push cases in the user profile interest library of the elderly user, interest-recommended content and emotion-regulating content are pushed to the elderly user through multimedia methods in combination with the interest-based push cases. Based on the current emotional state, match the elderly user's usual entertainment methods, and push interactive entertainment activities and interpersonal contact methods to the elderly user; Alternatively, based on the current emotional state, an emotional state dashboard for the elderly user can be generated and pushed to relevant medical staff to provide them with early warnings and care regarding the elderly user's emotional state. Real-time monitoring of the state fluctuations and duration corresponding to the current emotional state; When the fluctuation amplitude is greater than or equal to a predetermined amplitude threshold, and the emotional data of any modality in the multimodal emotional data triggers an abnormal warning signal, an abnormal alarm signal and the location status information of the elderly user are pushed to the medical staff. When the duration of the current emotional state within the abnormal state range is greater than or equal to a predetermined time threshold, the current emotional state and the root cause analysis results are used to match early warning cases in the early warning case library. Based on the expert experience decision tree and the aforementioned early warning cases, a risk suggestion report and anomaly warning signals are generated and pushed to the medical staff.

8. The method as described in claim 1, characterized in that, Following the step of pushing companionship and interaction services and / or medical assistance warning suggestions to the elderly user based on the current emotional state and the root cause analysis results, the method further includes: By combining the companionship and interaction services selected by the elderly users during the predetermined time period with their user profiles, a user profile of the elderly users is generated. The user profile feature vector is extracted from the user profile and cross-fused with the emotion feature recognition vector to obtain the user emotion feature vector; The fused user sentiment feature vector is input into the long-term risk prediction model, and the positive and negative time-series sentiment features of the user sentiment feature vector are captured through forward and backward propagation. The positive and negative temporal sentiment features are fused according to the temporal relationship to obtain the fused temporal dependency relationship. Risk factor features are then extracted from the user sentiment feature vector based on the temporal dependency relationship. The risk factor characteristics are classified and predicted, and risk prediction labels are output.

9. The method as described in claim 1, characterized in that, Following the step of collecting multimodal emotional data from elderly users, the following steps are also included: By fusing the multimodal emotion data according to the multidimensional evaluation criteria, a multidimensional flow state vector is obtained; The multidimensional flow state vector is input into a pre-built flow state assessment model. The flow state assessment model is then used in conjunction with the elderly user’s current user behavior to predict the location and trend of the elderly user’s flow channel. Based on the loss function of the aforementioned flow state assessment model, and considering the location and trend of the flow channels, the flow state of the elderly user is maintained or adjusted; wherein... Maintaining or adjusting the flow state of the elderly user includes: changing the difficulty of the current user behavior, decomposing or adding behavioral goals and / or providing behavioral reference suggestions; the loss function of the flow state assessment model is designed based on the target flow state and the location and trend of the flow channel.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the elderly companionship method based on multimodal emotion recognition as described in any one of claims 1 to 9.

Images