Elderly emotion bidirectional interaction method based on artificial intelligence technology

Abstract

The invention discloses an elderly emotion bidirectional interaction method based on an artificial intelligence technology, and relates to the technical field of elderly emotion interaction, and the method comprises the steps: 1, collecting the audio data and video data of the elderly; 2, respectively extracting audio feature data and visual feature data, and converting the audio data into audio transcription text data; 3, inputting the audio feature data, the visual feature data and the audio transcription text data into an emotion calculation model to obtain a context query version, a user emotion vector and a similarities vector; 4, inputting the features into a core chat model to obtain a reply table; and step 5, calculating the final score of each answer in the reply table and sorting, and selecting a random reply from the candidate replies with the final score greater than 0.7 and the total score ranking the top 10% as a final reply. The sentiment interaction method has the advantages of improving sentiment interaction accuracy and humanization of the elderly.

Description

Technical Field

[0001] The present invention relates to the technical field of emotional interaction for the elderly, and more specifically, to a two-way emotional interaction method for the elderly based on artificial intelligence technology. Background Art

[0002] As they age, their physical functions decline, and their children are unable to be around to take care of them for various reasons. Many elderly people living alone face great challenges in their later years, especially specific mental health problems, such as empty nest syndrome, which can lead to emotional disorders such as depression, anxiety, and loneliness. These problems not only reduce the quality of life, but may also have a negative impact on physical health.

[0003] The rapid development of large language models (LLMs), reasoning models, and agent-based AI methods has coincided with a growing global mental health crisis. The growing demand has not translated into adequate specialized support, especially for underserved populations, providing a unique opportunity for AI to complement human-led interventions, providing scalable and context-aware support while maintaining human connections in this sensitive area.

[0004] In recent years, AI has been increasingly integrated into mental health support. Its applications include the therapeutic chatbot Woebot launched by Fitzpatrick et al. in 2017. Research results show that users have high acceptance and usability scores for Woebot, and it has good feasibility and user experience among young people; Tutun et al. proposed an AI-based clinical decision support system (DSS) in their 2023 study. The system provides an efficient and accurate tool for mental health assessment, which can not only quickly identify psychological disorders, but also improve clinical efficiency by simplifying the assessment process; the AI-driven self-help tool proposed by van der Schyff et al. in 2023 provides immediate and personalized mental health support, acting as an understanding companion throughout the mental health journey. These AI companions can perform regular check-ins, provide emotional support and personalized resource recommendations. AI-driven self-tracking insights can support users by identifying patterns and suggesting coping strategies; the AI ​​launched by Li et al. in 2023 AI-driven conversational agents (CAs) can provide empathetic, context-aware support to individuals in distress. Leveraging LLM and multimodal input, CAs can engage in dynamic and coherent conversations, responding to text, voice, and even non-verbal cues to create a more comprehensive and natural support experience similar to that of an artificial supporter. Studies have shown that these CAs can reduce anxiety and depression symptoms in college students and adults. These AI-driven chatbots use an approach based on cognitive behavioral therapy to provide coping strategies.

[0005] While current technologies have a role to play in mental health intervention, they suffer from mechanical interactions, a lack of empathy, and cultural adaptability. Furthermore, these applications and products fail to fully consider the physiological and psychological characteristics of older adults in their design, resulting in inconvenient use. Furthermore, some emotional agents are limited in functionality, performing only simple tasks such as medication reminders and music playback, making them incapable of meeting the diverse emotional needs of older adults. For example, while smart mattresses and smart wristbands can monitor health data in real time, they lack emotional design and offer no emotional support or social interaction. While powerful, smart care robots lack humanistic design and fail to meet the emotional needs of older adults, leading to feelings of loneliness and loss during use. While smart home systems can be controlled by voice or mobile apps, they lack emotional design considerations and lack emotional interaction with older adults. While AI digital human social technology enables real-time communication, interaction, and emotional connection, it is limited in creating emotionally engaging content, often relying on a lexicon of words rather than truly engaging content. Consequently, existing products and applications have not gained widespread adoption in the field of mental health treatment for older adults.

[0006] Most existing technologies only adopt a unimodal analysis and modeling approach. However, changes in human facial expressions and voice intonation are also ways of expressing emotions to a certain extent. Therefore, unimodal technologies may have inaccurate analysis of human emotions. Moreover, the dialogue materials and intervention strategies of most existing technologies are mainly based on the European and American cultural background, which may not be appropriate for users from other cultural backgrounds. For example, it may appear inappropriate when discussing certain topics and cannot meet the needs of users from different cultural backgrounds. Summary of the Invention

[0007] The purpose of this invention is to design and develop a two-way emotional interaction method for the elderly based on artificial intelligence technology, construct emotional vectors through multiple modalities, extract more effective emotional features, combine emotion and context recognition, and improve the accuracy and humanization of emotional interaction for the elderly.

[0008] The technical solution provided by the present invention is: An artificial intelligence-based two-way emotional interaction method for the elderly, comprising the following steps: Step 1: Collect audio and video data of the elderly; Step 2: extracting audio feature data and visual feature data from the audio data and video data respectively, and converting the audio data into audio transcription text data; Step 3: Input the audio feature data, visual feature data, and audio transcription text data into an emotion computing model to obtain a context query version, a user emotion vector, and an empathy vector; Step 4: Input the context query version, user sentiment vector, and empathy vector into the core chat model to obtain a response table; Step 5: Pass the reply table to the response candidate ranker, and combine the context query version, user sentiment vector, and empathy vector to rank the final scores of each answer in the reply table. If the number of replies with a final score greater than 0.7 is less than 10, then any reply with a score greater than 0.7 is selected as the final reply. If there are more than 10 replies with a final score greater than 0.7, a random reply from the top 10% of candidate replies with the highest total score will be selected as the final reply.

[0009] Preferably, the step 1 further comprises pre-processing the audio data and video data; The preprocessing of the audio data includes: Step a: inputting the audio data into a VAD model based on energy threshold to perform actual speech segmentation, and then inputting the speech segment into an ASR model to obtain audio data after initial text recognition; Step b: boost the high-frequency part of the audio data by decibels and then input it into the RNNoise algorithm for denoising to obtain preprocessed audio data; The preprocessing of the video data includes: Step 1: If the user is over 75 years old, resample the original video frame rate to 1.2 times slower and then use the optical flow method to detect the facial motion area. Increase the sampling weight of the frame interval with a motion amplitude greater than 15%, and the weighted ratio is , matching the slow movements of the elderly; If the user is under 75 years old, the optical flow method is used to detect facial motion areas, and the sampling weight is increased for frame intervals with motion amplitude > 15%. The weighted ratio is ; in, is the detected motion amplitude; Step 2: Divide the video data into parts and sample from each part short segments of consecutive frames, Contains A short segment of consecutive frames is used as the data representation of the entire video; Step 3: Contains The short clips of consecutive frames are used to enhance the features of the elderly.

[0010] Preferably, extracting audio feature data from the audio data includes: Step c: Using Mel-frequency cepstral coefficients as the main feature representation of the preprocessed audio data to obtain a time-frequency feature matrix; Step d, inputting the time-frequency feature matrix into an audio feature extraction model to obtain a first feature matrix; The audio feature extraction model includes a 2DResNet-18 model, an average pooling layer, and a temporal attention layer connected in sequence; Step e: Multiply the time-frequency feature matrix and the first feature matrix and then pass them through the average pooling layer to obtain audio feature data.

[0011] Preferably, extracting visual feature data from video data includes: Step 4: Contains The short segments of consecutive frames are input into the 3D ResNet-101 neural network model for feature extraction to obtain the second feature matrix; Step 5: Pass the second feature matrix into the spatial attention module to obtain the spatial attention weight; The spatial attention module includes a one-dimensional convolutional layer, a fully connected layer and a normalization layer connected in sequence; Step 6: Multiply the second feature matrix by the spatial attention weight to generate weighted spatial features; Step 7: Pass the weighted spatial features into the channel attention module to obtain the channel attention weight; The channel attention module includes a one-dimensional convolutional layer, a fully connected layer and a normalization layer connected in sequence; Step 8: Matrix multiply the weighted spatial features by the channel attention weights to obtain weighted channel features. Step 9: Pass the weighted channel features through the average pooling layer to obtain the channel features after dimensionality reduction; Step 10: Pass the channel features after dimensionality reduction into the temporal attention module to obtain the temporal attention weight; The temporal attention module includes a one-dimensional convolutional layer, a fully connected layer and an activation function layer connected in sequence; Step 11: Multiply the channel features after dimensionality reduction by the time attention weight to obtain the weighted time features; Step 12: Pass the weighted temporal features through an average pooling layer to obtain visual feature data.

[0012] Preferably, the converting of the audio data into audio transcription text data specifically includes: Step I: passing the audio feature data into the VAD module to separate the human voice and noise to obtain a valid speech segment; Step II: The output valid speech segment is passed into the ASR module for speech-to-text transcription.

[0013] Preferably, the context query version of the user query includes: Audio transcripts, user intent, current sentiment topics, and user personas.

[0014] Preferably, the user emotion vector includes: Intent labels, emotion labels, current sentiment topics, and user profiles.

[0015] Preferably, the empathy vector includes: Current sentiment, current sentiment themes, and user personas.

[0016] Preferably, the final score satisfies: ; Where, For the final rating, Rate the reply, Score for global coherence.

[0017] Preferably, the response score satisfies: ; Where, is the local consistency score, To retrieve the matching score, Score the empathy match.

[0018] The beneficial effects of the present invention are: (1) The present invention designs and develops a two-way emotional interaction method for the elderly based on artificial intelligence technology, which uses multimodal emotional features for emotion recognition, making up for the problem of insufficient or inaccurate emotional features on a single modality. The mutual influence between multiple modalities helps to extract more comprehensive and effective emotional features; (2) The present invention designs and develops an emotional two-way interaction method for the elderly based on artificial intelligence technology, which introduces empathy analysis and user emotion analysis strategies to better understand user emotions and intentions; (3) The present invention designs and develops a two-way emotional interaction method for the elderly based on artificial intelligence technology, which provides a two-way emotional interaction model framework and methodology for the elderly based on multimodality and deep learning, fully considering the unique emotional needs of the elderly. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1This is a flow chart of the method for two-way emotional interaction among the elderly based on artificial intelligence technology described in the present invention.

[0020] Figure 2 Schematic diagram of the extraction process of visual feature data according to the present invention.

[0021] Figure 3 The figure is a schematic diagram of the process of transcribing audio into text data according to the present invention.

[0022] Figure 4 This is a flow chart of the contextual query understanding sub-model of the present invention.

[0023] Figure 5 This is a flow chart of the user understanding sub-model of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below in detail with reference to the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0025] like Figure 1 As shown, the present invention provides an artificial intelligence technology-based method for elderly people's emotional two-way interaction, which includes the following steps: Step 1: Collect audio and video data of the elderly, including: Using high-definition cameras, microphones and other equipment, the audio and video data of elderly users are collected in real time to provide raw materials for subsequent processing; Furthermore, the audio data and video data are pre-processed: 1) To better address the issues of slow speech, frequent pauses, and complex background noise faced by the elderly, the audio data is preprocessed: Step a: Use the VAD model based on energy threshold to segment the audio into actual speech segments, avoiding mechanical segmentation at silent locations. At the same time, use the ASR model to perform initial audio text recognition to ensure that segmentation does not interrupt complete semantic units. Step b: Boost the high-frequency portion of the speech (>4kHz) by 3dB to compensate for the frequency band attenuation common in presbycusis. Use the RNNoise algorithm, specifically optimized for home ambient noise.

[0026] 2) In order to capture subtle changes in elderly people’s expressions (such as frowning, twitching of the corners of the mouth) and slow movements, the video data is preprocessed: Step 1: If the user is over 75 years old, resample the original video frame rate to 1.2 times slower and then use the optical flow method to detect the facial motion area. Increase the sampling weight of the frame interval with a motion amplitude greater than 15%, and the weighted ratio is , matching the slow movements of the elderly; If the user is under 75 years old, the optical flow method is used to detect facial motion areas, and the sampling weight is increased for frame intervals with motion amplitude > 15%. The weighted ratio is ; in, is the detected motion amplitude; In this embodiment, the original video has a frame rate of 25 fps, and the resampled video has a frame rate of 20.8 fps. In this embodiment, the motion amplitude detected by the model is 20%, and the final weighted weight is 1.33 times.

[0027] Step 2: Divide the video data into parts and sample from each part short segments of consecutive frames, Contains A short segment of consecutive frames is used as the data representation of the entire video, making it meet the input requirements of the 3D convolutional neural network model; in, The value of satisfies: If the video data is less than 10 seconds long, divide the video data into 3 segments, ensuring that each segment has at least 2 seconds of valid content. If the duration of the video data is 10-30 seconds, the video data is divided into 5 segments, simulating the opening, main part, and ending parts; If the video data is longer than 30 seconds, the video data will be divided into 8 or more segments, each of which is 3-5 seconds long, in order to more effectively process the attention mechanism module; The value of satisfies: The initial value is 5. If the average motion amplitude in consecutive frames is greater than 30%, Update, if there are multiple consecutive frames, then select the one with an average weight not less than the weight threshold consecutive frames; The weight threshold is: ; Where, is the weight threshold; Step 3: Contains Short clips of consecutive frames are used to enhance the elderly features: Apply local histogram equalization (CLAHE) to areas such as the corners of the eyes and forehead.

[0028] Step 2: extracting audio feature data and visual feature data from the audio data and video data respectively, and converting the audio data into audio transcription text data; The audio feature data includes time information of speech, and the visual feature data includes facial images; The audio data extraction audio feature data uses a 2D ResNet-18 neural network model as the basis and uses the attention mechanism to extract audio features, including: Step c: The pre-processed audio data is represented by Mel-frequency cepstral coefficients (MFCC) as the main feature to obtain the time-frequency feature matrix ; Step d: the time-frequency feature matrix Input the audio feature extraction model to obtain the first feature matrix ; The audio feature extraction model includes a 2DResNet-18 model, an average pooling layer, and a temporal attention layer connected in sequence; The temporal attention layer includes a one-dimensional convolution block, a fully connected layer, and a ReLU activation layer connected in sequence; The one-dimensional convolution block uses three parallel 1D convolution layers with kernel sizes of 3, 5, and 7, respectively, a step size of 1, and 64 output channels. Step e: transform the time-frequency feature matrix and the first characteristic matrix After multiplication, the final audio feature data is obtained through the average pooling layer .

[0029] like Figure 2 As shown, the video data extraction visual feature data uses a 3D ResNet-101 neural network model and an attention mechanism to extract visual feature data, specifically including: Step 4: Contains The short segments of consecutive frames are input into the 3D ResNet-101 neural network model as independent units for feature extraction to obtain the second feature matrix ; Step 5: The second feature matrix Pass the spatial attention module to get the spatial attention weight ,This module characterizes the importance of features in spatial dimensions; The spatial attention module includes a one-dimensional convolutional layer, a fully connected layer and a normalization layer connected in sequence. That is, the input feature matrix is ​​first passed into a one-dimensional convolutional layer, and then processed by a fully connected layer. At this time, the correlation features in the spatial dimension are generated, and finally the normalization layer ( softmax Function) normalizes the attention weights to obtain the spatial attention weight matrix ; Step 6: The second characteristic matrix and the spatial attention weight matrix Perform matrix multiplication to generate weighted spatial feature matrix ; Step 7: The weighted spatial feature matrix Enter the channel attention module to obtain the channel attention weight matrix ,This module characterizes the importance of features in spatial dimensions; The channel attention module includes a one-dimensional convolutional layer, a fully connected layer and a normalization layer connected in sequence, that is, the input features are first passed into a one-dimensional convolutional layer, and then processed by a fully connected layer, at which time the correlation features in the channel dimension are generated, and finally passed through the normalization layer ( softmax Function) normalizes the attention weights to obtain the channel attention weight matrix ; Step 8: The weighted spatial feature matrix and channel attention weight matrix Perform matrix multiplication to obtain the weighted channel feature matrix ; Step 9: The weighted channel feature matrix After the average pooling layer, the channel feature matrix after dimensionality reduction is obtained , thereby reducing the feature dimension and smoothing the feature information; Step 10: The channel feature matrix after dimension reduction Pass the temporal attention module to get the temporal attention weight matrix This module characterizes the importance of features in the time dimension; The temporal attention module includes a one-dimensional convolutional layer, a fully connected layer and an activation function layer connected in sequence. That is, the input feature is first passed into a one-dimensional convolutional layer, and then processed by a fully connected layer. At this time, the correlation feature in the time dimension is generated, and the feature is passed through the activation function layer ( ReLU Activation function) calculates the time attention weight matrix ; Step 11: Reduce the channel feature matrix and the temporal attention weight matrix Multiply to get the weighted time feature matrix ; Step 12: The weighted time feature matrix After the average pooling module, the feature dimension is reduced and the feature information is smoothed, and the final visual feature data is obtained. .

[0030] like Figure 3As shown, the conversion of audio data into audio transcribed text data is to perform speech processing and understanding on the audio data, comprehensively separate human voice and background sound, and convert the original audio data into audio transcribed text data, specifically including: Step I: the audio feature data The data is passed to the VAD (Video Activity Detection) module to separate the human voice and noise to obtain valid voice segments. , specifically: Step a-1: Input audio feature data Divide feature segments, feature segment sets ; Step a-2, Time voice segment The VAD model based on convolutional neural network is input to detect the speech segment from the divided audio feature segments to remove noise and silence, that is: ; Where, It's time The probability of speech activity, VAD () function uses convolutional neural network to calculate ; The VAD model based on convolutional neural network includes a 1-dimensional convolutional layer, a maximum pooling layer and a fully connected layer connected in sequence; Step a-3, X Output valid voice clips after screening X s ; Among them, when the probability of voice activity exceeds the probability threshold, it is considered that the corresponding moment is a voice signal, and the corresponding voice segment is a valid voice area. Otherwise, it is considered as silence or background noise. Its purpose is to segment the valid voice area. x t If a segment is considered invalid (silence or background noise), it will be removed from the X Deleted; In this embodiment, the probability threshold is set to 0.5, which is obtained through experience.

[0031] Step II: Output the valid speech segment X s The data is passed to the ASR (Automation Speech Recognition) module for speech-to-text transcription. Step a-4: inputting a valid voice segment X s Frame sequence and text sequence YEstablish a mapping between and define the conditional probability: ; Where, The first i_text characters, A are all possible frame-to-character alignment paths, representing the mapping of audio frames to text characters. A ( Y ) are all possible alignment paths that can generate Y, P ( A | X s ) indicates that the model is aligned to a certain path A The confidence level, P ( Y | X s ) means all can generate Y The sum of the path probabilities is calculated using the forward-backward algorithm to calculate the conditional probabilities and the total probability; Step a-5: Use the greedy decoding method to select the character with the highest probability at each time step, and use an external language model (such as N-gram) to hypothesize all possible text sequences generated during the ASR decoding process (i.e., candidate results). ) is rescored to avoid errors caused by segment context and generate text sequences , where for each candidate result , the joint score is calculated according to the following formula: ; Where, is the score calculated by the acoustic model CTC, is the model score calculated by the N-gram model using trigrams. λ is the weight value, take The largest for among, .

[0032] In this embodiment, the weight value is 0.3; Step a-6: Merge the recognition results of the speech segments, involving punctuation recovery, text formatting, etc., to obtain the final audio transcription text data.

[0033] Step 3: The audio feature data , visual feature data The audio transcription text data is input into the sentiment computing model using the multimodal sentiment recognition method to obtain the context query version Q c , user emotion vectore Q and empathy vector e R ; The sentiment computing model includes a contextual query understanding sub-model, a user understanding sub-model, and an interpersonal response generation sub-model; The audio transcription text data is fed into the context query understanding sub-model, the audio feature data is fed into the user understanding sub-model and the interpersonal response generation sub-model, and the visual feature data is fed into the user understanding sub-model and the interpersonal response generation sub-model. The context query understanding sub-model converts the input into a context query version that depicts the conversation context and semantics. Q c , the user understanding sub-model converts the input into a vector that describes the user's emotions e Q , the interpersonal response generation sub-model converts the input into a vector that describes the empathy of the reply e R , specifically including: 1) If Figure 4 As shown, the audio transcription text data is passed into the contextual query understanding sub-model (Contextual Query Understanding, CQU) to generate a contextual query version that depicts the conversation context and semantics. Q c , including reference resolution and sentence completion, specifically including: The reference resolution includes: Step b-1: Use the WordPiece word segmentation algorithm based on the HuggingFace platform to convert the input audio transcription text data into a token sequence. Each token consists of three parts: word embedding, sentence distinguishing embedding, and position embedding; Step b-2: Use a multi-layer Transformer encoder to calculate the context representation vector of each token; In this embodiment, the BERT-base model is used, which has a total of 12 layers of transformer encoders; Step b-3: Pass the context representation vector of each token through a linear layer and then through Softmax The function is normalized to obtain the predicted label probability distribution of each token. Here, the predicted label set is a set of named entity labels, such as names of people and places, which describes the nature of the token. The named entity information of each token is generated according to the label with the highest probability. Steps b-1 to b-3 implement named entity recognition (NER) on the input audio transcription text data. The purpose of this operation is to identify key entities (such as names of people, places, objects, etc.) in the user input.

[0034] Step b-4: Extract mention span information from the token’s named entity information (You can as a pronoun); The extraction process involves passing the segmentation results through NER to obtain the original NER tag sequence (in B / I / O format), then traversing the tag sequence, merging consecutive BI tag information, and generating span information for the merged results. Mention span information refers to a continuous sequence of words (i.e., a phrase or word) in the text that may point to a certain entity. These span information can be noun phrases, pronouns, named entities, etc., and they may point to the same entity with other mentions in the same text (i.e., co-reference). Furthermore, extracting mention span information does not work for longer texts. Context text with more than 512 tokens needs to be segmented. Two specific segmentation strategies are adopted: Strategy 1: Divide the document into non-overlapping segments. Specifically, divide the document into non-overlapping paragraphs of fixed length (such as 512 tokens), input each paragraph independently into the transformer encoder, and generate a local hidden vector. , the mention score is calculated using the hidden vector only within the paragraph; Strategy 2: Generate overlapping segments and fuse the representations of the overlapping parts through interpolation. Specifically, the segments are still divided into 512 token segments, and then a sliding window with a step size of 256 and a length of 512 is generated to scan the segments. This design allows for segment overlap. For the overlapping parts, the hidden vector is reconstructed using the interpolation method, which is calculated according to the following formula: ; Where, It is The final hidden vector of the overlapping parts, is the weight, which increases with It decreases linearly with the increase of Indicates the Overlapping segments in the segment k The hidden vector of is calculated in the same way as the local hidden vector in strategy 1. For the Overlapping segments in the segment The hidden vector of and The hidden vectors of the left and right segments corresponding to the overlapping part; Step b-5: Calculate mention span information and candidate antecedents through scoring function The compatibility between the candidate antecedents is the named entity that the pronoun may point to, according to the following formula: ; Where, For rating, Indicates the mention of span information and The probability scores of candidate antecedents pointing to the same entity, is the score of the mention span information, indicating its score as pointing to a valid entity. It is The mention score of a candidate antecedent is calculated in the same way as similar; The score of the mentioned span information is calculated according to the following formula: ; Where, It is the feature vector of the span information. The context vector of the token is obtained by passing it through a transformer encoder. FFN() is a feedforward neural network processing. 、 is a learnable parameter; The probability score between the mention span information and the candidate antecedent referring to the same entity is calculated as follows: ; Where, 、 They are the mention span information and the The feature vectors of candidate antecedents, is a trainable parameter matrix.

[0035] The step b-5 is completed for each segment.

[0036] Step b-6, pass softmax Function calculates all candidate pairs 、 and its probability distribution of ratings, if P ( y_span l | x_span ) is the largest, then use y_span l replace x_span The pronoun in the middle, if the span information is mentioned, is the probability of a new entity Maximum, then it means x_span It is a new entity information; Mention of the current x_span , candidate antecedents y_span l Including the same paragraph in x_span All possible representatives before x_span Nouns and special symbols that represent meanings (express x There is no co-reference and it is a new entity). Specifically, it is calculated according to the following formula: ; in, is a learnable air finger score, For the candidate antecedents, For the candidate antecedents; In particular, , the calculation method is: ; Steps b-1 to b-6 complete the process of reference resolution of the text, the purpose of which is to resolve the pronouns (such as "he" and "this") in the user input into specific referents.

[0037] The sentence completion includes: Step c-1: Use a strategy combining rules and models to identify incomplete sentences in the input audio transcription text data. , calculate the distribution probability of candidate words, select the word with the highest distribution probability to fill in the sentence to complete the content; Identify incomplete sentences specifically: The rule method refers to the use of linguistically based rules to quickly filter out obviously incomplete sentences. Among them, the judgment rules for incomplete sentences are artificially specified, including grammatical rules, semantic logic rules and contextual information rules. Grammatical rules check the integrity of sentence components, and incomplete sentences are eliminated; semantic logic rules check whether there are contradictions or interruptions in the semantics of the sentence, and those with contradictions are eliminated; contextual rule information checks whether the sentence is coherent with the historical dialogue, and incoherent ones are eliminated.

[0038] The model method calculates the perplexity PPL of a sentence according to the following formula: ; Where, Is the first tokens, yes All previous tokens, is the model prediction The conditional probability of N is the total number of tokens in the sentence.

[0039] In this embodiment, a PPL threshold is set through experiments, and sentences above the threshold are defined as incomplete sentences. For the selection of the threshold, 1,000 sentences of conversations among the elderly are marked (50% complete, 50% incomplete), the PPLs of different sentences are counted, and the median of the intersection of the PPL intervals of complete sentences and incomplete sentences, 80, is taken as the threshold.

[0040] The scope of the candidate words is a predefined language model vocabulary, which is usually built based on a large-scale corpus (such as Chinese Wikipedia, news, conversation data, etc.), covering common words, phrases and some professional terms. For conversation scenarios with the elderly, special vocabulary in the fields of medical care, daily care, etc. can be added.

[0041] For each incomplete sentence , the distribution probability of candidate words is calculated by the following formula to complete the content: ; Where, is the distribution probability of candidate words, The first words, For Transformer The hidden state of the layer, is a learnable parameter matrix.

[0042] The hidden state aggregates contextual information through a multi-head attention mechanism: ; in, Q , K , V Represent the query, key and value matrices respectively, which are obtained by linear transformation of the hidden state H of the previous layer. d k is the dimension scaling factor, which is obtained as follows : ; Where, is the first intermediate parameter, is the second intermediate parameter; Step c-2: The maximum sequence length is set to 1024. By concatenating documents and adding a "document end" marker, the capture of long-distance dependencies is ensured. The generation process continues until the end marker is output or the maximum sequence length is reached.

[0043] In this embodiment, the maximum sequence length is 1024.

[0044] Steps c-1 to c-2 complete the completion of incomplete sentences in the text.

[0045] According to the steps b-1 to c-2, the final audio transcription text after contextual query understanding is obtained. Ctx , which is filled in the context query version Q c .

[0046] 2) If Figure 5 As shown, the audio generated by speech processing and understanding is transcribed into text Ctx , audio feature data , visual feature data The user input static attributes are input into the user understanding sub-model (User Understanding) to generate the emotion vector that describes the user's emotional attributes e Q , including intent tags, emotion tags, current emotional themes and user portraits, specifically: The acquisition of the intention label and the emotion label includes the following steps: Step d-1: transcribe the audio into text data Ctx Using MacBERT pre-trained model for Chinese text sequence encoding W Ctx ; The MacBERT pre-trained model includes an embedding layer, a 12-layer transformer encoder, and a Deep Pyramid CNN connected in sequence. Step d-2: Use 1D convolution to project the features of different modalities (text encoding and audio and video feature data) into the same dimension, and use the cross-modal fusion model MulT to concatenate the speech and text features; Step d-3: Use a multi-task learning strategy architecture for intent recognition and sentiment classification. This strategy optimizes the machine learning paradigm of multiple related tasks simultaneously by sharing the underlying model parameters. Among them, intent recognition and sentiment classification are joint tasks, sharing the front-end feature extraction layer and using independent task heads on the back-end.

[0047] For shared features, specifically, the concatenated features are input into the BiLSTM module and the self-attention pooling module to generate shared features h shared .

[0048] For the task head of intent recognition, the intent types are divided into medical consultation (0), daily companionship (1), life assistance (2), memory review (3), and other purposes (4). The shared features are passed through a fully connected layer and normalized to generate a probability distribution, and the intent label with the highest probability is selected.

[0049] For the emotion recognition task head, the emotion types are divided into happy (0), sad (1), anxious (2), angry (3), and neutral (4). The shared features are passed through a fully connected layer and normalized to generate a probability distribution, and the emotion label with the highest probability is selected.

[0050] Steps d-1 to d-3 complete the intent detection and emotion recognition tasks based on the input text combined with audio and video features.

[0051] The acquisition of the current emotional theme includes the following steps: Step e-1: Decompose the audio transcription text data into sentences according to the period and input them into the SBERT model to generate an embedding vector. The embedding vector here refers to the feature vector of the sentence. For sentences of different lengths, the feature vector length is also different. The longest feature vector is used as the standard, and feature vectors shorter than this length are padded with zeros. Step e-2: Calculate the cosine similarity of different embedding vectors to compare the similarity between embedding vectors, and then calculate the cosine similarity between all sentences to form a similarity matrix. Use the hierarchical clustering method to divide the similarity matrix into several categories. The cosine similarity is calculated according to the following formula: ; Where u and v are two embedding vectors, and is the component of the corresponding embedding vector; Step e-3: For each sentence category, calculate the word frequency and inverse document frequency (prevalence in the corpus) of all words in the sentence to form a TF-IDF value. Select the word with the highest TF-IDF in the category as the current sentiment theme, prioritizing verbs and nouns while merging synonyms. Steps e-1 to e-3 complete the task of performing topic detection on the input speech transcription text.

[0052] The acquisition of the user portrait specifically includes the following steps: Step f-1: Encode the user-authorized static attributes of the user input into a continuous vector using the SBERT model; The user input static attributes are user default information stored in the database before the interaction begins. These default information are generally static attributes related to the user, and their characteristics are that these attributes will not change or can be changed according to rules, such as the name, gender, age and other information of the elderly; Step f-2: Input the continuous vector into the LSTM to perform temporal dynamic modeling of the user's attribute vector. The final hidden state output is a compact representation of the user's behavior patterns and long-term preferences. This is a fixed-dimensional vector that captures long-term dependencies through a gating mechanism. Here, the user's behavior patterns, including preferences and personality, are inferred from their static features, which is the so-called user profile. Among them, the forget gate in LSTM determines whether to retain or discard historical information, the input gate is used to update newly occurring events, and the output gate is used to generate the current hidden state. The cell state update mechanism is used to combine the historical state and the current input to form an updated long-term memory.

[0053] Step f-3: Set multiple key time points (such as medication time) and extract key features, including time features (specific time period), behavioral features (what the user is doing through voice feedback during that time period), and emotional features (user characteristics during that time period). Use the time-aware attention mechanism to assign higher weights to key time points. Specifically, the following steps are included: Convert time points into period codes according to the following formula to capture patterns such as morning and evening, day of the week, etc.: ; Where, is the timestamp, d is the feature dimension, From 0 to d -1, for t_code Position encoding of the even-indexed positions of the time point feature vector, for t_code Position encoding of the odd-indexed positions of the time point feature vector; The importance score of a time point is calculated according to the following formula: ; Where, For time point t The importance score of 、 The learnable parameter matrix, It's time t The behavioral and emotional characteristics of , σ is the Sigmoid function, and the key time points are weighted according to the following formula to obtain the key time point features and suppress non-key noise: ; Where, For time point t Feature weighting of behavioral and emotional characteristics; Steps f-1 to f-3 complete the work of portraying the user portrait.

[0054] Fill the obtained intent label, emotion label, current emotion theme and user portrait into the user emotion vector e Q .

[0055] Improve the context query version Q c , including audio transcripts Ctx , user intention, current sentiment topics and user portraits, e Q Provide emotional context, Q c Providing semantic context, the two work together to generate empathetic responses.

[0056] 3) Transcribe the audio generated by speech processing and understanding into text Ctx , audio feature data , visual feature data The user input static attributes are passed into the interpersonal generation sub-model to generate an empathy vector that describes the emotional attributes. e R , e R The presence of a voice is key to being able to give emotional and empathetic responses; Empathy Vector e R In fact, it is a feature representation of the human setting for the response model, which is composed of the same e Q The difference is that the former describes the "personality" of a model, while the latter represents the personality of a real person. For example, the setting chosen for this invention is a gentle, kind, and patient "old friend," portrayed as a middle-aged or slightly older, approachable character who conveys care, encouragement, and positive emotions during conversations, and is adept at listening, comforting, and providing timely encouragement and advice. Gentle humor can be used to lighten the mood when appropriate, and the overall expression style is simple and straightforward, avoiding complex terminology or radical expressions.

[0057] Filling out the empathy vector using components from user understanding e R Specifically, we first need to set a personality template for the model, then encode the personality template into an emotion vector, and dynamically adjust the emotion vector of the model based on the user's current emotion, and then convert the adjusted emotion vector into the model's current emotion and fill it in. e R This part of the work is completed by the emotion recognition component in user understanding.

[0058] Next, the model’s preferred themes, such as memories, encouragement, or daily care, are preset, and the user’s emotion vector e Q Get the current emotional theme and fill it in e R The current emotional theme section in .

[0059] Finally, preset some key time point information, such as morning = encouragement, before bed = comfort, assign higher weights to these key time points, and fill in the key features e R This part of the work is completed by the user portrait component in user understanding.

[0060] Step 4: Query the context version Q c , user emotion vector e Q and empathy vector e R Enter the core chat model and get the reply table ; The core chat model specifically includes a paired data retrieval generator sub-model, a neural generator sub-model and an unpaired data retrieval generator sub-model. Q c 、 e Q 、 e R Each of these should be passed into the three sub-models respectively. These three sub-models generate responses based on their respective algorithms and store them in the response table. The so-called generator is a component that generates output content based on the characteristics of user input content. Based on the generation method, it can be divided into retrieval generators based on paired data, neural generators, and unpaired data retrieval generators. Specifically, they include: 1) The context query version of the user query Q c , user emotion vector e Q , empathy vector e R The steps that are passed as input to the paired data query generator to generate responses include: Step h-1: Cleanse 3 billion historical conversations (from internet social data and historical conversation records) to filter out sensitive information and low-quality content and create an index. The resulting data will be used for search in subsequent steps. Index establishment specifically includes: setting data cleaning rules to remove sensitive information and low-quality content, and using classic models such as IK word segmenter, FAISS, and cluster bucketing to establish indexes; Low-quality content includes duplicate and invalid data; Step h-2: Query the version based on the current conversation state (such as the context of the user query) Q c , user emotion vector e Q ) Use the index to retrieve candidate responses, output the retrieved historical conversation data, and generate a reply .

[0061] 2) The context query version of the user query Q c , user emotion vector e Q , empathy vector e R As input, the neural generator generates a response; Among them, the GRU-RNN model is used as the neural generator to convert the context query version of the user query Q c , user emotion vector e Q and empathy vector e R The input is passed to the GRU-RNN neural network for processing. The neural network generator is trained with common social conversation datasets such as OpenSubtitles and Reddit, that is, the user query context records Q c , user empathy vector e Q , empathy vector e R Splicing to generate intermediate vectors v , the intermediate vector v Input the neural generator and get a natural language response .

[0062] 3) The context query version of the user query Q c , user emotion vector e Q , empathy vector e R The unpaired data retrieval generator is fed as input to generate a response, specifically: Step j-1, from Q c Extract the input entity information and the relationship between entities; The entity information includes the entity's attribute information and the attributes described in the text. The entity's attribute information includes the entity's visual attributes such as color, shape, size, etc. By calculating the similarity between entities, we can solve the problem of ambiguous entity references in multimodal data. Then, we can align entities in different modalities to identify whether they refer to the same real-world object. This can be accomplished by using the reference resolution in steps b-4 to b-6. Step j-2: Use the extracted entities as nodes and relationships as edges to construct the graph structure of the knowledge graph; Step j-3: Use single sentences from news and speeches as unpaired data and query versions based on the constructed knowledge graph and the context of the user query Q c , user emotion vector e Q , empathy vector e R As the input of the unpaired data retrieval generator, find the relevant unpaired data; Step j-4: Search based on the principle of retaining only the topics that co-occur frequently in the conversation, and finally output the knowledge graph and the retrieved unpaired data to generate a response ; The high-frequency co-occurrence refers to the topics that appear in the top 20% of the time. The answers generated by the three sub-models 1), 2), and 3) 、 、 Save reply form .

[0063] Step 5: The reply table is passed to the response candidate ranker. The response scores in the calculation table are ranked based on the context query version, user sentiment vector, and empathy vector. A response with a final score higher than 0.7 is randomly selected as the final response. If there are more than 10 responses with a score higher than 0.7, a response with a total score in the top 10% of the candidate responses is randomly selected as the final response. Random selection is used to retain a certain degree of randomness so that the response does not appear dull and rigid. Specifically, the following steps are performed: Step k-1: Response form For each candidate response in the response candidate ranker, the response candidate ranker uses the DSSN model and the Entity Grid model to calculate the four major indicators of local consistency, global coherence, retrieval matching, and empathy matching to score and screen the candidate responses; Among them, local consistency is used to calculate the relevance of the response and the current user input. It uses the DSSN model for calculation. Specifically: The contextual query version of the user query Q c Each candidate response is encoded into a fixed-dimensional vector q and r through the BERT model, and the cosine similarity is calculated ; Use the following formula to calculate the word-level relevance between queries and responses, increasing sensitivity to entity words: ; In the formula, IDF( w ) is the search word w The frequency of occurrence in a specified corpus, f ( w , R ) is a word w In the reply form The frequency of occurrence in , avg_len is the average length of all replies, k 1=1.5, b =0.75 is an empirical parameter.

[0064] Local consistency score Calculate using the following formula: ; Global coherence is used to ensure that responses are consistent with the conversation history (for example, when a user is continuously discussing music, avoid suddenly switching to the topic of movies). It is calculated using the Entity Grid model. Specifically: Query version from the context of the user query Q c Extract entities and their grammatical roles (different roles for the same word) from the candidate responses, construct the entity-role transfer matrix, and calculate the coherence score according to the following formula: ; The LDA model is used to calculate the JS divergence Theme_Score of the historical conversation topic distribution and the candidate reply topic, and the global coherence score is obtained according to the following formula : ; Search matching calculates the matching scores of keywords (BM25 / TF-IDF) and semantics (DSSM). Specifically: The contextual version of the user query Q c Each candidate response is encoded into fixed-dimensional vectors q and r through the BERT model, and the matching score is calculated through the deep network. The formula is: ; Where, 、 、 and These are all learnable parameters used to calculate the matching score. Score the match; calculate Q c and R The TF-IDF vector angle is obtained by combining the word frequency and inverse document frequency of each word in the two texts. The cosine value of the angle between the two is TFIDF_Score. The final retrieval matching score Retrieval_Matching is calculated according to the following formula: ; Empathy matching by comparing response empathy vectors With the expected vector To verify whether the response is consistent with the model personality and user emotional needs. Expected vector The empathy score is encoded by the manually specified response characteristics, and the empathy_score formula is: ; In step k-2, the DSSN model outputs a response score by measuring local consistency, retrieval matching, and empathy matching. The response score is calculated according to the following formula: ; Where, Score responses; Step k-3: Add the response score and the global coherence score to obtain the final score, which is: ; Where, For the final rating.

[0065] The present invention designs and develops a two-way emotional interaction method for the elderly based on artificial intelligence technology. It uses multimodal emotional features for emotion recognition, which makes up for the problem of insufficient or inaccurate emotional features on a single modality. The mutual influence between multiple modalities helps to extract more comprehensive and effective emotional features and improve the accuracy; introduces empathy analysis and user emotion analysis strategies to better understand user emotions and intentions; uses a multimodal two-way emotional interaction dataset for the elderly for training, fully considering the unique emotional needs of the elderly; provides a two-way emotional interaction model framework and methodology for the elderly based on multimodality and deep learning, and through deep learning and neural network technology, communicates with elderly people living alone to timely understand the psychological conditions of the elderly, and intervenes in the imbalance of mental health of the elderly in a timely manner, thereby improving the mental health level of elderly people living alone.

[0066] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and embodiments shown and described herein.

Claims

1. A two-way emotional interaction method for the elderly based on artificial intelligence technology, characterized in that: The steps include: Step 1: Collect audio and video data of the elderly; Step 2: extracting audio feature data and visual feature data from the audio data and video data respectively, and converting the audio data into audio transcription text data; Step 3: Input the audio feature data, visual feature data, and audio transcription text data into an emotion computing model to obtain a context query version, a user emotion vector, and an empathy vector; Step 4: Input the context query version, user sentiment vector, and empathy vector into the core chat model to obtain a response table; Step 5: Pass the reply table to the response candidate ranker, and combine the context query version, user sentiment vector, and empathy vector to rank the final scores of each answer in the reply table. If the number of replies with a final score greater than 0.7 is less than 10, then any reply with a score greater than 0.7 is selected as the final reply. If there are more than 10 replies with a final score greater than 0.7, a random reply from the top 10% of candidate replies with the highest total score will be selected as the final reply.

2. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 1, characterized in that: The step 1 further comprises pre-processing the audio data and the video data; The preprocessing of the audio data includes: Step a: inputting the audio data into a VAD model based on energy threshold to perform actual speech segmentation, and then inputting the speech segment into an ASR model to obtain audio data after initial text recognition; Step b: boost the high-frequency part of the audio data by decibels and then input it into the RNNoise algorithm for denoising to obtain preprocessed audio data; The preprocessing of the video data includes: Step 1: If the user is over 75 years old, resample the original video frame rate to 1.2 times slower and then use the optical flow method to detect the facial motion area. Increase the sampling weight of the frame interval with a motion amplitude greater than 15%, and the weighted ratio is , matching the slow movements of the elderly; If the user is under 75 years old, the optical flow method is used to detect facial motion areas, and the sampling weight is increased for frame intervals with motion amplitude > 15%. The weighted ratio is ; in, is the detected motion amplitude; Step 2: Divide the video data into parts and sample from each part short segments of consecutive frames, Contains A short segment of consecutive frames is used as the data representation of the entire video; Step 3: Contains The short clips of consecutive frames are used to enhance the features of the elderly.

3. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 2, characterized in that: Extracting audio feature data from the audio data includes: Step c: Using Mel-frequency cepstral coefficients as the main feature representation of the preprocessed audio data to obtain a time-frequency feature matrix; Step d, inputting the time-frequency feature matrix into an audio feature extraction model to obtain a first feature matrix; The audio feature extraction model includes a 2DResNet-18 model, an average pooling layer, and a temporal attention layer connected in sequence; Step e: Multiply the time-frequency feature matrix and the first feature matrix and then pass them through the average pooling layer to obtain audio feature data.

4. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 3, characterized in that: The visual feature data extracted from video data includes: Step 4: Contains The short segments of consecutive frames are input into the 3D ResNet-101 neural network model for feature extraction to obtain the second feature matrix; Step 5: Pass the second feature matrix into the spatial attention module to obtain the spatial attention weight; The spatial attention module includes a one-dimensional convolutional layer, a fully connected layer and a normalization layer connected in sequence; Step 6: Multiply the second feature matrix by the spatial attention weight to generate weighted spatial features; Step 7: Pass the weighted spatial features into the channel attention module to obtain the channel attention weight; The channel attention module includes a one-dimensional convolutional layer, a fully connected layer and a normalization layer connected in sequence; Step 8: Matrix multiply the weighted spatial features by the channel attention weights to obtain weighted channel features. Step 9: Pass the weighted channel features through the average pooling layer to obtain the channel features after dimensionality reduction; Step 10: Pass the channel features after dimensionality reduction into the temporal attention module to obtain the temporal attention weight; The temporal attention module includes a one-dimensional convolutional layer, a fully connected layer and an activation function layer connected in sequence; Step 11: Multiply the channel features after dimensionality reduction by the time attention weight to obtain the weighted time features; Step 12: Pass the weighted temporal features through an average pooling layer to obtain visual feature data.

5. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 4, characterized in that: The converting of the audio data into audio transcription text data specifically includes: Step I: passing the audio feature data into the VAD module to separate the human voice and noise to obtain a valid speech segment; Step II: The output valid speech segment is passed into the ASR module for speech-to-text transcription.

6. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 5, characterized in that: The context query version of the user query includes: Audio transcripts, user intent, current sentiment topics, and user personas.

7. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 6, characterized in that: The user emotion vector includes: Intent labels, emotion labels, current sentiment topics, and user profiles.

8. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 7, characterized in that: The empathy vector includes: Current sentiment, current sentiment themes, and user personas.

9. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 8, characterized in that: The final score satisfies: ； Where, For the final rating, Rate the reply, Score for global coherence.

10. The method for two-way emotional interaction among the elderly based on artificial intelligence technology as claimed in claim 9, characterized in that: The response score satisfies: ； Where, is the local consistency score, To retrieve the matching score, Score the empathy match.

Images