Interactive digital person accompanying system for old people with cognitive impairment

Abstract

The invention, which relates to the field of the intelligent accompanying system, discloses an interactive digital person accompanying system for the old with cognitive impairment, comprising a data acquisition module, an emotion recognition module, a first matching module, a second matching module, a personalized interaction generation module and a reinforcement learning adaptive module. The data acquisition module acquires multi-modal sensing data of a target old person; the emotion recognition module outputs an emotion label and an intensity value through feature extraction and Transform fusion; a first matching module screens corpus fragments according to emotion intensity triggering conditions; the second matching module performs secondary screening based on the retrieval weight; the personalized interaction generation module generates digital twin interaction data through probability distribution sampling; the reinforcement learning adaptive module dynamically adjusts the corpus weight according to the interaction reward value; according to the method, the dual guarantee of emotion guarding and physiological safety is realized, the anxiety emotion of the old with cognitive impairment is remarkably relieved, and the novelty of interaction and high emotion relieving efficiency can be kept.

Description

Technical Field

[0001] This invention relates to the field of intelligent companionship systems, specifically an interactive digital human companionship system for elderly people with cognitive impairment. Background Technology

[0002] Patients with cognitive impairment typically experience memory loss, emotional fluctuations, and behavioral abnormalities, severely impacting their daily lives and social interactions. To improve the quality of life for the elderly and enhance the efficiency of care services, smart health technologies are gradually becoming a focus of attention, particularly in the areas of emotional care and cognitive support for the elderly, where digital companionship technology and artificial intelligence technology are being widely applied.

[0003] Currently, AI-based digital human companion systems are widely used to provide emotional support and psychological counseling. These systems not only facilitate human-computer interaction in emotional exchange but also respond in real-time to users' emotional changes and needs, effectively alleviating loneliness among the elderly.

[0004] However, existing digital companionship systems mostly focus on standardized and single interaction modes, lacking personalized emotional adaptation capabilities and failing to effectively alleviate patients' emotional stress. In particular, patients have the highest level of trust in the voices and appearances of their children, but traditional tools cannot provide real-time "children's companionship," causing them to miss the best opportunity to relieve anxiety and enhance their sense of security. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an interactive digital human companion system for elderly people with cognitive impairment.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] This invention discloses an interactive digital human companion system for elderly people with cognitive impairment, comprising:

[0008] The data acquisition module is used to acquire multimodal perception data of the target elderly person;

[0009] The emotion recognition module is used to extract features from the multimodal perception data and perform Transformer fusion to obtain emotion labels and corresponding emotion intensity values;

[0010] The first matching module is used to determine whether the emotion intensity value meets the preset emotion triggering condition. If so, it matches the emotion tag in the preset personalized corpus to obtain the first candidate corpus fragment.

[0011] The second matching module is used to obtain the retrieval weight of the first candidate corpus fragment under the current sentiment tag, and to filter the first candidate corpus fragment according to the retrieval weight to obtain the second candidate corpus fragment.

[0012] The personalized interaction generation module is used to convert the retrieval weights of all second candidate corpus fragments into a probability distribution, and to sample several matching corpus fragments according to the probability distribution, and to generate interactive data of the child digital twin according to the matching corpus fragments.

[0013] The reinforcement learning adaptive module is used to acquire the interaction performance data of the target elderly person in relation to the digital twin of their children, and to calculate the interaction reward value by weighting the interaction performance data; it is then determined whether the interaction reward value is greater than 0. If it is, the retrieval weight of the matching corpus fragment in the personalized corpus is increased; otherwise, the retrieval weight is decreased.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. By simultaneously capturing multimodal information such as facial micro-expressions, speech prosody and language semantics of the elderly through a multimodal emotion recognition module, and fusion outputting emotion labels and intensity coefficients through a multi-head self-attention Transformer, the system can accurately identify the emotional state of the elderly under complex conditions such as noisy environments or insufficient lighting, which improves the accuracy of emotion recognition compared to a single-modal system.

[0016] 2. Through the digital twin interaction module of children, combined with emotion tags and the characteristics of children's twin roles, matching comforting, encouraging, prompting or casual conversation segments are retrieved from the local personalized knowledge base. The voiceprint adaptive TTS engine synthesizes the children's voice and synchronously drives the digital human's mouth shape, facial expressions and body movements, providing the elderly with an immersive companionship experience of "children by their side".

[0017] 3. Through reinforcement learning adaptive mechanism, the system can dynamically adjust the retrieval weight and emotion threshold of each knowledge unit in the personalized local knowledge base according to the elderly’s interaction performance, so that the knowledge base content is increasingly in line with the elderly’s interests, life experience and cognitive state, maintaining the novelty of interaction and high emotional relief efficiency.

[0018] 4. The safety alarm module continuously monitors posture and emotional intensity. When a fall posture or emotional intensity exceeds a preset extreme threshold, an alarm is immediately pushed to multiple channels, achieving dual protection of emotional well-being and physical safety. This significantly alleviates anxiety in elderly people with cognitive impairment and reduces physical risks such as falls. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an overall system block diagram of an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of the multimodal emotion recognition process in an embodiment of the present invention;

[0022] Figure 3 This is a flowchart illustrating the security alarm processing in an embodiment of the present invention;

[0023] Figure 4 This is a flowchart illustrating the construction process of a personalized corpus in an embodiment of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Application Overview: In existing technologies, digital human companionship systems are widely used in the field of emotional support for the elderly, alleviating feelings of loneliness through human-computer interaction. However, existing systems mostly employ standardized interaction patterns, lacking personalized adaptation capabilities and struggling to effectively respond to the emotional fluctuations of patients with cognitive impairment. Especially when patients experience anxiety or anger, traditional systems cannot access personalized language data related to their children for comfort, resulting in limited interaction effectiveness. For example, when an elderly person experiences emotional fluctuations due to memory confusion, standardized comforting phrases fail to trigger their emotional dependence on their children and may instead exacerbate feelings of distrust.

[0026] To address the aforementioned issues, the inventors discovered that elderly individuals with cognitive impairment exhibit a unique emotional dependence on their children's voiceprints, facial features, and shared memories, but existing technologies have failed to effectively utilize this characteristic. Analysis revealed that consistently exceeding emotional limits is a key trigger for personalized interaction, and dynamically adjusting corpus retrieval weights can optimize interaction strategies. Therefore, they proposed constructing a personalized corpus using voiceprint-facial twins, combined with reinforcement learning mechanisms to continuously optimize interaction strategies.

[0027] Example:

[0028] like Figure 1-4As shown, an interactive digital human companion system for elderly people with cognitive impairment includes:

[0029] The data acquisition module is used to acquire multimodal perception data of the target elderly person;

[0030] The emotion recognition module is used to extract features from the multimodal perception data and perform Transformer fusion to obtain emotion labels and corresponding emotion intensity values;

[0031] The first matching module is used to determine whether the emotion intensity value meets the preset emotion triggering condition. If so, it matches the emotion tag in the preset personalized corpus to obtain the first candidate corpus fragment.

[0032] The second matching module is used to obtain the retrieval weight of the first candidate corpus fragment under the current sentiment tag, and to filter the first candidate corpus fragment according to the retrieval weight to obtain the second candidate corpus fragment.

[0033] The personalized interaction generation module is used to convert the retrieval weights of all second candidate corpus fragments into a probability distribution, and to sample several matching corpus fragments according to the probability distribution, and to generate interactive data of the child digital twin according to the matching corpus fragments.

[0034] The reinforcement learning adaptive module is used to acquire the interaction performance data of the target elderly person in relation to the digital twin of their children, and to calculate the interaction reward value by weighting the interaction performance data; it is then determined whether the interaction reward value is greater than 0. If it is, the retrieval weight of the matching corpus fragment in the personalized corpus is increased; otherwise, the retrieval weight is decreased.

[0035] Multimodal perception data refers to physiological and behavioral data collected through multiple dimensions such as vision and hearing. Specifically, it can be achieved by using a combination of cameras and microphones to simultaneously capture facial video and audio, comprehensively capturing the user's emotional expression characteristics. Emotional triggering conditions refer to the logical rules for determining whether an emotional state meets the standards for interactive intervention. This can be implemented using a threshold comparison algorithm, triggering a matching mechanism when the emotional intensity exceeds a set threshold and persists for a certain duration. The personalized corpus is a database containing children's voiceprint features and shared memory fragments. This can be constructed through voice sample analysis and mining of historical activity records, used to generate interactive content with emotional resonance. Retrieval weight adjustment refers to the calculation process of optimizing corpus priority based on reinforcement learning mechanisms. This can be implemented using a reward signal-driven weight update algorithm, enhancing the retrieval probability of effective corpus through positive feedback.

[0036] Specifically, the system collects users' facial expressions and voice data in real time using multimodal sensors, and identifies their current emotional state after feature fusion. When the detected emotional intensity continuously exceeds the threshold, the system prioritizes matching personalized speech fragments related to children. A two-stage screening mechanism is used: first, the speech fragments are initially selected based on emotion tags, and then they are weighted and sorted according to historical interaction effects. A probabilistic sampling method is used to generate a natural dialogue stream, and realistic interaction is constructed by combining voiceprint synthesis and expression-driven technology. After each interaction, the system calculates a reward value based on the user's physiological feedback and dynamically adjusts the speech fragment weights to optimize subsequent matching strategies.

[0037] Compared to existing technologies, this solution constructs a personalized corpus using twin technology based on child characteristics, addressing the issue of insufficient emotional adaptability in traditional systems. It introduces a reinforcement learning mechanism to dynamically optimize interaction strategies, overcoming the rigidity of interaction caused by fixed-weight matching patterns. Furthermore, it employs a combination of multi-stage screening and probability sampling to improve the naturalness of interaction while ensuring response speed.

[0038] Through the above technical solutions, this application achieves precise emotional intervention for elderly people with cognitive impairment. It enhances trust through the use of a twin with the characteristics of their children and improves the emotional soothing effect through a dynamic weight adjustment mechanism. The system can trigger personalized responses based on real-time emotional intensity, improve the accuracy of emotion recognition through multimodal data fusion, and ultimately form a virtuous cycle of interaction that aligns with the user's cognitive characteristics.

[0039] This application further proposes a multimodal perception data system, including facial video data and audio data of the target elderly person; the process of obtaining emotion labels and corresponding emotion intensity values ​​includes: extracting facial expression temporal features through the visual channel, extracting speech prosody features through the audio channel, and extracting semantic polarity features through the language channel; concatenating the facial expression temporal features, speech prosody features, and semantic polarity features and inputting them into a multi-head self-attention Transformer model for fusion, outputting emotion labels and emotion intensity values ​​in the range of 0-1; the emotion labels include joy, calmness, anxiety, sadness, and anger.

[0040] Among these, facial video data refers to dynamic image sequences of elderly people's faces captured by cameras. Specifically, facial ROIs can be extracted from the visual stream using a cascaded face detection network, and a lightweight residual network combined with a two-layer LSTM is used to model changes in facial expressions over time to capture continuous changes in facial expressions. Speech audio data refers to speech signals captured by microphones. Specifically, a short-time Fourier transform can be performed on the speech signal to generate a Mel spectrogram, which is then used by a convolutional network to extract prosodic sentiment features. Semantic polarity features refer to the sentiment analysis of speech-to-text using natural language processing techniques. Specifically, pre-trained models such as BERT can be used to calculate positive and negative sentiment scores for the text. The multi-head self-attention Transformer model refers to a multimodal feature fusion framework based on a self-attention mechanism. For example, a cross-modal attention layer can be used to model the association between features from different channels, and a comprehensive sentiment judgment result can be generated through weighted fusion.

[0041] Specifically, facial expression temporal features are extracted using a 3D convolutional neural network to capture dynamic change patterns between consecutive frames, such as the duration and intensity of micro-expressions like frowning and drooping corners of the mouth. Speech prosody features are extracted using short-time Fourier transform to extract fundamental frequency trajectories and spectral envelopes, such as identifying emotional arousal states corresponding to pitch increases or speech speed increases. Semantic polarity features are obtained by combining sentiment dictionary matching with contextual semantic analysis, such as detecting recurring negative words and their degree of modification in dialogue. These three features are concatenated and input into a multi-head self-attention layer containing positional encoding. Information complementarity is achieved by calculating the association weights between cross-modal features. Finally, a fully connected layer outputs a sentiment intensity value normalized to the 0-1 range and its corresponding classification label.

[0042] Through the above technical solution, this application can achieve refined identification of the emotional state of elderly people with cognitive impairment. For example, when an elderly person repeats words due to memory confusion, by combining the frequency of speech pauses, facial confusion, and semantic logical breaks, it can accurately determine that they are in an anxious state rather than normal communication, thus providing a reliable basis for generating childlike comforting content. This multi-dimensional feature fusion mechanism effectively improves the robustness of emotion recognition and solves the technical defects of traditional methods, such as high misjudgment rate and large response delay due to reliance on a single data source.

[0043] This application further proposes preset emotional triggering conditions, including an emotional intensity value exceeding a preset emotional intensity threshold and a duration exceeding a preset first time threshold.

[0044] The emotion intensity threshold is a pre-set critical value used to determine whether an emotion has reached the trigger condition for a response. A value of 0.7 can be used as an example. Its purpose is to filter out low-intensity emotional fluctuations to avoid invalid triggers. The first time threshold is the minimum duration for which the emotion intensity value must exceed the threshold. A value of 30 seconds can be used as an example. Its purpose is to eliminate interference from momentary emotional fluctuations in the system's judgment.

[0045] Specifically, when the target elderly person's emotional intensity value is detected to exceed the emotional intensity threshold, the system will activate a timing mechanism. If this state continues to exceed a first time threshold, the emotional trigger condition is determined to be met. For example, if the emotional intensity threshold is 0.7 and the first time threshold is 30 seconds, the system will only trigger the subsequent corpus matching process if the emotional intensity value is above 0.7 for 30 consecutive seconds. Therefore, brief emotional fluctuations (such as brief anxiety caused by environmental noise) will not be misjudged as emotional events requiring a response, thus reducing ineffective interactions.

[0046] Through the above technical solution, this application can accurately identify the emotional events that need to be responded to, avoid misjudgment caused by momentary emotional fluctuations or environmental interference, and thus ensure that the system only triggers interaction when the elderly really need emotional support, thereby improving the effectiveness and reliability of companionship behavior.

[0047] This application further proposes a method to construct a voiceprint-facial twin using voice samples and photos uploaded by children, and to generate a localized corpus containing titles, interest keywords and shared memory fragments by combining the elderly’s historical activity records.

[0048] Specifically, voice samples and photos are processed using voiceprint feature extraction models and facial feature extraction models to generate voiceprint vectors and facial vectors, respectively. The voiceprint vectors and facial vectors are concatenated and input into a twin generation model to output a voiceprint-facial twin. The elderly's historical activity records are processed using natural language processing technology to extract high-frequency titles, interest keywords, and shared memory fragments. These are then associated with the voiceprint-facial twin and stored in a localized corpus. The localized corpus is stored in an encrypted manner and runs only on local devices to avoid privacy data leakage. After the voice text is uploaded, it undergoes noise reduction and word segmentation. Keywords are extracted and matched with the existing corpus for similarity. If the similarity is lower than a preset threshold, new keywords and associated text fragments are added to the corpus.

[0049] Specifically, the voice samples uploaded by children are converted into 128-dimensional voiceprint vectors by a voiceprint encoder, and the photos are used to extract 64-dimensional facial vectors using a face recognition model. The two are concatenated to form a 192-dimensional feature vector, which is then input into a fully connected layer to generate a voiceprint-facial twin. Historical activity records are used to extract keywords that appear more than 5 times as interest keywords using the TF-IDF algorithm, and a shared memory fragment containing time, place, and event is extracted using an LSTM model. The localized corpus is stored in the form of an SQLite database, and each corpus fragment is associated with a voiceprint-facial twin feature vector, interest keywords, and shared memory fragment tags. When children upload new voice text containing "birthday party," the system detects that this keyword does not exist in the existing corpus, so "birthday party" is used as a new interest keyword, and the associated text fragment is stored in the database. The resulting interactive data contains the children's real voiceprints and facial features, combined with the elderly's historical preferences, to improve emotional identification and alleviate anxiety caused by unfamiliarity.

[0050] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0051] The personalized corpus constructs voiceprint-facial twins using voice samples and photos uploaded by children. First, multiple voice samples from the children are collected, including voice segments with different emotions, speaking speeds, and tones. Voiceprint recognition algorithms are used to extract voiceprint features and build a voice model of the children. Simultaneously, high-resolution photos of the children from multiple angles are collected, and facial recognition and 3D modeling technologies are used to construct facial models.

[0052] A localized corpus containing titles, interest keywords, and shared memory fragments is generated by combining the elderly's historical activity records. The system analyzes the elderly's daily activity trajectories, social records, and hobbies to extract key information. Shared experiences between the elderly and their children, such as important holidays and trips, are compiled into memory fragments. This information is combined with the children's commonly used ways of addressing the elderly to generate personalized corpus.

[0053] Children can upload voice text at any time to update the local corpus. The system provides a convenient voice input interface, allowing children to record new voice content via mobile devices. After natural language processing, this new content is automatically integrated into the existing corpus, achieving dynamic updates.

[0054] Through the aforementioned technical solution, this application realizes a highly personalized digital human companionship system. By constructing a voiceprint-facial twin, the system can accurately simulate the voice and appearance of children, enhancing the elderly's sense of intimacy and trust. Combined with the elderly's historical activity records, the generated corpus contains rich personalized content, making the interaction more closely resemble real-life scenarios. Children are allowed to update the corpus at any time, ensuring the system's real-time performance and adaptability, continuously providing companionship content that meets the elderly's current needs. This highly personalized approach significantly improves the system's emotional adaptability, effectively alleviating the emotional stress of the elderly, enhancing their sense of security, and thus improving the quality of life for elderly people with cognitive impairment.

[0055] This application further proposes a process for obtaining the second candidate corpus fragment, which includes sorting the first candidate corpus fragments in descending order of retrieval weight, marking the first candidate corpus fragments with the top N retrieval weights as the second candidate corpus fragments, where N is an integer not less than 20, and the initial value of the retrieval weight is 0.5.

[0056] The retrieval weight is a quantitative indicator used to measure the degree of matching between a corpus segment and the current sentiment label. It can be implemented using a floating-point number between 0 and 1, with an initial value of 0.5 to provide a baseline for subsequent reinforcement learning adjustments. Ranking refers to prioritizing candidate corpus segments based on their retrieval weights. This can be implemented using quicksort or heapsort algorithms, with descending order ensuring that high-weight segments are selected first. The top N candidate corpus segments are those with the highest weights from the ranking results. This can be achieved by setting a threshold or a fixed-quantity truncation method. A value of N not less than 20 balances selection efficiency and candidate diversity.

[0057] Specifically, after the personalized corpus completes initial matching and generates the first candidate corpus fragments, the system sorts them in descending order according to the retrieval weight corresponding to each fragment to form an ordered list. For example, when N is set to 20, the system automatically selects the top 20 high-weight fragments in the list as the second candidate corpus fragments. The initial value of the retrieval weight is uniformly set to 0.5, so that all matching fragments have the same priority in the first screening. As the reinforcement learning module dynamically adjusts the weights based on interactive feedback, the system gradually optimizes the personalized adaptation capability of the screening results.

[0058] Traditional digital human systems typically use a fixed number or random sampling method to select candidate corpora, resulting in insufficient matching between response content and user emotional state. This solution, however, employs dynamic sorting and weighted threshold truncation mechanisms to prioritize the retention of highly relevant corpus fragments. Simultaneously, by setting a lower limit for the N value, it ensures sufficient diversity in the candidate pool, preventing rigid response patterns caused by over-selection.

[0059] Through the above technical solutions, this application can improve corpus screening efficiency while ensuring response diversity, ensuring that high-weight segments are prioritized for subsequent processing, thereby shortening system response latency and enhancing the emotional adaptability of interactive content. Simultaneously, the unified setting of initial retrieval weight values ​​simplifies the system initialization configuration process and provides operable benchmark parameters for subsequent reinforcement learning-based dynamic optimization.

[0060] This application further proposes a process for obtaining matching corpus fragments, including converting the retrieval weights of the second candidate corpus fragments into corresponding sampling probabilities, calculated using the following formula:

[0061]

[0062] in, The square of the retrieval weight of the second candidate corpus segment. The sum of squares of the retrieval weights of all second candidate corpus segments. The sampling probability corresponding to the second candidate corpus segment;

[0063] A probability distribution is constructed from the sampling probabilities of all second candidate corpus fragments. One fragment is randomly selected from the second candidate corpus fragments according to the probability distribution as a matching corpus fragment. This matching corpus fragment is then removed from the second candidate corpus fragments, and the probability distribution is recalculated. Based on the new probability distribution, another second candidate corpus fragment is randomly selected as a matching corpus fragment. This random selection is repeated until the number of matching corpus fragments reaches M, where M is a positive integer not less than 3 and M is less than N.

[0064] Here, retrieval weight refers to a quantitative indicator that prioritizes candidate segments during the matching process. It can be dynamically adjusted using reinforcement learning algorithms based on historical interaction results, reflecting the applicability of different segments to the target elderly user. Probability distribution refers to the set of possibilities for selecting candidate segments based on retrieval weights. Specifically, it uses squaring operations to amplify the probability of high-weight segments, thereby increasing the output probability of high-quality segments. Random sampling refers to sampling without replacement according to the probability distribution, which can be implemented using a roulette wheel algorithm to ensure that the probability distribution of the remaining candidate set is dynamically updated after each sampling.

[0065] Specifically, when generating the matching corpus, the retrieval weight of each second candidate corpus segment is first squared, and the corresponding sampling probability is obtained through normalization. For example, when a segment's weight is 0.8, its square value is 0.64. If the sum of squares of all segments is 4, then the sampling probability of that segment is 0.16. Then, an initial random sampling is performed according to this probability distribution, and the selected segments are removed from the candidate set. At this point, the sum of squares of the remaining segments is updated to 3.36, and the new sampling probability of each segment needs to be recalculated. This process is repeated until a preset number of matching corpora is obtained; for example, when M is set to 5, five dynamically updated sampling processes are required. This gradual elimination of selected segments avoids the repeated use of similar corpora in the same dialogue scenario, while strengthening the output probability of high-value corpora through weight square calculation.

[0066] Traditional digital human systems typically employ fixed-weight sorting or simple probability sampling, which can easily lead to the frequent reuse of the same corpus fragments. This solution, however, uses a dynamically updated probability distribution and a sampling mechanism without replacement. This preserves the priority output of high-weight corpora while effectively increasing the diversity of corpus combinations through weight square calculation and multiple redistributions, thus avoiding the confusion caused by mechanical repetition of responses in elderly people with cognitive impairments.

[0067] Through the above technical solution, this application can achieve optimized combination and output of personalized corpora. While ensuring the high-frequency use of core corpora, it generates diverse natural dialogue content through dynamic adjustment of probability distribution. This technical approach is particularly suitable for scenarios where elderly people with cognitive impairment are sensitive to repetitive information. By avoiding the mechanical repetition of single corpora, it effectively reduces anxiety during the interaction process and enhances the authenticity and credibility of digital companionship.

[0068] This application further proposes a method for generating interactive data of a digital twin of a child, including refining the matching knowledge fragments through a large language model to generate colloquial text, synthesizing the child's speech through a voiceprint adaptive TTS engine, and synchronously driving the lip movements, facial expressions and body movements of the digital human to achieve multimodal alignment.

[0069] Among them, the large language model adopts a dialogue generation model based on the GPT-3 architecture, which expands the context of titles and interest keywords in the corpus fragments to generate coherent dialogue text containing greetings and questions. For example, "walking in the park" is expanded to "Last time we walked in the park, you pointed to the willows by the lake and said they were so beautiful." The voiceprint adaptive TTS engine establishes a voiceprint mapping model by comparing the fundamental frequency and formant parameters of the voice samples uploaded by children with those of the standard voice library. When synthesizing speech, the speech rate is adjusted to 120-150 words per minute, and a 0.5-1 second sentence interval is added to simulate the rhythm of real dialogue. The multimodal alignment uses an LSTM network to establish a temporal mapping relationship between speech features and facial action encoding. During speech synthesis, the mouth shape parameters corresponding to phonemes are calculated in real time, and the pose estimation model drives the digital human's head to tilt slightly by 5-10 degrees to enhance friendliness.

[0070] Specifically, when the matched corpus fragment is "Remember to take your medicine on time," the large language model refines it into "Mom, you should take your blood pressure medication. Let me pour you a glass of warm water." The voiceprint adaptive TTS engine generates a speech waveform with specific dialect endings based on the child's voiceprint characteristics. Simultaneously, the digital human's mouth closes its lips when pronouncing "medicine," and its right hand makes a cup-holding gesture when saying "pour a glass of water," maintaining a smiling expression with slightly raised eyebrows. When the matched corpus fragment is "It's time to check the news," the large language model optimizes it into "Mom, do you want to know what's new today? Let me bring up the local news push list for you." The voiceprint adaptive TTS engine generates a speech waveform with a friendly tone based on the child's voiceprint characteristics. At the same time, when the digital human pronounces "smell," its mouth is half-open, its face has a gentle smile, and its eyes show concern.

[0071] This process uses a timestamp alignment algorithm to ensure that the time deviation between speech, lip movements and actions is less than 200 milliseconds. The resulting interactive data can reproduce the timbre characteristics of children's voices in terms of hearing and achieve facial expression and action matching with a realism of over 90% in terms of visuals, effectively improving the acceptance of digital twins by elderly people with cognitive impairment.

[0072] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0073] The system refines matched knowledge fragments using a large language model to generate conversational text. Specifically, it employs the GPT-3.5 model as the large language model, takes matched knowledge fragments as input, and sets the prompt to "convert the following text into a more conversational expression" to generate conversational text.

[0074] Furthermore, child-specific speech is synthesized using a voiceprint-adaptive TTS engine. Tacotron2 is used as the base model for the TTS engine, and the model is fine-tuned through transfer learning to adapt to the voice characteristics of the child speakers. Specific steps include: collecting speech samples of the child speakers and extracting voiceprint features; adding the voiceprint features as conditional input to the Tacotron2 model; and fine-tuning the model using the collected speech samples.

[0075] Therefore, the lip movements, facial expressions, and body movements of the digital human are synchronously driven to achieve multimodal alignment. For example, an LSTM network is used to establish a speech-vision mapping model, taking synthesized speech features as input and outputting corresponding lip movement parameters, facial expression parameters, and body movement parameters. Specifically, lip movement parameters include lip shape and tongue position; facial expression parameters include the position and movement trajectory of key points such as eyebrows, eyes, and corners of the mouth; and body movement parameters include the position and rotation angle of major joints such as the head and arms.

[0076] Through the aforementioned technical solution, this application achieves highly realistic interaction with the digital twin of a child. This increases the acceptance and trust of elderly people with cognitive impairment in the digital companion system, and enhances the affinity and naturalness of emotional interaction. Furthermore, the realization of multimodal alignment makes the digital human's performance more realistic and expressive, effectively improving the elderly's immersion and interactive experience.

[0077] This application further proposes a calculation process for the interaction reward value, which includes obtaining the duration of emotional relief, the number of smile detections, and the number of rejections from the interaction performance data. The formula for calculating the interaction reward value is as follows: ,in These are empirical weighting coefficients. For the duration of emotional relief, Number of smile detections To refuse the number of interactions.

[0078] The duration of emotional relief refers to the length of time the target elderly person's emotional intensity value drops below a preset threshold. This can be calculated using the curve integral of the emotional intensity value, quantifying the system's effectiveness in alleviating negative emotions. The number of smile detections refers to the number of times the corners of the mouth turn upwards, counted by a facial expression recognition model. This can be achieved using the facial motion unit recognition function of the OpenFace framework, objectively reflecting the intensity of positive emotional feedback. The number of interaction rejections refers to the number of interaction termination requests captured through voice command recognition or posture detection. This can be achieved through keyword matching combined with head movement posture recognition, assessing interaction acceptance. The experience weight coefficient refers to a pre-set weighting parameter based on different interaction scenarios. This can be achieved using the analytic hierarchy process (AHP) to assign weights to different interaction indicators, balancing the influence of multi-dimensional evaluation indicators.

[0079] This application further proposes a process for adjusting retrieval weights, including:

[0080] If the retrieval weight is increased, the formula for calculating the change in retrieval weight is: Change = Preset learning rate * Interaction reward value * (1 - Current retrieval weight);

[0081] If the retrieval weight is reduced, the formula for calculating the change in retrieval weight is: Change = Preset learning rate * |Interaction reward value| * Current retrieval weight.

[0082] Specifically, during system operation, three indicators are collected in real time: duration of emotional relaxation, number of smile detections, and number of rejection interactions. These are weighted and summed using empirical weighting coefficients to obtain the interaction reward value. For example, when the system detects that an elderly person's emotional relaxation lasts for 120 seconds, with a total of 5 smiles and no rejection behavior, a positive reward value is calculated according to the preset weighting coefficients. Based on the sign of this reward value, different weighting adjustment formulas are used to dynamically update the retrieval weights of the used corpus fragments. For example, when the reward value is positive, the weight increment decreases as the current weight value increases, ensuring that the adjustment range of high-weight corpora tends to be gradual; when the reward value is negative, the weight reduction increases as the current weight value increases, accelerating the elimination of inefficient corpora.

[0083] Traditional digital human systems typically rely on a single interaction metric for feedback evaluation, such as simply counting the number of dialogue turns or the duration of voice responses, which cannot comprehensively reflect the effectiveness of emotional interaction. This solution, however, integrates three types of heterogeneous data—emotional relief, facial expression feedback, and behavioral rejection—to construct a multi-dimensional evaluation system. Combined with a dynamic weight adjustment mechanism, this system can more accurately identify effective interaction patterns.

[0084] Through the above technical solution, this application realizes multi-dimensional quantitative evaluation of interaction effect. By introducing an empirical weight coefficient to balance the influence weight of different evaluation indicators, and combining it with a non-linear adjustment strategy based on the current retrieval weight, it effectively avoids excessive fluctuation of weight value, so that the personalized corpus can continuously optimize the matching strategy according to the actual interaction effect, and significantly improves the emotional response adaptability and behavioral guidance effectiveness of the digital human companion system.

[0085] This application further proposes that the reinforcement learning adaptive module also includes recording interaction reward values ​​greater than 0 as positive rewards and interaction reward values ​​less than or equal to 0 as negative rewards; if there are P consecutive positive rewards, the emotion intensity threshold is increased by adjusting the step size according to a preset; if there are P consecutive negative rewards, the emotion intensity threshold is decreased by adjusting the step size according to a preset, where P is a positive integer not less than 3.

[0086] Positive rewards refer to feedback signals obtained when the system-generated interaction data effectively improves the elderly's emotional state. Specifically, this can be calculated by weighting positive indicators such as the duration of emotional relief and the number of smile detections, and is used to identify effective interaction patterns. Negative rewards refer to feedback signals when interaction data fails to achieve the expected results or triggers negative reactions. Specifically, this can be calculated by negative indicators such as the number of rejected interactions, and is used to identify interaction strategies that need optimization. The emotion intensity threshold is a critical condition parameter that triggers corpus matching. This can be implemented using a dynamic adjustment algorithm, such as automatically calibrating the trigger sensitivity based on historical interaction effects. The preset adjustment step size is a parameter controlling the magnitude of each threshold adjustment. Specifically, it can be set to a fixed increment of 0.05-0.1 to balance adjustment speed and stability.

[0087] Specifically, when the system receives multiple consecutive positive feedbacks, it indicates that the current emotion intensity threshold setting may be too sensitive, leading to frequent triggering of unnecessary interactions. In this case, gradually increasing the threshold can reduce the number of responses to minor emotional fluctuations and concentrate resources on processing more significant emotional changes. Conversely, when receiving multiple consecutive negative feedbacks, it indicates that the threshold setting is too high, causing missed opportunities for effective intervention. In this case, lowering the threshold can enhance the ability to capture subtle emotional changes. This adjustment process controls the magnitude of change through a preset step size. For example, when P is set to 3 times, the system increases the threshold by 0.1 after 3 consecutive positive rewards, ensuring the reliability of the adjustment while avoiding drastic parameter fluctuations that could affect system stability.

[0088] In some implementations, the threshold adjustment process can be combined with a time decay factor, for example, automatically reducing the adjustment step size during nighttime care to avoid excessive sleep intervention. The value of P can be set differently according to the user's cognitive level, for example, setting it to 5 times for elderly people with moderate cognitive impairment and 3 times for patients with severe cognitive impairment to accelerate adaptation. The threshold adjustment range can be limited to between 0.4 and 0.7 to prevent extreme values ​​from causing system failure.

[0089] Traditional digital human systems often use fixed emotion intensity thresholds, which cannot adapt to the dynamic changes in the emotional perception abilities of the elderly. For example, as Alzheimer's patients progress, the intensity of their emotional expression may gradually weaken, and a fixed threshold would lead to a continuous decline in the interaction trigger rate. This solution uses a reinforcement learning mechanism to achieve automatic threshold calibration, which can optimize trigger sensitivity based on long-term interaction effects while avoiding the operational burden of frequent manual parameter adjustments.

[0090] Through the above technical solution, this application effectively solves the problem of inaccurate interactive response caused by fixed thresholds in existing systems. By using a dynamic adjustment mechanism, the system continuously adapts to the changing emotional expression characteristics of the elderly, improving the timeliness and accuracy of emotional support. Especially in scenarios where the emotional fluctuation patterns of elderly people with cognitive impairment are unstable, this technology can reduce interactive omissions or over-interventions caused by improper threshold settings, maintaining the long-term effectiveness of the companionship system.

[0091] This application further proposes a safety alarm module for acquiring the target elderly person's body posture data and analyzing the shoulder, elbow, and wrist key points through a human key point posture recognition model to detect abnormal posture events. When an abnormal posture event is detected and continues for a preset second time threshold or when the emotional intensity value exceeds the emotional intensity threshold and continues for a preset third time threshold, a multi-channel safety alarm is triggered. The multi-channel safety alarm includes pushing alarm information to the children's mobile devices, the nurse station, and the smart call device, and issuing a prompt sound on-site by a digital twin.

[0092] Body posture data refers to real-time information reflecting the elderly person's limb movements collected by sensors or cameras. This can be achieved using a 3D motion capture system or a depth camera, and is used to identify abnormal posture events. The human key point posture recognition model is a model built based on machine learning algorithms to locate the coordinates of the shoulder, elbow, and wrist joints. This can be implemented using frameworks like OpenPose or MediaPipe, and by analyzing changes in key point positions, it determines whether there are abnormal behaviors such as falls, stiffness, or violent swinging. Abnormal posture events refer to limb movements that may pose safety risks, such as sudden falls, prolonged curling up, or violent arm swinging. These can be determined by jointly considering the displacement velocity and angle thresholds of key points. Multi-channel safety alarms refer to simultaneously sending alarm information to multiple terminals, such as pushing notifications to children's phones via mobile applications, notifying nurse station terminals via local area networks, triggering smart call button vibrations via IoT protocols, and having a digital twin play voice prompts locally, ensuring that alarm information covers the response needs of different scenarios.

[0093] Specifically, the safety alarm module continuously monitors the elderly person's body posture data, using a human key point posture recognition model to analyze the movement trajectories of the shoulder, elbow, and wrist joints in real time. When the displacement velocity of a key point exceeds a preset threshold or the joint angle deviates from the normal range, it is determined to be an abnormal posture event. If the event continues to exceed a second time threshold (e.g., 5 seconds), or if the emotional intensity value exceeds the threshold and continues to exceed a third preset time threshold (e.g., 10 seconds), a multi-channel alarm mechanism is triggered.

[0094] Alarms are categorized into three levels based on severity: high, medium, and low. High-level alarms simultaneously play a loud local TTS notification and flash an LED ring, while also uploading 3 frames of video to the cloud. Medium-level alarms are only pushed to family members' mobile devices. Low-level alarms are logged. If no user confirmation is received within 30 seconds, the system automatically escalates the alarm and notifies the nurses' station.

[0095] The criteria for classifying alarms into three severity levels include:

[0096] A high-level alarm is triggered when any of the following conditions are met:

[0097] A high-risk postural event (such as a fall or severe limb twitching) was detected and lasted for ≥2 seconds;

[0098] An emotion intensity value > 0.85 and lasting ≥ 10 seconds;

[0099] No user confirmation was received within 30 seconds after the medium-level alarm was triggered.

[0100] A medium-level alarm is triggered when any of the following conditions are met:

[0101] A medium-risk posture event (such as prolonged stiffness or repetitive tapping) was detected and lasted for ≥5 seconds;

[0102] An emotion intensity value >0.7 and lasting ≥30 seconds.

[0103] A low-level alarm is triggered when any of the following conditions are met:

[0104] Low-risk postural events (such as sitting imbalance or abnormal arm swing) are detected and last for ≥8 seconds;

[0105] An emotion intensity value greater than 0.6 and lasting for ≥60 seconds.

[0106] Alarm information is pushed to the child's mobile device, nurse station, and smart call device via mobile network, local area network, and IoT protocol, respectively. At the same time, the digital twin plays preset prompts in the interactive interface, such as "Please be careful" or "Do you need help?"

[0107] Through the aforementioned technical solution, this application can quickly trigger multi-dimensional alarms when an elderly person falls, experiences severe emotional fluctuations, or engages in other dangerous behaviors, thus shortening the rescue response time. For example, when an elderly person continuously poundes on a table due to anxiety and the intensity of their emotions exceeds a threshold, the system can simultaneously notify caregivers and initiate on-site voice reassurance to prevent further deterioration of their emotional state. Simultaneously, the multi-channel alarm mechanism ensures that even if one terminal fails to respond promptly, other channels can still effectively transmit alarm information, enhancing the reliability of safety assurance.

[0108] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0109] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0110] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An interactive digital human companion system for elderly people with cognitive impairment, characterized in that, include: The data acquisition module is used to acquire multimodal perception data of the target elderly person; The emotion recognition module is used to extract features from the multimodal perception data and perform Transformer fusion to obtain emotion labels and corresponding emotion intensity values; The first matching module is used to determine whether the emotion intensity value meets the preset emotion triggering condition. If so, it matches the emotion tag in the preset personalized corpus to obtain the first candidate corpus fragment. The second matching module is used to obtain the retrieval weight of the first candidate corpus fragment under the current sentiment tag, and to filter the first candidate corpus fragment according to the retrieval weight to obtain the second candidate corpus fragment. The personalized interaction generation module is used to convert the retrieval weights of all second candidate corpus fragments into a probability distribution, and to sample several matching corpus fragments according to the probability distribution, and to generate interactive data of the child digital twin according to the matching corpus fragments. The reinforcement learning adaptive module is used to acquire the interaction performance data of the target elderly person in relation to the digital twin of their children, and to calculate the interaction reward value by weighting the interaction performance data; it is then determined whether the interaction reward value is greater than 0. If it is, the retrieval weight of the matching corpus fragment in the personalized corpus is increased; otherwise, the retrieval weight is decreased.

2. The interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The multimodal perception data includes facial video data and voice audio data of the target elderly person; The process of obtaining the emotion label and the corresponding emotion intensity value includes: Facial expression temporal features are extracted through the visual channel, speech prosody features are extracted through the audio channel, and semantic polarity features are extracted through the language channel. Facial expression temporal features, speech prosody features, and semantic polarity features are concatenated and then input into a multi-head self-attention Transformer model for fusion, outputting an emotion label and an emotion intensity value in the range of 0-1; The emotion labels include joy, calm, anxiety, sadness, and anger.

3. The interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The preset emotional triggering conditions include: The emotional intensity value exceeds a preset emotional intensity threshold and the duration exceeds a preset first time threshold.

4. The interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The personalized corpus is constructed by using voice samples and photos uploaded by children to create a voiceprint-facial twin, and combined with the elderly's historical activity records to generate a localized corpus containing titles, interest keywords and shared memory fragments. Children can upload voice text at any time to update the localized corpus.

5. An interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The process of obtaining the second candidate corpus fragment includes: sorting the first candidate corpus fragments in descending order of the search weight, marking the first candidate corpus fragments with the top N search weights as the second candidate corpus fragments, where N is an integer not less than 20, and the initial value of the search weight is 0.

5.

6. An interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The process of obtaining the matching corpus fragments includes: The retrieval weight of the second candidate corpus segment is converted into the corresponding sampling probability, and the calculation formula is as follows: in, The square of the retrieval weight of the second candidate corpus segment. The sum of squares of the retrieval weights of all second candidate corpus segments. The sampling probability corresponding to the second candidate corpus segment; A probability distribution is constructed from the sampling probabilities of all second candidate corpus fragments. One fragment is randomly selected from the second candidate corpus fragments according to the probability distribution as a matching corpus fragment. This matching corpus fragment is then removed from the second candidate corpus fragments, and the probability distribution is recalculated. Based on the new probability distribution, another second candidate corpus fragment is randomly selected as a matching corpus fragment. This random selection is repeated until the number of matching corpus fragments reaches M, where M is a positive integer not less than 3 and M is less than N.

7. An interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The process of generating interactive data for the digital twin of the child based on the matched corpus fragments includes: The matching knowledge fragments are polished by a large language model to generate conversational text, which is then synthesized into childlike speech by a voiceprint adaptive TTS engine. Simultaneously, the lip movements, facial expressions, and body movements of the digital human are driven to achieve multimodal alignment.

8. An interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The calculation process for the interaction reward value includes: The interaction performance data includes the duration of emotional relief, the number of smile detections, and the number of times interaction was rejected. The formula for calculating the interaction reward value is: ,in These are empirical weighting coefficients. For the duration of emotional relief, Number of smile detections To refuse the number of interactions; The process of adjusting the retrieval weights includes: If the retrieval weight is increased, the formula for calculating the change in retrieval weight is: Change = Preset learning rate * Interaction reward value * (1 - Current retrieval weight); If the retrieval weight is reduced, the formula for calculating the change in retrieval weight is: Change = Preset learning rate * |Interaction reward value| * Current retrieval weight.

9. An interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: The reinforcement learning adaptive module also includes: Interaction reward values ​​greater than 0 are recorded as positive rewards, and interaction reward values ​​less than or equal to 0 are recorded as negative rewards; If there are P consecutive positive rewards, the emotional intensity threshold is increased by adjusting the step size according to the preset value; if there are P consecutive negative rewards, the emotional intensity threshold is decreased by adjusting the step size according to the preset value, where P is a positive integer not less than 3.

10. An interactive digital human companion system for elderly people with cognitive impairment according to claim 1, characterized in that: Also includes: The safety alarm module is used to acquire the body posture data of the target elderly person and analyze the shoulder, elbow and wrist key points through the human body key point posture recognition model to detect abnormal posture events; when the abnormal posture event is detected to last for a preset second time threshold or the emotional intensity value exceeds the emotional intensity threshold and lasts for a preset third time threshold, a multi-channel safety alarm is triggered. The multi-channel security alarm includes pushing alarm information to children's mobile devices, nurse stations, and smart call devices, and issuing prompts on-site by a digital twin.

Images