Multi-modal emotion large model interaction method, system, device and medium for emotional companionship for the elderly

Abstract

This invention discloses a multimodal emotional big data model interaction method, system, device, and medium for emotional companionship with the elderly, relating to the field of artificial intelligence. It includes: processing the elderly person's current speech, facial video, and text during the interaction process using a multimodal emotional big data model to obtain current text semantic cues, current speech emotional cues, and current facial expression cues; inferring the elderly person's emotional state based on these cues and historical human-computer interaction information; determining whether to trigger proactive question-based emotional companionship based on the elderly person's emotional state; and, if triggered, generating proactive question text, converting it into the robot's current speech, and playing it to complete the interaction between the emotional companionship robot and the elderly person, thereby improving the initiative and accuracy of emotional companionship for the elderly.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a multimodal emotional big model interaction method, system, device and medium for emotional companionship for the elderly. Background Technology

[0002] As the aging trend deepens, the number of elderly people living alone, in empty nests, or whose children are not around for extended periods is increasing. In their daily lives, the elderly not only have needs such as access to information, reminders, and health care, but also have more long-term and implicit needs for emotional companionship, including daily chats, loneliness relief, family connection, sharing memories, being listened to, and being cared for.

[0003] However, current smart elderly care systems primarily focus on functions such as security alarms, health monitoring, location tracking, and medication reminders. While related smart speakers or voice assistants can provide simple question-and-answer sessions and music playback, they typically rely on users' active activation and explicit commands. Furthermore, the emotional expression of the elderly is unique; for example, some elderly people do not directly say "I am lonely" but indirectly express their emotional state through other forms of expression. Current smart elderly care systems only recognize information based on single verbal text content, easily overlooking the actual emotional signals displayed by the elderly. Consequently, the accuracy and initiative of current emotional companionship for the elderly are relatively low. Summary of the Invention

[0004] Based on the above-mentioned technical problems, the present invention provides a multimodal emotional big model interaction method, system, device and medium for emotional companionship for the elderly, aiming to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of this invention provides a multimodal emotional large-scale model interaction method for emotional companionship of the elderly, applied to an emotional companion robot, the method comprising: Simultaneously collect the elderly person's current voice and corresponding current facial video, and obtain historical human-computer interaction information with the elderly person; Input the current elderly person's voice, current facial video, and current elderly person's text corresponding to the current elderly person's voice into the multimodal emotion model to obtain the current text semantic clues, current voice emotion clues, and current facial expression clues; The emotional state of the elderly is determined based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information. Based on the elderly person's emotional state, determine whether to trigger proactive questioning-based emotional support; When proactive questioning-based emotional companionship is triggered, proactive question text is generated and used as the current robot text; Convert the current robot text into the current robot speech and play it.

[0006] A second aspect of this invention provides a multimodal emotional large-scale interactive system for emotional companionship of the elderly, applied to an emotional companion robot, the system comprising: The data acquisition module is used to simultaneously collect the current voice and corresponding facial video of the elderly, and to obtain historical human-computer interaction information with the elderly. The data analysis module is used to input the current elderly person's voice, current facial video, and current elderly person's text corresponding to the current elderly person's voice into the multimodal emotion model to obtain the current text semantic clues, current voice emotion clues, and current facial expression clues. The state determination module is used to determine the emotional state of the elderly based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information. The proactive companionship module is used to determine whether to trigger proactive questioning-based emotional companionship based on the elderly person's emotional state. The first generation module is used to generate proactive question text and use it as the current robot text when proactive questioning-based emotional companionship is triggered. The voice playback module is used to convert the current robot's text into the current robot's voice and play it.

[0007] A third aspect of the present invention provides an electronic device, the electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, the steps of the multimodal emotional big model interaction method for emotional companionship for the elderly as described in the first aspect of the present invention are implemented.

[0008] A fourth aspect of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the multimodal emotional big model interaction method for emotional companionship for the elderly as described in the first aspect of the present invention.

[0009] In the multimodal emotional big data interaction method for emotional companionship of the elderly proposed in this invention, firstly, the multimodal emotional big data model is pre-trained to process the current elderly person's speech, current facial video, and current elderly person's text during the interaction process to obtain current text semantic cues, current speech emotion cues, and current facial expression cues. Then, the multimodal emotional big data model infers the elderly person's emotional state based on the current text semantic cues, current speech emotion cues, current facial expression cues, and historical human-computer interaction information. Finally, based on the elderly person's emotional state, it is determined whether to trigger proactive question-based emotional companionship. If proactive question-based emotional companionship is triggered, proactive question text is generated, converted into the current robot speech, and played to complete the interaction between the emotional companionship robot and the elderly person. Thus, this invention no longer relies solely on single-session verbal text for identification. Instead, it utilizes multimodal information such as facial video, voice, verbal text, historical interactions, and life context to proactively identify the emotional state of the elderly based on continuous emotional signals such as tone of voice, facial expressions, changes in historical interactions, and duration of silence contained within this multimodal information. This improves the accuracy of emotional state identification for the elderly. Furthermore, based on the emotional state of the elderly, it determines whether to trigger proactive questioning-based emotional companionship, further transforming the emotional recognition results into proactive greetings and other companionship behaviors. This eliminates the need for the user's active wake-up call or explicit instructions, expanding the interaction from single-session question-and-answer sessions to continuous companionship. This enhances the initiative of emotional companionship for the elderly and increases the practical value of the multimodal emotional model in elderly care companionship scenarios. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the steps of a multimodal emotional big model interaction method for emotional companionship for the elderly, as shown in an embodiment of the present invention. Figure 2 This is a structural block diagram of a multimodal emotional large-scale interactive system for emotional companionship of the elderly, provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] This invention proposes a multimodal emotional large-scale model interaction method for emotional companionship of the elderly, and this method is applied to emotional companion robots. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal emotional large-scale interaction method for emotional companionship of the elderly, as shown in an embodiment of the present invention. Figure 1 As shown, the multimodal emotional big model interaction method for emotional companionship for the elderly provided in this embodiment includes at least the following steps: Step S11: Synchronously collect the current voice and corresponding facial video of the elderly person, and obtain historical human-computer interaction information with the elderly person.

[0014] In this embodiment, the emotional companion robot is a robot specifically designed for long-term interaction with the elderly. During interaction, the emotional companion robot can simultaneously collect the elderly person's current voice and corresponding facial video, and obtain historical human-computer interaction information between the robot and the elderly person. In this embodiment, the elderly person is the target user of the emotional companion robot, referring to an elderly user with whom the robot has long-term interaction, identified through methods such as account binding, device binding, wake-up confirmation, or user self-reporting. The current elderly person's voice is the voice emitted by the elderly person during the current interaction, and the current facial video is a facial video captured during the current interaction. The current elderly person's voice and corresponding facial video are collected synchronously, meaning their time steps are aligned. The historical human-computer interaction information in this embodiment is the human-computer interaction information collected between the emotional companion robot and the elderly person during historical interactions. This information includes: the elderly person's voice, the elderly person's facial video, the elderly person's text corresponding to the voice, the robot's text, and contextual information about the elderly person's life scenario.

[0015] Step S12: Input the current elderly person's voice, current facial video, and current elderly person's text corresponding to the current elderly person's voice into the multimodal emotion model to obtain the current text semantic clues, current voice emotion clues, and current facial expression clues.

[0016] In this embodiment, a multimodal emotion model is pre-trained and deployed in the emotional companion robot. This trained multimodal emotion model is used to infer the emotional state of the elderly based on the collected multimodal information during interactions, and to generate corresponding companionship responses based on that emotional state. In one embodiment, the multimodal emotion model is trained based on multimodal interaction data samples of the elderly, corresponding emotional state samples, and companionship response samples.

[0017] It is understood that the current elderly person's voice, current facial video, current elderly person's text, and historical human-computer interaction information in this embodiment all belong to the multimodal interaction data of the elderly person during the interaction process. Furthermore, the various "samples" in this embodiment represent the relevant data used during the training of the multimodal emotion model.

[0018] After acquiring the speech of the elderly person in this embodiment, the speech can be converted (e.g., through ASR conversion) to obtain the corresponding language text information, which serves as the text for the elderly person's speech. Then, the elderly person's speech, facial video, and text are input into a pre-trained multimodal emotion model. The multimodal emotion model processes the speech to obtain current speech emotion cues, processes the facial video to obtain current facial expression cues, and processes the text to obtain current text semantic cues. The current text semantic cues, speech emotion cues, and facial expression cues are respectively the text semantic cues, speech emotion cues, and facial expression cues obtained during the current interaction.

[0019] Step S13: Determine the emotional state of the elderly person based on the current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information.

[0020] In this embodiment, historical human-computer interaction information can be analyzed to determine the long-term interaction status of the elderly person. For example, it can be determined whether the elderly person's communication frequency has decreased over a period of time, whether they remain silent continuously, whether they frequently repeat certain topics, or whether they repeatedly mention family members or past experiences. The long-term interaction status in this embodiment includes at least one of the following: interaction frequency within a preset time period, duration of silence, number of proactive questions, repeated topics, emotional state change trends, preferred chat topics, and family interaction records.

[0021] In this embodiment, emotional inference is performed based on current text semantic cues, current voice emotion cues, current facial expression cues, and the elderly person's long-term interaction status to obtain the elderly person's emotional state. The elderly person's emotional state includes at least one of the following: calm, joy, loneliness, depression, anxiety, irritability, longing, nostalgia, fatigue, silence, and a need for communication.

[0022] Step S14: Determine whether to trigger proactive questioning-based emotional support based on the elderly person's emotional state.

[0023] In this embodiment, an emotional companionship response tailored to the elderly person can be generated based on their determined emotional state. The decision to trigger proactive question-based emotional companionship is then determined based on this response. The emotional companionship response includes one or more of the following: proactive greeting response, casual conversation response, reminiscing response, loneliness-comforting response, family connection suggestion response, family member message prompt response, schedule reminder response, and follow-up care prompt response. If the generated emotional companionship response for the elderly person includes a proactive greeting response, then proactive question-based emotional companionship is triggered.

[0024] Step S15: When proactive questioning-based emotional companionship is triggered, generate proactive question text and use it as the current robot text.

[0025] In this embodiment, when it is determined that proactive questioning-based emotional companionship has been triggered, the robot can generate proactive question text based on a proactive greeting response strategy, and use the generated proactive question text as the current robot text. The current robot text refers to the text reply, text question, or reminder text generated by the robot during the current interaction.

[0026] In this embodiment, corresponding response strategies can be pre-set for various emotional companionship responses, such as generating a response strategy for an initiative greeting response, generating a response strategy for a casual conversation response, and so on. The response strategy includes at least one of the following: response tone, response method, response topic, and response length.

[0027] Step S16: Convert the current robot text into the current robot speech and play it.

[0028] In this embodiment, after obtaining the current robot text, the current robot text can be converted into the current robot voice through TTS and the current robot voice can be played to complete the personalized interaction between the robot and the elderly.

[0029] In this embodiment, instead of relying solely on single-session verbal text content for recognition, multimodal information such as facial video, voice, verbal text, historical interactions, and life context is utilized. Based on continuous emotional signals such as tone of voice, facial expressions, changes in historical interactions, and duration of silence contained within the multimodal information, the emotional state of the elderly is proactively identified, improving the accuracy of emotional state recognition. Furthermore, based on the emotional state of the elderly, it is determined whether to trigger proactive questioning-based emotional companionship, further transforming the emotional recognition results into proactive greetings and other companionship behaviors. This eliminates the need for the user's active wake-up call or explicit instructions, expanding from single-session question-and-answer interactions with the elderly to continuous companionship, enhancing the initiative of emotional companionship for the elderly, and increasing the practical value of the multimodal emotional model in elderly care companionship scenarios.

[0030] In conjunction with the above embodiments, in one implementation, the present invention also provides a multimodal emotional large-scale model interaction method for emotional companionship for the elderly. In addition to the steps described above, this method may further include step S21: Step S21: Without triggering proactive questioning-based emotional support, generate a response text for the current elderly person's text and use it as the current robot text.

[0031] In this embodiment, if the generated emotional companionship response for the elderly does not include an active greeting response, and it is determined that an active question-based emotional companionship will not be triggered, then a response text for the current elderly person's text can be generated based on the response strategy of the generated emotional companionship response for the elderly person, and the generated response can be used as the current robot text.

[0032] For example, if the generated emotional companionship response for the elderly does not include an initiating greeting but includes a casual conversation response, chat topics and / or companion-style question text can be generated based on the current elderly person's text as the current robot text; if the generated emotional companionship response for the elderly does not include an initiating greeting but includes a reminiscing / exchanging response, reminiscing / exchanging topics and / or family story guidance text can be generated based on the current elderly person's text as the current robot text; if the generated emotional companionship response for the elderly does not include an initiating greeting but includes a loneliness-relief response, comforting statements, relaxation guidance text, and / or family contact suggestions text can be generated based on the current elderly person's text as the current robot text. These are just some examples, and the comparison is not limited.

[0033] In conjunction with any of the above embodiments, the present invention also provides a multimodal emotional large-scale interaction method for emotional companionship for the elderly. In this method, step S12 may specifically include the following steps S31 to S35: Step S31: Process the current facial video using a multimodal emotion model to obtain the current facial expression trajectory.

[0034] In this embodiment, the robot can process the current facial video using a multimodal emotion model, representing the facial dynamics in the current facial video as frame-by-frame facial motion unit trajectories to obtain the current facial expression trajectory. This current facial expression trajectory is used to identify fine-grained expression evidence such as changes in the corners of the mouth, tension in the eyebrows, eye state, and cheek changes.

[0035] Step S32: Analyze the changes in the current facial expression trajectory and identify the facial expression with the largest change as the salient expression.

[0036] In this embodiment, the multimodal emotion model determines the amount of change in the current facial expression trajectory and analyzes the changes, identifying the facial expression with the largest change as the salient expression. The amount of change in this embodiment includes at least one of the following: changes in the corners of the mouth, the amount of eyebrow tightening, the amount of eye changes, and the amount of cheek changes; and the amount of change is the amount of change between preset frame intervals, which can be freely set, such as 0 frames, 1 frame, 3 frames, 5 frames, etc. Specifically, when the preset frame interval is 0 frames, the amount of change in this embodiment represents the amount of change between two adjacent frames, such as the amount of eye changes between any two adjacent video frames; when the preset frame interval is 3 frames, the amount of change in this embodiment represents the amount of change between any two video frames separated by 3 frames.

[0037] Step S33: Take the facial video frame corresponding to the significant expression in the current facial video as the current expression cue.

[0038] In this embodiment, after determining the salient expression in the current facial video, the next video frame corresponding to the salient expression is determined as the facial video frame corresponding to the salient expression, and the facial video frame corresponding to the salient expression in the current facial video is determined as the current expression cue. For example, based on the 3rd and 5th frames in the current facial video, the largest change in the corner of the mouth is determined, and the facial expression with the largest change in the corner of the mouth is determined to be sadness and a salient expression. At this time, the facial video frame corresponding to the salient expression (i.e., the 5th frame in the current facial video) is taken as the current expression cue.

[0039] Step S34: Use a multimodal emotion model to identify whether the current elderly text contains expressions of loneliness, reminiscence, family-related expressions, expressions of worry, and expressions of need for companionship, thereby obtaining semantic clues for the current text.

[0040] In this embodiment, the multimodal emotion big data model can identify whether the current text of the elderly contains expressions of loneliness, reminiscence, family-related expressions, expressions of worry, and expressions of need for companionship, and determine the identification results as semantic cues of the current text. For example, in one example, the semantic cue of the current text is: it only includes reminiscence, and the reminiscence is "It used to be so lively at home during the Spring Festival, XX was still making dumplings."

[0041] Step S35: Determine the speech rate, pitch, pauses, volume, and tone changes of the current elderly person's speech through a multimodal emotion model to obtain the current speech emotion cues.

[0042] In this embodiment, the multimodal emotion big data model can perform emotional semantic analysis on the current elderly person's speech, and obtain the speech rate, pitch, pauses, volume and tone changes of the current elderly person's speech, which can be used as emotional cues of the current speech.

[0043] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional large-scale model interaction method for emotional companionship for the elderly. In this method, step S13 may specifically include steps S41 to S44: Step S41: Using a multimodal emotion model, reason about the semantic cues of the current text, the emotional cues of the current voice, and the historical human-computer interaction information to obtain the emotional state output by the multimodal emotion model.

[0044] In this embodiment, the historical human-computer interaction information is first analyzed to determine the long-term interaction state of the elderly through a multimodal emotion model. Then, the multimodal emotion model is used to perform emotion reasoning on the semantic cues of the previous text, the current voice emotion cues, and the long-term interaction state of the elderly to obtain the emotion state output by the multimodal emotion model.

[0045] Step S42: Verify whether the emotional state output by the multimodal emotion model matches the current facial expression cue.

[0046] In this embodiment, the determined current facial expression cue in the multimodal emotion model serves as a reasoning constraint, ensuring consistency between emotion prediction and observable facial behavior. This verifies whether the emotional state output by the multimodal emotion model matches the current facial expression cue. Specifically, the match between the emotional state output by the multimodal emotion model and the current facial expression cue can be determined based on whether the emotional state is reasonably expressed as a manifestation of the current facial expression cue.

[0047] Step S43: If the emotional state output by the multimodal emotion model does not match the current facial expression cue, update the multimodal emotion model until the emotional state output by the updated multimodal language model matches the current facial expression cue.

[0048] In this embodiment, if the emotional state output by the multimodal emotion model is determined to be reasonable in relation to the current facial expression cue, then the emotional state output by the multimodal emotion model is determined to match the current facial expression cue. For example, if the emotional state output by the multimodal emotion model is depressed, and the prominent expression corresponding to the current facial expression cue is sadness, then the emotional state output by the multimodal emotion model is determined to be reasonable in relation to the current facial expression cue. When the emotional state output by the multimodal emotion model matches the current facial expression cue, the emotional state output by the multimodal emotion model can be directly determined as the emotional state of the elderly person.

[0049] In this embodiment, if the emotional state output by the multimodal emotion model is determined to be unreasonable in relation to the current facial expression cue, then the emotional state output by the multimodal emotion model is determined to be mismatched with the current facial expression cue. For example, if the emotional state output by the multimodal emotion model is loneliness, and the salient expression corresponding to the current facial expression cue is happiness, then the emotional state output by the multimodal emotion model is determined to be unreasonable in relation to the current facial expression cue. In cases where the emotional state output by the multimodal emotion model does not match the current facial expression cue, this embodiment can update the multimodal emotion model until the emotional state output by the updated multimodal language model matches the current facial expression cue, thus obtaining the last updated multimodal language model.

[0050] Step S44: Determine the emotional state output by the last updated multimodal large language model as the emotional state of the elderly.

[0051] In this embodiment, the current text semantic cues, current voice emotion cues, and current facial expression cues can be determined by the last updated multimodal large language model. The current text semantic cues, current voice emotion cues, and historical human-computer interaction information are used to infer the emotional state output by the last updated multimodal large language model, so as to determine the emotional state output by the last updated multimodal large language model as the emotional state of the elderly.

[0052] In this embodiment, the current expression cues generated by the facial motion unit trajectory are introduced into the latent inference stage of the multimodal emotion model. Significant visual evidence is retained before the emotion state is generated, so that the emotion prediction is consistent with the observable facial behavior. This can improve the recognition accuracy of states such as loneliness, depression, anxiety, and longing. Thus, for the companionship scenario of the elderly, fine-grained emotion states such as loneliness, depression, longing, and need for communication of the elderly can be more accurately identified.

[0053] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional large-scale model interaction method for emotional companionship of the elderly. In this method, in addition to the steps described above, it may also include step S51, and the "generating proactive question text" in step S15 may specifically include step S52, the "generating reply text for the current elderly person's text" in step S21 may specifically include step S53, and step S16 may specifically include step S54. Step S51: Based on historical human-computer interaction information, determine the elderly person's expression habits, listening comprehension ability, preferred topics, and the most recent interaction conversation with the elderly person's current speech.

[0054] In this embodiment, the robot can also determine the elderly person's expression habits, hearing comprehension ability, preferred topics, and the most recent interaction between the robot and the elderly person based on their historical human-computer interaction information, so as to make the robot's relevant responses more age-friendly.

[0055] Step S52: Generate an active question text based on the elderly person's preferred topics and the most recent interactive conversation with the elderly person's voice at the time interval.

[0056] In this embodiment, interaction preferences and context can be determined based on the elderly person's preferred topics and the most recent interaction session with the elderly person's voice. Then, based on the interaction preferences, context and response strategy of proactive greeting, proactive question text can be generated, thereby prioritizing the use of topics familiar to the elderly person and proactively generating continuous care prompts when there is continuous silence or multiple negative emotions.

[0057] Step S53: Based on the current elderly person's text, combined with the elderly person's preferred topics and the most recent interactive conversation with the current elderly person's voice, generate a reply text for the current elderly person's text.

[0058] In this embodiment, a response text can be generated based on the elderly person's preferred topics and the most recent interactive conversation with the elderly person's voice, prioritizing topics familiar to the elderly person to achieve age-friendly processing.

[0059] Step S54: Convert the current robot text into the current robot speech according to the elderly person's expression habits and hearing comprehension ability, and play it.

[0060] In this embodiment, the robot can convert the generated current robot text into current robot speech and play it according to the elderly person's expression habits and hearing comprehension ability. For example, if the elderly person's expression habits are polite or gentle, a clear, polite, and gentle expression can be used to convert the generated current robot text into current robot speech; if the elderly person's hearing comprehension ability is weak, a slower speaking speed or shorter sentence structures and avoidance of complex terminology can be used to convert the generated current robot text into current robot speech.

[0061] In this embodiment, the robot-generated answers have been processed to be age-friendly, making them more suitable for the understanding, communication, expression, and emotional acceptance habits of the elderly.

[0062] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional large-scale model interaction method for emotional companionship for the elderly. In addition to the steps described above, this method may further include steps S61 to S63: Step S61: If the current text semantic clues contain expressions of recollection or expressions related to family members, record the subsequent human-computer interaction information of the current elderly person's speech.

[0063] In this embodiment, when the robot determines that the current text semantic clues contain expressions of recollection or related to family members, it can record subsequent human-computer interaction information of the elderly person's speech. This subsequent human-computer interaction information refers to the interaction information between the elderly person and the robot that occurs after the current speech.

[0064] Step S62: Generate a family memoir based on the current elderly person's text and voice, and subsequent human-computer interaction information.

[0065] In this embodiment, text related to the topics of "family affection" and / or "family members" can be extracted from the current elderly person's text and subsequent human-computer interaction information of the current elderly person's voice in order to generate a family affection memoir.

[0066] Step S63: Send the family memoir to the elderly person's corresponding family member's user terminal.

[0067] In this embodiment, the elderly person has pre-linked a corresponding family member's user terminal to the robot and authorized the family member's user terminal to receive relevant messages sent by the robot under specific conditions. After generating a family memoir, the robot can send the memoir to the elderly person's corresponding family member's user terminal.

[0068] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional big-model interaction method for emotional companionship for the elderly. In this method, step S14 may specifically include step S71, and, in addition to the above steps, may also include steps S72 to S73: Step S71: When the elderly person's emotional state is loneliness, silence, or a need for communication, identify the trigger for proactive questioning-based emotional companionship.

[0069] In this embodiment, when the emotional state of the elderly is determined to be loneliness, silence, or a need for communication, an emotional companionship response for the elderly is generated, including: proactive greeting response, casual conversation response, and loneliness comforting response. Thus, it is determined that the generated emotional companionship response for the elderly includes a proactive greeting response, and proactive questioning-based emotional companionship is triggered.

[0070] Step S72: Based on multiple interactive conversations with the elderly, determine the number of times to trigger proactive questioning-based emotional companionship within a preset time period.

[0071] In this embodiment, the robot can record its interactions with the elderly person and, based on these interactions, determine the number of times it will trigger proactive, question-based emotional companionship within a preset time period. The preset time period is a customizable duration, such as 12 hours, 4 hours, etc.

[0072] Step S73: If the number of times proactive questioning-based emotional companionship is triggered within a preset time period is greater than the preset number, then a family member attention suggestion is sent to the family member user terminal corresponding to the elderly person.

[0073] In this embodiment, a preset number of times is set. This preset number represents the maximum number of times proactive questioning-based emotional companionship can be triggered within a preset time period. This preset number can be customized, such as 2 times, 5 times, etc. When the emotional companionship robot determines that the number of times proactive questioning-based emotional companionship is triggered within the preset time period exceeds the preset number, it sends a family member attention suggestion to the elderly person's corresponding family member user terminal to remind the family member to provide relevant communication and companionship to the elderly person, such as reminding the family member to meet with the elderly person, make a phone call, send a text message, etc.

[0074] In addition, in one embodiment, if the robot detects that an elderly person is silent for a long time, is persistently depressed, frequently expresses loneliness, or repeatedly requests family members to contact them, it can generate suggestions for family members to pay attention to and send them to the corresponding family member user terminal.

[0075] In a specific example, a scenario for alleviating depression is illustrated: an elderly person repeatedly expresses "It's meaningless" and "Nobody's talking to me," and facial cues indicate a depressed expression, while vocal cues indicate a low voice volume. The robot determines that the person may be in a depressed and lonely state and generates a loneliness-alleviating response: "I'm here to keep you company. We can talk slowly, no rush. Would you like to listen to a familiar old song first, or would you like to tell me what happened today?" Simultaneously, if this state is detected to persist, suggestions for family members to pay attention to can be generated and sent to the family member's user terminal.

[0076] In conjunction with any of the above embodiments, in one embodiment, based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information, in addition to determining the emotional state of the elderly, their companionship needs can also be determined. These companionship needs include at least one of the following: needs for proactive greetings, needs for conversational companionship, needs for comforting loneliness, needs for reminiscing and exchanging memories, needs for emotional connection with family, needs for schedule reminders, needs for health care, and needs for continuous attention.

[0077] In addressing the companionship needs of the elderly, this can be achieved by determining the elderly's long-term interaction status based on historical human-computer interaction information, and then performing emotional inference based on current text semantic cues, current voice emotional cues, current facial expression cues, and the elderly's long-term interaction status to arrive at their companionship needs. Specifically, current facial expression cues can be introduced into the latent inference stage of a multimodal emotional model to constrain its output, ensuring that the companionship needs predicted by the multimodal emotional model based on current text semantic cues, current voice emotional cues, and long-term interaction status match the current facial expression cues, thus obtaining the elderly's companionship needs. In this way, considering the unique emotional expressions of the elderly (such as repeatedly mentioning their children, prolonged silence, slower speech, depressed expression, or frequent recollections of the past to demonstrate a need for companionship), this embodiment can proactively identify emotional needs based on the elderly's tone of voice, facial expression, duration of silence, and changes in historical interactions.

[0078] In one embodiment, an emotional companionship response can be generated based on the elderly person's emotional state and / or companionship needs. In a specific example, when the elderly person's emotional state is loneliness or silence, an initiative greeting response and / or a small talk response are generated; when the elderly person's emotional state is depression or anxiety, a small talk response is generated; when the elderly person's emotional state is nostalgia or longing, a reminiscing and communication response is generated; when the elderly person's companionship needs are schedule reminders or health care needs, a schedule reminder response and / or follow-up care prompts are generated, responding to the elderly person based on gentle reminders, confirmatory questions, or follow-up care statements.

[0079] In a specific example, a scenario of companionship for the elderly is illustrated: An elderly person's interactions with the emotional companionship robot significantly decrease over a period of time, and on that day, they only say, "It's quite quiet at home today." The emotional companionship robot acquires the elderly person's speech, the corresponding text content, and historical human-computer interaction information. Speech analysis shows a slow speaking speed and long pauses; historical interaction analysis shows a recent decrease in the frequency of proactive communication; combining this information, it is determined that the elderly person's emotional state may be one of loneliness or a need for communication, with companionship requirements including proactive greetings and conversational interaction. At this point, a proactive question-based emotional companionship is triggered, and the robot generates a proactive greeting response: "It's a bit quiet today, let me chat with you for a while. Would you like to listen to the news today, or would you like to talk about things from your family in the past?" This response is not a simple "yes," but rather proactively provides an entry point for companionship to proactively care for the elderly person, enhancing the emotional companionship robot's ability to provide continuous companionship.

[0080] In a specific example, a scenario of nostalgia and family connection is illustrated: an elderly person mentions, "Our home used to be so lively during Chinese New Year," while their expression softens but their tone is hesitant. The robot recognizes that their emotional state may include nostalgia and longing, and that the elderly person's needs for companionship are the need for reminiscing and exchanging memories, as well as the need for family connection. The robot can then generate a reminiscing response: "It sounds like it must have been very lively back then. Do you remember who was the best at preparing the New Year's Eve dinner that year? Would you like me to help you organize this memory into a short passage and send it to your family?" This transforms the ordinary chat between the elderly person and the robot into a family memoir sent to the corresponding family member's terminal, thus achieving reminiscing companionship and family connection.

[0081] In one embodiment, multimodal input from the elderly can be collected to obtain multimodal information of the elderly during the interaction process (i.e., to obtain multimodal input from the elderly). The multimodal information includes: facial video, speech, language text, historical interactions, and life context. The multimodal information is preprocessed and then input into a multimodal emotion model. The multimodal emotion model extracts the facial motion unit trajectory of the elderly based on the facial video and performs expression change analysis to obtain expression cues. The multimodal emotion model performs speech intonation and language semantic analysis on the speech and language text to obtain speech emotion cues and language semantic cues. Furthermore, the multimodal emotion model performs long-term interaction state analysis on historical interaction information and life context information to obtain long-term interaction state, wherein the long-term interaction state includes at least: silence duration, communication frequency, and preferred topics. Next, the multimodal emotion model performs latent spatial emotion inference based on facial expression cues, vocal emotion cues, linguistic semantic cues, and long-term interaction states to identify the emotional state and companionship needs of the elderly, thus obtaining their emotional state and companionship needs. The emotional state includes at least: calmness, loneliness, depression, anxiety, joy, and longing; the companionship needs include chatting, comforting, reminiscing, emotional connection with family, reminders, and continuous care. Finally, based on the elderly's emotional state and companionship needs, an age-friendly companionship response is generated, and based on this response, multiple rounds of care are provided to the elderly and / or suggestions for family contact are generated. This companionship response includes at least one of the following: proactive greetings, conversational companionship, loneliness comfort, reminiscing, family contact suggestions, schedule reminders, and continuous care.

[0082] Thus, this embodiment models long-term emotional changes and identifies them by combining historical interaction states. This allows for the identification of high-frequency emotional needs in the elderly, such as loneliness, longing, silence, depression, anxiety, and the need for communication. It avoids misinterpreting short sentences, pauses, or silence as a lack of need. Furthermore, it transforms the emotional recognition results into companionship behaviors such as proactive greetings, conversations, reminiscences, and emotional connections. This enables the generation of family attention suggestions when the elderly are persistently silent, chronically depressed, or frequently express loneliness, thereby enhancing emotional connections and continuous care. This improves the accuracy, initiative, and affinity of the multimodal emotional model in scenarios such as alleviating loneliness, providing companionship, connecting with family, and daily care for the elderly, thus enhancing the practical value of the multimodal emotional model in elderly care companionship scenarios.

[0083] It should be noted that the robot of the present invention (i.e., the emotional companion robot) can be a desktop companion robot, a smart speaker-shaped robot, a robot with a screen, a mobile robot, or a robot system that is linked with a mobile application.

[0084] It should also be noted that all privacy-related data, such as "interaction" and "conversation," in this invention were obtained with the user's consent and comply with relevant laws, regulations, and privacy requirements.

[0085] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0086] Based on the same inventive concept, one embodiment of this invention provides a multimodal emotional large-scale interactive system for emotional companionship of the elderly, applied to emotional companion robots. (Reference) Figure 2 , Figure 2 This is a structural block diagram of a multimodal emotional large-scale interactive system for emotional companionship of the elderly, provided by an embodiment of the present invention. Figure 2 As shown, the system includes: The data acquisition module is used to simultaneously collect the current voice and corresponding facial video of the elderly, and to obtain historical human-computer interaction information with the elderly. The data analysis module is used to input the current elderly person's voice, current facial video, and current elderly person's text corresponding to the current elderly person's voice into the multimodal emotion model to obtain the current text semantic clues, current voice emotion clues, and current facial expression clues. The state determination module is used to determine the emotional state of the elderly based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information. The proactive companionship module is used to determine whether to trigger proactive questioning-based emotional companionship based on the elderly person's emotional state. The first generation module is used to generate proactive question text and use it as the current robot text when proactive questioning-based emotional companionship is triggered. The voice playback module is used to convert the current robot's text into the current robot's voice and play it.

[0087] Optionally, the system further includes: The second generation module is used to generate a response text for the current elderly person's text and use it as the current robot text without triggering proactive questioning-based emotional companionship.

[0088] Optionally, the data analysis module includes: The first processing module is used to process the current facial video using a multimodal emotion model to obtain the current facial expression trajectory; The expression determination module is used to analyze the amount of change in the current facial expression trajectory and identify the facial expression with the largest change as the salient expression. The amount of change includes at least one of the following: change in the corner of the mouth, tightening of the eyebrows, change in the eyes, and change in the cheeks. The first determining module is used to take the facial video frame corresponding to the significant expression in the current facial video as the current expression cue; The second determination module is used to identify whether the current elderly text contains expressions of loneliness, reminiscence, family-related expressions, expressions of worry, and expressions of companionship needs through a multimodal emotion model, thereby obtaining semantic clues of the current text. The third determination module is used to determine the speech rate, pitch, pauses, volume, and tone changes of the current elderly person's speech through a multimodal emotion model, thereby obtaining the current speech emotion cues.

[0089] Optionally, the state determination module includes: The first output module is used to reason about the current text semantic cues, current voice emotion cues, and historical human-computer interaction information through the multimodal emotion big model, and obtain the emotion state output by the multimodal emotion big model. The verification module is used to verify whether the emotional state output by the multimodal emotion model matches the current facial expression cue. The model update module is used to update the multimodal emotion model when the emotional state output by the multimodal emotion model does not match the current facial expression cues, until the emotional state output by the updated multimodal language model matches the current facial expression cues. The second output module is used to determine the emotional state of the elderly based on the emotional state output by the last updated multimodal large language model.

[0090] Optionally, the system further includes: The fourth determination module is used to determine the elderly person's expression habits, hearing comprehension ability, preferred topics, and the most recent interaction conversation with the elderly person's voice based on historical human-computer interaction information. The first generation module includes: The first friendly generation module is used to generate proactive question text based on the elderly person's preferred topics and the most recent interactive conversation with the elderly person's voice at the time interval. The second generation module includes: The second friendly generation module is used to generate a reply text for the current elderly person's text based on the elderly person's preferred topics and the most recent interactive conversation with the current elderly person's voice at the same time interval. The voice playback module includes: The user-friendly playback module is used to convert the current robot text into the current robot speech and play it according to the elderly's expression habits and hearing comprehension ability.

[0091] Optionally, the system includes: The recording module is used to record subsequent human-computer interaction information of the elderly person's speech when the current text semantic clues contain expressions of recollection or expressions related to family members; The third generation module is used to generate a family memoir based on the current elderly person's text and voice, and subsequent human-computer interaction information. The first sending module is used to send the family memoirs to the corresponding family member user terminal of the elderly.

[0092] Optionally, the proactive companionship module includes: The fifth module is used to determine when an elderly person's emotional state is one of loneliness, silence, or a need for communication, and to trigger proactive questioning-based emotional companionship. The system also includes: The sixth determining module is used to determine the number of times proactive questioning-based emotional companionship is triggered within a preset time period based on multiple interactive conversations with the elderly. The second sending module is used to send family member attention suggestions to the family member user terminal corresponding to the elderly person if the number of times proactive questioning-based emotional companionship is triggered within a preset time period is greater than a preset number.

[0093] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0094] The multimodal emotional big data interaction system for emotional companionship for the elderly in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The system can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment of the invention does not impose specific limitations.

[0095] The multimodal emotional big data interaction system for emotional companionship for the elderly in this embodiment of the invention can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0096] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the multimodal emotional big model interaction method for emotional companionship for the elderly as described in any of the above embodiments of the present invention.

[0097] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0098] Based on the same inventive concept, another embodiment of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the multimodal emotional big model interaction method for emotional companionship for the elderly as described in any of the above embodiments of the present invention. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0099] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0100] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0102] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A multimodal emotional large-scale model interaction method for emotional companionship of the elderly, applied to emotional companion robots, characterized in that, The method includes: Simultaneously collect the elderly person's current voice and corresponding current facial video, and obtain historical human-computer interaction information with the elderly person; Input the current elderly person's voice, current facial video, and current elderly person's text corresponding to the current elderly person's voice into the multimodal emotion model to obtain the current text semantic clues, current voice emotion clues, and current facial expression clues; The emotional state of the elderly is determined based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information. Based on the elderly person's emotional state, determine whether to trigger proactive questioning-based emotional support; When proactive questioning-based emotional companionship is triggered, proactive question text is generated and used as the current robot text; Convert the current robot text into the current robot speech and play it.

2. The multimodal emotional large-scale interaction method for emotional companionship for the elderly according to claim 1, characterized in that, The method further includes: Without triggering proactive questioning-based emotional support, generate a response text for the current elderly person's text and use it as the current robot text.

3. The multimodal emotional large-scale model interaction method for emotional companionship for the elderly according to claim 1, characterized in that, Inputting the current elderly person's voice, current facial video, and the corresponding text into a multimodal emotion model yields semantic cues for the current text, emotional cues for the current voice, and facial expression cues, including: The current facial video is processed using a multimodal emotion model to obtain the current facial expression trajectory; The changes in the current facial expression trajectory are analyzed, and the facial expressions with the largest changes are identified as significant expressions. The changes include at least one of the following: changes in the corners of the mouth, the amount of eyebrow tightening, the changes in the eyes, and the changes in the cheeks. The facial video frames corresponding to the most prominent expressions in the current facial video are used as current expression cues; By using a multimodal emotion model, we can identify whether the current text of the elderly contains expressions of loneliness, reminiscence, family-related expressions, expressions of worry, and expressions of need for companionship, and obtain semantic clues of the current text. By using a multimodal emotion model, we can determine the speech rate, pitch, pauses, volume, and tone changes of elderly people to obtain current speech emotion cues.

4. The multimodal emotional large-scale interaction method for emotional companionship for the elderly according to claim 1, characterized in that, Based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information, the emotional state of the elderly is determined, including: By using a multimodal emotion model, reasoning is performed on current text semantic cues, current voice emotion cues, and historical human-computer interaction information to obtain the emotion state output by the multimodal emotion model. Verify whether the emotional state output by the multimodal emotion model matches the current facial expression cues; If the emotional state output by the multimodal emotion model does not match the current facial expression cue, the multimodal emotion model is updated until the emotional state output by the updated multimodal language model matches the current facial expression cue. The emotional state output by the last updated multimodal large language model is identified as the emotional state of the elderly.

5. The multimodal emotional large-scale model interaction method for emotional companionship for the elderly according to claim 2, characterized in that, The method further includes: Based on historical human-computer interaction information, determine the elderly person's expression habits, listening comprehension ability, preferred topics, and the most recent interaction conversation with the elderly person's voice at the time interval. Generate proactive question text, including: Generate proactive question text based on the elderly person's preferred topics and the most recent interactive conversation with the elderly person's voice at the time interval; Generate a response text for the current elderly person's text, including: Based on the current elderly person's text, combined with the elderly person's preferred topics and the most recent interactive conversation with the current elderly person's voice, a response text is generated for the current elderly person's text. Convert the current robot text into the current robot speech and play it, including: Based on the elderly person's expression habits and hearing comprehension ability, the current robot text is converted into the current robot speech and played.

6. The multimodal emotional large-scale interaction method for emotional companionship of the elderly according to any one of claims 1 to 5, characterized in that, The method includes: When the current text semantic cues contain expressions of recollection or expressions related to family members, record the subsequent human-computer interaction information of the elderly person's current speech; Generate a family memoir based on the current text and voice of the elderly person. Send the family memoirs to the elderly person's corresponding family member's user terminal.

7. The multimodal emotional large-scale interaction method for emotional companionship for the elderly according to any one of claims 1 to 5, characterized in that, Based on the elderly person's emotional state, determine whether to trigger proactive questioning-based emotional support, including: When an older person's emotional state is characterized by loneliness, silence, or a need for communication, proactive questioning-based emotional support should be initiated. The method further includes: Based on multiple interactive conversations with the elderly, determine the number of times proactive questioning-based emotional companionship is triggered within a preset time period; If the number of times proactive questioning-based emotional support is triggered exceeds the preset number within a preset time period, a family member attention suggestion will be sent to the corresponding family member's user terminal.

8. A multimodal emotional large-scale interactive system for emotional companionship of the elderly, applied to emotional companion robots, characterized in that: The system includes: The data acquisition module is used to simultaneously collect the current voice and corresponding facial video of the elderly, and to obtain historical human-computer interaction information with the elderly. The data analysis module is used to input the current elderly person's voice, current facial video, and current elderly person's text corresponding to the current elderly person's voice into the multimodal emotion model to obtain the current text semantic clues, current voice emotion clues, and current facial expression clues. The state determination module is used to determine the emotional state of the elderly based on current text semantic cues, current voice emotion cues, current facial expression cues, and historical human-computer interaction information. The proactive companionship module is used to determine whether to trigger proactive questioning-based emotional companionship based on the elderly person's emotional state. The first generation module is used to generate proactive question text and use it as the current robot text when proactive questioning-based emotional companionship is triggered. The voice playback module is used to convert the current robot's text into the current robot's voice and play it.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the multimodal emotional big model interaction method for emotional companionship for the elderly as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the multimodal emotional big model interaction method for emotional companionship of the elderly as described in any one of claims 1 to 7.

Images