Multi-modal emotional large model interaction method, system, device and medium for children emotional accompaniment

Abstract

The application discloses a kind of multi-modal emotional big model interaction methods, systems, equipment and medium for children emotional accompaniment, it is related to artificial intelligence field.The application includes: obtaining the multi-modal information of children in the interaction process;By multi-modal emotional big model, the facial action unit analysis of children facial video is carried out, and the trend of children expression change is obtained;By multi-modal emotional big model, the emotional semantic analysis of children voice information and children language text information is carried out, and the voice emotional clue and text semantic clue are obtained;Based on the trend of children expression change, voice emotional clue and text semantic clue, emotional state reasoning is carried out by multi-modal emotional big model, and the emotional state of children and / or children accompanying demand is obtained;According to the emotional state of children and / or children accompanying demand, generate and play emotional accompaniment response for children, so as to accurately identify the emotional category and accompanying demand of children, improve the accuracy of emotional accompaniment for children.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a multimodal emotional big model interaction method, system, device and medium for emotional companionship for children. Background Technology

[0002] Related smart companion products for children typically function primarily through voice-based question and answering, story playback, nursery rhyme playback, or learning assistance. These products usually rely on voice recognition and a fixed content library for interaction, and can answer some of the questions children ask, but their ability to understand children's true emotional states and deeper needs for companionship is limited.

[0003] Therefore, how to more accurately identify children's emotional categories and companionship needs in order to improve the accuracy of emotional companionship for children is a technical problem that this invention urgently needs to solve. Summary of the Invention

[0004] Based on the above-mentioned technical problems, the present invention provides a multimodal emotional big model interaction method, system, device and medium for children's emotional companionship, aiming to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of this invention provides a multimodal emotional big data interaction method for emotional companionship with children, the method comprising: Acquire multimodal information of children during the interaction process, wherein the multimodal information includes at least two of the following: children's facial video, children's voice information, children's language text information, and interaction context information; The facial motion unit analysis of the children's facial videos was performed using a multimodal emotion big data model to obtain the trend of children's facial expression changes; The multimodal emotion model is used to perform emotional semantic analysis on the children's speech information and children's language text information to obtain speech emotion cues and text semantic cues. Based on the trends in children's facial expressions, vocal emotion cues, and textual semantic cues, the emotional state is inferred through the multimodal emotion model to obtain the children's emotional state and / or their need for companionship. Based on the child's emotional state and / or the child's need for companionship, generate and play an emotional companionship response for the child.

[0006] A second aspect of the present invention provides a multimodal emotional big data interaction system for children's emotional companionship, the system comprising: The data acquisition module is used to acquire multimodal information of children during the interaction process, including at least two of the following: children's facial video, children's voice information, children's language text information, and interaction context information; The facial expression analysis module is used to perform facial motion unit analysis on the children's facial videos using a multimodal emotion model to obtain the trend of children's facial expression changes; The data analysis module is used to perform emotional semantic analysis on the children's speech information and children's language text information through the multimodal emotion big model to obtain speech emotion cues and text semantic cues; The state reasoning module is used to perform emotional state reasoning based on the child's facial expression change trend, voice emotion cues, and text semantic cues, through the multimodal emotion big model, to obtain the child's emotional state and / or the child's companionship needs; The companionship response module is used to generate and play emotional companionship responses for children based on the child's emotional state and / or the child's companionship needs.

[0007] A third aspect of the present invention provides an electronic device, the electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, the steps of the multimodal emotional big model interaction method for children's emotional companionship as described in the first aspect of the present invention are implemented.

[0008] A fourth aspect of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the multimodal emotional big model interaction method for children's emotional companionship as described in the first aspect of the present invention.

[0009] In the multimodal emotional big model interaction method for children's emotional companionship proposed in this invention, recognition is no longer based solely on children's language and text content. Instead, it utilizes multimodal information such as children's facial videos, voice, language and text, and interactive context. Through the multimodal emotional big model, emotional state inference is performed based on the voice emotion cues, facial expression change trends, and text semantic cues contained in the multimodal information. This identifies children's emotional states and / or their companionship needs. By observing subtle changes in children's facial expressions, it can better identify children's unique fine-grained emotional states that are not easily expressed accurately through language, such as curiosity, grievance, fear, frustration, and coquetry. Furthermore, this invention not only identifies children's emotion categories but also further identifies their companionship needs. It expands from "identifying emotions" to "generating companionship behaviors" as an emotional companionship response. This allows for more accurate identification of children's fine-grained emotional states and companionship needs in scenarios where children's language expression is incomplete, their emotions change rapidly, and they have strong play needs and curiosity. It also generates companionship content that is more in line with children's psychological characteristics, improving the applicability and accuracy of emotional companionship for children. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the steps of a multimodal emotional big model interaction method for children's emotional companionship, as shown in an embodiment of the present invention. Figure 2 This is a structural block diagram of a multimodal emotional large-scale interactive system for children's emotional companionship, provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] This invention takes into account the distinct characteristics of children's emotional expression. On the one hand, children's language expression abilities are not yet fully mature, and many emotions cannot be expressed through complete and accurate language. On the other hand, when children are playing, asking questions, feeling frustrated, afraid, being affectionate, or seeking attention, they often express emotions simultaneously through facial expressions, tone of voice, actions, and dialogue. Judging emotions solely based on the written content spoken by children can easily overlook their true feelings. For example, a child might say, "I don't want to play anymore," but their voice and facial expressions might suggest affection or a desire for continued interaction; a child might not explicitly say "I'm scared," but their facial expressions, pauses, and changes in tone indicate a need for reassurance. Furthermore, related multimodal emotion recognition methods primarily rely on natural language reasoning, which can easily lose subtle facial expression evidence during the language generation process.

[0014] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes a multimodal emotion big model interaction method for children's emotional companionship. This method analyzes children's facial action units through a multimodal emotion big model to obtain the trend of children's facial expression changes. Based on the trend of children's facial expression changes, it uses latent space reasoning to make children's emotion prediction closer to observable facial behavior. This can alleviate the problem of children's subtle facial expressions being diluted in linear text descriptions, and thus better identify children's unique fine-grained emotional states that are not easily expressed accurately through language, such as curiosity, grievance, fear, frustration, and coquetry. It also expands from "identifying emotions" to "generating companionship behavior" in emotional companionship responses. In scenarios where children's language expression is incomplete, their emotions change rapidly, and they have strong play needs and curiosity, it can more accurately identify children's fine-grained emotions and companionship needs, and generate companionship content that is more in line with children's psychological characteristics, thereby improving the applicability and accuracy of emotional companionship for children.

[0015] This invention proposes a multimodal emotional large-scale model interaction method for emotional companionship with children, and this method is applied to emotional companion robots. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal emotional big-model interaction method for emotional companionship with children, as shown in an embodiment of the present invention. Figure 1 As shown, the multimodal emotional big model interaction method for children's emotional companionship provided in this embodiment includes at least the following steps: Step S11: Obtain multimodal information about the child during the interaction process.

[0016] In this embodiment, the emotional companion robot is a robot specifically designed for long-term interaction with children. During this interaction, the robot can acquire multimodal information about the child. This multimodal information includes at least two of the following: child facial video, child voice information, child language text information, and interaction context information. Specifically, the emotional companion robot can simultaneously collect the child's voice information and corresponding facial video, convert the voice information into child language text information, and acquire the interaction context information between the robot and the child. It is understood that the time steps of the child's voice information and the child's facial video acquired in this embodiment are aligned.

[0017] In this embodiment, the children are the target users of the emotional companion robot, referring to child users who have long-term interactions with the emotional companion robot, identified through methods such as account binding, device binding, wake-up confirmation, and user self-reporting. The interaction context information in this embodiment consists of human-computer interaction information collected during the historical interactions between the emotional companion robot and the child. This human-computer interaction information includes: historical child voice information, historical child facial video, historical child language text information, historical robot text, and contextual information about the child's daily life scenarios.

[0018] This embodiment inputs multimodal information into a pre-trained multimodal emotion model, which then performs multimodal encoding and extracts emotional evidence. The pre-trained multimodal emotion model is deployed in an emotional companionship robot. This trained model is used to infer the multimodal information collected from children during interactions, determining the children's emotional states and / or companionship needs, and generating child-oriented emotional companionship responses based on these needs. In one embodiment, the multimodal emotion model is trained based on samples of children's multimodal interaction data and corresponding samples of emotional states, companionship needs, and companionship responses. The various "samples" in this embodiment represent the relevant data used during the training of the multimodal emotion model.

[0019] Step S12: Perform facial motion unit analysis on the child's facial video using a multimodal emotion big data model to obtain the trend of the child's facial expression changes.

[0020] In this embodiment, after multimodal information is input into the multimodal emotion model, facial motion unit analysis can be performed on children's facial videos using the multimodal emotion model to obtain the trend of children's facial expression changes. This trend represents the dynamic changes in areas such as the eyebrows, eyes, cheeks, corners of the mouth, and lips. Compared to simple coarse-grained expression labels such as "happy" or "sad," the trend of children's facial expression changes can retain more fine-grained evidence of facial expressions.

[0021] Step S13: Perform emotional semantic analysis on the child's speech information and child's language text information using the multimodal emotion big data model to obtain speech emotion cues and text semantic cues.

[0022] In this embodiment, after multimodal information is input into the multimodal emotion model, emotional semantic analysis can be performed on children's speech information and children's language text information to obtain speech emotion cues and text semantic cues. Specifically, regarding the speech side, the multimodal emotion model can analyze children's speech information to obtain volume, speech rate, pauses, pitch variations, and emotional intensity, serving as speech emotion cues. Regarding the text content side, the multimodal emotion model can analyze children's language text information to obtain intent, question topic, emotion words, and contextual meaning, serving as text semantic cues.

[0023] Step S14: Based on the child's facial expression change trend, voice emotion cues, and text semantic cues, perform emotion state inference through the multimodal emotion big data model to obtain the child's emotional state and / or the child's companionship needs.

[0024] In this embodiment, after obtaining the child's facial expression change trends, vocal emotion cues, and textual semantic cues, the multimodal emotion model can perform emotion state inference based on these cues and combined with interaction context information to obtain the child's emotional state and / or companionship needs. Here, the child's emotional state represents the child's emotional category, and the child's companionship needs represent the category of needs the robot requires.

[0025] Step S15: Generate and play an emotional companionship response for the child based on the child's emotional state and / or the child's need for companionship.

[0026] In this embodiment, the multimodal emotion model can generate corresponding emotional companionship responses for children based on the determined emotional state and / or companionship needs of the children. Then, based on the generated emotional companionship responses, corresponding robot text is generated, converted into robot speech, and played to complete personalized interaction between the robot and the child. The robot text consists of text replies, text questions, or reminders generated by the robot during the interaction.

[0027] In one optional example, the emotional support response includes one or more of the following: game interaction response, story support response, curiosity question and answer response, encouragement to express oneself response, emotional comfort response, learning and exploration response, parent-child communication prompt response, and multi-round follow-up question response.

[0028] In this embodiment, instead of relying solely on the child's spoken text for identification, it comprehensively utilizes multimodal information such as the child's facial video, voice, spoken text, and interactive context. Through a multimodal emotion model, it infers the child's emotional state based on the phonological emotion cues, facial expression trends, and textual semantic cues contained within the multimodal information. This allows for the identification of the child's emotional state and / or companionship needs. By observing subtle changes in the child's facial expressions, it more accurately identifies the child's unique, fine-grained emotional states that are difficult to express accurately through language, such as curiosity, grievance, fear, frustration, whining, and seeking attention. Furthermore, this embodiment not only identifies the child's emotion categories but also further identifies the child's companionship needs. This extends from "identifying emotions" to "generating companionship behaviors"—an emotional companionship response. As a result, it can more accurately identify the child's fine-grained emotions and companionship needs in scenarios where the child's language expression is incomplete, emotions change rapidly, play needs are strong, and curiosity is high. It also generates companionship content that is more in line with the child's psychological characteristics, improving the applicability and accuracy of emotional companionship for children.

[0029] In conjunction with the above embodiments, in one implementation, the present invention also provides a multimodal emotional big model interaction method for emotional companionship with children, in which step S12 may specifically include steps S21 to S24: Step S21: Extract multiple frames of children's facial images from the children's facial video.

[0030] In this embodiment, the multimodal emotion big data model can first extract multiple frames of children's facial images from children's facial videos.

[0031] Step S22: Identify changes in facial muscle movements in each frame of the child's facial image.

[0032] In this embodiment, for the obtained multiple frames of children's facial images, changes in facial muscle movements in each frame can be identified. The change in facial muscle movements in a single frame refers to the amount of change in facial muscle movements in that frame compared to the previous frame. The amount of change in facial muscle movements includes at least one of the following: changes in the corners of the mouth, the amount of eyebrow tightening, the amount of eye movement, and the amount of cheek movement.

[0033] Step S23: Generate a sequence of changes in the intensity of facial muscle movements in children in chronological order.

[0034] In this embodiment, based on the changes in facial muscle movements in each frame of the child's facial image, a sequence of the intensity of changes in the child's facial muscle movements can be generated according to the temporal order of the facial image frames. This sequence of the intensity of changes in the child's facial muscle movements characterizes the direction and magnitude of the changes in the child's facial muscle movements.

[0035] Step S24: Determine the trend of the child's facial expression changes based on the sequence of intensity changes in the child's facial muscle movements.

[0036] In this embodiment, the trend of a child's facial expression changes can be obtained based on the sequence of intensity changes in the child's facial muscle movements.

[0037] In conjunction with any of the above embodiments, the present invention also provides a multimodal emotion big model interaction method for emotional companionship with children. In this method, the step S14 above, "based on the child's facial expression change trend, voice emotion cues, and text semantic cues, performing emotion state inference through the multimodal emotion big model to obtain the child's emotional state," specifically includes the following steps S31 to S33: Step S31: Determine the keyframes of the child's facial expressions based on the trend of the child's facial expression changes.

[0038] In this embodiment, the facial video frame with the greatest change in facial muscle movement can be determined based on the trend of the child's facial expression changes. The expression in this facial video frame is the child's prominent expression, and the determined facial video frame can be identified as the child's expression keyframe.

[0039] Step S32: The child's facial expression keyframes are used as visual emotional cues and embedded into the latent representation space of the multimodal emotional big model to obtain a latent representation space constrained by visual emotional cues.

[0040] In this embodiment, the child's facial expression keyframe can be embedded as a visual emotional cue (i.e. visual emotional evidence) into the latent representation space of the multimodal emotion model. With the visual emotional cue as the reasoning constraint, a latent representation space constrained by the visual emotional cue is obtained, so that subtle facial expression evidence is preserved in the model's internal representation.

[0041] Step S33: Based on phonological emotion cues and textual semantic cues, the child's emotional state is determined by performing emotional state reasoning in the latent representation space constrained by visual emotion cues through the multimodal emotion big model.

[0042] In this embodiment, the multimodal emotion big model performs emotion state reasoning in a latent representation space constrained by visual emotion cues based on phonological emotion cues, textual semantic cues, and interactive context information to determine the child's emotional state.

[0043] In one embodiment, the method for determining a child's need for companionship is the same as or similar to the method for determining a child's emotional state. For details, please refer to the method for determining a child's emotional state, which is obtained through processing by a multimodal emotional big data model.

[0044] In an optional specific example, an initial child's emotional state can be obtained by performing emotional inference on phonological emotional cues, textual semantic cues, and interactive context information using a multimodal emotion model. Then, within the multimodal emotion model, the identified visual emotional cues serve as inference constraints, ensuring consistency between the emotional prediction and observable facial behavior, and verifying whether the initial child's emotional state matches the visual emotional cue. Specifically, the match between the initial child's emotional state and the visual emotional cue can be determined based on whether the salient expression corresponding to the visual emotional cue is reasonable. If the salient expression corresponding to the visual emotional cue is reasonable, then the initial child's emotional state is determined to match the visual emotional cue. For example, if the initial child's emotional state is disappointment, and the salient expression corresponding to the visual emotional cue is sadness, then the salient expression corresponding to the visual emotional cue is determined to be reasonable. When the initial child's emotional state matches the visual emotional cue, this initial child's emotional state can be directly determined as the child's emotional state.

[0045] If it is determined that the initial emotional state of a child is not reasonably represented by the salient expression corresponding to the visual emotional cue, then the initial emotional state and the visual emotional cue are mismatched. For example, if the initial emotional state of a child is excitement, and the salient expression corresponding to the visual emotional cue is panic, then it is determined that the initial emotional state is not reasonably represented by the salient expression corresponding to the visual emotional cue. In cases of mismatch between the initial emotional state and the visual emotional cue, the initial emotional state is modified using the visual emotional cue to obtain the modified emotional state, which is then used as the child's emotional state.

[0046] In this embodiment, key visual emotional evidence is retained before generating a natural language companionship response through latent spatial emotion reasoning. This allows for the correction of emotion judgments using facial expression evidence when a child's language expression is incomplete or the semantics are inconsistent with the facial expression, thereby reducing the possibility of the multimodal emotion model misjudging a child's emotions based solely on textual semantics.

[0047] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional big-model interaction method for children's emotional companionship. In this method, the child's emotional state includes at least one of excitement, curiosity, confusion, frustration, grievance, fear, loss, boredom, and seeking attention; the child's companionship needs include at least one of play and interaction needs, curiosity-based questioning needs, story companionship needs, emotional soothing needs, encouragement and expression needs, learning and exploration needs, and parent-child communication needs; and, step S15 above, "generating a child-oriented emotional companionship response based on the child's emotional state and / or child's companionship needs," specifically includes the following steps S41 to S44: Step S41: When the child's emotional state is curiosity or excitement, and / or when the child's companionship need is a curiosity-based question-and-answer need or a learning and exploration need, generate exploratory questions and answers, fun explanations, game tasks, or interactive questions.

[0048] In this embodiment, when the multimodal emotion model determines that the child's emotional state is curiosity or excitement, and / or determines that the child's companionship need is a curiosity-based question-and-answer need or a learning-exploration need, it generates game interaction responses, learning-exploration responses, and / or curiosity-based question-and-answer responses. The game interaction response includes generating game tasks and / or interactive questions, the curiosity-based question-and-answer response includes generating exploratory questions and / or fun explanations, and the learning-exploration response includes generating exploratory questions and answers and interactive questions.

[0049] In a specific example, a scenario of children's curiosity-driven question-and-answer interaction is illustrated: While a child interacts with a companion robot, the robot acquires video images, audio, and text recordings of the child. For instance, the child points to a toy and asks, "Why did dinosaurs become extinct?" while simultaneously exhibiting wide eyes, increased speech speed, and a rising tone. The multimodal emotion model extracts trends in the child's facial expressions based on video information, identifies linguistic cues of excitement and curiosity based on audio information, and recognizes that the child is asking knowledge-exploration questions based on text information. The model further uses latent spatial emotion reasoning to determine that the child is currently in a state of curiosity and excitement, and that their companionship needs are "curiosity-driven question-and-answer" and "learning exploration." Subsequently, the model generates child-friendly responses, such as first explaining the reasons for the extinction of dinosaurs with a short story, and then asking a follow-up question: "Would you like to be a little archaeologist together and guess where dinosaur fossils were found?" This transforms the knowledge-based response into a sustained interaction.

[0050] In other words, the emotional companion robot in this embodiment can support multiple rounds of companionship and dynamically adjust the companionship strategy according to the emotional changes in the child's continuous interaction, thereby improving the continuity and naturalness of the companionship.

[0051] Step S42: When the child's emotional state is confused or frustrated, and / or when the child's need for companionship is a need for encouragement or emotional comfort, generate encouraging statements, step-by-step guidance, or low-difficulty exploratory tasks.

[0052] In this embodiment, when the multimodal emotion model determines that the child's emotional state is confusion or frustration, and / or determines that the child's need for companionship is a need for encouragement or emotional comfort, it generates an encouragement response and / or an emotional comfort response. The encouragement response includes generating encouraging statements, and the emotional comfort response includes generating encouraging statements, step-by-step guidance, and / or low-difficulty exploratory tasks.

[0053] In a specific example, a scenario for comforting a child experiencing frustration is illustrated: after failing multiple times in learning or playing, a child says "I can't," lowering their voice volume and exhibiting facial expressions such as downturned lips and droopy eyes. If the response is based solely on the text "I can't," a typical robot might only provide a knowledge hint. However, the multimodal emotion model in this embodiment combines facial motion unit trajectories and vocal cues to determine that the child needs not only an answer but also encouragement and emotional reassurance. The model's output of a supportive response could include: "It's okay, we can take it one step at a time. You've tried several times, which means you're trying hard. Now I'll give you an easier little hint." This response simultaneously satisfies the child's needs for emotional reassurance and task guidance.

[0054] Step S43: When the child's emotional state is fear, grievance, or loss, and / or when the child's companionship needs are story companionship needs, emotional comfort needs, or parent-child communication needs, generate comforting statements, companionship dialogues, story companionship content, or parent-child communication prompts.

[0055] In this embodiment, when the multimodal emotion model determines that the child's emotional state is fear, grievance, or loss, and / or determines that the child's companionship need is a need for story companionship, an emotional comforting need, or a need for parent-child communication, it generates a story companionship response, an emotional comforting response, and / or a parent-child communication prompt response. The story companionship response includes generating story companionship content and companionship dialogue. The emotional comforting response includes generating comforting statements. The parent-child communication prompt response includes generating parent-child communication prompts.

[0056] In a specific example, a scenario depicting a child's fear or distress is illustrated: when a child expresses "I don't want to sleep alone" at night, accompanied by signs such as slower speech, facial tension, and increased pauses, the multimodal emotion model can determine that the child is afraid or seeking companionship. The model can generate gentle reassuring statements, bedtime stories, or prompts for parent-child communication, and can also generate prompts for parental attention when necessary.

[0057] Step S44: When the child's emotional state is boredom or seeking attention, and / or when the child's companionship need is a need for play and interaction or a need for story companionship, generate game invitations, story recommendations, nursery rhyme interactions, or role-playing content.

[0058] In this embodiment, when the multimodal emotion model determines that the child's emotional state is boredom or seeking attention, and / or determines that the child's companionship need is a need for play and interaction or a need for story companionship, it generates a game interaction response and / or a story companionship response. The story companionship response at this time includes generating story recommendations, and the game interaction response at this time includes generating game invitations, nursery rhyme interactions, and / or role-playing content.

[0059] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional big-model interaction method for emotional companionship with children. In this method, in addition to the steps described above, it may also include step S51, and the "playing the emotional companionship response for children" in step S15 may specifically include step S52: Step S51: Based on the child's age, historical interaction preferences, parent-preset content boundaries, and current interaction scenario, perform child-adaptation processing on the emotional companionship response.

[0060] In this embodiment, the multimodal emotion model can acquire the child's age, historical interaction preferences, parent-preset content boundaries, and the current interaction scenario. The parent-preset content boundaries represent the range of allowed interactive content set by the parent, such as setting content unsuitable for children, like prohibiting interaction with children through horror games. Then, the multimodal emotion model can perform child-adaptation processing on the generated emotional companionship response based on the child's age, historical interaction preferences, parent-preset content boundaries, and the current interaction scenario, resulting in a child-adapted emotional companionship response.

[0061] Step S52: Play the child-adapted emotional support response for children.

[0062] In this embodiment, corresponding robot text can be generated based on the child-adapted emotional companionship response, and the generated robot text can be converted into robot speech and played to complete personalized interaction between the robot and the child.

[0063] Thus, this embodiment can generate child-friendly language based on the child's age, historical preferences, and parental settings, transforming the emotion recognition results into a companion response that is acceptable, understandable, and sustainable for the child, thereby improving the safety, friendliness, and comprehensibility of the robot's responses.

[0064] In conjunction with the above embodiments, in one embodiment, child adaptation processing includes at least one of the following: reducing the complexity of responses; using language that children can understand; filtering content that is inappropriate for children; controlling the length of responses; increasing encouraging expressions; and converting abstract knowledge into stories, metaphors, games, or everyday examples.

[0065] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal emotional large-scale model interaction method for emotional companionship with children. In addition to the steps described above, this method may further include steps S61 to S63: Step S61: Record the changes in the child's emotional state and companionship needs during multiple rounds of interaction.

[0066] In this embodiment, the emotional companion robot can interact with the child in multiple rounds and record the changes in the child's emotional state during these multiple rounds of interaction, as well as the changes in the child's companionship needs based on these multiple rounds of interaction.

[0067] Step S62: Generate a summary of child companionship based on the changes in emotional state and companionship needs.

[0068] In this embodiment, the emotional companionship robot can generate a child companionship summary based on the recorded changes in emotional state and companionship needs using a multimodal emotional big data model. This child companionship summary records the child's emotional state and companionship needs over a period of time.

[0069] Step S63: Detect the child's emotional state based on the child's companionship summary. When the child is detected to be in a state of fear, loss, grievance, or abnormal silence, a parental attention prompt is generated.

[0070] In this embodiment, the multimodal emotion big data model can continuously detect the child's emotional state based on the child's companionship summary, and determine whether the child is continuously in a state of fear, loss, grievance, or abnormal silence for a preset time period (which can be freely set, such as 2 hours). If it is determined that the child is continuously in a state of fear, loss, grievance, or abnormal silence for a preset time period, the multimodal emotion big data model can generate a parent attention prompt and send the parent attention prompt to the parent's user terminal corresponding to the child, so as to remind the parent to communicate and accompany the child, such as reminding the parent to pay attention to the child, comfort the child, play games, etc.

[0071] In one embodiment, multimodal input from children can be collected to obtain multimodal information about the children during the interaction process (i.e., acquiring children's multimodal input). This multimodal information includes: children's facial video, children's speech, children's spoken text, and interaction context. After multimodal preprocessing, this information is input into a multimodal emotion model. The multimodal emotion model performs facial motion unit analysis based on the children's facial video to obtain trends in children's facial expressions. The model then performs intonation and semantic analysis on the children's speech and spoken text to obtain emotional cues and semantic cues. Next, the multimodal emotion model performs latent spatial emotion inference based on the trends in children's facial expressions, emotional cues, semantic cues, and interaction context to preserve fine-grained facial evidence, identifying children's emotional states and determining their need for companionship, thus obtaining their emotional state and companionship needs. Finally, based on the children's emotional state and companionship needs, a child-friendly emotional companionship response is generated, and multiple rounds of interaction are conducted with the children based on this response, enabling personalized updates to the multimodal emotion model through multi-round interaction feedback.

[0072] It should be noted that the robot of the present invention (i.e., the emotional companion robot) can be a desktop companion robot, a smart speaker-shaped robot, a robot with a screen, a mobile robot, or a robot system that is linked with a mobile application.

[0073] It should also be noted that all privacy-related data, such as "interaction" and "dialogue," in this invention were obtained with the user's consent and comply with relevant laws, regulations, and privacy requirements.

[0074] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0075] Based on the same inventive concept, one embodiment of the present invention provides a multimodal emotional big-model interaction system for emotional companionship with children. (Reference) Figure 2 , Figure 2 This is a structural block diagram of a multimodal emotional large-scale interactive system for children's emotional companionship, provided by an embodiment of the present invention. Figure 2 As shown, the system includes: The data acquisition module is used to acquire multimodal information of children during the interaction process, including at least two of the following: children's facial video, children's voice information, children's language text information, and interaction context information; The facial expression analysis module is used to perform facial motion unit analysis on the children's facial videos using a multimodal emotion model to obtain the trend of children's facial expression changes; The data analysis module is used to perform emotional semantic analysis on the children's speech information and children's language text information through the multimodal emotion big model to obtain speech emotion cues and text semantic cues; The state reasoning module is used to perform emotional state reasoning based on the child's facial expression change trend, voice emotion cues, and text semantic cues, through the multimodal emotion big model, to obtain the child's emotional state and / or the child's companionship needs; The companionship response module is used to generate and play emotional companionship responses for children based on the child's emotional state and / or the child's companionship needs.

[0076] Optionally, the facial expression analysis module includes: The extraction module is used to extract multiple frames of children's facial images from the children's facial video; The recognition module is used to identify changes in facial muscle movements in each frame of a child's facial image. The sequence generation module is used to generate a sequence of changes in the intensity of facial muscle movements in children in chronological order. The expression determination module is used to determine the trend of a child's facial expression changes based on the intensity sequence of changes in the child's facial muscle movements.

[0077] Optionally, the state reasoning module includes: The keyframe determination module is used to determine the keyframes of the child's facial expressions based on the trend of the child's facial expression changes. The spatial generation module is used to embed the child's facial expression keyframes as visual emotional cues into the latent representation space of the multimodal emotional big model, so as to obtain a latent representation space constrained by visual emotional cues. The latent reasoning module is used to determine the child's emotional state by reasoning about the emotional state in a latent representation space constrained by visual emotional cues, based on phonological emotional cues and textual semantic cues, through the multimodal emotional big model.

[0078] Optionally, the child's emotional state includes at least one of excitement, curiosity, confusion, frustration, grievance, fear, loss, boredom, and seeking attention; the child's companionship needs include at least one of play and interaction needs, curiosity-answering needs, story companionship needs, emotional comfort needs, encouragement to express oneself needs, learning and exploration needs, and parent-child communication needs. The companion response module includes: The first generation module is used to generate exploratory questions and answers, fun explanations, game tasks or interactive questions when the child's emotional state is curiosity or excitement, and / or when the child's companionship needs are curiosity-based question-and-answer needs or learning-exploration needs. The second generation module is used to generate encouraging statements, step-by-step guidance, or low-difficulty exploratory tasks when the child's emotional state is confused or frustrated, and / or when the child's need for companionship is a need for encouragement or emotional comfort. The third generation module is used to generate comforting statements, companionship dialogues, story companionship or parent-child communication prompts when the child's emotional state is fear, grievance or loss, and / or when the child's companionship needs are story companionship needs, emotional comfort needs or parent-child communication needs. The fourth generation module is used to generate game invitations, story recommendations, nursery rhyme interactions, or role-playing content when the child's emotional state is bored or seeking attention, and / or when the child's companionship needs are play interaction needs or story companionship needs.

[0079] Optionally, the system further includes: The adaptation processing module is used to perform child-adaptation processing on the emotional companionship response based on the child's age, historical interaction preferences, parent-preset content boundaries, and current interaction scenario; The companion response module includes: The response playback module is used to play child-friendly emotional support responses that have been adapted for children.

[0080] Optionally, the child adaptation process includes at least one of the following: Reduce the complexity of the answer; Use language that children can understand; Filter out content that is inappropriate for children; Control the length of the answer; Increase encouraging expressions; Transform abstract knowledge into stories, metaphors, games, or everyday examples.

[0081] Optionally, the system further includes: The recording module is used to record changes in the child's emotional state and companionship needs during multiple rounds of interaction; The summary generation module is used to generate a child companionship summary based on the changes in emotional state and companionship needs. The parent alert module is used to detect the child's emotional state based on the child's companionship summary. When it detects that the child is in a state of fear, loss, grievance, or abnormal silence, it generates a parent attention alert.

[0082] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0083] The multimodal emotional big data interaction system for children's emotional companionship in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The system can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment of the invention does not impose specific limitations.

[0084] The multimodal emotional big data interaction system for children's emotional companionship in this embodiment of the invention can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0085] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the multimodal emotional big model interaction method for children's emotional companionship described in any of the above embodiments of the present invention.

[0086] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0087] Based on the same inventive concept, another embodiment of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the multimodal emotional big model interaction method for children's emotional companionship as described in any of the above embodiments of the present invention. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0088] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0089] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0091] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A multimodal emotional big data interaction method for children's emotional companionship, characterized in that, The method includes: Acquire multimodal information of children during the interaction process, wherein the multimodal information includes at least two of the following: children's facial video, children's voice information, children's language text information, and interaction context information; The facial motion unit analysis of the children's facial videos was performed using a multimodal emotion big data model to obtain the trend of children's facial expression changes; The multimodal emotion model is used to perform emotional semantic analysis on the children's speech information and children's language text information to obtain speech emotion cues and text semantic cues. Based on the trends in children's facial expressions, vocal emotion cues, and textual semantic cues, the emotional state is inferred through the multimodal emotion model to obtain the children's emotional state and / or their need for companionship. Based on the child's emotional state and / or the child's need for companionship, generate and play an emotional companionship response for the child.

2. The multimodal emotional big model interaction method for children's emotional companionship according to claim 1, characterized in that, Facial motion unit analysis was performed on the children's facial videos using a multimodal emotion model to obtain trends in children's facial expression changes, including: Extract multiple frames of children's facial images from the children's facial video; Identify changes in facial muscle movements in each frame of a child's facial image; Generate a sequence of changes in the intensity of children's facial muscle movements in chronological order; Based on the intensity sequence of changes in children's facial muscle movements, the trend of children's facial expression changes is determined.

3. The multimodal emotional large-scale interaction method for children's emotional companionship according to claim 1, characterized in that, Based on the trends in children's facial expressions, vocal emotion cues, and textual semantic cues, emotional state inference is performed using the multimodal emotion model to obtain the children's emotional state, including: Based on the described trends in children's facial expressions, keyframes of children's facial expressions are determined; The child's facial expression keyframes are used as visual emotional cues and embedded in the latent representation space of the multimodal emotional big model to obtain a latent representation space constrained by visual emotional cues. Based on phonological emotion cues and textual semantic cues, the emotional state of children is determined by reasoning about their emotional state in a potential representation space constrained by visual emotion cues through the multimodal emotion big model.

4. The multimodal emotional big model interaction method for children's emotional companionship according to claim 1, characterized in that, The child's emotional state includes at least one of the following: excitement, curiosity, confusion, frustration, grievance, fear, loss, boredom, and seeking attention; the child's need for companionship includes at least one of the following: need for play and interaction, need for answering questions about curiosity, need for story companionship, need for emotional comfort, need for encouragement to express oneself, need for learning and exploration, and need for parent-child communication. Based on the child's emotional state and / or companionship needs, generate a child-oriented emotional companionship response, including: When the child's emotional state is curiosity or excitement, and / or when the child's need for companionship is a need for curiosity-based questioning or a need for learning and exploration, exploratory questions and answers, fun explanations, game tasks, or interactive questions are generated. When the child's emotional state is confused or frustrated, and / or when the child's need for companionship is a need for encouragement or emotional comfort, generate encouraging statements, step-by-step guidance, or low-difficulty exploratory tasks. When the child's emotional state is fear, grievance, or loss, and / or when the child's need for companionship is a need for story companionship, emotional comfort, or parent-child communication, generate comforting statements, companionship dialogues, story companionship, or parent-child communication prompts. When the child's emotional state is boredom or seeking attention, and / or when the child's need for companionship is a need for play and interaction or a need for story companionship, generate game invitations, story recommendations, nursery rhyme interactions, or role-playing content.

5. The multimodal emotional big model interaction method for children's emotional companionship according to claim 1, characterized in that, The method further includes: The emotional companionship response is adapted to the child's age, historical interaction preferences, parent-preset content boundaries, and current interaction scenario. Play emotional support responses for children, including: Play child-friendly emotional support responses that have been adapted for children.

6. The multimodal emotional big model interaction method for children's emotional companionship according to claim 5, characterized in that, The child adaptation process includes at least one of the following: Reduce the complexity of the answer; Use language that children can understand; Filter out content that is inappropriate for children; Control the length of the answer; Increase encouraging expressions; Transform abstract knowledge into stories, metaphors, games, or everyday examples.

7. The multimodal emotional large-scale interaction method for children's emotional companionship according to any one of claims 1 to 6, characterized in that, The method further includes: Record changes in children's emotional state and companionship needs during multiple rounds of interaction; Based on the changes in emotional state and companionship needs, a summary of child companionship is generated; Based on the child companionship summary, the system detects the child's emotional state. When it detects that the child is in a state of fear, loss, grievance, or abnormal silence, it generates a prompt for parents to pay attention.

8. A multimodal emotional big data interaction system for children's emotional companionship, characterized in that, The system includes: The data acquisition module is used to acquire multimodal information of children during the interaction process, including at least two of the following: children's facial video, children's voice information, children's language text information, and interaction context information; The facial expression analysis module is used to perform facial motion unit analysis on the children's facial videos using a multimodal emotion model to obtain the trend of children's facial expression changes; The data analysis module is used to perform emotional semantic analysis on the children's speech information and children's language text information through the multimodal emotion big model to obtain speech emotion cues and text semantic cues; The state reasoning module is used to perform emotional state reasoning based on the child's facial expression change trend, voice emotion cues and text semantic cues, through the multimodal emotion big model, to obtain the child's emotional state and / or the child's companionship needs; The companionship response module is used to generate and play emotional companionship responses for children based on the child's emotional state and / or the child's companionship needs.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the multimodal emotional big model interaction method for children's emotional companionship as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the multimodal emotional big model interaction method for children's emotional companionship as described in any one of claims 1 to 7.

Images