A voice interaction system for an emotional-companion robot dog and a method thereof

Abstract

The application discloses a kind of machine dog voice interaction systems and methods for emotional accompaniment, it is related to human-computer intelligent interaction technical field, including multimodal emotion perception and state real-time construction module, it is used to gather user voice data and facial image data, generate speech emotion vector based on voice data, generate emotion label and its confidence probability based on facial image data, establish emotion state table based on speech emotion vector and emotion label;Intelligent decision and content generation module based on emotional context, generate the interactive action and voice data of machine dog based on speech emotion vector and emotion label;Real-time interaction management and safety interrupt module, according to user feedback data, interactive action and voice data are divided into effective interaction data and invalid interaction data;Effective interaction corresponding interactive action and voice data are updated to emotional interaction memory bank;Realize from "functional interaction" to "emotional interaction" leap.

Description

Technical Field

[0001] This invention relates to the field of human-computer intelligent interaction technology, specifically to a voice interaction system and method for a robot dog oriented towards emotional companionship. Background Technology

[0002] Intelligent interactive systems based on mobile robot platforms are gradually moving from the laboratory to practical applications, demonstrating broad potential in scenarios such as service robots, educational companionship, and remote assistance. Traditional human-computer interaction systems, especially those deployed on non-traditional carriers such as quadruped robots or robotic dogs, mostly focus on completing functional tasks, such as environmental navigation, object handling, or responding to preset commands. Their dialogue modules are often based on rule templates or general generation of large language models, lacking the ability to perceive and adaptively respond to the user's emotional state during the interaction. This makes the interaction process appear mechanical and rigid, making it difficult to establish a natural and warm emotional connection, greatly limiting its application effectiveness in scenarios requiring deep social interaction or emotional companionship.

[0003] In the field of affective computing, significant progress has been made in technologies that identify users' emotional states through multimodal signals such as speech and vision. For example, the invention patent CN120913557A discloses an intelligent speech recognition and natural language interaction method based on a quadruped robot, and the invention patent CN120056082B discloses an intelligent robot for emotional guidance based on speech interaction. Existing methods can extract and classify emotional features from the prosody and acoustic characteristics of speech, facial expressions and micro-expressions, and even the semantics of text. However, how to inject the identified emotional information into the dialogue generation process in real time and effectively, so that the machine's response is not only reasonable in content but also matches the emotional tone, intonation, and word choice, forming a closed-loop emotional interaction system, remains a challenge. Existing solutions often treat emotion as a single label or soft constraint in dialogue generation, failing to achieve deep integration and dynamic adjustment of emotional state with dialogue strategies and language style.

[0004] On the other hand, integrating such advanced intelligent interaction systems onto platforms like robotic dogs, which possess high mobility and anthropomorphic forms, presents both unique opportunities and challenges. The opportunity lies in the fact that the robotic dog's dynamic body language (such as shaking and posture changes) can form a multimodal collaborative output with emotional voice dialogue, greatly enhancing the richness and credibility of emotional expression. The challenge lies in the limited computing resources of mobile platforms, the susceptibility of sensors to motion noise interference, and the need to achieve low-latency local and remote voice interaction collaboration. Existing remote interaction systems often focus on one-way audio and video transmission or simple remote control commands, lacking real-time perception of the remote user's emotions and support for emotional two-way dialogue, making it difficult to support immersive, face-to-face, in-depth communication. Summary of the Invention

[0005] In order to overcome the above-mentioned technical problems, the purpose of this invention is to provide a voice interaction system and method for robot dogs oriented towards emotional companionship, so as to realize the leap from "functional interaction" to "emotional interaction".

[0006] The objective of this invention can be achieved through the following technical solutions: Specifically, the first aspect is to provide a voice interaction system for robot dogs designed for emotional companionship, including: The multimodal emotion perception and real-time state construction module is used to collect user data, including user voice data and facial image data. It generates voice emotion vectors based on voice data, generates emotion labels and their confidence probabilities based on facial image data, and establishes an emotion state table based on voice emotion vectors, emotion labels, and the robot dog's interaction context. The intelligent decision-making and content generation module based on emotional context generates interactive decisions for the robot dog based on user data and emotional state tables. The interactive decisions include interactive actions and / or interactive voice. The real-time interaction management and security interruption module collects user feedback data and divides interaction decisions into effective interaction data and invalid interaction data based on the user feedback data. The interaction decisions corresponding to the effective interaction data and invalid interaction data are transformed into an interaction context to be updated, and the sentiment state table is updated based on the interaction context to be updated and its corresponding user data. The personalized memory and long-term adaptation module stores user data and interaction decisions corresponding to effective interaction data in a database to form an emotional interaction memory bank, and adjusts interaction decisions by calling the emotional interaction memory bank.

[0007] A further solution for a robot dog voice interaction system aimed at emotional companionship: Preferably, the multimodal emotion perception and real-time state construction module extracts the speech text and speech acoustic features of the speech data through a speech recognition model, and generates a speech emotion vector based on the speech text and speech acoustic features.

[0008] Preferably, the multimodal emotion perception and real-time state construction module recognizes facial image data based on a convolutional neural network model, generates emotion labels and their confidence probabilities corresponding to the facial image data, and maps the speech emotion vector, emotion label and the robot dog's interaction context to a unified emotion embedding space through a multimodal fusion model and generates a multidimensional vector of speech emotion, a multidimensional vector of emotion label and a multidimensional vector of interaction context. An emotion state table is established based on the multidimensional vector of speech emotion, the multidimensional vector of emotion label and the multidimensional vector of interaction context.

[0009] Preferably, the intelligent decision-making and content generation module based on emotional context uses emotional LLM inference to process user data to generate interactive decisions, inputs the text responses in the interactive decisions into the speech synthesis engine, and the speech synthesis engine combines the emotional state table to adjust the interactive voice corresponding to the text responses in real time.

[0010] Preferably, the real-time interaction management and safety interruption module runs continuously in the background, identifies interruptions made by the user during the robot dog's response process, generates an interruption signal, interrupts the robot dog's interactive actions and voice data, and treats the interrupted interactive actions and voice data as invalid interactive data, while treating the uninterrupted interactive actions and voice data as valid interactive data.

[0011] Preferably, the real-time interaction management and safety interruption module is also equipped with a monitoring clock, which interrupts the robot dog's interactive actions and voice data when the multimodal emotion perception and real-time state construction module fails to collect user voice data and facial image data after a preset time period.

[0012] Preferably, the personalized memory and long-term adaptation module generates different similarity intervals based on the emotional interaction memory bank, calculates the cosine similarity of the voice emotion multidimensional vector, the emotion tag multidimensional vector, and the interaction context multidimensional vector, and calls the emotional interaction memory bank based on the cosine similarity and the similarity interval.

[0013] The second aspect specifically provides a voice interaction method for robot dogs oriented towards emotional companionship. This method is based on the aforementioned voice interaction system for robot dogs oriented towards emotional companionship and includes the following steps: S1: Collect user data, which includes user voice data and facial image data; S2: Generate speech emotion vectors based on speech data, and generate emotion labels and their confidence probabilities based on facial image data; S3: Establish an emotion state table based on speech emotion vectors, emotion tags, and the robot dog's interaction context; S4: Generate interaction decisions for the robot dog based on user data and emotional state table. Interaction decisions include interactive actions and / or interactive voice. S5: Based on user feedback data, the interaction decisions are divided into effective interaction data and invalid interaction data. The interaction decisions corresponding to the effective interaction data and invalid interaction data are transformed into an interaction context to be updated. The sentiment state table is updated according to the interaction context to be updated and its corresponding user data. S6: Store the user data and interaction decisions corresponding to the effective interaction data in the database to form an emotional interaction memory bank, and adjust the interaction decisions by calling the emotional interaction memory bank.

[0014] The beneficial effects of this invention are: 1. In this invention, compared to traditional companion robots that can only respond to explicit commands or execute preset dialogues, this invention, through dynamic emotional state modeling and contextual understanding, enables the robot dog to proactively perceive the user's nonverbal emotional cues (such as silence, sighs, and facial expressions) and generate responses with emotional appropriateness. This not only significantly enhances the "humanized" experience of the interaction, making the user feel truly understood and accompanied, but also proactively initiates caring interventions at critical moments (such as detecting persistent low mood), effectively alleviating loneliness and achieving a fundamental upgrade in the companionship model from passive to proactive, and from mechanical to empathetic.

[0015] 2. In this invention, by integrating multimodal perception and real-time interruption monitoring mechanisms, the system effectively solves problems such as "irrelevant answers," "incessant talking," or "missing user interruptions" that easily occur in traditional interactions. In elderly care scenarios, this manifests as follows: when the elderly speak slowly or lose focus, the system can patiently wait and provide timely prompts; when the elderly interrupt the interaction due to sudden discomfort, the system can immediately stop the current behavior and switch to a caregiver inquiry mode. This anthropomorphic interaction rhythm control and safety boundary awareness ensures a smooth, comfortable, and reliable companionship process, reducing the possibility of user confusion or risks caused by improper interaction.

[0016] 3. Unlike robots with fixed behavioral patterns, this invention utilizes long-term memory and learning mechanisms to enable the robot dog to gradually become familiar with each elderly person's lifestyle, emotional preferences, and health status. The system can remember the elderly person's emotional reactions to certain topics (such as past events and children), optimizing subsequent interaction strategies; and it can identify abnormal behavioral patterns (such as reduced daily activities and increased repetitive questioning) as early warning clues of potential health problems. This allows the robot dog to not only provide increasingly tailored daily companionship but also become a sustainable emotional partner and auxiliary health observation node, achieving the long-term accumulation and extension of companionship value. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of a voice interaction system for a robotic dog designed for emotional companionship, according to the present invention. Figure 2 This is a flowchart of a voice interaction method for a robot dog oriented towards emotional companionship, according to the present invention. Figure 3 This is a scenario diagram of a robot dog voice interaction system for emotional companionship according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1

[0021] See Figures 1-3 A voice interaction system for robot dogs designed for emotional companionship is disclosed, comprising a multimodal emotion perception and real-time state construction module, an intelligent decision-making and content generation module based on emotional context, a real-time interaction management and security interruption module, and a personalized memory and long-term adaptation module.

[0022] The multimodal emotion perception and real-time state construction module collects user voice data through the robot dog's built-in microphone, extracts the voice text and voice acoustic features (such as pitch, speech rate, and energy) from the voice data through a high-performance speech recognition model, and generates a voice emotion vector based on the voice text and voice acoustic features.

[0023] The robot dog is also equipped with an image acquisition module (high-definition camera) to collect facial image data of the user. It then uses a pre-trained convolutional neural network model to recognize the facial image data and generate corresponding emotion labels and their confidence probabilities. Emotion labels include happiness, sadness, and calmness, with the specific labels chosen adaptively by those skilled in the art.

[0024] The core component of the multimodal emotion perception and real-time state construction module, `context_aggregator`, resides on a high-performance computing unit (such as NX). `context_aggregator` receives speech emotion vectors, emotion tags, and the robot dog's interactive actions and speech data. Then, through a multimodal fusion model, it maps the speech emotion vectors, emotion tags, and the robot dog's interactive actions and speech data to a unified emotion embedding space, dynamically updating the emotion state table accordingly. This emotion state table is a multidimensional vector that includes not only the currently dominant emotion but also emotion intensity and historical trends, providing a panoramic emotion profile for decision-making. Specifically, it maps the speech emotion vectors, emotion tags, and the robot dog's interactive context to a unified emotion embedding space, generating multidimensional speech emotion vectors, multidimensional emotion tag vectors, and multidimensional interactive context vectors. Then, the emotion state table is generated from multiple sets of these multidimensional vectors.

[0025] The emotional context-based intelligent decision-making and content generation module uses emotional LLM inference to process user data and generate interactive decisions. The text responses from these decisions are then input into the speech synthesis engine, which, in conjunction with an emotional state table, adjusts the corresponding interactive speech in real time. The emotional LLM inference feeds real-time collected user data and the emotional state table into a large-scale language model (such as DeepSeek). This emotional LLM inference not only understands the literal semantics of the user's speech emotional vector but also infers from the emotional vectors in the emotional state table, guiding the generation of empathetic rather than literal responses.

[0026] Specifically, user data is processed through affective LLM inference to generate corresponding interaction decisions (including interactive actions and / or interactive voice). The text responses corresponding to the interactive voice in the interaction decisions are then input into the speech synthesis engine (which can use Minimax TTS). The style, speech rate, and intonation parameters of the speech synthesis engine are adjusted in real time based on the speech emotion vector in the emotion state table to ensure that the output speech matches the current user's emotion in terms of content and auditory perception, achieving a unified interaction between the user and the robot dog. An affective interaction memory bank is also established based on the generated robot dog interaction actions and voice data, as well as the emotion state table.

[0027] The real-time interaction management and safety interruption module ensures smooth and human-like interaction between the user and the robot dog, solving the problems of "the robot talking incessantly" or "missing user interruptions." Specifically, the real-time interaction management and safety interruption module runs continuously in the background, dedicated to real-time voice input interruption detection. It can quickly identify a sudden interruption by the user during the robot dog's response and immediately send an interruption signal to the decision center, causing the robot dog's interactive actions and voice data to be interrupted. This interruption indicates that the robot dog's interaction was ineffective, and therefore the interactive actions and voice data of this interaction can be considered invalid. If the robot dog's interactive actions and voice data are not interrupted, it indicates that the interaction was effective, and the interactive actions and voice data of this interaction can be considered valid.

[0028] The real-time interaction management and safety interruption module can also perform timeout interruption. A key monitoring clock is preset. For example, after an interaction is initiated, if the multimodal emotion perception and real-time state construction module fails to collect user voice data and facial image data within a preset time period (taking three minutes as an example), it will actively interrupt the robot dog's current interaction actions and voice data. This simulates the tacit understanding in human social interaction of "stopping in time when you notice that the other party is not paying attention" and avoids invalid or intrusive interactions.

[0029] The personalized memory and long-term adaptation module stores user data and interaction decisions corresponding to effective interaction data in a database, forming an emotional interaction memory bank. Interaction decisions are adjusted by calling this emotional interaction memory bank. Specifically, the personalized memory and long-term adaptation module generates different similarity intervals based on the emotional interaction memory bank, calculates the cosine similarity of the multi-dimensional vectors of speech emotion, emotion tags, and interaction context, and calls the emotional interaction memory bank based on the cosine similarity and similarity intervals. The personalized memory and long-term adaptation module generates different similarity intervals based on the emotional interaction memory bank in the following way: Extract the Softmax function corresponding to each set of voice emotion multidimensional vectors, emotion tag multidimensional vectors, and interaction context multidimensional vectors from the emotion interaction memory bank: i = 1, 2, ..., K; in, Let v, m, and n be the voice emotion vectors, where v is the multi-dimensional voice emotion vector mapped to the emotion state table. Let be the probability distribution of the multidimensional vector of speech emotion, with values ​​ranging from (0, 1), and m be the multidimensional vector of emotion labels mapped from the emotion label emotion state table. Let be the probability distribution of the sentiment label multidimensional vector, with values ​​ranging from (0, 1), and let n be the multidimensional vector of the interaction context mapped to the sentiment state table. Let be the probability distribution of the multidimensional vector of the interaction context, with values ​​ranging from (0, 1). ; The interval (0, 1) is divided into similarity intervals (happy, sad, calm) based on the sentiment label. Real-time collection of user data, and based on the emotional state table, derive corresponding voice emotion vectors, emotion tags, and the robot dog's interaction context, and calculate the data set. Then determine the The system determines which similarity interval a robot dog belongs to and then directly calls the interaction decision from the corresponding emotional interaction memory bank. Therefore, when the robot dog encounters similar emotional scenarios or users, the personalized memory and long-term adaptation module can retrieve historical cases from the memory bank to provide personalized references for the current emotional decision and response generation. This allows the robot dog's behavior to gradually align with the preferences and emotional patterns of specific users, achieving personalized growth that "understands you better the more you use it."

[0030] As one embodiment of the present invention, we take Grandpa Li, who lives alone, as an example. When Grandpa Li, who lives alone, sits quietly by the window for a long time due to mobility issues and has a lonely expression, the robot dog collects Grandpa Li's voice data and facial image data through a visual emotion analysis module, and generates voice emotion vectors and emotion tags corresponding to Grandpa Li's voice data and facial image data.

[0031] The intelligent decision-making and content generation module based on emotional context generates interactive actions and voice data for the robot dog based on Grandpa Li's voice emotion vector and emotion tags. The interactive actions and voice data can be obtained through pre-training or by retrieving and calling the emotional interaction memory bank. For example, the robot dog slowly approaches and gently touches the old man's calf, and simultaneously outputs in a gentle dialect: "Grandpa, shall we listen to the Huangmei Opera that you loved when you were young?" At this time, the display screen on the back of the robot dog lights up with a soft warm light and plays the opera list. If Grandpa Li responds to the interaction, the real-time interaction management and safety interruption module takes the interactive action and voice data as valid interaction data and updates it to the emotional interaction memory bank.

[0032] As one embodiment of the present invention, let's take Grandma Zhang, who suffers from mild cognitive impairment, as an example. When Grandma Zhang repeatedly asked "What day of the week is today?" for the third time, the personalized memory and long-term adaptation module detected the repetitive questioning pattern through the real-time voice interaction memory bank, and the emotional state table showed that her anxiety index was rising. The robot dog no longer mechanically repeated the date, but approached and gently wagged its tail in response: "Today is the beginning of winter, look, I found the red scarf you knitted," while simultaneously retrieving the fabric from the smart storage compartment. Once Grandma's attention was diverted, the robot dog automatically contacted the remote care platform to generate a "Cognitive Training Prompt," and subsequent dialogue would naturally incorporate date-oriented training: "The day you knitted the scarf was Thursday, and three days have passed now..." If Grandma suddenly got up and walked towards the kitchen, the motion monitoring immediately triggered the anti-fall protocol, and the robot dog would first go to the kitchen doorway to form a soft roadblock and issue a reminder: "The floor has just been cleaned, I'll walk slowly with you."

[0033] Example 2

[0034] See Figures 1-3 This paper discloses a voice interaction method for robot dogs designed for emotional companionship. The method is based on a voice interaction system for robot dogs designed for emotional companionship as described in Example 1, and includes the following steps: S1: Collect user data through a multimodal emotion perception and real-time state construction module. The user data includes user voice data and facial image data. A microphone and image acquisition module (high-definition camera) need to be installed in the robot dog. S2: Generates voice emotion vectors based on speech data and emotion labels and their confidence probabilities based on facial image data. A built-in microphone collects user speech data, and a high-performance speech recognition model extracts the speech text and acoustic features (such as pitch, speech rate, and energy) from the speech data. Voice emotion vectors are generated based on the speech text and acoustic features. An image acquisition module (high-definition camera) collects user facial image data and uses a pre-trained convolutional neural network model to recognize the facial image data, generating corresponding emotion labels and their confidence probabilities. Emotion labels include happiness, sadness, and calmness.

[0035] S3: An emotion state table is built based on speech emotion vectors, emotion tags, and the robot dog's interaction context. Specifically, the core component `context_aggregator` of the multimodal emotion perception and real-time state construction module resides on a high-performance computing unit (such as NX). `context_aggregator` receives speech emotion vectors, emotion tags, and the robot dog's interaction actions and speech data. Then, through a multimodal fusion model, it maps the speech emotion vectors, emotion tags, and the robot dog's interaction actions and speech data to a unified emotion embedding space, and dynamically updates the emotion state table accordingly. This emotion state table is a multidimensional vector that includes not only the currently dominant emotion but also emotion intensity, historical trends, etc., providing a panoramic emotion profile for decision-making.

[0036] S4: Based on user data and an emotional state table, the robot dog generates interaction decisions, including interactive actions and / or interactive speech. The intelligent decision-making and content generation module based on emotional context uses emotionally-oriented LLM inference to process user data and generate interaction decisions. The text responses in the interaction decisions are input into the speech synthesis engine, which, in conjunction with the emotional state table, adjusts the corresponding interactive speech in real time. The emotionally-oriented LLM inference feeds real-time collected user data and the emotional state table into a large-scale language model (such as DeepSeek). Emotionally-oriented LLM inference not only understands the literal semantics of the speech text in the user's speech emotional vector but also infers from the speech emotional vector in the emotional state table, guiding the emotionally-oriented LLM inference to generate empathetic rather than literal responses. By processing user data through emotionally-oriented LLM inference, corresponding interaction decisions (including interactive actions and / or interactive speech) are generated. Then, the text responses corresponding to the interactive speech in the interaction decisions are input into the speech synthesis engine (which can use Minimax TTS). The speech synthesis engine adjusts its style, speech rate, and intonation parameters in real time based on the speech emotion vector in the emotion state table to ensure that the output speech matches the current user's emotion in terms of content and auditory perception, thereby achieving a unified interaction between the user and the robot dog. It also establishes an emotion interaction memory bank based on the generated robot dog interaction actions, speech data, and emotion state table.

[0037] S5: Based on user feedback data, interaction decisions are categorized into valid and invalid interaction data. The interaction decisions corresponding to valid and invalid data are transformed into an interaction context to be updated. The emotional state table is updated based on this context and the corresponding user data. Specifically, the real-time interaction management and safety interruption module ensures the smoothness and human-like nature of the interaction between the user and the robot dog, resolving issues such as "the robot talking incessantly" or "missing user interruptions." Specifically, the real-time interaction management and safety interruption module runs continuously in the background, specifically for real-time voice input interruption detection. It can quickly identify sudden interruptions from the user during the robot dog's response and immediately send an interruption signal to the decision center, interrupting the robot dog's interaction actions and voice data. This interruption indicates a poor interaction effect, thus the interaction actions and voice data of that interaction can be considered invalid interaction data. If the robot dog's interaction actions and voice data are not interrupted, it indicates a good interaction effect, and the interaction actions and voice data of that interaction can be considered valid interaction data.

[0038] The real-time interaction management and safety interruption module can also perform timeout interruption. A key monitoring clock is preset. For example, after an interaction is initiated, if the multimodal emotion perception and real-time state construction module fails to collect user voice data and facial image data within a preset time period (taking three minutes as an example), it will actively interrupt the robot dog's current interaction actions and voice data. This simulates the tacit understanding in human social interaction of "stopping in time when you notice that the other party is not paying attention" and avoids invalid or intrusive interactions.

[0039] S6: Store the user data and interaction decisions corresponding to the effective interaction data in the database to form an emotional interaction memory bank. Adjust interaction decisions by calling the emotional interaction memory bank. Specifically, the personalized memory and long-term adaptation module generates different similarity intervals based on the emotional interaction memory bank, calculates the cosine similarity of the multi-dimensional vector of voice emotion, the multi-dimensional vector of emotion tag, and the multi-dimensional vector of interaction context, and calls the emotional interaction memory bank based on the cosine similarity and similarity interval. The calling method has been described in detail in Implementation Example 1 and will not be repeated here. When the robot dog encounters similar emotional scenarios or users, the personalized memory and long-term adaptation module can retrieve historical cases from the memory bank to provide personalized references for the current emotional decision and response generation, so that the robot dog's behavior gradually conforms to the preferences and emotional patterns of specific users, achieving personalized growth of "understanding you better the more you use it".

[0040] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A voice interaction system for robotic dogs designed for emotional companionship, characterized in that, include: The multimodal emotion perception and real-time state construction module is used to collect user data, including user voice data and facial image data. It generates voice emotion vectors based on voice data, generates emotion labels and their confidence probabilities based on facial image data, and establishes an emotion state table based on voice emotion vectors, emotion labels, and the robot dog's interaction context. The intelligent decision-making and content generation module based on emotional context generates interactive decisions for the robot dog based on user data and emotional state tables. The interactive decisions include interactive actions and / or interactive voice. The real-time interaction management and security interruption module collects user feedback data and divides interaction decisions into effective interaction data and invalid interaction data based on the user feedback data. The interaction decisions corresponding to the effective interaction data and invalid interaction data are transformed into an interaction context to be updated, and the sentiment state table is updated based on the interaction context to be updated and its corresponding user data. The personalized memory and long-term adaptation module stores user data and interaction decisions corresponding to effective interaction data in a database to form an emotional interaction memory bank, and adjusts interaction decisions by calling the emotional interaction memory bank.

2. The robot dog voice interaction system for emotional companionship according to claim 1, characterized in that, The multimodal emotion perception and real-time state construction module extracts speech text and speech acoustic features from speech data through a speech recognition model, and generates speech emotion vectors based on speech text and speech acoustic features.

3. The robot dog voice interaction system for emotional companionship according to claim 2, characterized in that, The multimodal emotion perception and real-time state construction module recognizes facial image data based on a convolutional neural network model, generates emotion labels and their confidence probabilities corresponding to the facial image data, and maps the speech emotion vector, emotion label and robot dog interaction context to a unified emotion embedding space through a multimodal fusion model and generates speech emotion multidimensional vector, emotion label multidimensional vector and interaction context multidimensional vector. Based on the speech emotion multidimensional vector, emotion label multidimensional vector and interaction context multidimensional vector, an emotion state table is established.

4. The robot dog voice interaction system for emotional companionship according to claim 1, characterized in that, The emotional context-based intelligent decision-making and content generation module uses emotional LLM inference to process user data to generate interactive decisions. The text responses in the interactive decisions are input into the speech synthesis engine, which combines the emotional state table to adjust the interactive voice corresponding to the text responses in real time.

5. A voice interaction system for a robotic dog oriented towards emotional companionship as described in claim 1, characterized in that, The real-time interaction management and safety interruption module runs continuously in the background, identifies interruptions made by the user during the robot dog's response process, generates an interruption signal, interrupts the robot dog's interactive actions and voice data, and treats the interrupted interactive actions and voice data as invalid interactive data, while treating the uninterrupted interactive actions and voice data as valid interactive data.

6. A voice interaction system for a robotic dog oriented towards emotional companionship as described in claim 5, characterized in that, The real-time interaction management and safety interruption module is also equipped with a monitoring clock. If the multimodal emotion perception and real-time state construction module fails to collect user voice data and facial image data within a preset time period, the robot dog's interactive actions and voice data will be interrupted.

7. A voice interaction system for a robotic dog oriented towards emotional companionship according to claim 3, characterized in that, The personalized memory and long-term adaptation module generates different similarity intervals based on the emotional interaction memory bank, calculates the cosine similarity of the voice emotion multidimensional vector, emotion tag multidimensional vector, and interaction context multidimensional vector, and calls the emotional interaction memory bank based on the cosine similarity and similarity interval.

8. A voice interaction method for robot dogs oriented towards emotional companionship, characterized in that, This method is implemented based on the voice interaction system for robot dogs oriented towards emotional companionship as described in any one of claims 1-7. Includes the following steps: S1: Collect user data, which includes user voice data and facial image data; S2: Generate speech emotion vectors based on speech data, and generate emotion labels and their confidence probabilities based on facial image data; S3: Establish an emotion state table based on speech emotion vectors, emotion tags, and the robot dog's interaction context; S4: Generate interaction decisions for the robot dog based on user data and emotional state table. Interaction decisions include interactive actions and / or interactive voice. S5: Based on user feedback data, the interaction decisions are divided into effective interaction data and invalid interaction data. The interaction decisions corresponding to the effective interaction data and invalid interaction data are transformed into an interaction context to be updated. The sentiment state table is updated according to the interaction context to be updated and its corresponding user data. S6: Store the user data and interaction decisions corresponding to the effective interaction data in the database to form an emotional interaction memory bank, and adjust the interaction decisions by calling the emotional interaction memory bank.

Images