Machine emotion interaction system

Abstract

The invention relates to the field of robots, and particularly discloses a robot emotional interaction system which comprises the steps of searching for an interaction object and confirming an interaction state; determining the current emotional state of the interaction object, carrying out interaction scene modeling based on the historical interaction record, the current interaction information and the environment semantic recognition result, deducing the user intention in the interaction, and outputting the current interaction scene parameters to understand the user intention; understanding and generating an interaction instruction set about voice, expression and action feedback of the robot in the next step based on the current interaction scene parameters and the user intention; facial expression generation, body movement control and emotion speech synthesis of the robot are carried out based on the interaction instruction set; and collecting real-time feedback of an emotional state and an environment of an interaction object, and adjusting an instruction in real time. The emotion interaction system capable of performing efficient semantic understanding is constructed, and the receiving mode of the system is divided into a user voice receiving stage and a real-time feedback receiving adjustment stage, so that different information emphasis is realized, and the operation efficiency of the interaction system is improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to robot interaction systems. Background Technology

[0002] With the development of humanoid robots, companion robots that emphasize human-computer interaction experience are showing broad application prospects. Their core lies in the robot's ability to adaptively integrate various social cues in social scenarios and respond appropriately. Traditional human-computer interaction relies too heavily on unimodal information and focuses more on task completion than on social interaction scenarios. Compared to task-oriented interactions with clear goals, experience-oriented human-computer interaction requires robots to collect relevant social cues and integrate them appropriately, necessitating a clear and adaptable system architecture.

[0003] The current robot execution strategy architecture is VLA (Visual Language Action Model). This architecture is suitable for language command-oriented task execution scenarios. However, due to the diversity of social scenarios, it is difficult to pre-train a unified strategy to cope with most scenarios. Furthermore, this architecture cannot identify and analyze user emotional feedback and output corresponding strategies in a targeted manner. Therefore, VLA has limited versatility in social scenarios with ambiguous objective functions.

[0004] In recent years, the rise of large language models has provided semantic interpretation tools for robot social scenarios. Speech models with emotion generation and recognition capabilities have made extracting user emotional cues from auditory modalities a viable approach to interaction cue extraction. Simultaneously, as social relationships become increasingly atomized, people's expectations for AI companionship have increased, placing higher demands on the capabilities of intelligent systems in social situations. However, humanoid robots integrating multimodal information still lack widely applicable paradigms in terms of interactive system architecture. How to reasonably integrate multimodal information while maintaining a reasonable latency for real-time interaction remains a challenge. Therefore, there is an urgent need to design a system architecture that incorporates emotion for social scenarios.

[0005] The technical problem to be solved by this application is: how to design a robot system structure that incorporates emotions and is tailored to social situations. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a robot emotion interaction system.

[0007] The technical solution adopted in this invention is: a robot emotion interaction system, comprising the following processes:

[0008] S1. Locate the interaction object and confirm the interaction status;

[0009] S2. Determine the current emotional state of the interactive object, and then perform interactive scenario modeling based on historical interaction records, current interaction information and environmental semantic recognition results. Infer the user's interaction intention and output the user's intention understanding and current interaction scenario parameters.

[0010] S3. Generate the robot's next set of interactive instructions regarding voice, facial expression, and action feedback based on the current interaction scenario parameters and user intent understanding;

[0011] S4. Generate facial expressions, control body movements, and synthesize emotional speech for robots based on interactive instruction sets;

[0012] S5: Collect real-time feedback on the emotional state of the interacting object and the environment, and adjust instructions in real time.

[0013] In some implementations, the search for the interactive object in S1 is achieved by combining a visual module and an auditory module for target detection. After the visual module identifies the interactive user, it confirms the interaction status through distance, eye tracking results, and specific gestures / keywords.

[0014] In some implementations, the current interaction information in S2 includes multimodal information obtained through user facial expression recognition, speech recognition, posture and gesture recognition.

[0015] In some implementations, the user facial expression recognition is achieved through an facial expression classification model trained on an facial expression database, and the model outputs the ratio of various emotion categories;

[0016] The speech recognition is achieved through Sense Voice, which can recognize tone and emotion, that is, recognize the content of the speech text and the emotion of the speech and label the speaker ID;

[0017] The posture and gesture recognition is achieved through predefined joint position state classification. The gesture recognition includes static gesture recognition and continuous frame dynamic gesture recognition. The posture recognition analyzes the degree of limb relaxation by joint position change trends and identifies specific actions with predefined corresponding text explanations.

[0018] In some implementations, S3 includes:

[0019] S31. By inputting user intent understanding and interaction scenario parameters into the response model, a response text with action and expression intensity labels and a text tone description are generated.

[0020] S32. Combine the pre-trained generative model to generate facial expression / body movement descriptions, input the descriptions into the pre-trained model to convert them into facial expression / body movement units, and insert the generated facial expression / body movement unit information into the interaction instruction set of the corresponding node in the text.

[0021] In some implementations, S3 includes:

[0022] S33. Input the text with inserted labels into the simple estimation model, segment it based on the labels, estimate the shortest time based on the number of words, and establish length limit parameter key-value pairs to prevent overlap of action planning. At the same time, for actions with longer time estimates, insert action checkpoint labels in the middle of the text. The strength of the action unit corresponding to the label is generated by weighting the strengths of the two units before and after the action according to the action progress.

[0023] In some implementations, the generation of the facial expressions includes speaking expressions and emotional expressions. The speaking expressions are implemented by an algorithm and, upon output, are subjected to temporal smoothing sequence weighted fusion with discretized emotional expressions and spatial weight control of different facial regions.

[0024] In some implementations, the output of facial expression generation and limb movement control of the robot in S4 first extracts the motion unit intensity of the facial expression and limb movement from the text, inputs the two parameters into the parameter model of the corresponding driving element, outputs the key-value pair of the original intensity label and the action of the driving element, performs path planning based on the preceding motion unit intensity, the target motion unit intensity and the corresponding length limit parameter, inserts the final planning parameter into the key-value pair at the start position of the action to cover the original single moment action parameter of the driving element, and adds a new action checkpoint key-value pair; replaces the intensity label with a sequence number and inserts it into the text, and replaces the intensity label in the key-value pair with the corresponding sequence number for easy comparison, outputs the reply text with the expression action sequence number and the key-value pair with the sequence number as the key and the action planning parameter of the specific driving element corresponding to the sequence number position as the value.

[0025] In some implementations, the real-time adjustment instructions in S5 include a real-time interruption module, facial expression detection, and voice-action-facial expression synchronization.

[0026] In some implementations, the judgment process of the real-time interruption module is as follows: after receiving a human voice, visual face detection is performed. If the visual recognition does not detect the speaker, the target search stage is entered. If the target is not identified within a set time and the sound source information has not disappeared, the robot voice interruption is implemented and the search continues. If the target is still not detected within a set time, the robot is reset and the sound source is ignored for a certain period of time according to the set buffer parameters. If the target is identified within a set time and the sound source information has not disappeared, the voice interruption is implemented immediately. If the sound source information disappears, the voice interruption is not implemented.

[0027] The beneficial effects of this invention are as follows:

[0028] (1) This invention improves the scalability and interpretability of the system by modularly encapsulating the entire process of the multimodal interaction system of the facial expression robot and adopting a system structure of multi-model collaboration.

[0029] (2) This invention integrates the interaction context through semantic context modeling and generates an interaction instruction set through the context, making the robot feedback more unified, the interaction more contextual, and enhancing the realism of the interaction;

[0030] (3) The present invention introduces a real-time feedback mechanism to actively search for and observe the speaker, and to provide feedback on the negative emotional tendencies of the speaker, thereby enhancing the human-like interaction and reducing the mechanical feeling.

[0031] (4) The present invention introduces real-time voice feedback after echo cancellation as a synchronization supervision, which reduces the model preprocessing time and improves the real-time performance of the interaction. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the system flowchart for an embodiment;

[0033] Figure 2 This is a schematic diagram of the target detection process;

[0034] Figure 3 This is a schematic diagram of the interaction status confirmation process;

[0035] Figure 4 A schematic diagram of the information collection and interaction instruction set generation process;

[0036] Figure 5 A schematic diagram illustrating the instruction generation process;

[0037] Figure 6 This is a diagram illustrating the real-time interruption process.

[0038] Figure 7 This is a schematic diagram of the facial expression and emotion detection process;

[0039] Figure 8 This is a schematic diagram of the voice, action, and facial expression synchronization process; Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention provides a technical solution: it introduces a large language model (LLM) and cutting-edge ASR and TTS models Sensevoice and Cozyvoice for efficient processing of voice interaction and extraction of emotional information; it introduces a visual language model (VLM) for semantic modeling of the environment; and in order to ensure the real-time performance of the interaction while processing multimodal information, the system's receiving mode is divided into a "user voice receiving stage" and a "real-time feedback receiving adjustment stage".

[0042] Please see Figure 1 A robot emotion interaction system includes the following processes:

[0043] S1. Locate the interaction object and confirm the interaction status;

[0044] S2. Determine the current emotional state of the interactive object, and then perform interactive scenario modeling based on historical interaction records, current interaction information and environmental semantic recognition results. Infer the user's interaction intention and output the user's intention understanding and current interaction scenario parameters.

[0045] S3. Generate the robot's next set of interactive instructions regarding voice, facial expression, and action feedback based on the current interaction scenario parameters and user intent understanding;

[0046] S4. Generate facial expressions, control body movements, and synthesize emotional speech for robots based on interactive instruction sets;

[0047] S5: Collect real-time feedback on the emotional state of the interacting object and the environment, and adjust instructions in real time.

[0048] Among them, steps S1-S4 are the user voice reception stage, and step S5 is the real-time feedback reception and adjustment stage; during the voice recognition, the system calls the user voice-robot expression algorithm to react in real time, and this feedback loop ends when the robot voice response begins.

[0049] Regarding the search for interactive objects in step S1, target detection is performed by combining the visual module and the auditory module. Specifically, the visual module searches for interactive objects. When there are no interactive objects in the robot's field of vision, but the auditory module identifies a potential user, the robot's head movement is controlled using the sound source angle information, while the visual module continues to search for the user. After the visual module identifies the interactive user, the interaction status is confirmed by distance, eye tracking results, and specific gestures / keywords.

[0050] Please see Figure 2The specific operation is as follows: After the system starts, it first performs face target detection in the initial field of view. If a target is detected, it enters the interaction state inference stage. If no target is detected, it enters the sound source priority mode to determine the sound location. If no sound source is detected, it observes the environment, stores the environmental semantic interpretation, and then enters the standby state, that is, it repeatedly detects every second. If a sound source is detected, it locates the sound source and then adjusts the viewing angle to find the target. If a target is detected, it enters the interaction state inference stage. If no face is detected, it resets the action and updates the environmental semantic interpretation.

[0051] Please see Figure 3 The specific process for inferring the interaction state based on distance determination, eye tracking results, and specific gestures / keywords is as follows:

[0052] When the system enters the interactive state inference phase, it first performs distance detection on the target. For example, using 2m as a boundary, when the distance is less than 2m, it performs gaze tracking and determines the gaze focus by the angle between the gaze normal and the screen normal. When the angle is less than 15°, it observes the user's gaze and enters the multimodal information acquisition phase; otherwise, it only observes the target. When the distance is greater than 2m, it uses gesture or keyword wake-up methods. The gesture method is implemented by determining the gesture coordinates of consecutive frames. When a gesture such as quickly shaking the palm or making a peace sign is recognized, it enters the multimodal information acquisition phase. The keyword method receives the sound source from the microphone and compares the speech recognition result with the keyword. If the comparison is successful, it enters the multimodal information acquisition phase; otherwise, it only observes the user's gaze.

[0053] Regarding step S2, after confirming the interaction object, facial expression recognition, speech recognition, and gesture recognition are performed on the interaction object, and the historical interaction records of the interaction object are called. The current interaction information recognition results, historical interaction records, and environmental semantic recognition results are combined to establish an interaction scenario model, and the current user intent understanding and interaction scenario parameters are output.

[0054] Please see Figure 4 The current interaction information in step S2 includes multimodal information obtained through user facial expression recognition, speech recognition, posture and gesture recognition. The intent inference is based on the current interaction information, environmental semantic recognition results, and historical interaction records to infer the interaction intent, specifically:

[0055] The multimodal information collection phase includes collecting current interaction information and environmental semantic information through user facial expression recognition, speech recognition, posture and gesture recognition, as well as environmental semantic information collection and modeling: User facial expression recognition is achieved through an facial expression classification model trained on an facial expression database, with the model outputting the ratio of various emotion categories; Speech recognition is achieved through Sense Voice, which can perform tone and emotion recognition, and simultaneously recognizes the speech text content and speech emotion and labels the speaker ID; Posture and gesture recognition is achieved through predefined joint position state classification, with gestures specifically divided into static gesture recognition, such as making a peace sign, and continuous frame dynamic gesture recognition, such as clapping; Posture recognition analyzes the degree of limb relaxation through joint position change trends and identifies specific actions with corresponding text explanations, such as resting hands on the chin or supporting the face; Due to the temporal continuity of the environment, this environmental semantic modeling will also use the previous modeling results as input, and integrate them with the current VLM modeling results through a large language model.

[0056] In the interaction intent inference stage, the above multimodal information and historical interaction records are input into the context modeling model based on the large language model to obtain the current user intent understanding inference results: attributes such as time, probability of possible scenarios, and user intent.

[0057] Interaction context parameters include multimodal information beyond intent understanding, which must include speech and text.

[0058] Regarding step S3, the user intent understanding and interaction scenario parameter input response model are combined with the pre-trained action unit generation model to generate a set of instructions for the robot's next steps in facial expression generation, limb movement control, and emotional voice interaction.

[0059] Specifically, step S3 includes:

[0060] S31. By inputting user intent understanding and interaction scenario parameters into the response model, a response text with action and expression intensity labels and a text tone description are generated.

[0061] S32. Combine the pre-trained action unit generation model to generate facial expression / body action descriptions, input the descriptions into the pre-trained model to convert them into facial expression action units / body action units, and insert the generated facial expression / body action action unit information into the interactive instruction set of the corresponding node of the text.

[0062] To prevent overlap in motion planning, step S3 also includes S33.

[0063] S33. Input the text with inserted labels into the simple estimation model, segment it based on the labels, estimate the shortest time based on the number of words, and establish length limit parameter key-value pairs to prevent overlap of action planning. At the same time, for actions with longer time estimates, insert action checkpoint labels in the middle of the text. The strength of the action unit corresponding to the label is generated by weighting the strengths of the two units before and after the action according to the action progress.

[0064] Regarding step S4, the generation of facial expressions includes speaking expressions and emotional expressions. The speaking expressions are implemented through an algorithm and, when output, are fused with the discrete emotional expressions through temporal smoothing sequence weighting and spatial weight control of different facial regions, so that the robot can make more diverse expressions when speaking, while ensuring the naturalness of the expression transition; where the spatial weights are preset parameters.

[0065] Please see Figure 5 Regarding the output of facial expression generation and limb movement control for robots, the first step is to extract the intensity of the motion units for facial expressions and limb movements from the text. These two parameters are then input into the parameter models of the corresponding driving elements. The output is a key-value pair containing the original intensity label and the motion of the driving element. Path planning is performed based on the preceding motion unit intensity, the target motion unit intensity, and the corresponding length constraint parameters. The final planned parameters are inserted into the key-value pair at the start of the motion, overriding the original motion parameters of the driving element at a single moment. A new motion checkpoint key-value pair is also added. The intensity label is replaced with a sequence number inserted into the text, and the intensity label in the key-value pair is replaced with the corresponding sequence number for easy comparison. The output is a response text with the expression / motion sequence number and a key-value pair containing the sequence number as the key and the motion planning parameters of the specific driving element corresponding to the sequence number position.

[0066] The real-time adjustment instructions in step S5 include the real-time interruption module, facial expression and emotion detection, and voice-action-facial expression synchronization.

[0067] Please see Figure 6 The judgment process of the real-time interruption module is as follows:

[0068] Upon receiving a human voice, visual face detection is performed. If the speaker is not detected by visual recognition, the target search phase begins. If the target is not identified within 2 seconds and the sound source information has not disappeared, the robot's voice is interrupted and the search continues to better identify the location of the sound source. If the target is still not detected within the next 2 seconds, the robot is reset and the sound source is ignored for a certain period of time according to the set buffer parameters. If the target is identified within 2 seconds and the sound source information has not disappeared, the robot's output is interrupted immediately. If the sound source information disappears, no interruption is implemented.

[0069] To better collect voice, when the robot plays voice, it receives ambient sound through an ambient sound microphone and its own voice through a microphone next to the speaker. An echo cancellation algorithm is used to separate the ambient sound from the robot's own voice. Optionally, to better distinguish its own voice from ambient sound, a separate soundproof chamber can be built inside the robot to reduce the impact of ambient sound on the speaker microphone's reception.

[0070] To further record, speech recognition is performed on the separated ambient sounds. If a human voice is recognized, speaker ID is identified, and continuous frame mouth movement detection is performed on the received visual images. If the ID matches the ID of an existing face, information comparison is performed; otherwise, a new ID is recorded.

[0071] If an existing ID is detected and the speaker is visually recognized, the facial information is compared with the existing information. If the confidence level is higher than a set threshold, the speaker is identified, the target is gazed at, and the speech is interrupted. The process then enters the multimodal information collection stage, and the speaker ID in the current speech recognition result is changed to the current ID. If the confidence level is lower than the set threshold, the visual module result is prioritized for gazing and speech interruption, and the speaker ID is set to unknown. If a new ID is detected and the direction of the sound source is consistent with the speaker detected by vision, the correspondence between the speaker's face and ID information is recorded.

[0072] Please see Figure 7 The specific steps of the facial expression and emotion detection module are as follows:

[0073] Before the start of this round of dialogue, the user's facial expressions and postures are recorded. The real-time facial expression and emotion detection results are compared with those before the start of this reply to calculate the facial expression and emotion trend. If the negative emotion trend exceeds the threshold, the voice is interrupted, the preset facial expression softening behavior is invoked, and a question is asked.

[0074] Please see Figure 8 The specific details of the synchronized voice, gestures, and facial expressions are as follows:

[0075] The robot's speech is received via a microphone next to the speaker, and text alignment is performed to confirm the speech playback progress. Simultaneously, speech rate is calculated, and the average speech rate from historical dialogues is used as the initial default speech rate. The smooth insertion interval for facial expressions and actions is determined by the speech rate and text word count. Considering that the percentage of connection parameters does not exceed 50%, a layered judgment method is adopted. For example, when the maximum interval length is <0.2s, the maximum interval is taken; when it is >0.2s, the interval estimated by the number of words in 0.2s is taken. This time threshold can be set by the user. Next, the interval threshold is compared with the length limit parameter * estimated speech rate from step S4 to proportionally adjust the speed of the action sequence. Emotional expressions are smoothly weighted based on the interval threshold. Using text labels as a reference, the matching progress of the current frame and the starting frame of the next action / expression is linearly or non-linearly smoothly aligned with the speech progress half an interval length in advance.

[0076] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot emotional interaction system, characterized in that, The process includes the following steps: S1. Locate the interaction object and confirm the interaction status; S2. Determine the current emotional state of the interactive object, and then perform interactive scenario modeling based on historical interaction records, current interaction information and environmental semantic recognition results. Infer the user's interaction intention and output the user's intention understanding and current interaction scenario parameters. S3. Generate the robot's next set of interactive instructions regarding voice, facial expression, and action feedback based on the current interaction scenario parameters and user intent understanding; S4. Generate facial expressions, control body movements, and synthesize emotional speech for robots based on interactive instruction sets; S5: Collect real-time feedback on the emotional state of the interacting object and the environment, and adjust instructions in real time.

2. The robot emotion interaction system according to claim 1, characterized in that, The search for interactive objects in S1 involves target detection through a combination of visual and auditory modules. After the visual module identifies the interactive user, it confirms the interaction status through distance, eye tracking results, and specific gestures / keywords.

3. The robot emotion interaction system according to claim 1, characterized in that, The current interaction information in S2 includes multimodal information obtained through user facial expression recognition, voice recognition, and gesture recognition.

4. The robot emotion interaction system according to claim 3, characterized in that, The user facial expression recognition is achieved through an facial expression classification model trained on an facial expression database, and the model outputs the ratio of various emotion categories; The speech recognition is achieved through Sense Voice, which can recognize tone and emotion, that is, recognize the content of the speech text and the emotion of the speech and label the speaker ID; The posture and gesture recognition is achieved through predefined joint position state classification. The gesture recognition includes static gesture recognition and continuous frame dynamic gesture recognition. The posture recognition analyzes the degree of limb relaxation by joint position change trends and identifies specific actions with predefined corresponding text explanations.

5. A robot emotion interaction system according to claim 1, characterized in that, S3 includes: S31. By inputting user intent understanding and interaction scenario parameters into the response model, a response text with action and expression intensity labels and a text tone description are generated. S32. Combine the pre-trained action unit generation model to generate facial expression / body action descriptions, input the descriptions into the pre-trained model to convert them into facial expression action units / body action units, and insert the generated facial expression / body action action unit information into the interaction instruction set of the corresponding node of the text.

6. A robot emotion interaction system according to claim 5, characterized in that, S3 includes: S33. Input the text with inserted labels into the simple estimation model, segment it based on the labels, estimate the shortest time based on the number of words, and establish length limit parameter key-value pairs to prevent overlap of action planning. At the same time, for actions with longer time estimates, insert action checkpoint labels in the middle of the text. The strength of the action unit corresponding to the label is generated by weighting the strengths of the two units before and after the action according to the action progress.

7. The robot emotion interaction system according to claim 1, characterized in that, The generation of the facial expressions includes speaking expressions and emotional expressions. The speaking expressions are implemented by an algorithm and, when output, are fused with the discretized emotional expressions through temporal smoothing and weighted fusion, as well as weight control of different facial regions in space.

8. A robot emotion interaction system according to claim 1, characterized in that, In S4, the output regarding the robot's facial expression generation and limb movement control first extracts the motion unit intensity of facial expressions and limb movements from the text. These two parameters are then input into the parameter models of the corresponding driving elements. The original intensity label and the key-value pair of the driving element's movement are output. Path planning is performed based on the preceding motion unit intensity, the target motion unit intensity, and the corresponding length constraint parameters. The final planning parameters are inserted into the key-value pair at the start of the movement, overwriting the original single-moment driving element's movement parameters, and a new movement checkpoint key-value pair is added. The intensity label is replaced with a sequence number inserted into the text, and the intensity label in the key-value pair is replaced with the corresponding sequence number for easy comparison. A response text with the facial expression movement sequence number and a key-value pair with the sequence number as the key and the movement planning parameter of the specific driving element corresponding to the sequence number position as the value are output.

9. A robot emotion interaction system according to claim 1, characterized in that, The real-time adjustment instructions in S5 include a real-time interruption module, facial expression and emotion detection, and voice-action-facial expression synchronization.

10. A robot emotion interaction system according to claim 9, characterized in that, The judgment process of the real-time interruption module is as follows: after receiving a human voice, visual face detection is performed. If the visual recognition does not detect the speaker, the target search stage is entered. If the target is not identified within a set time and the sound source information has not disappeared, the robot voice interruption is implemented and the search continues. If the target is still not detected within a set time, the robot is reset and the sound source is ignored for a certain period of time according to the set buffer parameters. If the target is identified within a set time and the sound source information has not disappeared, the voice interruption is implemented immediately. If the sound source information disappears, the voice interruption is not implemented.

Images