A contextualized interactive ai emotional companion robot system

Abstract

The application discloses a scene interactive AI emotional companion robot system, comprising a collection module for collecting multi-modal perception data of a target object in a physical interaction process. The application processes visual, auditory and tactile data through a multi-modal fusion neural network to generate a real-time emotion feature vector containing a degree of pleasure, a degree of arousal and a degree of dominance, significantly improving the recognition accuracy of complex micro-expressions and implicit emotions. Based on emotion state labels and historical logs, a customized script and logic tree are generated in real time using an AIGC engine, realizing a transition from "passive questioning and answering" to "active care", and enhancing the personalization and interest of the companionship. Through an ambient light dynamic compensation algorithm and depth occlusion calculation, the projection content adapts to changes in lighting and correctly occludes real objects, and combined with the synchronous physical feedback of the physical robot, the boundaries between virtual and reality are broken, thus realizing high-quality remote empathy and educational guidance.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a scenario-based interactive AI emotional companion robot system. Background Technology

[0002] Currently, with the deepening of societal aging, the expansion of the single-person living group, and the diversification of children's needs for companionship, emotional companionship smart devices have become a research hotspot in the field of human-computer interaction. Existing emotional companionship robots and intelligent care terminals have initially achieved basic functions such as voice dialogue, story playback, and timed reminders, alleviating the emotional companionship gap for specific groups to some extent. However, as users' demands for immersive companionship, personalization, and emotional resonance continue to increase, existing technological solutions are gradually revealing several technical shortcomings, as follows: First, the emotional perception dimension is too narrow, and the accuracy of multimodal fusion recognition is insufficient. Existing emotional companionship systems mostly rely on voice semantics or single facial expressions as the basis for emotion judgment, failing to effectively integrate multi-dimensional perception signals such as tactile interaction and body movements, making it difficult to capture users' implicit emotions and micro-expression changes. Especially in scenarios where users' emotional expression is relatively restrained and facial features are not obvious, the results of single-modal emotion recognition have significant deviations, failing to accurately map the user's true level of pleasure, arousal, and emotional dominance, thus leading to a mismatch between subsequent interaction strategies and the user's actual emotional needs.

[0003] Secondly, the interactive content is static and passive, lacking the ability to generate personalized and dynamic content. Existing companion devices mostly use pre-written scripts and fixed question-and-answer databases, employing a passive "user-triggered - system-responded" interaction model. This fails to dynamically generate customized, contextualized content based on the user's real-time emotional state and historical interests, making it difficult to achieve proactive emotional care. Furthermore, in existing solutions, content generation and interaction logic are independent, lacking a logical tree linking physical actions and visual presentation. This results in insufficient interaction coherence and immersion, leading to user fatigue after prolonged use and low product retention rates.

[0004] Third, the integration of virtual and real interaction is low, and the coordination between projection presentation and physical feedback is poor. Existing companion products mostly use either purely physical robots or purely virtual digital humans as the sole interaction medium: physical robots have limited interaction methods, making it difficult to construct rich virtual scenes; purely virtual solutions lack physical touch and physical feedback, resulting in insufficient realism. Some solutions that combine projection technology have not solved problems such as adaptive ambient lighting and realistic occlusion by physical obstacles. The projected image is easily affected by ambient light, and the occlusion relationship between virtual content and real-world objects does not conform to visual logic, resulting in a clear disconnect between virtual and real. Furthermore, the lack of a synchronized linkage mechanism between the physical robot's actions and the virtual scene content further reduces the immersive experience of the interaction.

[0005] Fourth, the development of behavioral habits lacks contextualization and proactive intervention mechanisms. Existing companionship systems often employ straightforward reminders and didactic methods for habit guidance and behavioral education, failing to transform daily behavioral education into engaging and contextualized interactive tasks, resulting in low user acceptance and participation. Furthermore, existing solutions lack real-time monitoring and proactive intervention mechanisms for user attention states and emotional shifts during interaction. When users experience distraction, negative emotional fluctuations, or interruptions in interaction, the system cannot proactively adjust its interaction strategies to regain user attention, leading to poor interaction continuity and companionship effectiveness.

[0006] Fifth, the forms of remote companionship are limited, and the depth of emotional transmission is insufficient. Existing remote companionship relies mainly on basic communication methods such as video calls and voice messages, which only achieve the direct transmission of information. It cannot deeply integrate the voice and content of the remote user with the local companionship scenario, nor can it synchronize the real-time emotional state and interaction process of the person being accompanied to the remote end. The emotional carrying capacity of remote companionship is weak, making it difficult to build an immersive remote companionship experience. Summary of the Invention

[0007] To address the aforementioned technical problems, a scenario-based interactive AI emotional companion robot system is provided. This technical solution resolves the issues raised in the background section.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A scenario-based interactive AI emotional companion robot system includes: The acquisition module is used to acquire multimodal perception data of the target object during physical interaction. The multimodal perception data includes tactile interaction signals acquired by the anthropomorphic physical robot, acoustic feature data acquired by the voice acquisition module, and facial micro-expression images and limb skeletal key point data acquired by the visual acquisition module. The extraction and fusion module is used to extract and fuse multimodal perception data in real time, generate a real-time emotion feature vector of the target object, and map the real-time emotion feature vector to a preset emotion state space to determine the current emotion state label. The construction module is used to dynamically construct a virtual scene script and interaction logic tree by calling the AIGC generation engine based on the current emotional state label and historical interaction logs; The monitoring module is used to transform daily habitual behavior education into fun tasks based on the generated virtual scene script and the historical behavior data of the target object. The projection module is used to control the projection device to project virtual scene content onto the projection medium according to the interaction logic tree, to perform remote companionship and data synchronization, and to synchronously control the drive unit of the physical robot to perform physical feedback actions that match the virtual scene content. The interaction module is used to continuously monitor the feedback data of the target object during the interaction process. If the feedback data indicates that the interaction is interrupted or the emotional state label has a negative shift, an active intervention strategy is triggered to adjust the generation parameters of the virtual scene content or the physical feedback action of the physical robot.

[0009] Preferably, the step of real-time feature extraction and fusion of multimodal perception data to generate a real-time emotion feature vector of the target object specifically includes: Key point detection is performed on facial micro-expression images acquired by the visual acquisition module. Motion amplitude parameters of the orbicularis oculi, zygomaticus major, and corrugator supercilii muscles are extracted. Combined with limb skeletal key point data, the acceleration and angular change rate of limb movements are calculated to generate the first emotional feature subset. ; The acoustic feature data acquired by the speech acquisition module is subjected to fundamental frequency extraction and Mel-frequency cepstral coefficient analysis to separate speech content and prosodic features. The slope of speech intonation, intensity variance, and pause duration are extracted to generate a second subset of emotion features. ; Temporal analysis of tactile interaction signals was performed to extract the pressure value, touch duration, and curvature parameters of the swipe trajectory, generating a third subset of emotional features. ; The first, second, and third subsets of emotion features are input into a pre-trained multimodal fusion neural network model, which outputs a normalized real-time emotion feature vector. The specific calculation formula is as follows:

[0010] in, It is a three-dimensional emotion feature vector. Represents pleasure level, Represents wakefulness. Represents dominance; , , These are the weight matrices for the visual, auditory, and tactile modalities, respectively. b For bias vectors, It is a non-linear activation function.

[0011] Preferably, the step of dynamically constructing a virtual scene script based on the current emotional state tag and historical interaction logs by calling the AIGC generation engine specifically includes: Extract the target object's set of interest keywords and high-frequency interaction time periods from historical interaction logs, and calculate the interest weight coefficient. ; Based on the coordinates of the current emotional state label in the emotional state space, a preset emotional-script mapping table is queried to obtain the basic script framework, which includes the story background, character settings and core conflict. The interest weight coefficient is input into the AIGC generation engine as a conditional control parameter to rewrite the non-core plots in the basic script framework and generate customized story text that includes the target object's name, recent experiences and preferred items. Based on natural language processing technology, customized story text is parsed, key interaction nodes are extracted, and a branch logic tree is constructed. Each branch node is associated with a specific set of physical robot action instructions and projection visual effect parameters. The interest weight coefficient The calculation formula is:

[0012] in, For the first k The weight coefficient of each interest keyword; For the first k The frequency of occurrence of each keyword within a unit of time; The average duration of a single interaction involving the k-th keyword; The time interval between the most recent occurrence of this keyword and the current time. This is the time decay factor, used to characterize the rate at which interest fades over time; This represents the total number of keyword sets.

[0013] Preferably, the construction of the virtual scene script includes a dedicated family theater scene, specifically including: Receive fragments of real-life events mentioned by the target object in natural language dialogue, perform named entity recognition and time series alignment on the event fragments, and extract the event subject, location and emotional color tags; The AIGC generation engine is invoked, and the extracted event subject and emotional color tags are used as generation conditions. A preset dramatic structure algorithm is applied to generate a first draft of the script, which includes four acts: opening, development, climax and ending. Based on the real-time voice feedback of the target audience, semantic similarity and emotional matching degree are calculated. If the emotional matching degree is lower than the preset threshold, the lines of the current scene are rewritten in real time to add humorous elements or reduce the difficulty of conflict, and the probability of the direction of subsequent scenes is updated simultaneously. The emotional compatibility The calculation formula is: in, The total number of statements in the current act; and The first j The values ​​of pleasure and arousal in the real-time emotional feature vector of the target object corresponding to each line of dialogue; and The first j The dialogue is pre-set with baseline values ​​for pleasure and arousal in terms of emotional intensity; when Below the threshold At that time, the script rewriting mechanism is triggered.

[0014] Preferably, the generation of the interaction logic tree specifically includes: On the physical robot side, the real-time emotional feature vector of the target object, the text record of the interaction content, and the visual perception image features are encrypted and encoded to generate an interaction feature data package. The interactive feature data package is uploaded to the cloud server and pushed to the bound associated authorized terminals in real time. After receiving the interactive feature data package, the associated authorized terminals decode and visualize it, generating an emotion fluctuation curve and an interactive content summary on the terminal display screen. Receive remote intervention commands from associated authorized terminals. These commands may include remotely recorded audio data, custom story text, or specific reminder timestamps. The audio data recorded remotely is used to extract voiceprint features. A voice cloning algorithm is used to train and generate a voice model of a specified speaker. The custom story text is then input into the AIGC generation engine for style adaptation to generate multimedia content to be played.

[0015] Preferably, the step of extracting voiceprint features from remotely recorded audio data and training a specified speaker's voice model using a voice cloning algorithm specifically includes: The remotely recorded audio data is subjected to noise reduction and silence removal, and then segmented into short frame sequences containing complete phonemes. Mel spectrograms of each short-time frame sequence are extracted as voiceprint feature maps; Input the voiceprint feature map into a pre-trained voiceprint encoder to extract the speaker embedding vector; The speaker embedding vector and the phoneme sequence of the target text are input together into the vocoder model to synthesize target speech waveform data with the timbre characteristics of the specified speaker. While the physical robot plays the target voice waveform data, the projection device is simultaneously controlled to display the associated images or dynamic effects uploaded by the authorized terminal. The speaker embedding vector The extraction calculation follows the formula below:

[0016] in, The final speaker embedding vector; The total number of short-time frame sequences input; For the first m Mel spectrogram data of the frame; is a nonlinear mapping function for the voiceprint encoder, used to map spectral features to a high-dimensional vector space.

[0017] Preferably, the process of cultivating daily habits and behaviors into engaging tasks specifically includes: Acquire historical behavioral data of the target object, identify recurring inefficient behavioral patterns, and generate a list of habits to be optimized; Retrieve educational data related to the list of habits to be optimized from the contextualized content ecosystem, and construct task scenarios, such as mapping the behavior of "brushing teeth" to the game scenario of "cleaning magic gems"; Set task triggering conditions and reward mechanisms, and calculate the action completion parameter when the target object is detected to perform a specific action; If the action completion parameters reach the preset standard, the AIGC generation engine is called to generate an instant reward animation, which is then projected onto the target object's hand or object surface via a projection device. At the same time, the physical robot outputs specific tactile vibration feedback. The action completion parameter The calculation formula is: in, Score the degree of completion of the action; The actual duration of the action. Recommended duration for standard movements; This refers to the real-time skeletal keypoint coordinate vectors extracted by the vision acquisition module. The coordinate vector of the skeletal key points for the standard movement; and These are the weighting coefficients, and ; It is a spatial distance penalty factor used to measure the standardization of action postures.

[0018] Preferably, the step of controlling the projection device to project virtual scene content onto the projection medium includes: Acquire a planar depth image of the projection medium and identify the coordinates of physical obstacles and surface texture features in the projection area; Generate a two-dimensional projection mapping matrix based on the character position information in the virtual scene content; The two-dimensional projection mapping matrix is ​​used to perform perspective transformation and occlusion calculation on the rendered image of the virtual character to ensure that the virtual character is visually behind physical obstacles or has a correct occlusion relationship with real objects. Real-time acquisition of ambient light sensor data dynamically adjusts the brightness and contrast parameters of the projected image, ensuring that the color gamut value of the projected content maintains a linear compensation relationship with the ambient light illuminance. The brightness of the projected image The dynamic adjustment calculation formula is as follows:

[0019] in, This is the adjusted projection brightness output value; Set the base brightness value for the projected content; The ambient illuminance value is collected in real time by the ambient light sensor. This is a standard ambient illuminance reference value; This is the brightness compensation coefficient, used to adjust the sensitivity of the projected brightness to changes in ambient light. hour ,on the contrary .

[0020] Preferably, the triggering of the active intervention strategy, adjusting the generation parameters of the virtual scene content or the physical feedback actions of the physical robot, specifically includes: Establish a boredom monitoring model to calculate the interaction frequency and attention distraction of the target object in real time per unit time. When the interaction frequency is lower than the first preset threshold and the attention distraction is higher than the second preset threshold, the target object is determined to be in a bored state. Open-ended questions related to the current virtual scene are randomly selected from the candidate question library of the AIGC generation engine, and the difficulty coefficient of the questions is adjusted according to the historical answer records of the target object. The robot is controlled to turn towards the target object and performs a wake-up action by slightly tilting its head and flashing the LED light source in its eyes, while generating guiding visual cues in the edge area of ​​the projection medium. The degree of attentional distraction The calculation formula is:

[0021] in, It is an index of attentional distractibility; The total number of frames within a unit time window; For the first The coordinates of the human eye gaze point detected in the frame; For the first The center coordinates of the frame virtual interactive object; The fixation deviation threshold; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0022] Preferably, it also includes model self-updating, specifically: Periodically collect the target object's dwell time, number of interactions, and changes in emotional feature vectors for different types of virtual scene content to construct a preference-feedback association dataset; The classification threshold of the emotion recognition model is fine-tuned using the preference-feedback association dataset. Specifically, the gradient descent value of the classification boundary between positive and negative emotions is calculated, and the weight parameters of the classifier are adjusted. The script structure of highly interactive virtual scenes is extracted as template features, stored in the preferred template library of the scenario-based content ecosystem, and the sampling probability of this template feature is increased when generating new scripts in the future. The classifier weight parameters The fine-tuning update formula is:

[0023] in, The weight parameters before the update. The updated weight parameters; The learning rate controls the update step size; For loss function Regarding weight The gradient, the loss function is based on the real sentiment label. Compared with model predictions Error calculation between; This is the regularization coefficient, used to prevent the model from overfitting; is the L2 norm of the weights.

[0024] Compared with existing technologies, the present invention provides a scenario-based interactive AI emotional companion robot system, which has the following beneficial effects: This invention processes visual, auditory, and tactile data through a multimodal fusion neural network to generate real-time emotional feature vectors containing pleasure, arousal, and dominance, significantly improving the accuracy of recognizing complex micro-expressions and implicit emotions. Based on emotional state tags and historical logs, it uses an AIGC engine to generate customized scripts and logic trees in real time, realizing a shift from "passive questioning" to "proactive care," enhancing the personalization and fun of companionship. Through dynamic ambient light compensation algorithms and depth occlusion calculations, the projected content adapts to changes in lighting and correctly occludes with real objects. Combined with synchronous physical feedback from a physical robot, it breaks down the boundaries between virtual and reality, thereby achieving high-quality remote empathy and educational guidance. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the system framework structure of the present invention; Figure 2 This is a schematic diagram of the method flow for S201-S204 in this invention; Figure 3 This is a schematic diagram of the method flow for S301-S304 in this invention; Figure 4 This is a schematic diagram of the method flow for S401-S403 in this invention. Detailed Implementation

[0026] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0027] Example 1 Please refer to Figure 1 As shown, a scenario-based interactive AI emotional companion robot system includes: The acquisition module 101 is used to acquire multimodal perception data of the target object during physical interaction. The multimodal perception data includes tactile interaction signals acquired by the anthropomorphic entity robot, acoustic feature data acquired by the voice acquisition module, and facial micro-expression images and limb skeletal key point data acquired by the vision acquisition module. The extraction and fusion module 102 is used to extract and fuse multimodal perception data in real time, generate a real-time emotion feature vector of the target object, and map the real-time emotion feature vector to a preset emotion state space to determine the current emotion state label. Module 103 is used to dynamically construct virtual scene scripts and interaction logic trees by calling the AIGC generation engine based on the current emotion state tags and historical interaction logs. The monitoring module 104 is used to transform daily habit behavior education into fun tasks based on the generated virtual scene script and combined with the target object's historical behavior data. Projection module 105 is used to control the projection device to project virtual scene content onto the projection medium according to the interaction logic tree, to provide remote companionship and data synchronization, and to synchronously control the drive unit of the physical robot to perform physical feedback actions that match the virtual scene content. Interaction module 106 is used to continuously monitor the feedback data of the target object during the interaction process. If the feedback data indicates that the interaction is interrupted or the emotional state label has a negative shift, an active intervention strategy is triggered to adjust the generation parameters of the virtual scene content or the physical feedback action of the physical robot.

[0028] As will be understood by those skilled in the art, this invention processes visual, auditory, and tactile data through a multimodal fusion neural network to generate real-time emotional feature vectors containing pleasure, arousal, and dominance, significantly improving the accuracy of recognizing complex micro-expressions and implicit emotions. Based on emotional state tags and historical logs, it utilizes an AIGC engine to generate customized scripts and logic trees in real time, realizing a shift from "passive questioning" to "proactive care," enhancing the personalization and fun of companionship. Through dynamic ambient light compensation algorithms and depth occlusion calculations, the projected content adapts to changes in lighting and correctly occludes with real objects, combined with synchronous physical feedback from the physical robot, breaking down the boundaries between virtual and reality.

[0029] Please refer to Figure 2 As shown, real-time feature extraction and fusion of multimodal perception data are performed to generate a real-time emotion feature vector of the target object, specifically including: S201. Perform key point detection on the facial micro-expression images acquired by the visual acquisition module, extract the motion amplitude parameters of the orbicularis oculi muscle, zygomaticus major muscle, and corrugator supercilii muscle, and calculate the acceleration and angular change rate of limb movements by combining the key point data of the limb skeleton, and generate the first emotional feature subset. ; S202. Perform fundamental frequency extraction and Mel-frequency cepstral coefficient analysis on the acoustic feature data acquired by the speech acquisition module to separate speech content and prosodic features, extract the slope of speech intonation, intensity variance, and pause duration, and generate a second emotional feature subset. ; S203. Perform time-domain analysis on the tactile interaction signal, extract the pressing pressure value, touch duration, and curvature parameters of the sliding trajectory, and generate a third emotional feature subset. ; S204. Input the first emotional feature subset, the second emotional feature subset, and the third emotional feature subset into the pre-trained multimodal fusion neural network model, and output a normalized real-time emotional feature vector. The specific calculation formula is as follows:

[0030] in, It is a three-dimensional emotion feature vector. Represents pleasure level, Represents wakefulness. Represents dominance; , , These are the weight matrices for the visual, auditory, and tactile modalities, respectively. b For bias vectors, It is a non-linear activation function.

[0031] Please refer to Figure 3 As shown, based on the current emotional state label and historical interaction logs, the AIGC generation engine is invoked to dynamically construct a virtual scene script, specifically including: S301. Extract the set of interest keywords and high-frequency interaction time periods of the target object from the historical interaction logs, and calculate the interest weight coefficient. ; S302. Based on the coordinates of the current emotional state label in the emotional state space, query the preset emotional-script mapping table to obtain the basic script framework, which includes the story background, character settings and core conflict. S303. Input the interest weight coefficient as a conditional control parameter into the AIGC generation engine to rewrite the non-core plots in the basic script framework and generate customized story text containing the target object's name, recent experiences, and preferred items. S304. Based on natural language processing technology, the customized story text is parsed, key interaction nodes are extracted, and a branch logic tree is constructed. Each branch node is associated with a specific set of physical robot action instructions and projection visual effect parameters. Interest weight coefficient The calculation formula is:

[0032] in, For the first k The weight coefficient of each interest keyword; For the first k The frequency of occurrence of each keyword within a unit of time; The average duration of a single interaction involving the k-th keyword; The time interval between the most recent occurrence of this keyword and the current time. This is the time decay factor, used to characterize the rate at which interest fades over time; This represents the total number of keyword sets.

[0033] Please refer to Figure 4 As shown, the construction of the virtual scene script includes a dedicated home theater scene, specifically including: S401. Receive fragments of real-life events mentioned by the target in natural language dialogue, perform named entity recognition and time series alignment on the event fragments, and extract the event subject, location and emotional color labels. S402. Call the AIGC generation engine, use the extracted event subject and emotional color tags as generation conditions, and apply the preset dramatic structure algorithm to generate the first draft of the script. The first draft of the script includes four acts: opening, development, climax and ending. S403. Based on the real-time voice feedback of the target object, calculate the semantic similarity and emotional matching degree. If the emotional matching degree is lower than the preset threshold, rewrite the lines of the current scene in real time, add humorous elements or reduce the difficulty of conflict, and update the probability of the direction of subsequent scenes in a synchronous manner. Emotional compatibility The calculation formula is: in, The total number of statements in the current act; and The first j The values ​​of pleasure and arousal in the real-time emotional feature vector of the target object corresponding to each line of dialogue; and The first j The dialogue is pre-set with baseline values ​​for pleasure and arousal in terms of emotional intensity; when Below the threshold At that time, the script rewriting mechanism is triggered.

[0034] Generate an interaction logic tree, specifically including: On the physical robot side, the real-time emotional feature vector of the target object, the text record of the interaction content, and the visual perception image features are encrypted and encoded to generate an interaction feature data package. The interactive feature data package is uploaded to the cloud server and pushed to the bound associated authorized terminals in real time. After receiving the interactive feature data package, the associated authorized terminals decode and visualize it, generating an emotion fluctuation curve and an interactive content summary on the terminal display screen. Receive remote intervention commands from associated authorized terminals. These commands may include remotely recorded audio data, custom story text, or specific reminder timestamps. The audio data recorded remotely is used to extract voiceprint features. A voice cloning algorithm is used to train and generate a voice model of a specified speaker. The custom story text is then input into the AIGC generation engine for style adaptation to generate multimedia content to be played.

[0035] The process involves extracting voiceprint features from remotely recorded audio data and using a timbre cloning algorithm to train and generate a timbre model for a specified speaker. Specifically, this includes: The remotely recorded audio data is subjected to noise reduction and silence removal, and then segmented into short frame sequences containing complete phonemes. Mel spectrograms of each short-time frame sequence are extracted as voiceprint feature maps; Input the voiceprint feature map into a pre-trained voiceprint encoder to extract the speaker embedding vector; The speaker embedding vector and the phoneme sequence of the target text are input together into the vocoder model to synthesize target speech waveform data with the timbre characteristics of the specified speaker. While the physical robot plays the target voice waveform data, the projection device is simultaneously controlled to display the associated images or dynamic effects uploaded by the authorized terminal. Speaker embedding vector The extraction calculation follows the formula below:

[0036] in, The final speaker embedding vector; The total number of short-time frame sequences input; For the first m Mel spectrogram data of the frame; is a nonlinear mapping function for the voiceprint encoder, used to map spectral features to a high-dimensional vector space.

[0037] Transforming daily habit and behavior education into fun tasks, specifically including: Acquire historical behavioral data of the target object, identify recurring inefficient behavioral patterns, and generate a list of habits to be optimized; Retrieve educational data related to the list of habits to be optimized from the contextualized content ecosystem, and construct task scenarios, such as mapping the behavior of "brushing teeth" to the game scenario of "cleaning magic gems"; Set task triggering conditions and reward mechanisms, and calculate the action completion parameter when the target object is detected to perform a specific action; If the action completion parameters reach the preset standard, the AIGC generation engine is called to generate an instant reward animation, which is then projected onto the target object's hand or object surface via a projection device. At the same time, the physical robot outputs specific tactile vibration feedback. Action completion parameters The calculation formula is: in, Score the degree of completion of the action; The actual duration of the action. Recommended duration for standard movements; This refers to the real-time skeletal keypoint coordinate vectors extracted by the vision acquisition module. The coordinate vector of the skeletal key points for the standard movement; and These are the weighting coefficients, and ; It is a spatial distance penalty factor used to measure the standardization of action postures.

[0038] The steps of controlling the projection device to project virtual scene content onto the projection medium include: Acquire a planar depth image of the projection medium and identify the coordinates of physical obstacles and surface texture features in the projection area; Generate a two-dimensional projection mapping matrix based on the character position information in the virtual scene content; A two-dimensional projection mapping matrix is ​​used to perform perspective transformation and occlusion calculation on the rendered image of the virtual character to ensure that the virtual character is visually behind physical obstacles or has a correct occlusion relationship with real objects. Real-time acquisition of ambient light sensor data dynamically adjusts the brightness and contrast parameters of the projected image, ensuring that the color gamut value of the projected content maintains a linear compensation relationship with the ambient light illuminance. Projection screen brightness The dynamic adjustment calculation formula is as follows:

[0039] in, This is the adjusted projection brightness output value; Set the base brightness value for the projected content; The ambient illuminance value is collected in real time by the ambient light sensor. This is a standard ambient illuminance reference value; This is the brightness compensation coefficient, used to adjust the sensitivity of the projected brightness to changes in ambient light. hour ,on the contrary .

[0040] Triggering proactive intervention strategies to adjust the generation parameters of virtual scene content or the physical feedback actions of physical robots, specifically including: Establish a boredom monitoring model to calculate the interaction frequency and attention distraction of the target object in real time per unit time. When the interaction frequency is lower than the first preset threshold and the attention distraction is higher than the second preset threshold, the target object is determined to be in a bored state. Open-ended questions related to the current virtual scene are randomly selected from the candidate question library of the AIGC generation engine, and the difficulty coefficient of the questions is adjusted according to the historical answer records of the target object. The robot is controlled to turn towards the target object and performs a wake-up action by slightly tilting its head and flashing the LED light source in its eyes, while generating guiding visual cues in the edge area of ​​the projection medium. Distraction The calculation formula is:

[0041] in, It is an index of attentional distractibility; The total number of frames within a unit time window; For the first The coordinates of the human eye gaze point detected in the frame; For the first The center coordinates of the frame virtual interactive object; The fixation deviation threshold; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0042] This also includes model self-updating, specifically: Periodically collect the target object's dwell time, number of interactions, and changes in emotional feature vectors for different types of virtual scene content to construct a preference-feedback association dataset; The classification threshold of the emotion recognition model is fine-tuned using a preference-feedback association dataset. Specifically, the gradient descent value of the classification boundary between positive and negative emotions is calculated, and the weight parameters of the classifier are adjusted. The script structure of highly interactive virtual scenes is extracted as template features, stored in the preferred template library of the scenario-based content ecosystem, and the sampling probability of this template feature is increased when generating new scripts in the future. Classifier weight parameters The fine-tuning update formula is:

[0043] in, The weight parameters before the update. The updated weight parameters; The learning rate controls the update step size; For loss function Regarding weight The gradient, the loss function is based on the real sentiment label. Compared with model predictions Error calculation between; This is the regularization coefficient, used to prevent the model from overfitting; is the L2 norm of the weights.

[0044] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A scenario-based interactive AI emotional companion robot system, characterized in that, include: The acquisition module is used to acquire multimodal perception data of the target object during physical interaction. The multimodal perception data includes tactile interaction signals acquired by the anthropomorphic physical robot, acoustic feature data acquired by the voice acquisition module, and facial micro-expression images and limb skeletal key point data acquired by the visual acquisition module. The extraction and fusion module is used to extract and fuse multimodal perception data in real time, generate a real-time emotion feature vector of the target object, and map the real-time emotion feature vector to a preset emotion state space to determine the current emotion state label. The construction module is used to dynamically construct a virtual scene script and interaction logic tree by calling the AIGC generation engine based on the current emotional state label and historical interaction logs; The monitoring module is used to transform daily habitual behavior education into fun tasks based on the generated virtual scene script and the historical behavior data of the target object. The projection module is used to control the projection device to project virtual scene content onto the projection medium according to the interaction logic tree, to provide remote companionship and data synchronization, and to synchronously control the drive unit of the physical robot to perform physical feedback actions that match the virtual scene content. The interaction module is used to continuously monitor the feedback data of the target object during the interaction process. If the feedback data indicates that the interaction is interrupted or the emotional state label has a negative shift, an active intervention strategy is triggered to adjust the generation parameters of the virtual scene content or the physical feedback action of the physical robot.

2. The scenario-based interactive AI emotional companion robot system according to claim 1, characterized in that, The process of real-time feature extraction and fusion of multimodal perception data to generate a real-time emotion feature vector of the target object specifically includes: Key point detection is performed on facial micro-expression images acquired by the visual acquisition module to extract the motion amplitude parameters of the orbicularis oculi, zygomaticus major, and corrugator supercilii muscles. Combined with limb skeletal key point data, the acceleration and angular change rate of limb movements are calculated to generate the first emotional feature subset. ; The acoustic feature data acquired by the speech acquisition module is subjected to fundamental frequency extraction and Mel-frequency cepstral coefficient analysis to separate speech content and prosodic features. The slope of speech intonation, intensity variance, and pause duration are extracted to generate a second subset of emotion features. ; Temporal analysis of tactile interaction signals was performed to extract the pressure value, touch duration, and curvature parameters of the swipe trajectory, generating a third subset of emotional features. ; The first, second, and third emotion feature subsets are input into a pre-trained multimodal fusion neural network model, which outputs a normalized real-time emotion feature vector. The specific calculation formula is as follows: ;in, It is a three-dimensional emotion feature vector. Represents pleasure level, Represents wakefulness. Represents dominance; , , These are the weight matrices for the visual, auditory, and tactile modalities, respectively. b For bias vectors, It is a non-linear activation function.

3. The scenario-based interactive AI emotional companion robot system according to claim 2, characterized in that, The process of dynamically constructing a virtual scene script based on the current emotional state label and historical interaction logs by calling the AIGC generation engine specifically includes: Extract the target object's set of interest keywords and high-frequency interaction time periods from historical interaction logs, and calculate the interest weight coefficient. ; Based on the coordinates of the current emotional state label in the emotional state space, a preset emotional-script mapping table is queried to obtain the basic script framework, which includes the story background, character settings and core conflict. The interest weight coefficient is input into the AIGC generation engine as a conditional control parameter to rewrite the non-core plots in the basic script framework and generate customized story text that includes the target object's name, recent experiences and preferred items. Based on natural language processing technology, customized story text is parsed, key interaction nodes are extracted, and a branch logic tree is constructed. Each branch node is associated with specific entity robot action indexes and projection visual effect parameters. The interest weight coefficient The calculation formula is: ;in, For the first k The weighting coefficient of each interest keyword; For the first k The frequency of occurrence of each keyword within a unit of time; The average duration of a single interaction involving the k-th keyword; The time interval between the most recent occurrence of this keyword and the current time. This is the time decay factor, used to characterize the rate at which interest fades over time; This represents the total number of keyword sets.

4. The scenario-based interactive AI emotional companion robot system according to claim 3, characterized in that, The constructed virtual scene script includes a dedicated home theater scene, specifically including: Receive fragments of real-life events mentioned by the target object in natural language dialogue, perform named entity recognition and time series alignment on the event fragments, and extract the event subject, location and emotional color tags; The AIGC generation engine is invoked, and the extracted event subject and emotional color tags are used as generation conditions. A preset dramatic structure algorithm is applied to generate a first draft of the script, which includes four acts: opening, development, climax and ending. Based on the real-time voice feedback of the target audience, semantic similarity and emotional matching degree are calculated. If the emotional matching degree is lower than the preset threshold, the lines of the current scene are rewritten in real time to add humorous elements or reduce the difficulty of conflict, and the probability of the direction of subsequent scenes is updated simultaneously. The emotional compatibility The calculation formula is: ;in, The total number of statements in the current act; and The first j The values ​​of pleasure and arousal in the real-time emotional feature vector of the target object corresponding to each line of dialogue; and The first j The dialogue is pre-set with baseline values ​​for pleasure and arousal in terms of emotional intensity; when Below the threshold At that time, the script rewriting mechanism is triggered.

5. The scenario-based interactive AI emotional companion robot system according to claim 4, characterized in that, The generation of the interaction logic tree specifically includes: On the physical robot side, the real-time emotional feature vector of the target object, the text record of the interaction content, and the visual perception image features are encrypted and encoded to generate an interaction feature data package. The interactive feature data package is uploaded to the cloud server and pushed to the bound associated authorized terminals in real time. After receiving the interactive feature data package, the associated authorized terminals decode and visualize it, generating an emotion fluctuation curve and an interactive content summary on the terminal display screen. Receive remote intervention commands from associated authorized terminals. These commands may include remotely recorded audio data, custom story text, or specific reminder timestamps. The audio data recorded remotely is used to extract voiceprint features. A voice cloning algorithm is used to train and generate a voice model for a specified speaker. The custom story text is then input into the AIGC generation engine for style adaptation to generate multimedia content to be played.

6. The scenario-based interactive AI emotional companion robot system according to claim 5, characterized in that, The step of extracting voiceprint features from remotely recorded audio data and training a specified speaker's voice model using a voice cloning algorithm specifically includes: The remotely recorded audio data is subjected to noise reduction and silence removal, and then segmented into short frame sequences containing complete phonemes. Mel spectrograms of each short-time frame sequence are extracted as voiceprint feature maps; Input the voiceprint feature map into a pre-trained voiceprint encoder to extract the speaker embedding vector; The speaker embedding vector and the phoneme sequence of the target text are input together into the vocoder model to synthesize target speech waveform data with the timbre characteristics of the specified speaker. While the physical robot plays the target voice waveform data, the projection device is simultaneously controlled to display the associated images or dynamic effects uploaded by the authorized terminal. The speaker embedding vector The extraction calculation follows the formula below: ;in, The final speaker embedding vector; The total number of short-time frame sequences input; For the first m Mel spectrogram data of the frame; is a nonlinear mapping function for the voiceprint encoder, used to map spectral features to a high-dimensional vector space.

7. The scenario-based interactive AI emotional companion robot system according to claim 6, characterized in that, The method of transforming daily habit and behavior education into fun tasks specifically includes: Acquire historical behavioral data of the target object, identify recurring inefficient behavioral patterns, and generate a list of habits to be optimized; Retrieve educational data related to the list of habits to be optimized from the contextualized content ecosystem, and construct task scenarios, such as mapping the "brushing teeth" behavior to the game scenario of "cleaning magic gems"; Set task triggering conditions and reward mechanisms, and calculate the action completion parameter when the target object is detected to perform a specific action; If the action completion parameters reach the preset standard, the AIGC generation engine is called to generate an instant reward animation, which is then projected onto the target object's hand or object surface via a projection device. At the same time, the physical robot outputs specific tactile vibration feedback. The action completion parameter The calculation formula is: ;in, Score the degree of completion of the action; The actual duration of the action. Recommended duration for standard movements; This refers to the real-time skeletal keypoint coordinate vectors extracted by the vision acquisition module. The coordinate vector of the skeletal key points for the standard movement; and These are the weighting coefficients, and ; It is a spatial distance penalty factor used to measure the standardization of action postures.

8. The scenario-based interactive AI emotional companion robot system according to claim 7, characterized in that, The step of controlling the projection device to project virtual scene content onto the projection medium includes: Acquire a planar depth image of the projection medium and identify the coordinates of physical obstacles and surface texture features in the projection area; Generate a two-dimensional projection mapping matrix based on the character position information in the virtual scene content; The two-dimensional projection mapping matrix is ​​used to perform perspective transformation and occlusion calculation on the rendered image of the virtual character to ensure that the virtual character is visually behind physical obstacles or has a correct occlusion relationship with real objects. Real-time acquisition of ambient light sensor data dynamically adjusts the brightness and contrast parameters of the projected image, ensuring that the color gamut value of the projected content maintains a linear compensation relationship with the ambient light illuminance. The brightness of the projected image The dynamic adjustment calculation formula is as follows: ;in, This is the adjusted projection brightness output value; Set the base brightness value for the projected content; The ambient illuminance value is collected in real time by the ambient light sensor. This is a standard ambient illuminance reference value; This is the brightness compensation coefficient, used to adjust the sensitivity of the projected brightness to changes in ambient light. hour ,on the contrary .

9. A scenario-based interactive AI emotional companion robot system according to claim 8, characterized in that, The triggering of the active intervention strategy, adjusting the generation parameters of the virtual scene content or the physical feedback actions of the physical robot, specifically includes: Establish a boredom monitoring model to calculate the interaction frequency and attention distraction of the target object in real time per unit time. When the interaction frequency is lower than the first preset threshold and the attention distraction is higher than the second preset threshold, the target object is determined to be in a bored state. Open-ended questions related to the current virtual scene are randomly selected from the candidate question library of the AIGC generation engine, and the difficulty coefficient of the questions is adjusted according to the historical answer records of the target object. The robot is controlled to turn towards the target object and performs a wake-up action by slightly tilting its head and flashing the LED light source in its eyes, while generating guiding visual cues in the edge area of ​​the projection medium. The degree of attentional distraction The calculation formula is: ;in, It is an index of attentional distractibility; The total number of frames within a unit time window; For the first The coordinates of the human eye gaze point detected in the frame; For the first The center coordinates of the frame virtual interactive object; The fixation deviation threshold; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

10. A scenario-based interactive AI emotional companion robot system according to claim 9, characterized in that, This also includes model self-updating, specifically: Periodically collect the target object's dwell time, number of interactions, and changes in emotional feature vectors for different types of virtual scene content to construct a preference-feedback association dataset; The classification threshold of the emotion recognition model is fine-tuned using the preference-feedback association dataset. Specifically, the gradient descent value of the classification boundary between positive and negative emotions is calculated, and the weight parameters of the classifier are adjusted. The script structure of highly interactive virtual scenes is extracted as template features, stored in the preferred template library of the scenario-based content ecosystem, and the sampling probability of this template feature is increased when generating new scripts in the future. The classifier weight parameters The fine-tuning update formula is: in, The weight parameters before the update. The updated weight parameters; The learning rate controls the update step size; loss function Regarding weight The gradient, the loss function is based on the real sentiment label. Compared with model predictions Error calculation between; This is the regularization coefficient, used to prevent the model from overfitting; is the L2 norm of the weights.

Images