A control system for an embodied intelligent bionic personalized companion robot

Abstract

This invention belongs to the field of artificial intelligence robot control technology. The disclosed control system for an embodied intelligent bionic personalized companion robot includes a multimodal companion perception module, a user emotional state recognition module, a personalized companion profile modeling module, an embodied behavior decision generation module, a bionic interactive action control module, and a companion feedback self-learning update module. Utilizing multimodal deep learning emotion recognition, personalized profile modeling, deep sequence prediction, deep reinforcement learning decision-making, and bionic action control principles, it generates multimodal companion perception data, user emotional state recognition results, user personalized companion profiles, user companionship need prediction results, embodied companionship behavior decision results, and bionic interactive action control instructions. This achieves accurate user emotion recognition, companionship need prediction, personalized interactive decision-making, and bionic action execution, solving the problems of existing companion robots having limited emotion recognition, mechanical interactive actions, and insufficient long-term self-learning capabilities.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence robot control technology, and in particular relates to a control system for an embodied intelligent bionic personalized companion robot. Background Technology

[0002] With the development of artificial intelligence, service robots, and intelligent interaction technologies, companion robots are increasingly being applied in scenarios such as family companionship, emotional care, daily reminders, and human-computer interaction. Compared to ordinary voice assistants, embodied intelligent companion robots not only need to understand user language but also need to combine user voice, facial expressions, postures, interactive text, environmental conditions, and the robot's own state to determine the user's true emotions and companionship needs. They must then form more natural biomimetic interactions through language, facial expressions, body movements, and motion behaviors. Therefore, an embodied intelligent biomimetic personalized companion robot control system is needed to achieve multimodal emotion recognition, personalized companion profile modeling, embodied behavior decision-making, and biomimetic motion control.

[0003] The existing technology has at least the following problems that need to be improved: (1) Existing companion robots mostly rely on speech recognition or fixed question-and-answer logic for interaction. They lack a multimodal fusion recognition mechanism for user voice data, user facial image data, user posture and action data and user interaction text data. It is difficult to accurately judge the low state, anxiety state, loneliness state, irritability state and need for companionship state, which leads to problems of emotional misjudgment and inappropriate response in companion interaction.

[0004] (2) Existing companion robots mostly use preset dialogue and fixed action templates. They lack a mechanism for personalized learning by combining historical companion record data, interest preference data, daily routine data, emotional change record data and user real-time feedback data. They also lack a process for generating embodied companion behavior decision results based on the prediction results of user companion needs and performing bionic action control, which makes it difficult for robots to form the ability to accompany a single user in the long term. Summary of the Invention

[0005] To address the above issues and overcome the shortcomings of existing technologies, this invention provides an embodied intelligent bionic personalized companion robot control system. Utilizing multimodal deep learning emotion recognition, personalized profile modeling, deep sequence prediction, deep reinforcement learning decision-making, and bionic motion control principles, it generates multimodal companionship perception data, user emotional state recognition results, user personalized companionship profiles, user companionship need prediction results, embodied companionship behavior decision-making results, and bionic interactive motion control instructions. This achieves accurate user emotion recognition, companionship need prediction, personalized interactive decision-making, and bionic motion execution, solving the problems of existing companion robots' limited emotion recognition, insufficient personalization, mechanical interactive actions, and insufficient long-term self-learning ability.

[0006] This invention provides a control system for an embodied intelligent bionic personalized companion robot, including a multimodal companion perception module, a user emotional state recognition module, a personalized companion profile modeling module, an embodied behavior decision generation module, a bionic interactive action control module, and a companion feedback self-learning update module.

[0007] The multimodal companionship perception module collects user voice data, user facial image data, user posture and action data, user interactive text data, environmental sound data, environmental image data, robot body posture data, and robot contact sensor data in the companionship scenario to generate multimodal companionship perception data. The user emotion state recognition module receives multimodal companionship perception data, extracts voice emotion features, facial expression features, posture and action recognition features, and text semantic emotion features based on a voice emotion recognition network, a facial expression recognition network, a posture and action recognition network, and a text semantic emotion recognition network, and generates user emotion state recognition results through a multimodal attention fusion network. The personalized companionship profile modeling module receives multimodal companionship perception data and user emotional state recognition results. Based on user interaction habit data, historical companionship record data, interest preference data, daily routine data, emotional change record data, and behavioral feedback data, it constructs a personalized companionship profile of the user and generates a prediction result of the user's companionship needs based on the personalized companionship profile. The embodied behavior decision generation module receives the user's emotional state recognition results and the user's companionship needs prediction results, generates a companionship interaction strategy based on a deep reinforcement learning decision network, and generates embodied companionship behavior decision results based on the companionship interaction strategy. The bionic interactive motion control module receives the embodied companionship behavior decision results and the robot's body posture data. Based on the robot's head posture, robotic arm posture, torso posture, mobile chassis status, and safety distance constraints, it performs bionic motion decomposition, motion smoothing, posture constraint, and safety obstacle avoidance processing on the embodied companionship behavior decision results, and generates bionic interactive motion control commands. The companion feedback self-learning update module receives bionic interactive action control commands, robot execution status data, user real-time feedback data, user subsequent emotional change data, and manual review results, generates personalized companion feedback samples, and updates the parameters of the user emotional state recognition module, personalized companion profile modeling module, embodied behavior decision generation module, and bionic interactive action control module based on the personalized companion feedback samples.

[0008] Furthermore, the multimodal companionship perception module performs noise reduction, endpoint detection, speech segmentation, volume normalization, and voiceprint consistency detection on the user's voice data to obtain valid user voice data; it performs face detection, key point localization, illumination equalization, occlusion area marking, and image clarity filtering on the user's facial image data and environmental image data to obtain valid user facial image data and valid environmental image data; and it extracts head posture, shoulder posture, arm movements, torso tilt angle, and gait change information from the user's posture and movement data to obtain user posture and movement feature data.

[0009] Furthermore, the user emotion state recognition module inputs voice emotion features, facial expression features, posture behavior features, and text semantic emotion features into a multimodal attention fusion network, assigns fusion weights according to the reliability of each modality in the current companionship scenario, and generates a fused emotion feature vector; based on the fused emotion feature vector and emotion fusion confidence, it identifies stable state, pleasant state, depressed state, anxious state, lonely state, irritable state, and need for companionship state, and generates user emotion state recognition results.

[0010] Furthermore, the personalized companionship profile modeling module constructs a personalized companionship profile for users based on historical companionship record data, interest preference data, daily routine data, emotional change record data, behavioral feedback data, and profile stability score; it then performs time-series modeling on the personalized companionship profile based on a deep sequence prediction network to predict the user's companionship needs type, companionship intensity, suitable interaction methods, and unsuitable interaction methods within the current time period, generating a user companionship needs prediction result.

[0011] Furthermore, the embodied behavior decision generation module acquires current time data, current spatial location data, environmental sound data, environmental image data, robot body posture data, and the user's current activity state to generate the current companionship scenario state; based on the user's emotional state recognition results, user companionship need prediction results, and the current companionship scenario state, it generates language response strategies, facial expression strategies, body movement strategies, movement approach strategies, companionship reminder strategies, and soothing interaction strategies; based on the interaction value score, inappropriate user interaction methods, and safe distance constraints, it selects the target companionship interaction strategy from the candidate companionship interaction strategies, and generates embodied companionship behavior decision results based on the target companionship interaction strategy.

[0012] Furthermore, the bionic interactive motion control module decomposes the embodied companionship behavior decision result into head rotation motion, robotic arm swing motion, torso tilting motion, moving chassis motion, and facial light motion based on the target's facial expression presentation method, target limb movement type, target movement distance, and target soothing interaction method; and performs speed smoothing, angle smoothing, acceleration limiting, posture constraint, and safety obstacle avoidance processing on the head rotation motion, robotic arm swing motion, torso tilting motion, and moving chassis motion to generate bionic interactive motion control commands.

[0013] Furthermore, the companionship feedback self-learning update module binds multimodal companionship perception data, user emotional state recognition results, user personalized companionship profiles, user companionship need prediction results, embodied companionship behavior decision-making results, bionic interactive action control commands, robot execution state data, user real-time feedback data, user subsequent emotional change data, and manual review results to generate personalized companionship feedback samples; a personalized companionship feedback sample library is constructed based on the personalized companionship feedback samples; emotion recognition error samples, companionship profile deviation samples, embodied behavior decision mismatch samples, and bionic action execution deviation samples are selected from the personalized companionship feedback sample library, and the parameters of the user emotional state recognition module, personalized companionship profile modeling module, embodied behavior decision generation module, and bionic interactive action control module are updated based on the selected samples.

[0014] The beneficial effects of this invention are as follows: (1) Using the principle of multimodal deep learning emotion recognition, the multimodal companionship perception module collects user voice data, user facial image data, user posture and action data, user interactive text data, environmental sound data, environmental image data, robot body posture data and robot contact sensor data to generate multimodal companionship perception data; then, the user emotion state recognition module extracts voice emotion features, facial expression features, posture behavior features and text semantic emotion features based on the voice emotion recognition network, facial expression recognition network, posture and action recognition network and text emotion recognition network, and generates user emotion state recognition results through the multimodal attention fusion network. The technical effect is to improve the accuracy and stability of user emotion recognition and solve the problem that existing companion robots are prone to misjudging the user's true emotions in single-modal recognition.

[0015] (2) By utilizing the principles of personalized profile modeling and deep sequence prediction, the personalized companion profile modeling module constructs a personalized companion profile of the user based on historical companion record data, interest preference data, daily routine data, emotional change record data and behavioral feedback data. Based on the deep sequence prediction network, the user's companion needs prediction results are generated. The technical effect is to improve the long-term adaptability of the companion robot to the individual user's interaction habits, emotional change patterns and companion needs, and to solve the problem that the existing companion robots use fixed language and fixed actions and are difficult to form personalized companionship.

[0016] (3) Utilizing the principles of deep reinforcement learning decision-making and bionic action control, the embodied behavior decision generation module generates companionship interaction strategies based on the user's emotional state recognition results and the user's companionship needs prediction results. The target companionship interaction strategies are then selected based on the interaction value score, generating embodied companionship behavior decision results. The bionic interaction action control module then performs bionic action decomposition, action smoothing, posture constraint, and safety obstacle avoidance processing on the embodied companionship behavior decision results, generating bionic interaction action control instructions. The technical effect is to improve the coordination and naturalness between the companion robot's language, facial expressions, body movements, and movement behavior, and to solve the problems of mechanized interaction behavior, stiff movements, and insufficient safety obstacle avoidance in existing companion robots. Attached Figure Description

[0017] The accompanying drawings are provided to further understand the present invention and form part of the specification. They are used together with the embodiments of the present invention to explain the invention and do not constitute a limitation thereof.

[0018] Figure 1 This is a flowchart of the multimodal companionship perception and emotion recognition proposed in this invention; Figure 2 This is a flowchart of the personalized companionship profile and embodied behavior decision-making process proposed in this invention; Figure 3 This is a flowchart of the biomimetic interactive motion control and feedback update proposed in this invention. Detailed Implementation

[0019] Example 1, see Figures 1-3 The present invention provides a control system for an embodied intelligent bionic personalized companion robot, including a multimodal companion perception module, a user emotional state recognition module, a personalized companion profile modeling module, an embodied behavior decision generation module, a bionic interactive action control module, and a companion feedback self-learning update module.

[0020] The multimodal companionship perception module collects user voice data, user facial image data, user posture and movement data, user interactive text data, environmental sound data, environmental image data, robot body posture data, and robot contact sensor data in the companionship scenario, generates multimodal companionship perception data, and sends the multimodal companionship perception data to the user emotional state recognition module and the personalized companionship profile modeling module.

[0021] The user emotion state recognition module receives multimodal companionship perception data, extracts voice emotion features, facial expression features, posture and action recognition features, and text semantic emotion features based on a voice emotion recognition network, a facial expression recognition network, a posture and action recognition network, and a text semantic emotion recognition network, and generates user emotion state recognition results through a multimodal attention fusion network.

[0022] The personalized companionship profile modeling module receives multimodal companionship perception data and user emotional state recognition results. Based on user interaction habit data, historical companionship record data, interest preference data, daily routine data, emotional change record data, and behavioral feedback data, it constructs a personalized companionship profile for the user and generates a prediction result of the user's companionship needs based on the personalized companionship profile.

[0023] The embodied behavior decision generation module receives the user's emotional state recognition results and the user's companionship needs prediction results, and generates companionship interaction strategies based on a deep reinforcement learning decision network. The companionship interaction strategies include language response strategies, facial expression strategies, body movement strategies, movement approach strategies, companionship reminder strategies, and soothing interaction strategies, and generates embodied companionship behavior decision results based on the companionship interaction strategies.

[0024] The bionic interactive motion control module receives the embodied companionship behavior decision results and the robot's body posture data. Based on the robot's head posture, robotic arm posture, torso posture, mobile chassis status, and safety distance constraints, it performs bionic motion decomposition, motion smoothing, posture constraint, and safety obstacle avoidance processing on the embodied companionship behavior decision results, generates bionic interactive motion control commands, and sends the bionic interactive motion control commands to the companion robot's actuator.

[0025] The companion feedback self-learning update module receives bionic interactive action control commands, robot execution status data, user real-time feedback data, user subsequent emotional change data, and manual review results, generates personalized companion feedback samples, and updates the parameters of the user emotional state recognition module, personalized companion profile modeling module, embodied behavior decision generation module, and bionic interactive action control module based on the personalized companion feedback samples.

[0026] Through the cooperation of the above modules, this system can combine multimodal perception, deep learning emotion recognition, personalized profile modeling, deep reinforcement learning decision-making, and bionic motion control, enabling the companion robot to generate personalized, continuous, and embodied companion interaction behaviors based on the user's emotional state, interaction habits, and companionship needs.

[0027] Example 2: This example is based on all the above examples. The multimodal companion sensing module includes: Multimodal data acquisition; the multimodal companionship perception module acquires user voice data, user facial image data, user posture and movement data, user interaction text data, environmental sound data, environmental image data, robot body posture data, and robot contact sensing data.

[0028] Voice data preprocessing: The multimodal companion perception module performs noise reduction, endpoint detection, voice segmentation, volume normalization, and voiceprint consistency detection on the user's voice data to obtain effective user voice data.

[0029] Image data preprocessing; the multimodal companion perception module performs face detection, key point localization, illumination equalization, occlusion area marking, and image sharpness filtering on user facial image data and environmental image data to obtain effective user facial image data and effective environmental image data.

[0030] Posture and motion data preprocessing; the multimodal companion perception module extracts head posture, shoulder posture, arm movements, torso tilt angle and gait change information from the user's posture and motion data to obtain user posture and motion feature data.

[0031] Robot state data binding; the multimodal companion perception module obtains the robot's head posture, robotic arm posture, torso posture, mobile chassis status, and contact feedback status based on the robot's body posture data and robot contact sensor data.

[0032] Multimodal time alignment: The multimodal companionship perception module aligns effective user voice data, effective user facial image data, user posture and motion feature data, user interaction text data, effective environmental image data, robot body posture data, and robot contact sensor data according to timestamps to generate multimodal companionship perception data.

[0033] Example 3: This example is based on all the above examples. The user emotion state recognition module includes: Multimodal companionship perception data reception; the user emotional state recognition module receives the multimodal companionship perception data sent by the multimodal companionship perception module.

[0034] Voice emotion feature extraction: The user emotion state recognition module inputs effective user voice data into the voice emotion recognition network, extracts pitch change features, speech rate change features, energy change features, and speech pause features, and generates voice emotion features.

[0035] Facial expression feature extraction: The user emotion state recognition module inputs effective user facial image data into the facial expression recognition network, extracts eye state features, eyebrow state features, mouth corner change features, facial muscle tension features, and expression category features, and generates facial expression features.

[0036] Posture and behavior feature extraction: The user emotion state recognition module inputs the user's posture and action feature data into the posture and action recognition network, extracts features such as head drooping, trunk contraction, arm swinging, slow gait, and approaching and moving away, and generates posture and behavior features.

[0037] Text semantic emotion feature extraction: The user emotion state recognition module inputs user interaction text data into the text emotion recognition network, extracts emotion word features, negative semantic features, help-seeking semantic features, loneliness semantic features, and positive semantic features, and generates text semantic emotion features.

[0038] Multimodal attention fusion: The user emotion state recognition module inputs voice emotion features, facial expression features, posture behavior features and text semantic emotion features into the multimodal attention fusion network, and assigns fusion weights according to the reliability of each modality in the current companionship scenario to generate a fused emotion feature vector.

[0039] Emotion fusion confidence calculation: The user emotion state recognition module calculates the emotion fusion confidence based on voice emotion features, facial expression features, posture and behavior features, text semantic emotion features, and the fusion weights of each modality. The calculation formula is as follows:

[0040] in, It is the confidence level of emotional fusion; It is the emotional response value corresponding to the voice emotion features; It is the emotional response value corresponding to facial expression features; It is the emotional response value corresponding to the posture and behavioral characteristics; It is the emotional response value corresponding to the semantic emotional features of the text. These are the fusion weights corresponding to the speech emotion features; These are the fusion weights corresponding to facial expression features; These are the fusion weights corresponding to the posture and behavioral features; These are the fusion weights corresponding to the semantic sentiment features of the text; It is the stability constant.

[0041] User emotional state recognition results are generated; the user emotional state recognition module identifies stable state, happy state, depressed state, anxious state, lonely state, irritable state, and need for companionship state based on the fused emotional feature vector and emotional fusion confidence, and generates user emotional state recognition results.

[0042] Regarding parameter adjustments: Step 1: Adjusting the weights of speech modalities; when the user's speech data is clear and the speech pauses are obvious, increase the weight of speech emotion features in the multimodal attention fusion network; when the ambient sound data is strong and the speech signal-to-noise ratio is low, decrease the weight of speech emotion features and increase the weight of facial expression features and posture behavior features.

[0043] Step 2: Facial modality weight adjustment; when the user's facial image data is clear and the occlusion area is small, increase the weight of facial expression features; when the user is facing away from the robot or their face is significantly occluded, decrease the weight of facial expression features and increase the weight of text semantic emotion features.

[0044] Step 3: Adjusting posture modal weights; when the user is sitting still, looking down, with trunk contracted, or with a slow gait, increase the weight of posture behavior features; when the user's posture action data is affected by environmental occlusion, decrease the weight of posture behavior features and add continuous frame posture smoothing processing.

[0045] Step 4: Adjust the weight of text semantic modality; when user interaction text data contains semantic features of seeking help, loneliness, or negation, increase the weight of text semantic emotion features; when user interaction text data is short and semantically incomplete, decrease the weight of text semantic emotion features, and supplement the judgment by combining historical companionship record data.

[0046] By performing the above operations, the user emotion state recognition module can avoid misjudgment of a single modality and stably recognize the user's emotion state in complex family companionship scenarios.

[0047] Example 4: This example is based on all the above examples. The personalized companion profile modeling module includes: Multimodal companionship perception data reception; Personalized companionship profile modeling module receives multimodal companionship perception data.

[0048] The user emotion state recognition result is received; the personalized companion profile modeling module receives the user emotion state recognition result generated by the user emotion state recognition module.

[0049] Historical companionship record data acquisition; the personalized companionship profile modeling module acquires the user's historical interaction content, historical response preferences, historical action preferences, and historical companionship duration with the companion robot at different time periods, different spatial locations, and different emotional states.

[0050] Interest preference data extraction; the personalized companionship profile modeling module extracts music preferences, topic preferences, activity preferences, reminder preferences, and interaction frequency preferences based on user interaction text data and historical companionship record data.

[0051] Daily routine data extraction; the personalized companionship profile modeling module extracts patterns of wake-up time, rest time, meal time, activity time, and alone time based on users' activity records at different dates and time periods.

[0052] Emotional change record data generation; the personalized companion profile modeling module binds the user's emotional state recognition results with time period, spatial location, interaction content and robot companion behavior to generate emotional change record data.

[0053] Profile stability score calculation: The personalized companionship profile modeling module calculates the profile stability score based on historical companionship records, interest preference data, daily routine data, and emotional change records. The calculation formula is as follows:

[0054] in, It is a portrait stability score; It refers to the completeness of historical companionship records. It is the interest matching degree corresponding to interest preference data; It is the consistency of the patterns in the data corresponding to daily routines; It is the deviation in emotional fluctuation corresponding to the recorded data of emotional changes.

[0055] User personalized companionship profile construction: The personalized companionship profile modeling module constructs a user personalized companionship profile based on historical companionship record data, interest preference data, daily routine data, emotional change record data, behavioral feedback data, and profile stability score.

[0056] User companionship demand prediction results are generated; the personalized companionship profile modeling module performs time-series modeling of the user's personalized companionship profile based on a deep sequence prediction network, predicts the user's companionship demand type, companionship intensity, suitable interaction method and unsuitable interaction method in the current time period, and generates user companionship demand prediction results.

[0057] Regarding parameter adjustments: Step 1: Adjust the weight of historical companionship record data; when a user has recently received a certain type of companionship interaction and generated positive feedback, increase the weight of the corresponding historical companionship record data; when a user has recently rejected a certain type of companionship interaction multiple times, decrease the weight of the corresponding historical companionship record data and mark the corresponding interaction method as an inappropriate interaction method.

[0058] Step 2: Adjust the weight of interest preference data; when a user consistently provides positive feedback on a topic, music, or activity, increase the weight of the corresponding item in the interest preference data; when a user's interests change over different time periods, increase the weight of recent interest preference data based on the most recent interaction time.

[0059] Step 3: Adjust the weight of daily routine data; when users experience low mood or loneliness during fixed time periods, increase the weight of daily routine data in the prediction results of user companionship needs; when users are temporarily away from home or in non-routine activity states, decrease the weight of daily routine data and increase the weight of family scenario context data.

[0060] Step 4: Adjust the weight of emotional change recording data; when a certain type of companionship behavior can make the user's subsequent emotional changes tend to be stable or pleasant, increase the weight of this type of companionship behavior in the user's personalized companionship profile; when a certain type of companionship behavior causes the user's subsequent emotional changes to turn into an irritable state, decrease the weight of this type of companionship behavior.

[0061] By performing the above operations, the personalized companion profile modeling module can form a dynamic companion profile adapted to a single user, so that the companion robot no longer interacts only according to fixed scripts and fixed actions.

[0062] Example 5: This example is based on all the above examples. The embodied behavior decision generation module includes: The user emotion state recognition result is received; the embodied behavior decision generation module receives the user emotion state recognition result sent by the user emotion state recognition module.

[0063] The module receives the user companionship demand prediction results; the embodied behavior decision generation module receives the user companionship demand prediction results sent by the personalized companionship profile modeling module.

[0064] The current companionship scenario status is obtained; the embodied behavior decision generation module obtains current time data, current spatial location data, environmental sound data, environmental image data, robot body posture data, and the user's current activity status to generate the current companionship scenario status.

[0065] Candidate companionship interaction strategy generation; The embodied behavior decision generation module generates language response strategies, facial expression strategies, body movement strategies, movement approach strategies, companionship reminder strategies, and soothing interaction strategies based on the user's emotional state recognition results, user companionship need prediction results, and the current companionship scenario status.

[0066] Deep reinforcement learning decision-making; the embodied behavior decision generation module inputs the user's emotional state recognition results, user companionship demand prediction results, current companionship scenario status and candidate companionship interaction strategies into the deep reinforcement learning decision network, and calculates the interaction value score corresponding to each candidate companionship interaction strategy.

[0067] Interaction value score calculation: The embodied behavior decision generation module calculates the interaction value score based on the immediate emotion improvement value, long-term companionship matching value, action safety constraint value, and user refusal penalty value. The calculation formula is as follows:

[0068] in, It is an interaction value rating; It is an immediate mood improvement value; It is a long-term companionship matching value; These are motion safety constraint values; It is the user's rejection penalty value; , , and These are the reward weight coefficients for the corresponding items.

[0069] The companion interaction strategy selection module selects the target companion interaction strategy from the candidate companion interaction strategies based on the interaction value score, the user's inappropriate interaction methods, and the safe distance constraint.

[0070] Embodied companionship behavior decision generation: The embodied companionship behavior generation module generates embodied companionship behavior decision results based on the target companionship interaction strategy. The embodied companionship behavior decision results include the target's language response content, the target's facial expression presentation method, the target's body movement type, the target's movement distance, the target's companionship reminder content, and the target's comforting interaction method.

[0071] Regarding parameter adjustments: Step 1: Adjust the reward parameters for the interaction value score; when the user's immediate feedback data is positive and the user's subsequent emotional change data tends to be stable after the target companionship interaction strategy is executed, increase the reward value of the corresponding strategy; when the user's immediate feedback data is negative or the user's subsequent emotional change data tends to be irritable after the target companionship interaction strategy is executed, decrease the reward value of the corresponding strategy.

[0072] Step 2: Adjusting the constraints of the approach strategy; when the user's emotional state is identified as loneliness or need for companionship and the user does not show any rejection behavior, the target movement distance is appropriately increased; when the user's emotional state is identified as agitation or anxiety, the target movement distance is reduced, and verbal response strategies and low-amplitude physical movement strategies are given priority.

[0073] Step 3: Adjusting the soothing interaction strategy; when the user's companionship need prediction results show a high level of companionship, increase the interaction value score of the soothing interaction strategy; when the ambient sound data is strong or the user is having a phone conversation, decrease the interaction value score of the verbal response strategy and increase the weight of the facial expression strategy and the body language strategy.

[0074] Step 4: Adjusting the companionship reminder strategy; when the data on daily routines indicates that the current time period belongs to a meal, medication, rest, or activity period, increase the interaction value score of the companionship reminder strategy; when users have repeatedly ignored similar reminders recently, reduce the trigger frequency of the companionship reminder strategy and adjust the target companionship reminder content.

[0075] By performing the above operations, the embodied behavior decision generation module can dynamically select companionship behaviors based on the user's current mood, personalized profile, and actual scenario, enabling the companion robot to have stronger personalized and embodied intelligent interaction capabilities.

[0076] Example 6: This example is based on all the above examples. The bionic interactive motion control module includes: Embodied companionship behavior decision-making results reception; the bionic interactive action control module receives the embodied companionship behavior decision-making results sent by the embodied behavior decision-making generation module.

[0077] The robot body posture data is received; the bionic interactive motion control module receives the robot body posture data sent by the multimodal companion perception module.

[0078] Bionic motion decomposition: The bionic interactive motion control module decomposes the embodied companionship behavior decision-making results into head turning motion, robotic arm swinging motion, torso tilting motion, chassis movement motion, and facial expression lighting motion based on the target's facial expression presentation method, target limb movement type, target movement distance, and target soothing interaction method.

[0079] Motion smoothing: The bionic interactive motion control module performs speed smoothing, angle smoothing, and acceleration limiting on head rotation, robotic arm swinging, torso tilting, and mobile chassis movements based on the robot's head posture, robotic arm posture, torso posture, and mobile chassis status.

[0080] Posture constraint processing: The bionic interactive motion control module performs posture constraint processing on the bionic motion based on the robot's joint range of motion, the robotic arm's safe angle, the torso's balance range, and the mobile chassis's stability range.

[0081] Safety obstacle avoidance: The bionic interactive motion control module adjusts the movement of the mobile chassis and the swinging motion of the robotic arm based on environmental image data, robot contact sensor data and safety distance constraints to avoid obstacles.

[0082] The biomimetic interactive motion control module integrates the motion smoothing processing results, posture constraint processing results, and safety obstacle avoidance processing results to generate biomimetic interactive motion control commands, and then sends these commands to the companion robot's actuator.

[0083] Example 7: This example is based on all the above examples. The companion feedback self-learning update module includes: Control command reception; the companion feedback self-learning update module receives bionic interactive action control commands sent by the bionic interactive action control module.

[0084] Execution status data acquisition; The companion feedback self-learning update module acquires the robot execution status data of the companion robot's actuator during the execution process. The robot execution status data includes the head execution status, robotic arm execution status, torso execution status, mobile chassis execution status, and facial expression light execution status.

[0085] Real-time user feedback data acquisition; the companionship feedback self-learning update module acquires real-time user feedback data on the target language response content, target facial expression presentation method, target body movement type, target movement distance, target companionship reminder content, and target comforting interaction method.

[0086] The accompanying feedback self-learning update module acquires data on users' subsequent emotional changes after the target accompanying interaction strategy is executed.

[0087] Personalized companionship feedback sample generation; the companionship feedback self-learning update module binds multimodal companionship perception data, user emotional state recognition results, user personalized companionship profile, user companionship need prediction results, embodied companionship behavior decision results, bionic interactive action control instructions, robot execution status data, user real-time feedback data, user subsequent emotional change data, and manual review results to generate personalized companionship feedback samples.

[0088] Sample library construction; the companion feedback self-learning update module constructs a personalized companion feedback sample library based on personalized companion feedback samples.

[0089] Abnormal sample screening: The companion feedback self-learning update module filters out emotion recognition error samples, companion profile deviation samples, embodied behavior decision mismatch samples, and bionic action execution deviation samples from the personalized companion feedback sample library.

[0090] Parameter updates: The companion feedback self-learning update module updates the parameters of the user emotion state recognition module based on emotion recognition error samples, updates the parameters of the personalized companion profile modeling module based on companion profile deviation samples, updates the parameters of the embodied behavior decision generation module based on embodied behavior decision mismatch samples, and updates the parameters of the bionic interaction action control module based on bionic action execution deviation samples.

[0091] By performing the above operations, this system can continuously correct the parameters of emotion recognition, personalized profile, embodied decision-making, and bionic motion control based on long-term user interaction feedback, enabling the companion robot to gradually develop personalized companionship capabilities for individual users during long-term use.

Claims

1. A control system for a body-worn intelligent bionic personalized companion robot, characterized in that: It includes a multimodal companionship perception module, a user emotional state recognition module, a personalized companionship profile modeling module, an embodied behavior decision generation module, a biomimetic interactive action control module, and a companionship feedback self-learning update module; The multimodal companionship perception module collects user voice data, user facial image data, user posture and action data, user interactive text data, environmental sound data, environmental image data, robot body posture data, and robot contact sensor data in the companionship scenario to generate multimodal companionship perception data. The user emotion state recognition module receives multimodal companionship perception data, extracts voice emotion features, facial expression features, posture and action recognition features, and text semantic emotion features based on a voice emotion recognition network, a facial expression recognition network, a posture and action recognition network, and a text semantic emotion recognition network, and generates user emotion state recognition results through a multimodal attention fusion network. The personalized companionship profile modeling module receives multimodal companionship perception data and user emotional state recognition results. Based on user interaction habit data, historical companionship record data, interest preference data, daily routine data, emotional change record data, and behavioral feedback data, it constructs a personalized companionship profile of the user and generates a prediction result of the user's companionship needs based on the personalized companionship profile. The embodied behavior decision generation module receives the user's emotional state recognition results and the user's companionship needs prediction results, generates a companionship interaction strategy based on a deep reinforcement learning decision network, and generates embodied companionship behavior decision results based on the companionship interaction strategy. The bionic interactive motion control module receives the embodied companionship behavior decision results and the robot's body posture data. Based on the robot's head posture, robotic arm posture, torso posture, mobile chassis status, and safety distance constraints, it performs bionic motion decomposition, motion smoothing, posture constraint, and safety obstacle avoidance processing on the embodied companionship behavior decision results, and generates bionic interactive motion control commands. The companion feedback self-learning update module receives bionic interactive action control commands, robot execution status data, user real-time feedback data, user subsequent emotional change data, and manual review results, generates personalized companion feedback samples, and updates the parameters of the user emotional state recognition module, personalized companion profile modeling module, embodied behavior decision generation module, and bionic interactive action control module based on the personalized companion feedback samples.

2. The control system for a body-worn intelligent bionic personalized companion robot according to claim 1, characterized in that: The multimodal companionship perception module performs noise reduction, endpoint detection, speech segmentation, volume normalization, and voiceprint consistency detection on the user's voice data to obtain valid user voice data. Face detection, key point localization, illumination equalization, occlusion area marking, and image sharpness filtering are performed on user facial image data and environmental image data to obtain valid user facial image data and valid environmental image data. Based on user posture and movement data, head posture, shoulder posture, arm movement, torso tilt angle, and gait change information are extracted to obtain user posture and movement feature data.

3. The control system for a body-worn intelligent bionic personalized companion robot according to claim 2, characterized in that: The multimodal companionship perception module obtains the robot's head posture, robotic arm posture, torso posture, mobile chassis status, and contact feedback status based on the robot's body posture data and robot contact sensor data. It aligns effective user voice data, effective user facial image data, user posture and motion feature data, user interaction text data, effective environmental image data, robot body posture data, and robot contact sensor data according to timestamps to generate multimodal companionship perception data.

4. The control system for a body-worn intelligent bionic personalized companion robot according to claim 3, characterized in that: The user emotion state recognition module inputs effective user voice data into the voice emotion recognition network, extracts pitch change features, speech rate change features, energy change features, and speech pause features to generate voice emotion features; it inputs effective user facial image data into the facial expression recognition network, extracts eye state features, eyebrow state features, mouth corner change features, facial muscle tension features, and expression category features to generate facial expression features; and it inputs user posture and movement feature data into the posture and movement recognition network, extracts head drooping features, trunk contraction features, arm swing features, gait slowness features, and approach and departure behavior features to generate posture and behavior features. User interaction text data is input into a text emotion recognition network to extract emotion word features, negative semantic features, help-seeking semantic features, loneliness semantic features, and positive semantic features, generating text semantic emotion features.

5. The control system for a body-worn intelligent bionic personalized companion robot according to claim 4, characterized in that: The user emotion state recognition module inputs voice emotion features, facial expression features, posture behavior features, and text semantic emotion features into a multimodal attention fusion network. It assigns fusion weights based on the reliability of each modality in the current companionship scenario, generating a fused emotion feature vector. Based on the voice emotion features, facial expression features, posture behavior features, text semantic emotion features, and the fusion weights of each modality, it calculates the emotion fusion confidence score. Based on the fused emotion feature vector and the emotion fusion confidence score, it identifies stable state, pleasant state, depressed state, anxious state, lonely state, irritable state, and need for companionship state, generating the user emotion state recognition result.

6. The control system for a body-worn intelligent bionic personalized companion robot according to claim 5, characterized in that: The personalized companion profile modeling module obtains the user's historical interaction content, historical response preferences, historical action preferences, and historical companionship duration with the companion robot at different time periods, different spatial locations, and different emotional states. Based on user interaction text data and historical companionship records, music preferences, topic preferences, activity preferences, reminder preferences, and interaction frequency preferences are extracted. Based on users' activity records at different dates and time periods, patterns of wake-up time, rest time, meal time, activity time, and time spent alone are extracted. The user's emotional state recognition results are bound to time period, spatial location, interaction content, and robot companionship behavior to generate emotional change record data.

7. The control system for a body-worn intelligent bionic personalized companion robot according to claim 6, characterized in that: The personalized companionship profile modeling module calculates a profile stability score based on historical companionship record data, interest preference data, daily routine data, and emotional change record data; and constructs a personalized companionship profile for the user based on historical companionship record data, interest preference data, daily routine data, emotional change record data, behavioral feedback data, and profile stability score. Based on deep sequence prediction networks, time-series modeling of personalized companionship profiles of users is performed to predict the type, intensity, appropriate and inappropriate interaction methods of users' companionship needs in the current time period, and generate user companionship need prediction results.

8. The control system for a body-worn intelligent bionic personalized companion robot according to claim 7, characterized in that: The embodied behavior decision generation module acquires current time data, current spatial location data, environmental sound data, environmental image data, robot body posture data, and user's current activity state to generate the current companionship scenario state; based on the user's emotional state recognition results, user companionship need prediction results, and the current companionship scenario state, it generates language response strategies, facial expression strategies, body movement strategies, movement approach strategies, companionship reminder strategies, and soothing interaction strategies. The user's emotional state recognition results, user companionship demand prediction results, current companionship scenario status, and candidate companionship interaction strategies are input into a deep reinforcement learning decision network to calculate the interaction value score corresponding to each candidate companionship interaction strategy. Based on the interaction value score, inappropriate interaction methods for the user, and safe distance constraints, the target companionship interaction strategy is selected from the candidate companionship interaction strategies, and the embodied companionship behavior decision results are generated based on the target companionship interaction strategy.

9. The control system for a body-worn intelligent bionic personalized companion robot according to claim 8, characterized in that: The bionic interactive motion control module decomposes the embodied companionship behavior decision-making results into head rotation, robotic arm swing, torso tilt, mobile chassis, and facial light movements based on the target's facial expression presentation method, target limb movement type, target movement distance, and target soothing interaction method. Based on the robot's head posture, robotic arm posture, torso posture, and mobile chassis state, the module performs speed smoothing, angle smoothing, and acceleration limiting on the head rotation, robotic arm swing, torso tilt, and mobile chassis movements. Finally, it performs posture constraint processing on the bionic movements based on the robot's joint range of motion, robotic arm safety angle, torso balance range, and mobile chassis stability range. Based on environmental image data, robot contact sensor data, and safety distance constraints, obstacle avoidance adjustments are made to the motion of the mobile chassis and the swinging motion of the robotic arm. The motion smoothing processing results, posture constraint processing results, and safety obstacle avoidance processing results are integrated to generate biomimetic interactive motion control commands.

10. The control system for a body-worn intelligent bionic personalized companion robot according to claim 9, characterized in that: The companion feedback self-learning update module acquires robot execution status data during the execution process of the companion robot actuator; and acquires real-time user feedback data on the target language response content, target facial expression presentation method, target body movement type, target movement distance, target companion reminder content, and target soothing interaction method. After the target companionship interaction strategy is executed, data on subsequent user emotional changes are acquired. Multimodal companionship perception data, user emotional state recognition results, user personalized companionship profiles, user companionship need prediction results, embodied companionship behavior decision-making results, bionic interactive action control commands, robot execution status data, user real-time feedback data, user subsequent emotional change data, and manual review results are bound together to generate personalized companionship feedback samples. A personalized companionship feedback sample library is constructed based on these samples. Emotion recognition error samples, companionship profile deviation samples, embodied behavior decision-making mismatch samples, and bionic action execution deviation samples are selected from the personalized companionship feedback sample library. Based on the selected samples, the parameters of the user emotional state recognition module, personalized companionship profile modeling module, embodied behavior decision generation module, and bionic interactive action control module are updated.

Images