A robot control method and system based on brain-heart-body integration

Abstract

The application provides a robot control method and system based on brain-heart-body integration, belonging to the technical field of embodied robot control, comprising obtaining the current state of the robot, a semantic target and a sentiment behavior modulation signal; performing dynamic weighted fusion and object analysis on multi-modal observation data to generate a scene semantic graph containing operable attributes and spatial relationships; combining the semantic target, the environment state and the behavior modulation signal, using an optimization model with semantic constraints to generate a desired trajectory, and performing adaptive control based on a learning strategy and online parameter update; embedding constraint monitoring based on a safety barrier function and residual-based fault detection throughout the execution process, and performing hierarchical response and classified abnormal recovery. The integration of task driving, sentiment modulation and closed-loop control is realized, and the situational adaptability, interactive flexibility and operational safety of the robot in a complex human-robot interaction environment are improved.

Description

Technical Field

[0001] This invention belongs to the field of embodied robot control technology, specifically relating to a robot control method and system based on the integration of brain, mind, and body coordination. Background Technology

[0002] With the rapid development of embodied intelligence and human-robot collaboration technologies, robots not only need to perform repetitive tasks in structured environments, but also need to interact naturally and collaborate safely with humans in open, dynamic, real-world scenarios, while understanding task semantics. Existing robot control systems typically adopt a layered or modular architecture, separating and designing modules such as perception, planning, control, and safety monitoring, allowing each to operate independently.

[0003] In traditional robot control frameworks, the connection between low-level actuator control and high-level decision-making systems is typically limited to a simple instruction interface. Sensors output geometric information (such as obstacle positions and target coordinates), the motion planning module focuses on generating geometric trajectories, the control module relies on precise kinematic and dynamic models for feedback adjustment, and the safety monitoring module is often independent of the control loop, triggering an emergency stop only after an anomaly occurs. This fragmented design often lacks semantic constraints between the structured motion planning and control execution, making it difficult to generate a continuous flow of actions that conforms to the task intent. Furthermore, faults often result in direct task failure, leading to frequent task interruptions and insufficient adaptability in complex dynamic environments. Summary of the Invention

[0004] To address the shortcomings of embodied robots in semantic understanding and motion control in dynamic scenarios, this invention provides a robot control method and system based on the integration of brain, mind, and body.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A robot control method based on brain-heart-body coordination, wherein the robot comprises a brain layer, a heart layer, and a body layer, includes the following steps: The robot's current motion state and raw multimodal sensing data are acquired; the raw multimodal sensing data are feature-encoded and weighted to generate a unified environmental state; based on the unified environmental state, the human body, obstacles, and target objects in the scene are analyzed to obtain a semantic relationship graph containing nodes and edges; wherein, nodes represent human bodies, obstacles, or manipulated objects, and edges represent distance relationships, occlusion relationships, contact relationships, or interaction relationships between nodes; The semantic target transmitted from the brain layer and the emotional modulation signal from the heart layer are acquired. Based on the semantic relationship graph, the semantic target transmitted from the brain layer is parsed to obtain the target state. Based on the target state, the robot's current state, the emotional modulation signal from the heart layer, and the unified environmental state, the trajectory optimization function is solved using model predictive control to obtain the preliminary control command. The robot's current motion state and unified environment state are input into a parameterized strategy function, which outputs a compensation control command. The compensation control parameters are superimposed on the initial control command, and the superimposed control parameters are input into the forward prediction model inside the volume layer, which outputs the robot's predicted state at the next moment.

[0006] Preferably, the trajectory optimization function is specifically: ; in, The reference state for the semantic target. To predict control inputs in the time domain, , For the emotional modulation signal of the heart layer, This is the robot's current state. For semantic constraints, For semantic goals, Environmental state; and These are the weight matrices for the state error term and the control input term, respectively. Semantic constraint weight coefficients Preferably, the original multimodal sensing data is subjected to feature encoding and weighted fusion to generate a unified environmental state. Specifically, the image information, depth information, force or tactile feedback, and robot body state in the original multimodal sensing data are respectively input into separate encoders and converted into multiple high-dimensional semantic feature vectors. The multiple high-dimensional semantic feature vectors are multiplied by the modal fusion weights, and the weighted feature vectors are then concatenated according to the channel dimension to obtain the unified environmental state.

[0007] Preferably, the parameterized strategy function is specifically: ;in, For parameterized strategy functions, To control the output, In robot state, To unify the environment state, train using any pair of parameterized policy functions from supervised learning, imitation learning, or reinforcement learning.

[0008] Preferably, robot safety monitoring is performed by calculating the residual between the robot's predictive control and the actual sensor observations. When the residual exceeds a preset threshold, the system is determined to be abnormal, and an abnormal response mechanism is triggered.

[0009] Preferably, the anomaly response mechanism includes a tiered response strategy and an anomaly recovery strategy; The graded response strategy includes: Category 1 minor risk, the robot continues its task; Category 2 medium risk, the robot decelerates, pauses, or replans; Category 3 severe risk, the robot must immediately stop its current action and reverse; Category 4 emergency risk, an emergency stop is triggered and manual confirmation is required. The anomaly recovery strategy includes: if the anomaly is target loss, re-sensing and updating the object's position; if the anomaly is grasping failure, adjusting the grasping posture and retrying; if the anomaly is path blockage, performing local replanning; if the anomaly is user sudden approach, pausing and generating explanation or confirmation feedback; if the anomaly is sensor failure or actuator malfunction, entering safe mode and stopping execution; when anomalies occur consecutively, the number of recovery attempts exceeds a threshold, or safety constraints cannot be satisfied again, the current task is determined to have failed.

[0010] This invention also provides a robot control system based on the integration of brain, mind, and body, specifically including: An environmental perception module is used to acquire the robot's current motion state and raw multimodal sensing data; to perform feature encoding and weighted fusion on the raw multimodal sensing data to generate a unified environmental state; based on the unified environmental state, to analyze the human body, obstacles and target objects in the scene to obtain a semantic relationship graph containing nodes and edges; wherein, nodes represent human bodies, obstacles or operation objects, and edges represent distance relationships, occlusion relationships, contact relationships or interaction relationships between nodes.

[0011] The motion planning module is used to acquire semantic targets transmitted from the brain layer and emotional modulation signals from the heart layer. Based on the semantic relationship graph, the semantic targets transmitted from the brain layer are parsed to obtain the target state. Based on the target state, the robot's current state, the emotional modulation signals from the heart layer, and the unified environmental state, the trajectory optimization function is solved using model predictive control to obtain preliminary control commands.

[0012] The execution control module is used to input the robot's current motion state and unified environment state into a parameterized strategy function, and output a compensation control command; to superimpose the compensation control parameters onto the initial control command, to input the superimposed control parameters into the forward prediction model inside the body layer, and to output the robot's predicted state at the next moment.

[0013] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps described in the robot control method based on brain-mind-body coordination.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, which, when loaded by a processor, is capable of executing the steps described in the robot control method based on brain-mind-body coordination.

[0015] The robot control method based on the integration of brain, mind, and body coordination provided by this invention has the following beneficial effects: This invention enables a robot to receive semantic targets, emotion-modulated signals, and interaction risk warnings, forming an embodied execution system that integrates high-level task-driven, emotion-constrained, and low-level closed-loop control. A semantic relationship graph explicitly associates the human body, obstacles, and targets, providing clear geometric and semantic basis for subsequent near-human deceleration and local replanning trigger conditions. Model predictive control incorporating emotion-modulated signals solves the trajectory optimization function, outputting the expected trajectory. A reinforcement learning / imitation learning-based control strategy complements traditional motion planning and feedback trajectories. The learning strategy only corrects local compensations, absorbing the generalization advantages of data-driven approaches while retaining the safety baseline of model-based control, achieving an adaptive balance between task efficiency and interactive comfort. This improves the robot's reliability and adaptability in open, dynamic environments with frequent human-robot interaction. Attached Figure Description

[0016] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a mathematical model diagram illustrating the process from multimodal observation to unified environmental representation and then to the construction of an object relationship graph in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the embodied control and adaptive adjustment framework in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0020] Example This invention provides a robot control method based on brain-mind-body coordination, comprising: In the overall robot architecture, the body layer is a crucial link connecting high-level cognitive decision-making with actual physical execution. Its core task is to transform the task objectives output by the brain layer and the emotional modulation signals given by the heart layer into executable, feedback-enabled, and constrained motion control commands, thereby enabling the robot to operate stably and interact safely in complex environments. Unlike traditional discrete control frameworks, the body layer does not only undertake the control functions of the lower-level actuators, but also emphasizes the integrated collaboration of perception, planning, control, and safety monitoring. It achieves the unification of motion generation, execution correction, and anomaly response through multi-sensor feedback and closed-loop adjustment mechanisms.

[0021] Structurally, the body layer mainly consists of four sub-modules: environment and body perception module, motion planning module, execution control module, and safety monitoring module. The perception module is responsible for collecting information such as vision, depth, force, touch, and joint states to form a real-time estimate of the robot's external environment and its own state. The motion planning module generates the desired trajectory based on the target state, task semantics, and constraints. The execution control module implements joint-level and end-effector-level control based on kinematic models, dynamic models, and feedback errors. The safety monitoring module assesses collision risks, force thresholds, posture anomalies, and execution deviations in real time to ensure the interpretability, safety, and recoverability of the execution process.

[0022] Let the robot's current state be The target state is The control input is Then the basic closed-loop control form of the body layer can be written as: ; in, This represents behavioral modulation signals from the body layer, used to adjust movement speed, compliance, safety distance, and response style. For example, when a user is anxious, tense, or hesitant, the body layer can reduce movement speed and robotic arm amplitude, and increase safety distance constraints; when the user is calm and the environmental risk is low, the system maintains a normal execution rhythm. Thus, body layer execution is not a mechanical output with a fixed trajectory, but a dynamic control process constrained by task semantics, environmental state, and user interaction state.

[0023] ; in, This is a coupled kinematics and dynamics model of the robot. To ensure trajectory tracking accuracy, an error term is typically defined:

[0024] ; And this error is continuously reduced through feedback control. In actual systems, the body layer typically operates on different time scales: the perception fusion module is responsible for updating the state of the environment and objects, the motion planning module generates or corrects local trajectories based on semantic goals, the execution control module completes joint or end-effector control, and the safety monitoring module continuously performs constraint checks.

[0025] The body layer receives semantic goals, subtask sequences, and environmental understanding results from the brain layer, receives emotional modulation signals and interaction risk warnings from the heart layer, and outputs chassis motion, robotic arm trajectory, end effector actions, and safety status feedback.

[0026] (1) Multimodal perception fusion and operable object understanding are the foundation for safe execution and precise interaction at the body level. Its core objective is to jointly utilize visual, depth, force, tactile and proprioceptive information in a dynamic environment to uniformly identify, locate and estimate the state of human body, obstacles, target objects and operable areas. Unlike traditional single visual detection, embodied robots not only need to determine what the object is, but also need to further understand whether it is approachable, graspable, avoidable and cooperative, so as to provide structured input for motion planning and execution control.

[0027] like Figure 1 As shown, the multimodal perception fusion and operable object understanding process includes six parts: multimodal observation, modality-specific encoding, multimodal fusion and unified environment representation, object parsing and availability perception understanding, semantic relation graph construction, and downstream volumetric task input.

[0028] During the multimodal observation phase, time is acquired. Image information Depth information Force or tactile feedback information and robot status information And form a multimodal observation set: ; Among them, image information Used to reflect scene appearance, target category, and texture information; depth information Used to reflect target distance, spatial hierarchy, and three-dimensional structure; force or tactile feedback information. Used to reflect contact strength, force direction, and interaction feedback; robot status information. It is used to reflect the robot's pose, motion state, and body perception information.

[0029] In the modality-specific encoding stage, the system uses a visual encoder, a depth encoder, a force or tactile encoder, and a state encoder to extract features from the four types of modal data, respectively, to obtain the corresponding modal features. , , and Among them, visual features It mainly includes object category, texture, edge and spatial semantic information; deep features It mainly includes distance, depth level, and three-dimensional structural information; force or tactile features. It mainly includes contact strength, force direction, and interactive feedback information; state characteristics. It mainly includes robot pose, motion state, and body perception information.

[0030] In the multimodal fusion and unified environment characterization stage, the system will , , and Input fusion module A unified environmental state is generated through weighted fusion. It is represented as: ; In one implementation, the fusion feature can be represented as: ; in, Indicates the first Modal features, This represents the fusion weights for the corresponding modalities. The unified environmental state obtained through fusion. It should include at least object category, spatial location, motion trend, and interaction attributes to provide a unified input for subsequent object parsing, semantic relationship graph construction, motion planning, and safety monitoring.

[0031] During the object parsing and availability-aware understanding phases, the system is based on a unified environmental state. The system analyzes human figures, obstacles, target objects, and operable areas in a scene and models them at three levels: target category, spatial relationships, and operability attributes. The target category layer identifies people, obstacles, target objects, and operable areas; the spatial relationships layer describes distance, occupancy, relative position, and occupied area; and the operability layer determines whether an object is reachable, graspable, obstacle-avoidable, cooperative, and whether there are contact surfaces, grasping points, and operational constraints. For human figures, the system records key points, posture direction, safe distance, and range of motion; for obstacle figures, the system records geometric boundaries, occupied areas, and collision risks; and for target object figures, the system records grasping points, contact surfaces, operational constraints, and reachability information.

[0032] Subsequently, in the semantic relationship graph construction phase, the system constructs a semantic relationship graph from the object parsing results: ; in, This represents a set of nodes, which include humans, obstacles, robots, target objects, and operable areas. This represents a set of edges, where edges describe the distance relationships, occlusion relationships, interaction relationships, and collision risk relationships between nodes. Based on this semantic relationship graph, the system can generate obstacle avoidance decisions, approach strategies, and cooperative action selection results. For example, the distance relationship between human nodes and robot nodes can be used to generate minimum safe distance constraints and near-human deceleration strategies; the occlusion relationship between obstacle nodes and target object nodes can be used to generate local path replanning conditions; and the interaction relationship between target object nodes and operable area nodes can be used to determine the approach direction, grasping posture, and end effector action constraints.

[0033] Finally, in the downstream body layer input stage, the system organizes the unified environment state, object set, and semantic relationship graph at consecutive time steps into environment state sequence, object set, and relationship graph set, respectively: ; ; ; Among them, the environmental state sequence A collection of objects used to provide continuous scene states for motion planning. A set of relationship diagrams used to provide information on human bodies, obstacles, target objects, and operable areas for execution control. Used to provide structured constraints for safety monitoring, collision avoidance, local replanning, and anomaly recovery. Therefore, attached... Figure 1 A continuous data flow is formed between multimodal observation, modality-specific coding, unified environment representation, object parsing, semantic relationship graph construction, and downstream volume layer input, enabling the multimodal perception results to be further applied to motion planning, execution control, and safety monitoring, rather than just serving as isolated perception outputs.

[0034] The basic implementation logic of Algorithm 1 involves first performing modal encoding on the image, depth, force perception, and robot state; then forming a unified environmental representation through a fusion module; and finally, using an object parsing module to identify human bodies, obstacles, and manipulable targets, and constructing a scene semantic relationship graph. This process is consistent with the unified environmental state representation calculation process, reflecting the computational chain of multimodal input, unified representation, object parsing, and relationship modeling. During perception, greater emphasis is placed on physical executability, i.e., whether the object is reachable, whether the action is safe, whether contact is stable, and whether the current environment allows continued execution.

[0035] To enhance the feasibility of the method, the volume perception output is not directly used as input to the controller as unstructured features. Instead, it is further organized into a set of objects, spatial relationships, and operable attributes. Specifically, each object node can contain information such as category label, 3D position, bounding box, velocity estimate, graspable region, and safety level; each relationship edge can contain distance, relative direction, occlusion degree, and potential collision risk. This avoids the motion planning module relying solely on low-level pixel features, giving the subsequent control process a more explicit semantic interpretability and constraint source.

[0036] (2) Semantic goal-driven motion planning and control is a key link in understanding tasks, generating actions, and executing stably. Unlike traditional geometric trajectory-based planning methods, this strategy no longer focuses solely on how to reach the target location, but further combines task semantics, object attributes, environmental constraints, and human-computer interaction requirements to generate motion control sequences that are goal-oriented, situation-adaptive, and safe and interpretable. In other words, the robot not only needs to know where to go, but also needs to understand why it goes, how it goes, and under what constraints.

[0037] Let the semantic target of the brain layer output be The environmental condition is The robot's current state is The planning strategy can then be expressed as: ; in, This refers to the action or control command at the current moment. Here, the semantic target... It includes not only the target location, but also information such as object category, operation method, priority constraints, and interaction preferences. For example, in the task of handing a water cup to the user, It should not only correspond to the location of the water cup, but also include constraints such as the graspable area of ​​the water cup, delivery direction, user position, safe distance, and interaction rhythm. Therefore, the motion planning process can usually be divided into four stages: semantic target parsing, sub-target generation, trajectory optimization, and closed-loop control, in order to achieve a layer-by-layer mapping from high-level task semantics to low-level execution actions.

[0038] At the trajectory planning level, model predictive control is specifically employed as the trajectory optimization model. Taking the robot's current state, the reference state corresponding to the semantic target, the unified environment state, the semantic relationship graph, and safety constraints as inputs, and combining them with the robot's kinematic or dynamic model, the model predicts changes in the robot's state over multiple future control cycles. The solution variables of the model are the control input sequence in the prediction time domain. By comparing the trajectory tracking error, control cost, and semantic constraint cost corresponding to different control input sequences, the optimal control sequence that satisfies both the task objective and safety constraints is determined.

[0039] The model uses the following trajectory optimization function as the cost function: in, The reference state for the semantic target. To predict control inputs in the time domain, These are semantic constraints used to characterize target approach methods, obstacle avoidance requirements, human-machine safety distance, motion smoothness, and constraints on the operated object. and These are the weight matrices for the state error term and the control input term, respectively. These are the semantic constraint weight coefficients.

[0040] By minimizing the above cost function The optimal predicted state sequence and optimal control input sequence are obtained by solving the problem. The system uses the first control input in the optimal control input sequence as the nominal control command at the current time. The data is then sent to the subsequent control fusion module. In the next control cycle, the robot state and environment state are reacquired, and the model is solved again, thus forming a rolling optimization process. The subsequent parameterized strategy function generates compensation control instructions based on real-time environmental changes and execution deviations. The compensation control instructions are fused with the nominal control instructions, and after safety constraint verification and amplitude limiting, the final control instructions are formed and sent to the robot actuator.

[0041] At the control execution level, feedback errors are further utilized to correct the control strategy in real time, enabling the robot to have dynamic replanning capabilities when faced with environmental disturbances, target movement, or changes in task status. ; In actual operation, when the target position changes significantly, the local path is blocked by dynamic obstacles, the user suddenly approaches the robot, or the end effector's contact state is inconsistent with the expected state, the system will trigger local replanning, deceleration and pause, or safe retreat, instead of continuing to execute the original trajectory. This avoids the problem of correct high-level planning but unstable low-level execution.

[0042] To further clarify the volumetric execution process, semantically target-driven motion control is divided into several typical states: standby, target approach, grasp preparation, grasp, delivery, pause, backtrack, and emergency stop. Each state corresponds to different control objectives, safety constraints, and adjustable control parameters. Specifically, in standby mode, the system primarily maintains the stability of the robot chassis, robotic arm, and end effector. Adjustable parameters include position holding, joint holding torque, sensor refresh rate, and safety monitoring threshold. During target approach, the system mainly controls the chassis or robotic arm to move towards the target's operable area. Adjustable parameters include chassis linear velocity, angular velocity, robotic arm end effector approach velocity, acceleration limit, path smoothness coefficient, obstacle avoidance weight, and minimum safety distance. In grasping preparation mode, the system focuses on adjusting the end effector's posture and gripper pre-opening degree. Adjustable parameters include end effector pose, approach direction, gripper opening / closing width, end effector posture error threshold, and low-speed approach distance. In grasping mode, the system combines force or tactile feedback for compliant control. Adjustable parameters include gripper closing speed, gripping force threshold, and end effector contact force. The adjustable parameters include: limit, impedance control stiffness, damping coefficient, and number of retries for grasping failure; in delivery mode, the system moves towards the user or designated location while maintaining the stability of the target object, with adjustable parameters including delivery speed, end-effector trajectory smoothness, object posture maintenance error, safe distance between the human and the robot, and voice prompt frequency; in pause mode, the system stops the current motion and maintains a safe waiting state, with adjustable parameters including speed zeroing time, pause duration, safe distance threshold, and task confirmation trigger condition; in retreat mode, the system exits the current operating area along a safe direction, with adjustable parameters including retreat direction, retreat speed, retreat distance, obstacle avoidance constraint weight, and recovery planning trigger condition; in emergency stop mode, the system immediately cuts off or suppresses motion output, with adjustable parameters including actuator braking signal, joint torque limit, end-effector motion lock state, and manual confirmation recovery condition.

[0043] Therefore, different states do not only correspond to different action stages, but also to a set of configurable control parameters and safety constraints. The system can switch between different states based on semantic goals, environmental states, emotion modulation signals, and safety monitoring results, and dynamically adjust parameters such as speed, acceleration, end-effector pose, gripper opening and closing, contact force, compliance, safety distance, pause duration, backtracking distance, and emergency stop conditions. This ensures that the robot has clear control criteria and safety boundaries during target approach, grasping, delivery, and anomaly handling. Therefore, semantic goal-driven motion planning and control strategies are essentially a hierarchical decision-making process that integrates task semantics, environmental states, and feedback control. Figure 2 As shown, this method enables the robot's body layer to transform abstract task intentions into stable, safe, and interpretable embodied behaviors, providing core support for subsequent interactions.

[0044] (3) Learning-based embodied control strategies are an important method for achieving autonomous execution and online adaptation in complex environments. Compared with traditional control methods that rely on precise models, learning-based control emphasizes extracting state and action mapping relationships from interaction data, enabling robots to handle problems such as model uncertainty, environmental disturbances, and task changes in unstructured scenarios. Especially in tasks such as grasping, obstacle avoidance, collaboration, and dynamic tracking, learning-based control strategies can effectively improve the system's generalization and adaptive capabilities.

[0045] Let the robot state be The result of perception fusion is The parameterized strategy function can be expressed as: ; in, For parameterized policy functions, To control the output. Specifically, to control the robot's state. and perception fusion results The input parameterized policy function is used for feature encoding and fusion to generate compensation quantities for velocity, end-effector pose, or contact force. These compensation quantities are then superimposed with the nominal control command output by the aforementioned trajectory optimization model, and after safety constraint verification, the final control command is formed. This policy can be trained through supervised learning, imitation learning, or reinforcement learning. Supervised learning is suitable for data scenarios with clearly defined states and action labels; imitation learning is suitable for learning operational policies from manually taught trajectories; and reinforcement learning is suitable for tasks that optimize long-term behavioral performance through reward functions. In volumetric approach design, the learned policy does not replace all controllers but rather serves as a supplement to traditional motion planning and feedback control, enabling the system to have stronger adaptive capabilities in unknown disturbances, complex contact, and unstructured environments. Reinforcement learning typically models the embodied control process as a Markov decision process, learning the optimal control policy by maximizing long-term cumulative rewards.

[0046] ; in, This represents the expectation of the state transitions and action trajectories generated during the execution of the control strategy. For instant rewards, As a discount factor, Indicates the first The reward is weighted in the long-term return, with each moment having a discounted weight. The reward function is typically constructed by considering factors such as task completion, trajectory smoothness, energy consumption, and safety constraints. For example, in a grasping task, the reward can be composed of target proximity, successful grasp status, contact force stability, and collision penalty; in an obstacle avoidance task, the reward can be determined by target arrival efficiency, minimum safe distance, path smoothness, and the number of abnormal triggers. In this way, parameterized policy functions can balance task efficiency and safety constraints.

[0047] In actual execution, due to the continuous changes in environmental conditions, target location, and contact conditions, the control strategy also needs to possess online adaptive adjustment capabilities. It should be noted that the parameter update here is not an additional process independent of the aforementioned control model, but rather an update to the aforementioned parameterized strategy function. Medium parameters Local adaptive correction. Among them, This represents the updatable parameters in the parameterized policy function, which can specifically correspond to local compensation terms, control gain parameters, or lightweight adaptation module parameters. Its function is to make limited corrections to the control output based on the robot's real-time execution error, rather than to perform large-scale updates to the entire policy network during actual robot operation.

[0048] Therefore, a parameter update mechanism can be introduced: ; in, This indicates the control error or task loss at the current moment. For learning rate, Indicates the current task loss Regarding local parameters The gradient is used to determine the update direction and adjustment magnitude for reducing the loss of parameters. Here... This directly corresponds to the aforementioned model and error calculation, and can be reflected by trajectory tracking error, target state deviation, control input cost, safety constraint state, and the residual between the system's predicted output and actual observation. In other words, when end-point position deviation, target approach error, contact force instability, insufficient safety distance, or abnormally high residuals occur during the aforementioned motion planning and control process, these execution deviations can all be reflected in… This is further used to correct local parameters in the parameterized strategy function.

[0049] For example, in a grasping task, if the robot's end effector deviates from its position when approaching the target, or if the contact force fluctuates significantly during the grasping process, then It can reflect the grasping error, contact stability, and safety constraint status, and the system adjusts the compliance control gain, gripper closure compensation parameters, or end effector motion compensation terms accordingly. During obstacle avoidance tasks, if a person or obstacle gets too close, resulting in insufficient safety distance, then... It can reflect path deviation and safety constraint pressure, and the system adjusts obstacle avoidance weights, speed modulation parameters, or local path compensation terms accordingly. Thus, the parameter update process forms a closed loop with the aforementioned control strategy, error calculation, and safety monitoring, rather than being an isolated parameter adjustment step.

[0050] This update process enables the robot to continuously adjust its control parameters based on real-time feedback, thereby improving its adaptability to unknown scenarios and dynamic disturbances. It should be noted that in real-world robot systems, online updates do not mean unlimited real-time changes to all policy parameters. Instead, they typically apply to local compensation terms, gain parameters, or lightweight adaptation modules, and set upper limits for parameter update magnitudes and safety constraints to prevent policy drift that could lead to execution instability.

[0051] (4) Safety constraints, fault detection and abnormal response mechanisms.

[0052] In the real-world deployment of embodied intelligent robots, safety constraints, fault detection, and emergency response are crucial for ensuring stable system operation and reliable human-robot collaboration. Because robots need to continuously interact with humans, obstacles, and dynamic targets in open environments, their execution process not only faces collision risks, control instability, and sensor anomalies, but may also be affected by communication delays, actuator failures, and sudden environmental changes. Embedding safety mechanisms into the entire process of perception, planning, and control constructs a closed-loop protection system encompassing constraint monitoring, anomaly identification, tiered response, and resumption of execution.

[0053] Let the robot state be The control input is The set of security constraints is Then the safe execution conditions can be expressed as: ;in, This can include upper speed limits, joint torque thresholds, minimum safe distances, end-effector contact force thresholds, attitude constraints, and collision avoidance constraints. To ensure the trajectory always meets safety conditions, constraints can be added to the control optimization process.

[0054] ; in, This is a safety barrier function used to characterize whether the state and control inputs are within the safe domain. When approaching constraint boundaries, the system can reduce speed in advance, increase the safety distance, or switch to a conservative control strategy; when constraints are violated or there is an imminent risk of collision, the system enters a pause, rollback, or emergency stop state. Thus, the safety mechanism is no longer a passive protection after mission failure, but rather plays a continuous constraining role throughout the planning and control process.

[0055] For fault detection, a residual-based anomaly monitoring mechanism is adopted. Let the system predict the output as... The actual observation is The residual is then defined as:

[0056] ; in, This represents the residual, which has a different meaning from the reward item mentioned earlier. When this happens, it can be determined that the system has an anomaly, among which, This is a preset threshold. The anomaly may originate from sensor drift, actuator failure, model mismatch, or external impact. The system triggers different levels of response strategies based on the anomaly type, such as deceleration, pause, avoidance, reversal, or emergency braking. Its emergency control can be represented as:

[0057] ; in, This serves as the protective control input for the safety mode. For ease of system implementation, safety responses are categorized into four types: The first type is minor risk, such as short-term target occlusion or localized positioning fluctuations; the system can maintain the task and make minor trajectory adjustments. The second type is moderate risk, such as approaching dynamic obstacles or blocked paths; the system should decelerate, pause, and replan. The third type is severe risk, such as a human entering a dangerous distance, contact force exceeding a threshold, or abnormal end-effector posture; the system should immediately stop the current action and execute a reversal. The fourth type is emergency risk, such as an impending collision, actuator malfunction, or persistent safety constraint failure; the system should trigger an emergency stop and await manual confirmation.

[0058] Regarding anomaly recovery, the body layer does not simply classify all anomalies as task failures, but rather categorizes and handles them based on their source. If the target is lost, the system prioritizes re-sensing and updating the object's position; if the grasping fails, the system can adjust the grasping posture and retry; if the path is blocked, the system performs local replanning; if the user suddenly approaches, the system pauses and generates explanation or confirmation feedback through the core layer; if a sensor fails or an actuator malfunctions, it enters a safety mode and stops execution. Only when anomalies occur consecutively, the number of recovery attempts exceeds a threshold, or safety constraints cannot be satisfied again, does the system classify the current task as a failure. Therefore, safety constraints, fault detection, and anomaly response mechanisms essentially construct a real-time, verifiable, interventionist, and recoverable safety control closed loop during robot execution. Figure 2As shown, this mechanism enables robots not only to perform tasks, but also to ensure human-machine safety in complex environments, thus providing key support for the stable operation, anomaly recovery, and practical implementation of embodied intelligence systems.

[0059] This invention also provides a robot control system based on brain-heart-body coordination, comprising: The environmental perception module is used to acquire the robot's current motion state and raw multimodal sensing data; it performs feature encoding and weighted fusion on the raw multimodal sensing data to generate a unified environmental state; based on the unified environmental state, it performs object parsing on human bodies, obstacles and target objects in the scene to construct a semantic relationship graph containing nodes and edges; where nodes represent human bodies, obstacles or manipulated objects, and edges represent distance relationships, occlusion relationships, contact relationships or interaction relationships between nodes.

[0060] The motion planning module is used to acquire semantic targets transmitted from the brain layer and emotional modulation signals from the heart layer. Based on the semantic relationship graph, the semantic targets transmitted from the brain layer are analyzed to obtain the target state. Based on the target state, the robot's current state, the emotional modulation signals from the heart layer, and the unified environmental state, the trajectory optimization function is solved using model predictive control to obtain preliminary control commands.

[0061] The execution control module is used to input the robot's current motion state and unified environment state into the parameterized strategy function, and output the compensation control command; it superimposes the compensation control parameters onto the initial control command, inputs the superimposed control parameters into the forward prediction model inside the volume layer, and outputs the robot's predicted state at the next moment.

[0062] The various modules in the aforementioned brain-heart-body integrated robot control system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0063] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a robot control method based on brain-mind-body coordination. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0064] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the aforementioned method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a robot control method based on brain-mind-body coordination. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0065] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A robot control method based on brain-heart-body coordination, wherein the robot comprises a brain layer, a heart layer, and a body layer, characterized in that, Includes the following steps: The robot's current motion state and raw multimodal sensing data are acquired; the raw multimodal sensing data are feature-encoded and weighted to generate a unified environmental state; based on the unified environmental state, the human body, obstacles, and target objects in the scene are analyzed to obtain a semantic relationship graph containing nodes and edges; wherein, nodes represent human bodies, obstacles, or manipulated objects, and edges represent distance relationships, occlusion relationships, contact relationships, or interaction relationships between nodes; The semantic target transmitted from the brain layer and the emotional modulation signal from the heart layer are acquired. Based on the semantic relationship graph, the semantic target transmitted from the brain layer is parsed to obtain the target state. Based on the target state, the robot's current state, the emotional modulation signal from the heart layer, and the unified environmental state, the trajectory optimization function is solved using model predictive control to obtain the preliminary control command. The robot's current motion state and unified environment state are input into a parameterized strategy function, which outputs a compensation control command. The compensation control parameters are superimposed on the initial control command, and the superimposed control parameters are input into the forward prediction model inside the volume layer, which outputs the robot's predicted state at the next moment.

2. The robot control method based on brain-mind-body coordination integration according to claim 1, characterized in that, The trajectory optimization function is specifically as follows: ； in, The reference state for the semantic target. To predict control inputs in the time domain, , For the emotional modulation signal of the heart layer, This is the robot's current state. For semantic constraints, For semantic goals, Environmental state; and These are the weight matrices for the state error term and the control input term, respectively. These are the semantic constraint weight coefficients.

3. The robot control method based on brain-mind-body coordination integration according to claim 1, characterized in that, The original multimodal sensing data is subjected to feature encoding and weighted fusion to generate a unified environmental state. Specifically, the image information, depth information, force or tactile feedback, and robot body state in the original multimodal sensing data are input into separate encoders and converted into multiple high-dimensional semantic feature vectors. The multiple high-dimensional semantic feature vectors are multiplied by the modal fusion weights, and the weighted feature vectors are then concatenated according to the channel dimension to obtain the unified environmental state.

4. The robot control method based on brain-mind-body coordination integration according to claim 1, characterized in that, The parameterization strategy function is specifically as follows: ;in, For parameterized strategy functions, To control the output, In robot state, To unify the environment state, train using any pair of parameterized policy functions from supervised learning, imitation learning, or reinforcement learning.

5. The robot control method based on brain-mind-body coordination integration according to claim 1, characterized in that, Robot safety monitoring is performed by calculating the residual between the robot's predictive control and the actual sensor observations at the next moment. When the residual exceeds a preset threshold, the system is determined to be abnormal, and an abnormal response mechanism is triggered.

6. The robot control method based on brain-mind-body coordination integration according to claim 5, characterized in that, The anomaly response mechanism includes a tiered response strategy and an anomaly recovery strategy; The graded response strategy includes: Category 1 minor risk, the robot continues its task; Category 2 medium risk, the robot decelerates, pauses, or replans; Category 3 severe risk, the robot must immediately stop its current action and reverse; Category 4 emergency risk, an emergency stop is triggered and manual confirmation is required. The anomaly recovery strategy includes: if the anomaly is target loss, re-sensing and updating the object's position; if the anomaly is grasping failure, adjusting the grasping posture and retrying; if the anomaly is path blockage, performing local replanning; if the anomaly is user sudden approach, pausing and generating explanation or confirmation feedback; if the anomaly is sensor failure or actuator malfunction, entering safe mode and stopping execution; when anomalies occur consecutively, the number of recovery attempts exceeds a threshold, or safety constraints cannot be satisfied again, the current task is determined to have failed.

7. A robot control system based on brain-heart-body coordination, characterized in that, include: The environmental perception module is used to acquire the robot's current motion state and raw multimodal sensing data; The original multimodal sensing data is feature-encoded and weighted to generate a unified environmental state. Based on the unified environmental state, the human body, obstacles and target objects in the scene are analyzed to obtain a semantic relationship graph containing nodes and edges. Among them, nodes represent human bodies, obstacles or operation objects, and edges represent distance relationships, occlusion relationships, contact relationships or interaction relationships between nodes. The motion planning module is used to acquire semantic targets transmitted from the brain layer and emotional modulation signals from the heart layer. Based on the semantic relationship graph, the semantic targets transmitted from the brain layer are parsed to obtain the target state. Based on the target state, the robot's current state, the emotional modulation signals from the heart layer, and the unified environmental state, the trajectory optimization function is solved using model predictive control to obtain preliminary control commands. The execution control module is used to input the robot's current motion state and unified environment state into a parameterized strategy function, and output a compensation control command; to superimpose the compensation control parameters onto the initial control command, to input the superimposed control parameters into the forward prediction model inside the body layer, and to output the robot's predicted state at the next moment.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 6.

Images