Control method, system, storage medium and program product for a robot

Abstract

This disclosure provides a robot control method, system, storage medium, and program product. The control method includes: receiving natural language instructions; parsing and planning the natural language instructions using a first model to output a structured task sequence; receiving the structured task sequence and, in a physical simulation environment, simulating and verifying multiple candidate operation schemes generated based on the structured task sequence according to preset physical feasibility judgment rules; responding to the existence of verified candidate operation schemes, determining the optimal operation scheme from the verified candidate operation schemes according to preset scheme optimization rules; and generating basic control instructions for driving the robot based on the optimal operation scheme. The positive improvement effect of this disclosure is that, through the synergy of the above-mentioned technical means, this method improves the operational safety, environmental adaptability, and overall task success rate of the robot when executing complex and fuzzy natural language instructions.

Description

Technical Field

[0001] This disclosure relates to the field of robot control, and more particularly to a robot control method, system, storage medium, and program product. Background Technology

[0002] In existing technologies, intelligent control systems for robots often adopt a modular and phased design approach. A common implementation method is to utilize semantic understanding modules based on Large Language Models (LLM) or Multimodal Large Language Models (MLLM) to parse natural language instructions issued by humans into machine-executable action sequences. However, such solutions typically focus on the logical decomposition of the task itself during parsing, making it difficult to transform the ambiguous semantics inherent in the instructions, which concern operational safety and physical feasibility (e.g., "gentle" or "careful"), into precise and verifiable physical operational constraints. This can lead to physical risks in the generated instruction sequences, such as applying excessive force to fragile objects or causing collisions in complex environments. Another technical approach focuses on physical simulation and verification, using a precise world model (WM) to pre-run or generate operational schemes in a virtual environment to select physically feasible solutions. While such methods can ensure the safety of operations in a simulation environment, their planning process usually relies on fixed-format instruction inputs or predefined parameter spaces, lacking a deep understanding and adaptability to open and ambiguous natural language instructions, which limits their application in diverse and unstructured scenarios. Summary of the Invention

[0003] The technical problem to be solved by this disclosure is to overcome the defect in the prior art that robots cannot automatically convert the ambiguous semantics in the natural language instructions into precise control instructions that meet physical safety constraints when executing natural language instructions, and to provide a robot control method, system, storage medium and program product.

[0004] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0005] This disclosure provides a robot control method, the control method comprising:

[0006] The system receives natural language instructions, parses and plans the natural language instructions through a first model, and outputs a structured task sequence. The first model is configured as a semantic understanding and task planning model. The structured task sequence contains at least one sub-task, and each sub-task is associated with at least one quantized physical constraint parameter.

[0007] The system receives the structured task sequence and performs simulation verification on multiple candidate operation schemes generated based on the structured task sequence in a physical simulation environment according to a preset physical feasibility judgment rule. If there is a candidate operation scheme that passes the verification, the system determines the optimal operation scheme from the candidate operation schemes that pass the verification according to a preset scheme optimization rule.

[0008] Based on the optimal operation scheme, generate basic control commands for driving the robot;

[0009] During the process of the robot executing operations according to the basic control instructions, data sets from tactile, visual, and force sensors are collected in real time. The data sets are then fused using a second model to generate fused perception features. Based on these fused perception features, the basic control instructions are adjusted in real time to obtain the final control instructions to drive the robot.

[0010] Optionally, after the step of simulating and verifying multiple candidate operation schemes generated based on the structured task sequence, the method further includes:

[0011] If all of the candidate operation schemes fail to pass the verification, a feedback signal containing specific physical constraint conflict information is generated, and an adjustment to the structured task sequence is triggered based on the feedback signal.

[0012] Optionally, the preset physical feasibility determination rules include at least one of the following: the deformation rate of the manipulated object does not exceed a first threshold, the grasping stability coefficient is not lower than a second threshold, the operating force value does not exceed the corresponding physical constraint threshold, and the motion trajectory is collision-free.

[0013] Optionally, the preset scheme selection rule is a multi-objective weighted scoring rule, used to score the candidate operation schemes that have passed the verification; the scoring dimensions include at least one of the following: constraint satisfaction, operation efficiency, and operation energy consumption.

[0014] Optionally, the second model is a multimodal Transformer model based on a cross-attention mechanism; the fused perception feature is a feature representation that integrates multimodal information, and the feature representation simultaneously encodes the semantic category information and physical state information of the perceived object.

[0015] Optionally, the real-time adjustment of the basic control commands based on the fused sensing features includes:

[0016] The optimal operation scheme and the fused sensing features are input into the slow control network to generate the basic control commands;

[0017] The fused sensing features, updated in real time, are input into a fast control network to generate control parameter fine-tuning instructions;

[0018] The basic control command and the control parameter fine-tuning command are combined to obtain the final control command.

[0019] Optionally, the slow control network operates at a first frequency to output the basic control command containing the joint motion trajectory and basic force control parameters;

[0020] The fast control network operates at a second frequency higher than the first frequency, and is used to output the control parameter fine-tuning command for fine-tuning the force control parameters and joint angles;

[0021] The synthesis is achieved by superimposing the control parameter fine-tuning instructions onto the basic control instructions.

[0022] This disclosure provides a control system for a robot, the control system comprising:

[0023] A semantic planning module is used to receive natural language instructions, parse and plan the natural language instructions through a first model, and output a structured task sequence; the first model is configured as a semantic understanding and task planning model, and the structured task sequence contains at least one sub-task, and each sub-task is associated with at least one quantized physical constraint parameter;

[0024] The pre-simulation optimization module is used to receive the structured task sequence, and perform simulation verification on multiple candidate operation schemes generated based on the structured task sequence in a physical simulation environment according to a preset physical feasibility judgment rule; in response to the existence of a candidate operation scheme that has passed the verification, the optimal operation scheme is determined from the candidate operation schemes that have passed the verification according to a preset scheme selection rule.

[0025] The control module is used to generate basic control commands for driving the robot based on the optimal operation scheme.

[0026] The perception adjustment module is used to collect data sets from tactile, visual, and force sensors in real time during the process of the robot performing operations according to the basic control instructions. The second model is used to fuse the dataset to generate fused perception features. Based on the fused perception features, the basic control instructions are adjusted in real time to obtain the final control instructions to drive the robot.

[0027] Optionally, the pre-performance optimization module is further configured to:

[0028] If all of the candidate operation schemes fail to pass the verification, a feedback signal containing specific physical constraint conflict information is generated, and an adjustment to the structured task sequence is triggered based on the feedback signal.

[0029] Optionally, the preset physical feasibility determination rules include at least one of the following: the deformation rate of the manipulated object does not exceed a first threshold, the grasping stability coefficient is not lower than a second threshold, the operating force value does not exceed the corresponding physical constraint threshold, and the motion trajectory is collision-free.

[0030] Optionally, the preset scheme selection rule is a multi-objective weighted scoring rule, used to score the candidate operation schemes that have passed the verification; the scoring dimensions include at least one of the following: constraint satisfaction, operation efficiency, and operation energy consumption.

[0031] Optionally, the second model is a multimodal Transformer model based on a cross-attention mechanism; the fused perception feature is a feature representation that integrates multimodal information, and the feature representation simultaneously encodes the semantic category information and physical state information of the perceived object.

[0032] Optionally, the sensing adjustment module is specifically used for:

[0033] The optimal operation scheme and the fused sensing features are input into the slow control network to generate the basic control commands;

[0034] The fused sensing features, updated in real time, are input into a fast control network to generate control parameter fine-tuning instructions;

[0035] The basic control command and the control parameter fine-tuning command are combined to obtain the final control command.

[0036] Optionally, the slow control network operates at a first frequency to output the basic control command containing the joint motion trajectory and basic force control parameters;

[0037] The fast control network operates at a second frequency higher than the first frequency, and is used to output the control parameter fine-tuning command for fine-tuning the force control parameters and joint angles;

[0038] The synthesis is achieved by superimposing the control parameter fine-tuning instructions onto the basic control instructions.

[0039] This disclosure provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the robot control method described in any one of the above-described methods.

[0040] This disclosure provides a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the robot control method as described in any one of the above statements.

[0041] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0042] The positive advancements of this disclosure lie in the following: by deeply integrating semantic understanding with physical pre-simulation and introducing closed-loop adjustments based on real-time multimodal perception, it achieves end-to-end intelligent operation of the robot, from "understanding instructions" to "safe execution." Specifically, firstly, this method ensures that the task planning generated from the source of instructions has a physically feasible basis by imposing physical constraints on the semantic parsing and planning process. Secondly, through physical pre-simulation and scheme optimization in a virtual environment, it achieves automatic verification and optimization of the planning results, proactively mitigating physical risks. Finally, in the execution phase, real-time multimodal perception information is used to dynamically fine-tune the control instructions, enabling the robot to flexibly adapt to real-time changes in the physical environment and effectively compensate for model errors and external disturbances. Through the synergy of the above technical means, this method improves the operational safety, environmental adaptability, and overall task success rate of the robot when executing complex and ambiguous natural language instructions. Attached Figure Description

[0043] Figure 1 A flowchart illustrating a robot control method provided as an exemplary embodiment of this disclosure;

[0044] Figure 2 A schematic diagram of a control system for a robot provided as an exemplary embodiment of this disclosure;

[0045] Figure 3 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. Detailed Implementation

[0046] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0047] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0048] Example 1

[0049] Figure 1 A flowchart of a robot control method provided as an exemplary embodiment of this disclosure is included. The control method includes:

[0050] Step 101: Receive natural language instructions, parse and plan the natural language instructions through the first model, and output a structured task sequence; the first model is configured as a semantic understanding and task planning model, the structured task sequence contains at least one sub-task, and each sub-task is associated with at least one quantized physical constraint parameter.

[0051] Step 101 utilizes a specially trained first model to achieve the transformation from "instruction semantic understanding" to "constrained task planning". Specifically:

[0052] 1. Input: Natural Language Instructions. Instructions originate from the user and are in the form of human natural language. They typically include the operation goal, the object, and modifying requirements. For example, in home care or rehabilitation nursing, an instruction might be "Pass me the water glass on the table" or "Perform gentle knee flexion and extension movements for the patient." These instructions are characterized by containing action verbs with clear intent (such as "pass," "flex," etc.), but also often include qualitative descriptions that are difficult to quantify directly (such as "gently").

[0053] 2. Core Processing: The First Model (Semantic Understanding and Task Planning Model). Its core functions are twofold: first, understanding the semantic meaning of instructions, identifying actions, objects, and implicit conditions; second, planning, organizing the understood semantics into a series of logically ordered and executable atomic operation steps. To achieve these dual functions, this model is typically built upon large-scale multimodal language models (such as Llama, GPT, etc.). Its key capabilities stem from fine-tuning training of instructions for specific application domains (such as home services and rehabilitation care as described in the document). This training allows the model to internalize prior knowledge of task flows in relevant scenarios, as well as experience in mapping specific semantics to physical parameters. For example, through training, the model can learn that the word "gentle" in the context of "massage" may be associated with a lower force threshold.

[0054] 3. Output: A structured task sequence and its key attributes. "Structured" refers to a data structure with a fixed format, clearly defined fields, and seamless machine parsing (e.g., JSON, XML, or a specific protocol buffer format). It transforms ambiguous instructions into a clear, step-by-step "work list." Detailed component description:

[0055] Subtasks: A structured task sequence consists of one or more "subtasks." Each subtask represents the unit of operation required to achieve the final instruction goal. For example, for the instruction "pick up the water glass," the subtask sequence might include "locate the water glass," "plan the grasping posture," "execute the grasp," and "lift and move." Each subtask typically contains fields such as "task ID," "action type," and "target object" in the data structure.

[0056] Quantitative physical constraint parameters: These are physical constraints, expressed in numerical values ​​and units, explicitly associated with one or more subtasks. These parameters transform the qualitative descriptions (adjectives) in the instructions into quantitative indicators that the control system can directly comply with. Their role is to provide specific verification criteria for the next step of "physical rehearsal." These parameters are automatically inferred and associated by the first model based on its understanding of the instruction semantics. For example, for the modifier "gently," the model might associate a constraint of "maximum clamping force ≤ 2.0 Newtons" for the "grab" subtask; for "stable," it might associate a constraint of "maximum acceleration ≤ 0.5 m / s²" for the "move" subtask. 2 The constraint is "". Specifically, it takes the form: "physical_constraints": {"max_force": 2.0, "unit": "N"}.

[0057] Specific examples are explained below:

[0058] Taking the common instruction in a home service scenario, "Place that fragile glass gently on the other side of the table," as an example, after processing in step 101: the natural language instruction "Place that fragile glass gently on the other side of the table." The first model identifies the core verb "place," the target object "glass" (with the attribute "fragile"), the destination "the other side of the table," and the modifier "gently." Based on its training knowledge, it understands that "fragile" means that the force needs to be controlled during the operation to prevent breakage, and "gently" means that violent shaking should be avoided during the movement. Output structured task sequence (exemplary description): The model outputs a structured data object, which may contain the following sub-task sequence:

[0059] Subtask 1 (Identification and Location):

[0060] Action: IDENTIFY;

[0061] Target object: GLASS_CUP;

[0062] Physical constraints: {} (This task may not require specific physical constraints);

[0063] Subtask 2 (Fetch):

[0064] Action: GRASP;

[0065] Target object: GLASS_CUP;

[0066] Physical constraints: {"max_grasp_force": 5.0, "unit": "N"} (The model maps "fragile" to a smaller maximum gripping force limit);

[0067] Subtask 3 (Move):

[0068] Action: MOVE;

[0069] Target location: OPPOSITE_SIDE_OF_TABLE;

[0070] Physical constraints: {"max_acceleration": 0.3, "unit": "m / s"} 2 ", "max_jerk": 0.5, "unit": "m / s 3 The model maps "smoothly" to constraints on motion acceleration and jerk.

[0071] Subtask 4 (Placement):

[0072] Action: PLACE;

[0073] Target location: OPPOSITE_SIDE_OF_TABLE;

[0074] Physical constraints: {"placement_velocity": 0.1, "unit": "m / s"} (The model sets a velocity limit for placement to ensure smooth contact).

[0075] Step 102: Receive the structured task sequence and, in the physical simulation environment, perform simulation verification on multiple candidate operation schemes generated based on the structured task sequence according to the preset physical feasibility judgment rules.

[0076] Step 102 serves to perform a physical feasibility test and screening of the abstract task plan generated in the preceding step (Step 101) in a high-fidelity digital virtual environment before the robot performs any real physical actions. Its purpose is to transform the task plan, which is "theoretically consistent with semantic intent," into one or more "physically absolutely safe, stable, and efficient" executable operations. This allows for the early detection and avoidance of all potential physical risks (such as object damage, operation failure, and mechanical collisions) within the virtual domain, ensuring the absolute reliability of subsequent real operations. Specifically:

[0077] 1. Input: Receive a structured task sequence. This input comes directly from the output of step 101. It is a structured task description that has been given preliminary quantitative physical constraints (such as maximum operating force and speed limits).

[0078] 2. Working Environment: The environment is a physical simulation environment. A "physical simulation environment" refers to a virtual digital space constructed by a physics engine (such as Bullet, ODE, MuJoCo, or domain-specific simulation software). This environment simulates the core physical laws of the real world, including but not limited to rigid body dynamics, collision detection, friction, and soft body deformation. The environment must be loaded with high-precision geometric and physical models (mass, inertia, elastic modulus, etc.) of the robot, the manipulated object, and the working scene. Its parameters should be aligned with the real world as closely as possible to ensure the validity of the simulation results.

[0079] 3. Core judgment criteria: Based on the preset physical feasibility judgment rules.

[0080] Optionally, the preset physical feasibility judgment rules include at least one of the following: the deformation rate of the manipulated object does not exceed a first threshold, the grasping stability coefficient is not lower than a second threshold, the operating force value does not exceed the corresponding physical constraint threshold, and the motion trajectory is collision-free.

[0081] The "preset physical feasibility judgment rules" is a set of rules used to monitor and evaluate the execution process of candidate operation schemes in a physical simulation environment. Its purpose is to pre-identify and eliminate all physical risks that could lead to operation failure, object damage, equipment collision, or personnel injury within the virtual domain. Four typical but non-exclusive verification dimensions are listed here, and the system can be configured to have one or more of them active simultaneously. A candidate scheme is considered "verified" only when it meets the judgment conditions of all enabled rules throughout the entire simulation execution process.

[0082] Rule 1. The deformation rate of the operated object does not exceed the first threshold.

[0083] This rule focuses on the mechanical integrity of the target object during operation. "Deformation rate" is a physical quantity used to quantify the degree of shape change of an object under external force; in simulation, it can be derived by calculating the object's mesh vertex displacement, volume change rate, or stress-strain relationship. "First threshold" is a pre-set value characterizing the maximum safe deformation limit that the object can withstand. The purpose of this rule is to prevent plastic deformation, structural damage, or functional failure of the target object due to excessive force or improper handling. For example, when grasping a soft fruit, it is necessary to ensure that its compressive deformation is within a recoverable elastic range to avoid the peel cracking.

[0084] Judgment Method: During simulation execution, the physics engine calculates or post-processes the deformation rate of the manipulated object in real time. If, at any moment, the deformation rate exceeds a preset "first threshold," it is determined that this rule has been violated, and the candidate solution is not feasible. The specific value of the "first threshold" depends on the material properties of the object (such as elastic modulus and yield strength) and safety design requirements. For example, for a specific silicone massage ball, its deformation rate safety threshold may be set to no more than 10%; while for a precision electronic component, this threshold may be close to 0%.

[0085] Rule 2. The crawling stability coefficient shall not be lower than the second threshold.

[0086] This rule assesses the reliability of a robot's end effector (such as a gripper or suction cup) in grasping or holding a target object. The "grasping stability coefficient" is an indicator used to quantify the degree of grasping stability; its calculation can be based on the principles of force closure and form closure, or through simulation analysis of the gripping system's anti-slip and anti-rotation capabilities under disturbances. The "second threshold" is a preset minimum stability requirement. The purpose of this rule is to ensure that during subsequent operations such as movement and handling, the object will not accidentally detach from the gripper or undergo unexpected relative motion, thereby preventing operational failure or secondary risks.

[0087] Judgment Method: In the simulation, after the robot performs a grasping action and applies a preset force, the system calculates the stability coefficient under that grasping posture. This coefficient may take into account the effects of friction cone conditions, contact force distribution, and external disturbances (such as gravity and inertial forces). If the calculated stability coefficient is lower than a preset "second threshold," the grasping scheme is determined to be unstable and infeasible. For example, for a smooth-surfaced glass, using planar clamping and using an envelope grasping method will correspond to different stability coefficients, with the latter typically having a higher coefficient.

[0088] Rule 3. The operating force value shall not exceed the corresponding physical constraint threshold.

[0089] This rule ensures that the physical forces actually applied in the simulation strictly adhere to the "quantified physical constraint parameters" associated with each subtask in the "structured task sequence" generated in step 101. "Operating force value" refers to the dynamic force values ​​such as contact force and clamping force generated between the robot's end effector and the manipulated object during the simulation. "Corresponding physical constraint threshold" refers to the specific force constraint value associated with the aforementioned subtask, such as "maximum clamping force ≤ 5N". This rule serves as a direct bridge between high-level semantic planning (abstract constraints) and low-level physical simulation (specific force values), mandating that simulation behavior must not exceed the safety boundaries set during the planning phase.

[0090] Judgment method: During the simulation of a specific subtask (such as grasping or pressing), the relevant operational force value is monitored in real time. If at any moment the monitored operational force value exceeds the physical constraint threshold associated with that subtask, it is immediately determined that this rule has been violated, and the solution is not feasible. This is a compliance check for input instructions.

[0091] Rule 4. The trajectory of motion is collision-free.

[0092] This rule checks whether the robot body, end effector, and manipulated object geometrically interfere with other static or dynamic obstacles (including the robot's own links) in the simulation environment throughout the planned motion path. "Collision-free" means that there is no intersection between the bounding boxes or precise geometric models of all relevant moving parts at discrete time steps. The purpose of this rule is to prevent equipment damage, operational failures, or safety accidents caused by mechanical collisions.

[0093] Judgment Method: Physics simulation engines typically have built-in continuous collision detection functionality. The collision detection module runs continuously as the simulated robot moves along the candidate trajectory. If any collision event that should not occur is detected, it is considered a violation of this rule, and the proposed solution is deemed infeasible. Trajectory planning must consider the robot's workspace, joint constraints, and the three-dimensional geometry of the environment.

[0094] Example explanation:

[0095] Taking the instruction "wipe a wooden tabletop with a cloth" in a home service scenario as an example, step 101 parsing may generate a sub-task sequence containing "move to the starting point," "wipe repeatedly with a specific force," and "move away." A quantization constraint of "downward pressure ≤ 15N" is associated with the "wipe" sub-task to prevent damage to the tabletop's finish. In the simulation verification of step 102, for a generated candidate wiping scheme (including the specific robotic arm motion trajectory, end effector posture, and downward pressure control curve), the system makes a judgment based on the above rules:

[0096] Deformation rate rule: Simulation calculation of the microscopic deformation of the contact area between the cloth and the table, as well as the possible deflection of the table itself. If the preset "first threshold" is set for the table to have a deformation rate of no more than 0.5%, and the simulation shows that the local deformation rate reaches 0.6% due to concentrated downward pressure, then this rule is considered to be violated.

[0097] Grasping stability rule: Simulation evaluates the robot gripper's gripping of the cloth. If the calculated "grasping stability coefficient" is 0.7 in the rapid reciprocating motion simulation, which is lower than the preset "second threshold" of 0.85, then the cloth is deemed to be at risk of slipping, violating this rule.

[0098] Operating force rule: The applied downward pressure is monitored in real time during simulation. If the downward pressure reaches 16N at a certain moment in the scheme, exceeding the subtask constraint of 15N, then this rule is directly determined to be violated.

[0099] Motion trajectory rules: Simulation detects the movement path of the robotic arm. If the trajectory planning is improper, causing the robotic arm to collide with the table lamp model on the table during movement, this rule is considered to be violated.

[0100] Only if the candidate solution meets all enabled judgment rules in the simulation (i.e., the deformation rate and gripping stability meet the requirements, the operating force does not exceed 15N, and there is no collision throughout the process), can the solution pass the physical feasibility verification at this stage and enter the subsequent "solution selection" stage.

[0101] 4. Core processing action: Simulation verification of multiple candidate operation schemes generated based on structured task sequences.

[0102] For each generated candidate operation plan, the system rigorously simulates its entire execution process in a physical simulation environment. The simulation engine calculates the robot and environment states at each step based on physical laws. The verification module checks the data throughout the simulation process synchronously or afterwards according to the "preset physical feasibility judgment rules." A candidate plan is marked as "verified" only if it satisfies all the preset physical feasibility judgment rules throughout the entire simulation process. If any rule is violated at any time, the plan is judged as "infeasible."

[0103] Specific example: Taking the instruction "pick up a fragile ceramic cup from the table and put it in the cabinet" in a home service scenario as an example, after step 101, a structured task sequence is obtained, where the "grab" subtask is associated with the quantitative constraint "maximum gripping force ≤ 8N" (derived from the semantic mapping of "fragile"). In step 102, the input is: receiving the above structured task sequence with constraints. Candidate solutions are generated: Based on the "grab" task, under the constraint of "maximum gripping force ≤ 8N", the system may generate multiple candidate grabbing solutions. For example:

[0104] Option A: The gripping position is the middle of the cup, and the clamping force is 5N.

[0105] Option B: The gripping position is the cup handle, and the clamping force is 4N.

[0106] Option C: The gripping position is the middle of the cup, and the clamping force is 7N.

[0107] Option D: The gripping position is at the edge of the cup, and the clamping force is 6N.

[0108] (At the same time, different parameter schemes may also be generated for the velocity curve of the movement trajectory).

[0109] Simulation Verification: In a physical simulation environment, perform complete grabbing and initial movement simulations of schemes A, B, C, and D sequentially. Verification rules may include:

[0110] Rule 1 (Object Integrity): The maximum stress value at the stress concentration point of the ceramic cup in the simulation is less than the compressive strength threshold of the ceramic.

[0111] Rule 2 (Operational Stability): In the simulation, the ceramic cup does not slip relative to each other during the entire grasping process (i.e., the clamping force and the coefficient of friction satisfy the static friction condition).

[0112] Rule 3 (Constraint Compliance): The peak clamping force in the actual simulation is ≤ 8N.

[0113] Rule 4 (Trajectory Safety): The robot arm's movement path should not collide with the table or itself.

[0114] Verification results:

[0115] Options A and B may successfully pass all rule validations.

[0116] Option C was deemed infeasible because the clamping force of 7N caused the stress on the cup body in the simulation to approach or exceed the threshold (violating rule 1).

[0117] Option D is deemed infeasible because its posture becomes unstable due to grasping the edge of the cup, resulting in swaying or slippage during simulated movement (violating rule 2).

[0118] Step 1021: In response to the existence of a verified candidate operation scheme, the optimal operation scheme is determined from the verified candidate operation schemes according to the preset scheme selection rules.

[0119] Step 1021 is executed after step 102, which completes the physical feasibility simulation verification of multiple candidate operation schemes. Specifically:

[0120] 1. Triggering conditions and inputs: In response to the existence of a validated candidate operation scheme.

[0121] Logical Connection: This condition clarifies the execution prerequisite for step 1021. It directly depends on the output of the physical simulation verification step in step 102. Step 102 performs parallel simulation and rule determination on all candidate operation schemes generated based on the structured task sequence. Its output is a set of feasible schemes that have passed all the "physical feasibility determination rules" checks. This set may be empty (no feasible schemes), contain one scheme, or contain multiple schemes. If the set is empty, the feedback adjustment process is triggered (see step 1022), not this step. If there is only one feasible scheme in the set, that scheme can be directly determined as the output.

[0122] 2. Decision-making basis: Based on the preset scheme selection rules.

[0123] Optionally, the preset scheme selection rule is a multi-objective weighted scoring rule, which is used to score the candidate operation schemes that have passed the verification; the scoring dimensions include at least one of the following: constraint satisfaction, operation efficiency, and operation energy consumption.

[0124] The scoring dimensions are specific aspects used to measure and compare the performance of candidate operation schemes. The rules explicitly list "constraint satisfaction," "operational efficiency," and "operational energy consumption" as core scoring dimensions, and the system can be configured to enable one or more of these dimensions. The definition of each dimension is as follows:

[0125] Dimension 1. Constraint Satisfaction: This dimension measures the degree to which the actual physical parameters of a solution conform to the "quantified physical constraint parameters" associated with the "structured task sequence" in step 101, and the safety margin during solution execution. It is not a simple "satisfied / unsatisfied" binary judgment, but rather a quantitative score of the degree of satisfaction. For example, for a subtask with the constraint "grasping force ≤ 5N", the peak actual grasping force of solution A is 4N, while that of solution B is 2N. Although both satisfy the constraint, solution B has a larger safety margin (3N margin vs. 1N margin). The constraint satisfaction score can be designed to be positively correlated with the "margin" of actual parameter values ​​falling below the constraint threshold, or negatively correlated with the risk of exceeding the limit (such as variance).

[0126] Dimension 2. Operational Efficiency: This dimension measures the simplicity of the time or steps required for a solution to complete a predetermined task. Generally, the shorter the completion time, the higher the operational efficiency score. In simulation, the estimated completion time of the task can be directly calculated or simulated using the preset motion trajectory parameters (such as velocity and acceleration of each axis). In some scenarios, operational efficiency may also be reflected in the shortest possible motion path or the minimization of ineffective actions.

[0127] Dimension 3. Operational Energy Consumption: This dimension measures the estimated energy consumed by the robot system during the execution of the solution. Energy consumption is related to factors such as the robot's joint torque, movement speed, and running time. In physical simulation, energy consumption can be estimated by integrating the joint torque and angular velocity over the simulation period. Lower estimated energy consumption corresponds to a higher operational energy consumption score. Optimizing this dimension can help extend the robot's endurance or reduce operating costs.

[0128] The core operation of the "multi-objective weighted scoring rule" is to combine the scores of the above multiple dimensions into a comprehensive total score, which specifically includes the following steps:

[0129] Single-dimensional score calculation: For each enabled scoring dimension of each candidate operation plan, a raw score is calculated based on a predefined mathematical function or lookup table method. For example, the operation efficiency score S_efficiency = k / T (where T is the completion time and k is the normalization coefficient); the constraint satisfaction score S_constraint = M (where M is the minimum safety margin); and the operation energy consumption score S_energy = E_max / E (where E is the estimated energy consumption and E_max is the reference maximum energy consumption).

[0130] Weighting: Each scoring dimension is assigned a positive weight coefficient (e.g., W_constraint, W_efficiency, W_energy). The weights reflect the relative importance of different performance metrics in the current task or general configuration. Weight coefficients can be pre-set based on domain experience or configured during system deployment. For example, in precision assembly tasks, "constraint satisfaction" might have a higher weight; in material handling tasks, "operational efficiency" might have a higher weight. The sum of all weight coefficients is typically 1, but this is not mandatory.

[0131] Overall score calculation: For each candidate solution, multiply its original scores in each dimension by the corresponding dimension's weight, and then sum them to obtain the overall score of the solution. The formula is expressed as: Overall score = W_constraint * S_constraint + W_efficiency * S_efficiency + W_energy * S_energy.

[0132] Scheme Comparison and Determination: Calculate the overall score of all candidate operation schemes. The scheme with the highest overall score is determined as the "optimal operation scheme".

[0133] Example explanation:

[0134] Taking "assisting patients in performing passive upper limb lifting exercises" in the rehabilitation nursing scenario as an example, after analysis in step 101, the task includes sub-tasks such as "grasping the patient's forearm" and "smoothly lifting to the target angle", and is associated with quantitative constraints such as "maximum lifting force ≤ 30N" and "joint angular velocity ≤ 10 degrees / s".

[0135] After simulation verification in step 102, it is assumed that two feasible solutions are obtained:

[0136] Option A: Employs a faster lifting speed (8 degrees / s), resulting in a direct trajectory, short completion time, and moderate estimated energy consumption. Simulation shows a maximum lifting force of 25 N.

[0137] Option B: A slower lifting speed (5 degrees / s) is used, and the motion trajectory includes a slow start and a slow stop segment. The completion time is longer, and the estimated energy consumption is lower. Simulation shows that the maximum lifting force is 18N.

[0138] The preset scoring rules utilize three dimensions, with the following weights: constraint satisfaction (W1=0.5), operational efficiency (W2=0.3), and operational energy consumption (W3=0.2). The scoring function is simplified as follows:

[0139] Constraint satisfaction score (S1) = (Constraint threshold - Simulation maximum force value) = Margin (N). Scheme A has a margin of 5N, so it gets 5 points; Scheme B has a margin of 12N, so it gets 12 points.

[0140] Operational efficiency score (S2) = 1 / completion time (s). For example, if A takes 3 seconds, it gets 0.333 points; if B takes 5 seconds, it gets 0.2 points.

[0141] Operating energy consumption score (S3) = 1 / Estimated energy consumption (relative unit). For example, if A consumes 1.2 units of energy, it will receive 0.833 points; if B consumes 1.0 units of energy, it will receive 1.0 points.

[0142] Calculate the overall score:

[0143] Total score for Plan A = (5 * 0.5) + (0.333 * 0.3) + (0.833 * 0.2) = 2.5 + 0.1 + 0.1666 ≈ 2.7666

[0144] Total score for Plan B = (12 * 0.5) + (0.2 * 0.3) + (1.0 * 0.2) = 6.0 + 0.06 + 0.2 = 6.26

[0145] Option B's overall score is significantly higher than Option A's. Although Option A is faster, Option B performs better in terms of constraint satisfaction (greater safety margin) and energy consumption, and its larger safety margin is given a higher weight (0.5). Therefore, based on this selection rule, the system will determine Option B as the "optimal operation plan." This selection reflects the priority given to operational safety and compliance (reflected by a high constraint satisfaction weight) in the rehabilitation nursing scenario.

[0146] Optionally, after the step of simulating and verifying multiple candidate operation schemes generated based on the structured task sequence, the method further includes: step 1022, in response to all candidate operation schemes failing verification, generating a feedback signal containing specific physical constraint conflict information, and triggering adjustments to the structured task sequence based on the feedback signal. Specifically:

[0147] 1. Triggering condition: Response to all candidate operation schemes failing verification.

[0148] Following the output of step 102 (physical simulation verification), "All candidate operation schemes failed verification" means that, based on the currently received "structured task sequence", the system explored all the candidate operation schemes it generated in the physical simulation environment (e.g., through different combinations of force parameters and trajectory parameters), but each scheme violated at least one "preset physical feasibility judgment rule" during the simulation execution process (such as object deformation exceeding the limit, grasping instability, operation force exceeding the constraint threshold, or collision occurring).

[0149] 2. Core processing: Generate feedback signals containing specific physical constraint conflict information.

[0150] The feedback signal typically needs to include the following elements:

[0151] Conflict identification: Clearly identify which one or more "subtasks" in the "structured task sequence" caused the physical feasibility conflict.

[0152] Conflict rules: Indicate which one or more "physical feasibility judgment rules" were violated (e.g., "grabbing stability coefficient is below the threshold" or "the trajectory of motion collides with environmental obstacles").

[0153] Conflict Quantification Description: Provides the key physical quantities and their values ​​that cause the conflict. For example, if the conflict fails due to unstable gripping, it can report, "Under the set maximum gripping force of 5N, the simulation-calculated gripping stability coefficient is 0.6, which is lower than the preset threshold of 0.8"; if the conflict fails due to excessive deformation, it can report, "The simulation measured the maximum deformation rate of the object to be 7%, which exceeds the threshold of 5%." It can also link back the "quantified physical constraint parameters" associated with the subtask that triggered the conflict and point out the contradiction between them and physical reality. For example, "When adhering to the subtask constraint 'maximum gripping force ≤ 5N', stable gripping cannot be achieved."

[0154] Correction suggestions: Based on simulation analysis, potential directions for parameter adjustment can be provided. For example, "It is recommended to relax the 'maximum clamping force' constraint to ≥6N", or "It is recommended to adjust the target pose to avoid vulnerable areas".

[0155] 3. Feedback Execution: Adjustments to the structured task sequence are triggered based on feedback signals.

[0156] The content and method of adjustment: Targeted revisions based on feedback. Adjustments may occur at two levels:

[0157] Adjusting quantifiable physical constraint parameters: Based on conflict information, modify the constraint values ​​associated with relevant subtasks. For example, if feedback indicates that a stable gripping cannot be achieved under the current force constraint, the constraint threshold of "maximum gripping force" may be appropriately increased (within the overall safety boundary); if feedback indicates that a collision will occur at the current motion speed, the constraint of "maximum motion speed" may be decreased.

[0158] Adjustments to task logic or sequence: In some cases, simply adjusting parameters may not be sufficient to solve the problem, and it may be necessary to modify the task decomposition method. For example, if feedback indicates that approaching from a single direction will inevitably result in a collision, the planning module may adjust the subtask sequence, adding a "relocation" or "detour approach" subtask.

[0159] Closed-loop process formation: This "trigger adjustment" action transforms the entire control flow from a unidirectional chain of step 101 → step 102 into a closed loop including the reverse feedback of step 1022. After adjustment, a new "structured task sequence" is generated, which will then re-enter step 102 for physical pre-performance verification. This iterative process can be repeated until a feasible solution is found or a preset iteration limit is reached.

[0160] Example explanation:

[0161] Take the example of "retrieving a hardcover book from the middle shelf of a crowded bookshelf" in a home service scenario. The initial structured task sequence generated in step 101 includes the subtask of "directly and horizontally extracting the book", and may be associated with the constraint of "pulling force ≤ 10N" (to prevent damage to the pages).

[0162] In step 102, the system generates multiple candidate extraction schemes (different gripping points and pull-out speed curves) based on this sequence. The physical simulation environment loads an accurate model of the bookshelf, the books, and other books in the surrounding area.

[0163] Simulation results: All candidate solutions failed in the simulation. Analysis revealed that, under the constraint of "pull force ≤ 10N," the static friction between the book and adjacent books exceeded the available pull force, preventing the book from moving (violating the implicit feasibility of "task completion," or categorized as "operational failure" under specific constraints). Furthermore, if a force greater than 10N was forcibly applied, the simulation showed that the stress at the page joints exceeded the limit (violating the "object deformation rate" rule).

[0164] Step 1022 is executed as follows:

[0165] Generate feedback signal: The system generates feedback information, which may include: "Conflict subtask: 'Horizontal extraction'; Rule violation: Unable to overcome static friction to start movement under the constraint 'pull force ≤ 10N', and page stress exceeds limit if the force value is increased; Key physical quantities: The current estimated minimum starting friction force is 12N, and the maximum allowable pull force corresponding to the page safety stress is 15N; Conflict description: The current constraint (10N) is lower than the minimum force required for movement (12N); Potential suggestion: The operation method needs to be re-evaluated or the force constraint relaxed."

[0166] Triggering Adjustment: This feedback signal is sent to the semantic planning module. After parsing the conflict information, this module may take one or more of the following adjustments:

[0167] Adjust the constraint: Modify the "pull force" constraint to a range, such as "≤ 15N" (within the safe range of the page), but this operation must ensure that it complies with the higher-level semantic "gentle" reassessment.

[0168] Adjusting task logic: The planning module may decide to modify the task sequence. For example, adding a pre-task "gently push the adjacent books on both sides outward to free up space" before performing "horizontal extraction" may reduce the required pulling force to below 10N. Alternatively, the action could be changed to "first tilt the book to one side, then pull it out".

[0169] Closed-loop iteration: The new, adjusted sequence of structured tasks (including the "open the neighboring book" subtask and corresponding new constraints) is submitted again to step 102 for a new round of physical pre-testing and verification. This process continues until a physically feasible operation plan is found.

[0170] Step 103: Based on the optimal operation scheme, generate basic control instructions for driving the robot.

[0171] Step 103 transforms the abstract operational plan, output from the preceding steps (physical pre-simulation and optimization) and verified for physical feasibility and optimized through comprehensive selection, into a complete set of specific, time-sequential, low-level motion and force control command sequences that can directly drive the robot's joints and end effectors. Specifically:

[0172] 1. Input: Based on the optimal operation plan. The plan may include:

[0173] Motion trajectory parameters: the desired sequence of path points for the robot's end effector or joints in the task space or joint space, the motion speed, acceleration, jerk, etc., of each path segment in time planning.

[0174] Force / torque control parameters: In sub-task phases that require force control interaction (such as grab, insert, wipe), specify the desired force / torque value, force control direction, impedance parameter, or compliance target.

[0175] Operation pose and timing: The precise start and end poses (position and attitude) for each subtask, as well as the time synchronization relationship between subtasks and path points.

[0176] Subtask logic and triggering conditions: The solution may contain embedded logical order of subtask execution, or triggering conditions based on specific states (such as contact detection).

[0177] 2. Core processing: Instruction generation.

[0178] The essence of processing behavior: "Generation" here means "calculated" or "mapped." This is a computational process whose core task is to convert the high-level, task-oriented "optimal operation plan" into low-level, robot-body-oriented "basic control commands." This conversion typically involves kinematic and dynamic calculations as well as the setting of control laws.

[0179] The generation process may involve the following technical aspects:

[0180] Kinematic solution: The motion trajectory of the end effector described in the scheme is decomposed into the sequence of angles, angular velocities, and angular accelerations of each joint of the robot through inverse kinematics calculation.

[0181] Control law parameterization: The force control target specified in the scheme (such as "maintaining a contact force of 10N") is converted into the set point or impedance / admittance control parameters of the specific force / torque control loop.

[0182] Command sequence arrangement: Based on the timing requirements of the scheme, the calculated joint trajectory points, control mode commands (position control / force control switching), force control parameters, etc., are arranged into a strictly timed command flow.

[0183] Low-level interface adaptation: The generated instruction format must match the interface protocol of the specific robot's low-level controller (such as a servo driver). This may involve data format conversion, communication protocol encapsulation, etc.

[0184] 3. Output: Basic control commands used to drive the robot.

[0185] The nature and function of the output: The "basic control instructions" are the final output of this step, and are a set of signals that directly act on the robot's drive system. The term "basic" emphasizes that they are the reference instructions before real-time fine-tuning in the subsequent execution phase (step 104). They constitute the "planned" or "nominal" drive signals for the robot's motion.

[0186] Typical form of instruction content: This instruction is usually a data packet sent periodically to the robot controller, and may include:

[0187] Joint space commands: Target position, target velocity, and target torque / current for each joint in each control cycle.

[0188] Task space commands: The target pose and target force / torque of the end effector are automatically decomposed into joint commands by the robot controller.

[0189] Control mode instructions: Specify the position control, speed control, force control, or hybrid control mode that the robot or a specific joint should be in during a specific time period.

[0190] State trigger command: Defines the triggering conditions for switching from the current command segment to the next command segment (such as time arrival, position arrival, contact force reaching the threshold).

[0191] Specific examples are explained below:

[0192] Taking the action of "assisting a patient from a sitting to a standing position" in rehabilitation nursing as an example, steps 101 and 102 ultimately determine an "optimal operation plan": the robot needs to support the patient's elbow from the side in a specific posture, and assist the patient to stand up along a specific arc trajectory with a steady speed and acceleration (e.g., an average vertical speed of 0.1 m / s and a maximum acceleration of 0.2 m / s²), while controlling the assisting force between 15N and 20N throughout the process, and ensuring that the direction of the force is always approximately perpendicular to the patient's back. In step 103, the system generates "basic control commands" based on this "optimal operation plan":

[0193] Trajectory conversion: The circular trajectory of the end effector (support pad) described in the scheme is solved by robot inverse kinematics into the time-varying angle sequence of the robot's six joints {a1(t), a2(t), ..., a6(t)}, and the corresponding joint velocity and acceleration sequence is calculated.

[0194] Force control parameter settings: During the core phase of assisted standing, the robot switches to force control mode. The force control target is set as follows: maintain a force of 17.5N (the average value in the scheme) in the Z-axis direction of the tool coordinate system, and set appropriate impedance parameters (such as stiffness and damping) to make the robot exhibit a certain degree of compliance in this direction, while maintaining position tracking in other directions.

[0195] Command orchestration: A timing command stream is generated. For the first 2 seconds, the robot operates in position control mode, driving each joint to move to the initial support pose {a1(0), ..., a6(0)}. Subsequently, after the trigger condition "position reached," the robot switches to force control mode and begins executing the calculated joint trajectory sequence, while simultaneously injecting the force control target parameters. The entire standing process is planned for 5 seconds; therefore, the trajectory sequence corresponds to 5 seconds of joint motion data. After 5 seconds, the command stream can instruct the robot to maintain the final pose or switch back to position control to prepare for the next action.

[0196] Output: The final generated "basic control instructions" is the complete drive data package mentioned above, which includes timing, control mode switching, target trajectories for each joint (in position control), and target force parameters (in force control). This instruction stream is sent to the robot's underlying controller, driving the robot to begin performing assisted standing actions.

[0197] Step 104: During the process of the robot executing operations according to the basic control instructions, the data sets of tactile, visual and force sensors are collected in real time. The data sets are fused through the second model to generate fused perception features. The basic control instructions are adjusted in real time based on the fused perception features to obtain the final control instructions to drive the robot.

[0198] The purpose of this step is to initiate a parallel process—one that involves real-time interaction with the physical world and online correction based on multimodal perception feedback—while the robot is actually performing physical operations according to the "basic control instructions" generated in step 103. Specifically:

[0199] 1. Data Source: Real-time acquisition of datasets from tactile, visual, and force sensors. The system utilizes three sensor modalities with different physical properties in parallel to obtain a comprehensive and complementary real-time description of operation and environment.

[0200] Tactile sensors: These are typically arrays or distributed sensors installed on the end effector of a robot (such as fingertips or palms) to measure pressure distribution, contact area, texture information, and slippage trends when in contact with the manipulated object. Their data reflects the local geometry and mechanical properties of the contact, and the sampling frequency is usually high (e.g., 100Hz or higher).

[0201] Visual sensors: typically cameras, which can be configured as monocular, binocular, or depth cameras, are used to acquire two-dimensional images or three-dimensional point clouds of the operating scene. Visual data provides global spatial context, object recognition, pose estimation, and a wide range of motion information, with a relatively fixed frame rate (e.g., 30fps).

[0202] Force sensors: These are typically six-dimensional force / torque sensors mounted on the robot's wrist or joints, used to accurately measure the resultant force and torque of the interaction between the end effector and the environment. Force data directly reflects the dynamic state of the interaction and is sampled at a very high frequency (e.g., 1000Hz).

[0203] Dataset: Here, "dataset" refers to the collection of raw or preliminarily processed data streams acquired synchronously or after time alignment from the three sensors mentioned above within a continuous time window. This collection is the direct input for multimodal fusion processing.

[0204] 2. Core processing: The dataset is fused using a second model to generate fused perceptual features.

[0205] The role of the second model: In this disclosure, the "second model" can be referred to as the "multimodal sensing fusion model". Its core function is to solve the problem of the inherent correlation and unified representation of multi-source heterogeneous sensor data.

[0206] Fusion processing flow:

[0207] 2-1. Preprocessing and Alignment: Timestamp synchronization, spatial coordinate unification, and data standardization (normalization) are performed on tactile, visual, and force data from different sampling frequencies and coordinate systems to form a time-aligned multimodal data stream.

[0208] 2-2. Feature-level Fusion: The second model receives the aligned multimodal data stream. Through its internal structure (e.g., designing independent encoding subnetworks for each modality and then exchanging information through cross-attention layers), the model learns and uncovers deep correlations between different modal data. For example, the model can learn the correspondence between "a specific deformation pattern of an object in a visual image," "the uneven distribution of pressure on a tactile array," and "a specific fluctuation pattern of wrist force sensor readings." This correlation is a manifestation of "semantic-physical" correlation at the real-time perception level.

[0209] 2-3. Output: The model's output is a unified "fusion-sensory feature." This feature is a vector representation with fixed or variable dimensions, and its key attribute is that it simultaneously encodes semantic-level information (such as "the currently grasped object is a glass" or "a wiping action is in progress") and physical state information (such as "the current slip of the glass relative to the clamp is 0.5 mm", "the real-time interactive force in the Y-axis component is 2.3 N", and "the pressure at the center of the contact area is 15 kPa"). This feature provides a rich and comprehensive description of the environmental state for subsequent instruction adjustments.

[0210] 4. Control Adjustment: Based on the fused perception features, the basic control commands are adjusted in real time to obtain the final control commands. The purpose is to use the deviation between the real-time state reflected in the "fused perception features" and the expected state (implied in the basic commands) or the predicted potential risks to correct the "basic control commands" generated in step 103 and being executed online.

[0211] Optionally, the basic control commands are adjusted in real time based on the fused sensing features, including: inputting the optimal operation scheme and the fused sensing features into the slow control network to generate basic control commands; inputting the real-time updated fused sensing features into the fast control network to generate control parameter fine-tuning commands; and synthesizing the basic control commands and the control parameter fine-tuning commands to obtain the final control commands.

[0212] Optionally, the slow control network operates at a first frequency to output basic control commands containing joint motion trajectories and basic force control parameters; the fast control network operates at a second frequency higher than the first frequency to output control parameter fine-tuning commands for fine-tuning the force control parameters and joint angles; synthesis is achieved by superimposing the control parameter fine-tuning commands onto the basic control commands.

[0213] The "fast and slow dual-network" collaborative control architecture is the key technical means for this solution to address the imbalance between semantic planning and physical real-time adaptation.

[0214] 1. Overall architecture and division of labor logic.

[0215] The core of this adjustment mechanism lies in functional decoupling. The slow control network and the fast control network share the same set of real-time environmental perception information (i.e., "fused perception features"), but receive different auxiliary inputs and assume distinctly different responsibilities:

[0216] Slow control network: Responsible for decisions related to the global semantics of the task, long-term trajectory, and steady-state control objectives. It focuses on a longer time scale and emphasizes the macroscopic intent of achieving the "optimal operation plan".

[0217] Fast control network: Specifically responsible for handling compensation related to transient environmental disturbances, high-frequency model errors, and dynamic interactive states. It focuses on an extremely short timescale, similar to a high-speed "reflection arc," and is specifically designed to suppress high-frequency interference.

[0218] Ultimately, both are unified and output as a "final control command" through a single instruction synthesis stage. This design, which decouples the control system based on time scale and functional concerns, improves the stability, real-time performance, and interpretability of the entire control system.

[0219] 2. Network input, functions, and operating characteristics.

[0220] 2-1. Slow Control Network:

[0221] Input: Receives two key inputs. The first is the "optimal operation plan", which includes the physically verified global motion trajectory, force control target, and task logic; the second is the "fusion perception features", which provides a unified understanding of the current environment and object state.

[0222] Functionality and Operation: This network operates at a relatively low initial frequency (e.g., a cycle time ≥ 1 Hz, or ≤ 1000 ms). Its core function is to perform calculations or inferences based on a global task framework defined by the "optimal operation scheme" and combined with the current environmental state reflected by "fused perception features" (such as the approximate pose of objects and the overall layout of the scene) to generate or update the "basic control commands" used to drive the robot. For example, when perception features indicate a small systematic deviation in the initial position of an object, the slow network can recalculate the offset of the entire motion trajectory; when switching task phases, the slow network is responsible for switching control modes and setting the corresponding basic parameters.

[0223] Output: Its output is the "basic control command", which specifically includes the joint motion trajectory (i.e., a series of target positions, velocities, and acceleration sequences within the joint space) and basic force control parameters (i.e., target values ​​for force / torque control, impedance, or admittance parameters). This output constitutes the macroscopic "baseline" or "reference trajectory" for robot execution.

[0224] 2-2. Quickly control the network:

[0225] Input: It only accepts "real-time updated fused sensing features" as the sole input. This allows it to focus on processing the highest frequency sensing information stream, unaffected by direct changes in the details of global task planning.

[0226] Functionality and Operation: This network operates at a significantly higher second frequency than the slow network (e.g., a cycle time ≤ 10ms, i.e., a frequency ≥ 100Hz). This high-frequency setting is to meet the requirements of dynamic disturbance suppression. Many physical disturbances (such as mechanical vibrations, high-frequency fluctuations in contact forces) have frequency components above tens of hertz. According to the sampling theorem and control theory, to effectively compensate for these disturbances, the update frequency of the control system needs to be several times higher than the dominant frequency component of the disturbance. The fast network is trained to directly detect errors or disturbances requiring instantaneous compensation (such as "small slip velocity" or "high-frequency force jitter") from the fused features and immediately output the corresponding "control parameter fine-tuning instructions".

[0227] Output: Its output is a "control parameter fine-tuning command," specifically referring to incremental corrections, including fine-tuning of joint angles (D_a) and force control parameters (D_F). Its amplitude is usually limited, and its purpose is to compensate for deviations instantaneously, rather than to reprogram.

[0228] 3. Quantitative definition and engineering significance of operating frequency

[0229] First frequency (slow network): Its setting needs to balance computational complexity and response time. Too low a frequency may lead to sluggish response to moderate changes; too high a frequency will increase unnecessary computational load. It is usually on the order of 1 Hz to tens of Hz to ensure sufficient time for task-level decisions.

[0230] The second frequency (fast network): must be significantly higher than the first frequency. Its setting needs to match or be close to the control frequency of the robot's underlying servo drive to ensure effective processing of high-frequency disturbance information captured by the sensor and to attenuate disturbances at frequencies of tens of hertz or even higher. It is typically in the hundreds of Hz range.

[0231] 4. Instruction synthesis method

[0232] "Synthesis" refers to merging the "basic control commands" of the slow network and the "control parameter fine-tuning commands" of the fast network into a single "final control command." A typical and preferred synthesis method is achieved through linear superposition. In each unified control command output cycle (usually synchronized with the second frequency of the fast network), the system performs the following operations:

[0233] Final joint target position / angle = Base joint target position / angle + Joint angle fine-tuning amount;

[0234] Final force / torque target value = basic force / torque target value + force control parameter fine-tuning amount;

[0235] Comprehensive example explanation:

[0236] Take, for example, "robot-assisted drinking water for patients with hand tremors" in rehabilitation nursing. Step 102, the "optimal operation plan," outlines a smooth circular trajectory from the table to the patient's mouth. The control system operates at 200Hz (second frequency), while the slow network operates at 10Hz (first frequency).

[0237] Slow network (10Hz): Every 100ms, based on fused features (global position of the cup, position of the patient's head), outputs a smooth joint trajectory fragment and basic grip force (e.g., 12N) for the next 100ms.

[0238] Fast network (200Hz): Every 5ms, it extracts high-frequency shaking caused by hand tremor (such as 8-12Hz jitter) from the fused features and immediately calculates the joint angle fine adjustment amount to counteract the jitter (such as D_a1 = +0.02 degrees, D_a2 = -0.01 degrees).

[0239] Command synthesis (200Hz): In every 5ms cycle, the current basic joint angles of the slow network (e.g., a1_slow=30.0 degrees, a2_slow=45.0 degrees) are superimposed with the fine-tuning amount of the fast network to obtain the final command (a1_final=30.02 degrees, a2_final=44.99 degrees) and sent to the driver.

[0240] The robot's end effector follows a smooth trajectory updated at 10Hz on a macroscopic scale, while simultaneously superimposing active anti-vibration motion at a frequency of 200Hz to counteract vibrations and keep the cup stable. The synthesis of force control commands follows the same principle. This mechanism is also applicable to other scenarios requiring suppression of high-frequency disturbances, such as in assisted lower limb training, where a fast network can actively dampen the micro-vibrations of the pedal caused by muscle tremors.

[0241] Optionally, the second model is a multimodal Transformer model based on a cross-attention mechanism; the fused perception feature is a feature representation that integrates multimodal information, and the feature representation simultaneously encodes the semantic category information and physical state information of the perceived object.

[0242] The second model's architecture is defined as a multimodal Transformer model based on a cross-attention mechanism. Its input is preprocessed, time-aligned real-time data streams from tactile, visual, and force sensors. Its core function is to replace simple data stitching or early fusion, achieving deep fusion at the feature or representation level.

[0243] How cross-attention works: This is key to achieving effective multimodal fusion. In this architecture, each modality typically has its own independent encoder branch. Cross-attention allows a sequence from one modality (as a "query") to actively "attention" to a sequence from another modality (as a "key" and a "value"). For example, a query vector of visual features can compute association weights with all locations of tactile features, thus selectively integrating tactile information most relevant to the current visual content. This operation is bidirectional or cyclic, allowing visual features to absorb tactile and force contexts, and vice versa. Through this mechanism, the model can learn cross-modal physical laws such as "specific visual texture patterns (e.g., smooth surfaces) are associated with low-friction tactile signals and slippery force fluctuation patterns."

[0244] 2. The property of fusing perceptual features: simultaneously encoding semantic category information and physical state information.

[0245] Function of feature representation: The "fusion-perceptual feature" is a holistic abstraction and summary of the current multimodal input by the second model, and is a vector with fixed or variable dimensions. It serves as the direct input for subsequent control networks (especially fast control networks) to make real-time decisions.

[0246] The meaning of dual coding:

[0247] Semantic category information: This refers to the high-level understanding of the perceived object and the intention of the operation contained in this feature. For example, in certain dimensions or subspaces of the feature space, activation patterns can distinguish whether the current object of operation is a "ceramic cup," a "soft fruit," or a "human arm"; and can distinguish whether the current action is "grabbing," "wiping," or "pulling." This information connects to the semantic planning in step 101, providing the context of "what is the task" for control adjustments.

[0248] Physical state information: This feature includes a real-time, quantified description of the physical properties of the object and its environment. For example, in other dimensions of the feature, its values ​​can characterize the object's "real-time deformation level," "slip velocity relative to the end effector," "vector and magnitude of the current contact force," and "local stiffness of the environmental surface," etc. This information provides precise feedback on "the current physical condition."

[0249] The implementation of "simultaneous encoding": In the cross-attention-based multimodal Transformer, the model, through training, spontaneously carries different types of information in different parts of the unified feature representation of the output, or through complex nonlinear combinations. This joint representation enables the control system to simultaneously receive semantic warnings of "handling fragile objects" and physical alarms of "current gripping force approaching the safety threshold" based on a unified feature vector, thereby making coordinated adjustment decisions.

[0250] Specific examples are explained below:

[0251] Take the scenario of "washing and handling a glass with detergent on it" in a household service as an example. During step 104:

[0252] Multimodal input:

[0253] Visual: The camera captures the outline of the glass, the wet reflections on its surface, and the highlights.

[0254] Tactile sensation: The fingertip tactile array measures the pressure distribution in the area in contact with the cup wall, which may be uneven and discontinuous due to foam, and detects weak vibrations.

[0255] Force sensing: Wrist force sensors measure the resultant force and torque of the gripping force, which may show slight torque fluctuations around the gripping axis.

[0256] Fusion processing of the second model (multimodal Transformer):

[0257] Inside the model, the feature sequences extracted by the visual branch (containing information such as "smooth surface" and "cylindrical shape") are used to query the feature sequences of the tactile branch through a cross-attention layer. The model has learned from the training data that the visual pattern of "wet reflection" needs to be associated with the tactile pattern of "dispersed and fluctuating pressure distribution" and the force pattern of "fluctuation of tangential force", both pointing to the physical state of "slippery surface".

[0258] At the same time, tactile and force features are also used as queries to focus on visual features in order to confirm the identity of the object (the glass) and its posture.

[0259] After multiple layers of cross-attention interaction, the model finally outputs a fused perceptual feature vector. The activation pattern of this vector in one subspace represents the semantic information "object: slippery glass; task: gripping". In another subspace, its value precisely represents the physical state information "currently estimated static friction coefficient: 0.1 (low)", "real-time slippage trend index: 0.15 (caution required)", and "current gripping force: 8N".

[0260] Feature Usage: This fused perceptual feature is fed into a fast control network. After parsing the feature, the network can simultaneously understand "the object to be grasped is slippery and fragile" (semantic) and quantify the perception of "high risk of slippage" (physical state). Therefore, it may instantaneously generate two sets of fine-tuning instructions: one is to slightly and smoothly increase the gripping force (e.g., +2N) to compensate for the decrease in the coefficient of friction; the other is to adjust the grasping posture to increase the enclosure. These adjustments are based on a unified understanding of the semantic (fragility) and physical (slip resistance) states, ensuring the reliability of the operation.

[0261] It should be noted that the embodiments described in the above-described method and apparatus embodiments of this disclosure are merely exemplary. Those skilled in the art, after understanding the concept of this disclosure, will realize that the embodiments described above are merely exemplary implementation methods used to explain the principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Specifically, the specific tools, means, components, and their models (such as specific large language model names, physics engine software, sensor models, and neural network structure names), specific parameters, thresholds, values, and their ranges (such as deformation rate thresholds, stability coefficient thresholds, specific Hertz numbers of operating frequencies, and the number of entries in the training dataset) recorded in the embodiments of this disclosure are all illustrative examples provided to more clearly illustrate the technical solutions and working principles of this disclosure. Those skilled in the art should understand that, based on the core concept disclosed in this disclosure—that is, achieving intelligent control of robots through a closed-loop architecture of semantic planning, physical pre-simulation, multimodal fusion, and hierarchical collaborative control—various modifications, substitutions, combinations, or equivalent changes can be made to the above-described exemplary specific implementation means. For example, models with different architectures or training methods can achieve the same semantic understanding and planning functions; different physical simulation platforms can be used; different quantization rules and scoring dimensions can be defined and applied; different network operating frequency parameters can be set; and different data fusion network structures or instruction synthesis algorithms can be used. As long as such modifications or changes do not depart from the substantive technical concept defined in this disclosure, they still fall within the scope of the technical solutions claimed in this disclosure.

[0262] Example 2

[0263] Corresponding to the aforementioned robot control method embodiments, this disclosure also provides embodiments of a robot control system.

[0264] Figure 2 A schematic diagram of a control system for a robot provided as an exemplary embodiment of this disclosure, the control system comprising:

[0265] The semantic planning module 21 is used to receive natural language instructions, parse and plan the natural language instructions through the first model, and output a structured task sequence. The first model is configured as a semantic understanding and task planning model. The structured task sequence contains at least one sub-task, and each sub-task is associated with at least one quantized physical constraint parameter.

[0266] The pre-simulation optimization module 22 is used to receive the structured task sequence and, in the physical simulation environment, perform simulation verification on multiple candidate operation schemes generated based on the structured task sequence according to the preset physical feasibility judgment rules; in response to the existence of a candidate operation scheme that has passed the verification, the optimal operation scheme is determined from the candidate operation schemes that have passed the verification according to the preset scheme selection rules.

[0267] Control module 23 is used to generate basic control commands for driving the robot based on the optimal operation scheme;

[0268] The perception adjustment module 24 is used to collect data sets from tactile, visual and force sensors in real time during the process of the robot performing operations according to basic control instructions. The second model is used to fuse the data sets to generate fused perception features. Based on the fused perception features, the basic control instructions are adjusted in real time to obtain the final control instructions to drive the robot.

[0269] Optionally, the pre-performance optimization module 22 is also used for:

[0270] If all candidate operation schemes fail to pass verification, a feedback signal containing specific physical constraint conflict information is generated, and an adjustment to the structured task sequence is triggered based on the feedback signal.

[0271] Optionally, the preset physical feasibility judgment rules include at least one of the following: the deformation rate of the manipulated object does not exceed a first threshold, the grasping stability coefficient is not lower than a second threshold, the operating force value does not exceed the corresponding physical constraint threshold, and the motion trajectory is collision-free.

[0272] Optionally, the preset scheme selection rule is a multi-objective weighted scoring rule, which is used to score the candidate operation schemes that have passed the verification; the scoring dimensions include at least one of the following: constraint satisfaction, operation efficiency, and operation energy consumption.

[0273] Optionally, the second model is a multimodal Transformer model based on a cross-attention mechanism; the fused perception feature is a feature representation that integrates multimodal information, and the feature representation simultaneously encodes the semantic category information and physical state information of the perceived object.

[0274] Optionally, the sensing adjustment module 24 is specifically used for:

[0275] The optimal operation plan and fused sensing features are input into the slow control network to generate basic control commands;

[0276] The real-time updated fused sensing features are input into the fast control network to generate control parameter fine-tuning instructions;

[0277] The basic control command and the control parameter fine-tuning command are combined to obtain the final control command.

[0278] Optionally, the slow control network operates at a first frequency to output basic control commands containing joint motion trajectories and basic force control parameters;

[0279] The fast control network operates at a second frequency higher than the first frequency, used to output fine-tuning commands for control parameters that fine-tune force control parameters and joint angles;

[0280] Synthesis is achieved by superimposing control parameter fine-tuning instructions onto basic control instructions.

[0281] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0282] Example 3

[0283] Figure 3 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the robot control method described in any of the above embodiments. Figure 3 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0284] like Figure 3 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0285] Bus 93 includes a data bus, an address bus, and a control bus.

[0286] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0287] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0288] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the robot control method provided in any of the above embodiments.

[0289] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0290] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0291] Example 4

[0292] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot control method provided in any of the above embodiments.

[0293] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0294] Example 5

[0295] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the robot control method described in any of the preceding embodiments.

[0296] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0297] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for controlling a robot, characterized in that, The control method includes: The system receives natural language instructions, parses and plans the natural language instructions through a first model, and outputs a structured task sequence. The first model is configured as a semantic understanding and task planning model. The structured task sequence contains at least one sub-task, and each sub-task is associated with at least one quantized physical constraint parameter. The system receives the structured task sequence and performs simulation verification on multiple candidate operation schemes generated based on the structured task sequence in a physical simulation environment according to a preset physical feasibility judgment rule. If there is a candidate operation scheme that passes the verification, the system determines the optimal operation scheme from the candidate operation schemes that pass the verification according to a preset scheme optimization rule. Based on the optimal operation scheme, generate basic control commands for driving the robot; During the process of the robot executing operations according to the basic control instructions, data sets from tactile, visual, and force sensors are collected in real time. The data sets are then fused using a second model to generate fused perception features. Based on these fused perception features, the basic control instructions are adjusted in real time to obtain the final control instructions to drive the robot.

2. The control method according to claim 1, characterized in that, The step of simulating and verifying multiple candidate operation schemes generated based on the structured task sequence further includes: If all of the candidate operation schemes fail to pass the verification, a feedback signal containing specific physical constraint conflict information is generated, and an adjustment to the structured task sequence is triggered based on the feedback signal.

3. The control method according to claim 1, characterized in that, The preset physical feasibility determination rules include at least one of the following: the deformation rate of the manipulated object does not exceed the first threshold, the grasping stability coefficient is not lower than the second threshold, the operating force value does not exceed the corresponding physical constraint threshold, and the motion trajectory has no collision.

4. The control method according to claim 1, characterized in that, The preset scheme selection rule is a multi-objective weighted scoring rule, which is used to score the candidate operation schemes that have passed the verification; the scoring dimensions include at least one of the following: constraint satisfaction, operation efficiency, and operation energy consumption.

5. The control method according to claim 1, characterized in that, The second model is a multimodal Transformer model based on a cross-attention mechanism; the fused perception feature is a feature representation that integrates multimodal information, and the feature representation simultaneously encodes the semantic category information and physical state information of the perceived object.

6. The control method according to claim 1, characterized in that, The real-time adjustment of the basic control commands based on the fused sensing features includes: The optimal operation scheme and the fused sensing features are input into the slow control network to generate the basic control commands; The fused sensing features, updated in real time, are input into a fast control network to generate control parameter fine-tuning instructions; The basic control command and the control parameter fine-tuning command are combined to obtain the final control command.

7. The control method according to claim 6, characterized in that: The slow control network operates at a first frequency to output the basic control commands containing joint motion trajectories and basic force control parameters. The fast control network operates at a second frequency higher than the first frequency, and is used to output the control parameter fine-tuning command for fine-tuning the force control parameters and joint angles; The synthesis is achieved by superimposing the control parameter fine-tuning instructions onto the basic control instructions.

8. A control system for a robot, characterized in that, The control system includes: A semantic planning module is used to receive natural language instructions, parse and plan the natural language instructions through a first model, and output a structured task sequence; the first model is configured as a semantic understanding and task planning model, and the structured task sequence contains at least one sub-task, and each sub-task is associated with at least one quantized physical constraint parameter; The pre-simulation optimization module is used to receive the structured task sequence, and perform simulation verification on multiple candidate operation schemes generated based on the structured task sequence in a physical simulation environment according to a preset physical feasibility judgment rule; in response to the existence of a candidate operation scheme that has passed the verification, the optimal operation scheme is determined from the candidate operation schemes that have passed the verification according to a preset scheme selection rule. The control module is used to generate basic control commands for driving the robot based on the optimal operation scheme. The perception adjustment module is used to collect data sets from tactile, visual, and force sensors in real time during the process of the robot performing operations according to the basic control instructions. The second model is used to fuse the dataset to generate fused perception features. Based on the fused perception features, the basic control instructions are adjusted in real time to obtain the final control instructions to drive the robot.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot control method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot control method as described in any one of claims 1 to 7.

Images