A method and system for task planning control of a legged robot

Abstract

The application provides a kind of foot type robot task planning control method and system, it is related to robot control technical field, the method includes: obtaining the task instruction and multi-source perception data of robot;Task instruction is extracted, semantic features are obtained, and structured environment representation is constructed according to multi-source perception data;Based on semantic features, retrieve from the preset historical task library, obtain the historical task experience fragment corresponding to task instruction;Through language model, according to historical task experience fragment, combined with structured environment representation, generate the hierarchical task plan of robot, and carry out feasibility and safety verification, obtain the target skill sequence of robot, and carry out skill arrangement, generate skill execution scheduling strategy, and drive robot to execute skill;At the same time, when robot executes skill, real-time closed loop correction is carried out to skill execution scheduling strategy according to the real-time sensing data of robot.The application improves the execution effect of foot type robot in complex long-term task.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically, to a task planning and control method and system for a legged robot. Background Technology

[0002] Currently, legged robots are widely used in inspection, warehousing, and rescue scenarios due to their high mobility in unstructured terrain. For complex scenarios, legged robots typically need to perform long-duration tasks involving multiple sub-steps, such as retrieving and delivering items, opening doors and moving objects, and inspection and retrieval. Therefore, a systematic design of the autonomous planning and mobility capabilities of legged robots is necessary to enable them to complete long-duration tasks across regions, using multiple tools, and involving multiple interactions in dynamic, unstructured environments.

[0003] In related technologies, language models are widely used in robot task planning due to their powerful semantic reasoning and task decomposition capabilities, enabling sub-task decomposition and action sequence generation through semantic reasoning. However, when legged robots perform long-duration tasks, the action sequences generated by language models lack clear prerequisites and effect descriptions. Therefore, they are difficult to align with the full-body control constraints, gait planning requirements, and strong motion-operation coupling characteristics of legged robots, resulting in poor planning feasibility for long-duration tasks and making it difficult to meet the application requirements of actual operational scenarios for long-duration tasks. Summary of the Invention

[0004] The problem addressed by this invention is how to improve the performance of legged robots for long-duration tasks.

[0005] To address the aforementioned problems, this invention provides a task planning and control method and system for a legged robot.

[0006] In a first aspect, the legged robot task planning and control method of the present invention includes:

[0007] Acquire task instructions and multi-source sensor data from the legged robot;

[0008] The task instructions are extracted to obtain their semantic features, and environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking are performed based on the multi-source perception data to construct a structured environment representation for the legged robot.

[0009] Based on the semantic features, a search is conducted from a preset historical task database to obtain historical task experience fragments corresponding to the task instruction.

[0010] Using a language model, based on the historical task experience fragments and combined with the structured environment representation, a hierarchical task plan for the legged robot is generated.

[0011] The feasibility and safety of the hierarchical task plan are verified to obtain the target skill sequence of the legged robot.

[0012] Skills are arranged according to the target skill sequence to generate a skill execution scheduling strategy for the legged robot;

[0013] The legged robot is driven to perform skills according to the skill execution scheduling strategy; at the same time, when the legged robot performs skills, the skill execution scheduling strategy is corrected in real time in a closed loop based on the real-time sensor data of the legged robot.

[0014] Optionally, acquiring the task instructions and multi-source perception data of the legged robot includes:

[0015] The visual perception data, laser point cloud data, inertial data, wrist perception data, and foot tactile perception data of the legged robot are acquired.

[0016] The visual perception data, the laser point cloud data, the inertial data, the wrist perception data, and the foot tactile perception data are used as the multi-source perception data, and the task instructions issued to the legged robot are obtained.

[0017] Optionally, the step of extracting the task instructions to obtain the semantic features of the task instructions includes:

[0018] The task instructions are preprocessed to obtain the preprocessed text of the task instructions;

[0019] The preprocessed text is analyzed using a semantic feature extraction model to determine the task objectives, operation objects, action requirements, and constraint information in the task instructions.

[0020] The operation object, the operation object, the action requirements, and the constraint information are vectorized and encoded to generate semantic feature vectors;

[0021] The semantic features of the task instruction are obtained based on the semantic feature vector.

[0022] Optionally, the step of constructing a structured environment representation for the legged robot based on the multi-source perception data for environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking includes:

[0023] The multi-source sensing data is subjected to time synchronization and external parameter calibration to obtain spatiotemporally consistent standardized sensing data;

[0024] Based on the standardized perception data, environmental objects are identified to generate an object attribute set for the legged robot. The object attribute set includes the category, pose, and grasping parameters of the manipulated object.

[0025] Based on the standardized perception data, terrain accessibility analysis is performed on the operating object to determine the terrain slope, step height, ground friction coefficient, and slip risk parameters within a preset area between the legged robot and the operating object.

[0026] A semantic map of the preset area is generated based on the terrain slope, the step height, the ground friction coefficient, and the slip risk parameter.

[0027] Dynamic obstacle detection and tracking are performed based on the time series of the standardized sensing data to generate a dynamic obstacle state sequence.

[0028] The object attribute set, the semantic map, and the dynamic obstacle state sequence are fused to obtain the structured environment representation of the legged robot within the preset area.

[0029] Optionally, the step of retrieving historical task experience fragments corresponding to the task instruction from a preset historical task database based on the semantic features includes:

[0030] The semantic features are normalized to obtain a standardized semantic feature vector.

[0031] Based on the standardized semantic feature vector, a retrieval vector adapted to the retrieval rules of the historical task database is generated;

[0032] Based on the retrieval vector, a cosine similarity matching retrieval is performed in a preset historical task database to obtain the historical task data;

[0033] The historical task data is structured and parsed to determine skill strategies, skill parameters, failure types, and recovery strategies.

[0034] The skill strategy, skill parameters, failure type, and recovery strategy are integrated to obtain the historical task experience fragment that matches the task instruction.

[0035] Optionally, the step of generating a hierarchical task plan for the legged robot using a language model, based on the historical task experience fragments and combined with the structured environment representation, includes:

[0036] The historical task experience fragments, the structured environment representation, and the task instructions are input into the language model, wherein the language model includes a semantic planning sub-model and a parameter calculation sub-model;

[0037] The semantic planning sub-model is used to break down the task objective in the task instruction into multiple stage sub-objectives corresponding to the task objective and multiple skills corresponding to each stage sub-objective. The skills are then integrated according to the order of the stage sub-objectives to obtain multiple skill sequences.

[0038] The sub-model is calculated using the parameters. For each skill in the skill sequence, the execution parameters corresponding to each skill are calculated using a tool function based on the structured environment representation.

[0039] Each skill sequence is associated with the execution parameters corresponding to each skill to obtain the hierarchical task plan corresponding to each skill sequence.

[0040] Optionally, the feasibility and safety verification of the hierarchical task plan to obtain the target skill sequence of the legged robot includes:

[0041] Obtain the standard skill entry corresponding to each skill in the skill sequence from the preset skill library. Each standard skill entry includes the prerequisites, effect definition, parameter interface and safety envelope of the skill.

[0042] Based on the prerequisites, effect definitions, parameter interfaces, and security envelopes, the feasibility of the execution parameters of each skill in the hierarchical task plan is verified and security constraints are checked to obtain the comprehensive cost of the hierarchical task plan.

[0043] The task plan at the level with the highest overall value is taken as the target skill sequence.

[0044] Optionally, the step of arranging skills according to the target skill sequence to generate a skill execution scheduling strategy for the legged robot includes:

[0045] Based on the execution parameters, skill attributes, and sequential relationships of each skill in the target skill sequence, determine the execution priority and timing requirements of each skill.

[0046] Set the switching logic for each skill, which includes the triggering conditions and execution thresholds for the skill;

[0047] Based on the dynamic obstacle state and terrain constraints in the structured environment representation, an execution timing window and action adjustment margin are configured for each of the skills.

[0048] The execution priority, timing requirements, switching logic, execution timing window, and action adjustment margin are integrated to construct the basic scheduling framework for the skill, and the skill execution scheduling strategy is generated through the basic scheduling framework.

[0049] Optionally, the step involves driving the legged robot to execute skills according to the skill execution scheduling strategy; simultaneously, when the legged robot executes a skill, the skill execution scheduling strategy is corrected in real-time using real-time sensor data of the legged robot, including:

[0050] According to the skill execution scheduling strategy, drive the executor of each skill to work;

[0051] Simultaneously, when the legged robot performs a skill, real-time sensing data of the legged robot is acquired; the real-time sensing data includes the real-time pose, real-time joint torque, and real-time foot contact force of the legged robot.

[0052] The real-time pose, real-time joint torque, and real-time foot contact force are analyzed in real time to determine the execution deviation value of each skill.

[0053] Determine whether the execution deviation value of the skill is greater than or equal to a preset correction threshold;

[0054] If so, the skill execution scheduling strategy is dynamically corrected based on the real-time sensing data according to the preset correction rules until the execution deviation value is less than the preset correction threshold.

[0055] If not, the skill execution scheduling strategy remains unchanged.

[0056] In a second aspect, the legged robot task planning and control system of the present invention includes:

[0057] The data acquisition unit is used to acquire task instructions and multi-source sensor data of the legged robot.

[0058] The data analysis unit is used to extract the task instructions, obtain the semantic features of the task instructions, and perform environmental object recognition, terrain accessibility analysis and dynamic obstacle tracking based on the multi-source perception data to construct a structured environment representation of the legged robot.

[0059] The retrieval unit is used to retrieve historical task experience fragments corresponding to the task instruction from a preset historical task database based on the semantic features.

[0060] The planning unit is used to generate a hierarchical task plan for the legged robot by using a language model, based on the historical task experience fragments, and in combination with the structured environment representation.

[0061] The verification unit is used to perform feasibility and safety verification on the hierarchical task plan to obtain the target skill sequence of the legged robot.

[0062] The orchestration unit is used to orchestrate skills according to the target skill sequence and generate a skill execution scheduling strategy for the legged robot.

[0063] An execution unit is used to drive the legged robot to perform skills according to the skill execution scheduling strategy; at the same time, when the legged robot performs skills, the skill execution scheduling strategy is corrected in real time in a closed loop based on the real-time sensor data of the legged robot.

[0064] The legged robot task planning and control method and system of this invention effectively solves the problem of the disconnect between action sequences and robot physical constraints in traditional language model planning by combining semantic understanding, environmental perception, historical experience retrieval, and real-time closed-loop correction. This significantly improves the performance of legged robots in complex, long-duration tasks. Specifically, before generating the task plan, this invention first constructs a structured environment representation based on multi-source perception data, including terrain accessibility and dynamic obstacles, and introduces historical task experience fragments as prior knowledge. This allows the hierarchical task plan generated by the language model to fully consider the strong coupling characteristics of the legged robot's movement and operation, gait constraints, and dynamic environmental changes, avoiding the infeasibility of actions caused by semantic reasoning alone, thereby improving the executability of the planning scheme in real-world scenarios. During skill execution, the scheduling strategy is corrected in a closed loop using real-time sensor data, enabling the robot to adjust its action sequence in a timely manner according to dynamic information such as terrain changes and obstacle movement. This perception-planning-execution closed-loop mechanism of this invention effectively improves the adaptability of legged robots in unstructured environments, ensuring that long-duration tasks can still be completed smoothly when the environment changes. Furthermore, the generated hierarchical task plan undergoes feasibility and safety verification, enabling the early elimination of action sequences that could lead to robot instability, collisions, or predicaments, thus ensuring operational safety. Simultaneously, efficient scheduling strategies are generated through skill orchestration, reducing unnecessary action switching and repetitive planning, thereby improving the overall efficiency of task execution. By retrieving similar experience fragments from the historical task database, targeted reference templates are provided for the language model, making the planning results more closely match the actual capabilities of the legged robot. This experience-based retrieval mechanism not only accelerates the planning process but also improves the planning success rate in new task scenarios through transfer learning. This invention decomposes long-term tasks into executable skill sequences through hierarchical task planning and combines structured environment representation with real-time correction, enabling robots to autonomously complete complex tasks involving multiple regions, tools, and interactions (such as door opening and carrying, inspection and retrieval), and also expanding the application potential of legged robots in practical scenarios such as inspection and rescue.

[0065] In summary, this invention significantly improves the planning feasibility, execution robustness, and environmental adaptability of legged robots for long-term tasks through multimodal information fusion, historical experience guidance, and closed-loop control, providing a reliable technical solution for the autonomous operation of legged robots in dynamic unstructured environments. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the task planning and control method for a legged robot according to an embodiment of the present invention.

[0067] Figure 2 This is a schematic diagram of the structure of the task planning and control system for a legged robot according to an embodiment of the present invention. Detailed Implementation

[0068] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0069] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0070] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0071] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0072] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0073] Combination Figure 1 As shown in the figure, an embodiment of the present invention provides a task planning and control method for a legged robot, comprising:

[0074] Acquire task instructions and multi-source sensor data for the legged robot.

[0075] Specifically, when acquiring task instructions and multi-source perception data for the legged robot, the system first receives long-term task instructions in natural language form through the user interface, such as fetching a bottle of water from the kitchen and placing it on the living room table. At the same time, it activates the multi-source perception and state estimation module, which integrates multimodal sensors such as visual sensors (e.g., RGB-D cameras), LiDAR, inertial measurement units (IMU), force / torque sensors, and foot tactile arrays. After synchronizing the time and calibrating the external parameters of each sensor, the system collects environmental images, point clouds, robot posture, joint states, foot contact forces, and other information in real time, and uses these raw observation data as input for subsequent processing.

[0076] The task instructions are extracted to obtain their semantic features. Based on the multi-source perception data, environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking are performed to construct a structured environmental representation of the legged robot.

[0077] Specifically, firstly, natural language processing technology is used to identify the intent of task instructions and extract key information, resulting in a task description containing semantic features such as target objects, actions, and locations. Simultaneously, the perception module fuses multi-source data: object detection and pose estimation algorithms identify objects in the environment, such as doors, water bottles, and tables, and their poses; 3D geometric reconstruction and terrain classification algorithms analyze ground slope, step height, and friction coefficient to generate a terrain accessibility layer; and multi-object tracking algorithms are used to track dynamic obstacles, such as pedestrians, in real time. Based on this, the world model and semantic map modules integrate this information into a structured environmental representation. This representation uses a combination of object-relationship graphs and semantic maps. Object nodes include category, pose, graspability, and reachability labels; relationship edges describe relative positions and topological connectivity; and the semantic map marks walkable areas, restricted areas, risk areas, and terrain costs on a 2D or 3D grid, thus forming a complete environmental model that includes both geometric information and semantic labels, providing a unified state space for subsequent planning.

[0078] In a preferred embodiment of the present invention, the world model and semantic map module construction process includes: semantic mapping: receiving environmental observations from a multi-source perception module. It identifies discrete objects in the environment (such as tables, doors, cups, etc.) through a preset semantic segmentation algorithm; state association: it associates the robot pose output by the state estimation module with the object state. (Including 3D coordinates, orientation, and contact state) are mapped to a unified global coordinate system; Relationship extraction: Spatial topology analysis is used to extract the relationships between objects, such as On(cup, table) and the reachability relationships between the robot and objects, such as Reachable(robot, door). Finally, these are integrated to generate a structured environment representation that includes object features, spatial relationships, and physical attributes. This structured representation transforms unstructured point cloud data into logical symbols that a language model planner can understand. Structured Context Representation It plays a crucial role in the system's operation, connecting the preceding and following stages. Its specific application is reflected in the planning phase (specifically for language model planners): It provides a semantic snapshot of the environment state for the language model. The planner is based on... The system uses 'object-relationship' information (such as determining whether the target object is inside a closed container) to decide whether prerequisite skills like 'opening the door' or 'removing the obstacle' need to be executed first, thus ensuring that the generated hierarchical task plan is logically complete. In the verification phase (for the feasibility and safety verification module): the system extracts... The geometric features and physical properties (such as slope, step height, passage width, etc.) are used as hard constraints input into the safety verification model. For example, through... The terrain slope information in the model is compared with the maximum static friction model of the legged robot. If If the slope exceeds the robot's physical limits, the verification module will immediately provide feedback to the planner, forcing it to replan the path or switch movement modes. Based on the semantic features, a search is performed in a pre-set historical task database to obtain historical task experience fragments corresponding to the task instruction.

[0079] Based on the semantic features, a search is performed from a pre-set historical task database to obtain historical task experience fragments corresponding to the task instruction.

[0080] Specifically, the Long Memory (LTM) unit in the memory module is invoked, which contains a vector index and a structured event library. First, the current task instructions and environment summary are encoded into query embedding vectors. Then, based on cosine similarity, the top-K most similar historical experiences are retrieved from the vector index. At the same time, a comprehensive retrieval score is calculated by combining feasibility priors and recency decay factors to filter out experience fragments that are highly relevant to the current task. These experience fragments are stored in a structured form, including fields such as task ID, sub-objective sequence, skill call record, parameter range, failure type, and recovery strategy. The retrieval results are then injected as prompts into the subsequent language model planner to enhance the targeting and reliability of the planning.

[0081] In a preferred embodiment of the present invention, the memory module consists of a Short-Term Memory (STM) and a Long-Term Memory (LTM). The STM records the current task context, completed sub-goals, current item status, and the reason for the most recent failure, with an update frequency consistent with the control loop. The LTM stores reusable experience across tasks, including successful plan templates, parameter ranges, failure attributions, and recovery strategies. To support enhanced retrieval planning, the LTM includes at least a vector index library and a structured event library: the vector index library stores embedded vectors of task summaries, environment summaries, and skill fragments; the structured event library stores data in a table structure (task ID, timestamp, sub-goal, skill k, parameter). Fields such as key observation summary, results, failure type, and recovery strategy.

[0082] During retrieval, the current query q is encoded into an embedding vector. And calculate with historical entries Similarity is used to obtain Top-K candidate experience: Where cos represents cosine similarity, the system can further incorporate feasibility scoring and recentity decay to form a comprehensive retrieval score: ;in, Indicates that a query will be performed. Encoded 3D embedding vector, This represents the i-th historical entry in the long-term memory. 3D embedding vector, express The dimension of the real space, i.e., the dimension of the vector space in which the embedded vectors reside. This represents the weighting coefficient of the base similarity score, controlling the proportion of cosine similarity in the overall score. The weighting coefficient represents the feasibility score and controls the proportion of the feasibility assessment in the overall score. The weighting coefficient representing the recency decay accounts for the proportion of the overall score in terms of the time elapsed. This represents the overall search score, used for the final selection of experience fragments that can be injected into an LLM. Indicates the first A feasibility score is assigned to each historical experience to assess its feasibility in the current scenario. The score represents the recentity score of the i-th historical experience, which decays over time. Newer experiences have higher scores. The above score is used to filter experience fragments that can be injected with LLM hints.

[0083] Using a language model, based on the historical task experience fragments and combined with the structured environment representation, a hierarchical task plan for the legged robot is generated.

[0084] Specifically, a language model planner is employed, which consists of a semantic planning sub-agent and a parameter calculation sub-agent working collaboratively. The semantic planning sub-agent receives user instructions, environmental summaries, and retrieved historical experience, and outputs discrete sub-target maps and skill skeleton sequences under the constraints of the prompt orchestration function, such as navigating to the kitchen door before opening it. The parameter calculation sub-agent generates specific continuous parameters for each skill based on the current structured environment representation and the prerequisites and parameter interfaces of skills in the skill library, such as target pose, gait speed, and grasping force threshold. The entire planning process adopts a hierarchical decomposition strategy, first generating high-level sub-targets at the room or area level, and then generating fine-grained skill sequences within each sub-target, ultimately forming a hybrid discrete-continuous hierarchical task plan. Each step in the plan includes precondition checks, skill names, parameters, success criteria, and failure fallback fields to ensure the executability and verifiability of the output.

[0085] The feasibility and safety of the hierarchical task plan are verified to obtain the target skill sequence of the legged robot.

[0086] Specifically, the feasibility and safety verification module performs physical constraint checks on the plan. This module first checks whether the prerequisites for each skill are met in the current environmental state, then calls the Task and Motion Planning (TAMP) unit to generate specific motion trajectories for the skill sequence and checks constraints such as joint limits, collisions, and accessibility. At the same time, the stability constraint unit evaluates the stability margin of the robot at each stage based on the distance between the foot support polygon and the centroid projection, especially for dynamic processes such as grasping, carrying, and crossing. In addition, the multi-objective cost evaluation unit calculates the comprehensive cost based on energy consumption, time, and risk costs, and sorts and filters candidate plans. If any check fails, the module feeds back the failure type and the reason for the violation of key constraints as structured information to the language model planner, triggering plan correction until a target skill sequence that passes all constraint verifications is generated.

[0087] Skills are arranged according to the target skill sequence to generate a skill execution scheduling strategy for the legged robot.

[0088] Specifically, the skill orchestrator retrieves corresponding reusable skill entries from the legged mobility-operation skill library based on the validated skill sequences and the current environmental state. Each skill entry includes a semantic description, prerequisites, effects, parameter set, underlying controller, and safety envelope. The orchestrator assigns specific parameters to each skill according to the task plan structure and maintains the execution stack and state machine, defining the entry conditions, running conditions, and exit conditions for each skill. At the same time, the orchestrator generates a macro-level scheduling strategy, including skill switching order, exception handling branches, and resource constraints such as power management, and pre-configures fallback skills and recovery strategies for possible failures, forming a complete skill execution scheduling strategy.

[0089] In a preferred embodiment of the present invention, the skill orchestrator compares the current environment observations in real time. and Preset skill effects .like The environmental state after the skill is performed does not meet expectations; for example, after performing the action of opening a door, If the gate in the memory is still in the Closed state, the system determines that the skill execution has failed and immediately calls the failure record in the memory module to roll back and retry or trigger a replanning process, thereby achieving closed-loop control at the task level. Each skill entry in the skill library includes at least: semantic description, task tag, and prerequisites. ,Effect Parameter set Policy / Controller Security envelope and embedding vectors .

[0090] in, and This can be expressed using predicates, for example:

[0091] Skill retrieval employs a joint scoring system of semantic similarity and feasibility priors: firstly, a query vector is obtained based on the task sub-target text and the environmental summary. Top-K skills are retrieved based on cosine similarity; then, the prerequisites of each candidate skill are checked. With security envelope Does it meet the requirements in the current state, and assign a feasibility score? The skills are merged into a comprehensive score, and the best skill is selected for the parameterization stage. To adapt to the strong coupling characteristics of foot-based movement and manipulation, the skill library can be organized in layers according to movement skills, posture / stability skills, manipulation skills, interaction skills, and composite skills (movement + manipulation). Composite skills can encapsulate typical foot-end coordination patterns, such as picking up / putting down while walking, or grasping after crossing obstacles, to reduce combinatorial explosion in high-level planning.

[0092] The legged robot is driven to perform skills according to the skill execution scheduling strategy; at the same time, when the legged robot performs skills, the skill execution scheduling strategy is corrected in real time in a closed loop based on the real-time sensor data of the legged robot.

[0093] Specifically, the underlying control and execution module uses a whole-body control framework to solve for foot contact constraints, body posture, end effector trajectory, and joint limits in a unified manner, outputting joint torque or position commands to drive the robot to execute the current skill. During execution, the skill orchestrator continuously monitors real-time feedback from sensors, including foot contact state, joint torque, posture changes, and end effector force. When L1 level anomalies such as parameter deviation, grasping failure, or stability abnormalities are detected, micro-correction is triggered, that is, the skill parameters are locally adjusted without changing the high-level skill sequence, such as adjusting the grasping posture or changing the gait speed. When multiple consecutive micro-corrections are ineffective or the planned structure fails due to sudden environmental changes, macro-replanning is triggered, and the language model planner is called again to generate new sub-goals and skill sequences. All execution feedback, abnormal events, and correction operations are recorded and written into the long-term memory of the memory module to support experience transfer and skill optimization for subsequent tasks.

[0094] In a preferred embodiment of the present invention, the underlying control and execution module can adopt a whole-body control (WBC) framework, unifying foot contact constraints, body posture, end effector trajectory, and joint limits into an optimization problem for solution, outputting joint torque or desired acceleration. In composite skills, foot contact patterns and end effector trajectories need to be coordinated: for example, when carrying heavy objects, gait speed needs to be reduced and stability margin increased; when opening drawers or doors, the center of mass shift caused by external forces needs to be considered and the foot placement dynamically adjusted. To achieve interface unification with upper-level skills, this embodiment defines a parameterized control interface: skill k receives parameters at execution time t. With local targets Controller output action (Joint torque or position / velocity command), and return execution feedback. (Including success criteria, contact events, energy consumption estimation, and anomaly indicators). The upper-level orchestrator is based on... Update the next moment Or it may trigger a rollback.

[0095] For example, the legged robot task planning and control method described in this embodiment can be implemented using a legged robot. Embodiment 1: Indoor cross-room retrieval and door opening task. Task description: The robot is located in room A. The user's instruction is to retrieve a bottle of water from the kitchen and deliver it to the living room table. There are doors, cabinets, and moving pedestrians in the environment.

[0096] Step E1-1: The perception module identifies the location of the water bottle, the pose of the door handle, and the walkable area, constructs a semantic map, and marks the open and closed state of the door.

[0097] Step E1-2: The memory module retrieves historical experience: In a kitchen-like scenario, first execute "open door - enter - locate water bottle - grab - exit - arrive at living room - place".

[0098] Steps E1-3: The LLM planner generates a hierarchical plan: G0 arrives at the kitchen door → G1 opens the door → G2 navigates to the counter → G3 grabs the water bottle → G4 navigates to the living room → G5 places the bottle.

[0099] Step E1-4: Feasibility verification module check: The door opening action has high stability requirements and requires the selection of the "three-legged support + end push-pull" composite skill; the grasping phase checks accessibility and collision.

[0100] Steps E1-5: Skill orchestrator executes and monitors: if the door handle recognition confidence decreases, a re-observation is triggered; if the grab fails, posture fine-tuning and retry are triggered; if the passage is blocked by pedestrians, macro replanning of detours is triggered.

[0101] Steps E1-6: After the task is completed, write the success parameter range (door handle push and pull force threshold, gait speed limit, etc.) and the exception handling strategy into long-term memory.

[0102] Example 2: Outdoor Unstructured Terrain Inspection and Pickup Task. Task Description: The robot finds scattered tools on the ground during its inspection route in the park. The user command is "pick up the wrench and put it in the recycling bin." The ground contains grass, steps, and some slippery areas.

[0103] Step E2-1: Overlay a terrain accessibility layer onto the semantic map to increase the risk cost J_R in slippery areas and limit gait speed.

[0104] Step E2-2: The LLM planner generates sub-goals: approach the tool → stand stably → identify the grasping posture → grasp → reach the recycling bin → place.

[0105] Step E2-3: The feasibility verification module adds stability margin constraints during the crawling phase. And during the transport phase, acceleration and turning radius are limited, among which, For distance measurement function, For the position of the robot's load center of mass, To capture the stable region, To stabilize the margin threshold, constraints are imposed. Used to ensure mechanical stability during the gripping phase.

[0106] Step E2-4: If slippage is detected at the foot during execution, micro-correction is triggered: switch to a more conservative gait and increase foot redundancy; if necessary, macro-replanning is triggered to avoid slippery areas.

[0107] Step E2-5: Write the successful gait parameters and failed segments under slippery terrain into memory to support subsequent similar tasks.

[0108] Example 3: Obstacle-crossing handling and stacking task in a warehouse scenario. Task description: The robot needs to move multiple boxes from area A to a shelf in area B. There are narrow passages and low obstacles along the way, which need to be crossed or detoured. The task is long and needs to be repeated.

[0109] Step E3-1: The skill library includes skills such as "passing through narrow passages", "crossing low obstacles", "carrying with both hands / one hand", and "aligning items in front of shelves".

[0110] Step E3-2: The LLM planner generates multi-cycle plans in batch mode and inserts the safety sub-goals of "energy consumption budget" and "energy threshold return to charging" into the plans.

[0111] Step E3-3: The multi-objective optimization module updates the weights w in each loop. E With wT Increase W when battery is low E Prioritize low-energy routes; increase w as the mission deadline approaches. T Prioritize efficiency.

[0112] Step E3-4: During repetitive handling, the memory module statistically analyzes the optimal route and placement posture to form a task template; when the environmental layout changes, the system quickly replans based on the template.

[0113] Step E3-5: Through skill version management and online self-update mechanism, continuously optimize high-frequency skills (positional placement) to improve long-term operational stability.

[0114] This embodiment of the legged robot task planning and control method effectively solves the problem of the disconnect between action sequences and robot physical constraints in traditional language model planning by combining semantic understanding, environmental perception, historical experience retrieval, and real-time closed-loop correction. This significantly improves the performance of legged robots in complex, long-duration tasks. Specifically, before generating the task plan, this embodiment first constructs a structured environment representation based on multi-source perception data, including terrain accessibility and dynamic obstacles, and introduces historical task experience fragments as prior knowledge. This allows the hierarchical task plan generated by the language model to fully consider the strong coupling characteristics of the legged robot's movement and operation, gait constraints, and dynamic environmental changes, avoiding the infeasibility of actions caused by semantic reasoning alone, thereby improving the executability of the planning scheme in real-world scenarios. During skill execution, the scheduling strategy is corrected in a closed loop using real-time sensor data, enabling the robot to adjust its action sequences promptly based on dynamic information such as terrain changes and obstacle movement. This perception-planning-execution closed-loop mechanism effectively enhances the adaptability of legged robots in unstructured environments, ensuring that long-duration tasks can still be completed smoothly when the environment changes. Furthermore, the generated hierarchical task plan undergoes feasibility and safety verification, enabling the early elimination of action sequences that could lead to robot instability, collisions, or predicaments, thus ensuring operational safety. Simultaneously, efficient scheduling strategies are generated through skill orchestration, reducing unnecessary action switching and repetitive planning, thereby improving overall task execution efficiency. By retrieving similar experience fragments from the historical task database, targeted reference templates are provided for the language model, making the planning results more closely aligned with the actual capabilities of the legged robot. This experience-based retrieval mechanism not only accelerates the planning process but also improves the planning success rate in new task scenarios through transfer learning. This embodiment decomposes long-term tasks into executable skill sequences through hierarchical task planning and combines structured environment representation with real-time correction, enabling the robot to autonomously complete complex tasks involving multiple regions, tools, and interactions (such as door opening and carrying, inspection and retrieval), and also expanding the application potential of legged robots in practical scenarios such as inspection and rescue.

[0115] In summary, this embodiment significantly improves the planning feasibility, execution robustness, and environmental adaptability of legged robots for long-term tasks through multimodal information fusion, historical experience guidance, and closed-loop control, providing a reliable technical solution for the autonomous operation of legged robots in dynamic unstructured environments.

[0116] Optionally, acquiring the task instructions and multi-source perception data of the legged robot includes:

[0117] The visual perception data, laser point cloud data, inertial data, wrist perception data, and foot tactile perception data of the legged robot are acquired.

[0118] The visual perception data, the laser point cloud data, the inertial data, the wrist perception data, and the foot tactile perception data are used as the multi-source perception data, and the task instructions issued to the legged robot are obtained.

[0119] Specifically, firstly, visual perception data of the legged robot is acquired. RGB and RGB-D cameras are used as visual acquisition units to perform visual imaging and depth acquisition of the environment, completing target / operable area recognition and 3D geometric reconstruction processing to obtain object category, pose, graspable area, and dynamic obstacle information, providing a visual observation basis for semantic mapping and task planning. Secondly, laser point cloud data is acquired. A lidar sensor completes 3D point cloud scanning of the environment. After noise reduction, ground segmentation, and walkable area estimation, environmental geometry, terrain slope, step height, restricted areas, and risk areas are obtained, forming a spatial constraint basis for movement planning and traversability assessment. Next, inertial data is acquired. The body's three-axis angular velocity, three-axis acceleration, and attitude angle information are collected at high frequency by the body's IMU sensor. After state estimation and fusion, the robot's real-time pose, tilt state, and motion trend are obtained for stability determination, centroid projection calculation, and whole-body control input. Finally, wrist perception data is acquired. A force / torque sensor is installed on the wrist of the robotic arm to collect real-time data on the interaction between the end effector and the control object. Information on contact forces, torques, and external disturbances between objects provides force feedback for adjusting the grasping force threshold, correcting the operating trajectory, collision protection, and anomaly detection. Next, tactile perception data from the feet is acquired. Tactile arrays and contact sensors are installed at the feet to detect the contact state, contact position, pressure distribution, slip events, and ground support characteristics between the feet and the ground. This completes contact event detection and ground friction estimation, providing underlying feedback for gait planning, support polygon calculation, stability margin determination, and fall prevention. After acquiring the above perception data, time-stamp alignment and multi-sensor extrinsic parameter calibration are performed on visual perception data, laser point cloud data, inertial data, wrist perception data, and foot tactile perception data. This unifies multimodal observations into the same spatiotemporal coordinate system, forming standardized multi-source perception data, which is input to the multi-source perception and state estimation module to output the robot's own state and the state of the surrounding objects. Finally, task instructions are acquired. Long-term task instructions issued by the user, spanning multiple rooms, terrains, steps, and tools, are received in natural language form and used as the top-level input for the language model planner to generate hierarchical task plans.

[0120] In this optional embodiment, by acquiring visual perception data, laser point cloud data, inertial data, wrist perception data, and foot tactile perception data and synchronizing them with time and calibrating their spatial extrinsic parameters, the system can construct a complete and high-fidelity multimodal environmental perception foundation. The visual and laser data complement each other to achieve accurate three-dimensional geometric reconstruction and target recognition, the inertial data provides the robot's own motion state and posture reference, and the wrist force perception and foot tactile data provide real-time feedback on contact force information during the interaction process. This deep fusion of multi-source heterogeneous data provides rich and reliable input for subsequent environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking, ensuring the accuracy and real-time performance of the structured environment representation, thereby laying a solid data foundation for legged robots to perform long-term tasks in complex unstructured terrain.

[0121] Optionally, the step of extracting the task instructions to obtain the semantic features of the task instructions includes:

[0122] The task instructions are preprocessed to obtain the preprocessed text of the task instructions;

[0123] The preprocessed text is analyzed using a semantic feature extraction model to determine the task objectives, operation objects, action requirements, and constraint information in the task instructions.

[0124] The operation object, the operation object, the action requirements, and the constraint information are vectorized and encoded to generate semantic feature vectors;

[0125] The semantic features of the task instruction are obtained based on the semantic feature vector.

[0126] Specifically, the task instructions are first preprocessed using word segmentation, stop word removal, format regularization, and instruction structure cleaning to process the long-term natural language task instructions input by the user. Redundant words and non-critical modifiers are removed, while core expressions such as task objectives, operation objects, action types, location areas, and constraints are retained, resulting in standardized and uniform preprocessed text suitable for model input. Secondly, a semantic feature extraction model is used to analyze the preprocessed text. Using the semantic understanding unit of the language model planner as a carrier, intent recognition and element decomposition are performed on the preprocessed text to accurately determine the task objectives, operation objects, action requirements, and constraint information in the task instructions. The task objectives are the endpoints of long-term tasks such as cross-room retrieval, inspection and pickup, and warehouse handling; the operation objects are interactive entities such as water bottles, boxes, tools, and doors; and the action requirements include navigation, grasping, placing, opening doors, crossing, and pushing / pulling. The action is constrained by conditions such as energy consumption limits, time limits, stability limits, obstacle avoidance limits, and terrain accessibility limits. The parsed operation objects, action requirements, and constraints are then vectorized. The embedded layer of the language model converts these structured semantic elements into dense, fixed-dimensional embedding vectors. Simultaneously, the semantic descriptions of skills from the skill library and the object attribute labels of the world model are aligned to generate semantic feature vectors compatible with the system memory module and skill retrieval module. Finally, the semantic features of the task instructions are derived from these feature vectors. These feature vectors serve as query vectors for the system memory module, input for the language model planner to generate hierarchical plans, and core features for the skill orchestrator to match skill entries and verify feasibility. This process transforms task instructions from natural language to structured semantic features, providing a unified semantic input for subsequent hierarchical planning, feasibility verification, and skill orchestration.

[0127] In a preferred embodiment of the present invention, the skill orchestrator is responsible for converting the verified skill sequences into executable schedules and performing closed-loop monitoring and error correction during execution. The skill orchestrator maintains an execution stack and state machine, defines entry conditions, running conditions, and exit conditions for each skill, and classifies and rolls back runtime anomalies. The micro-modification in the above embodiment refers to adjusting only the parameters without changing the high-level sub-objectives and skill structure. For example, adjusting the grasping posture, increasing the grasping force threshold, changing gait speed, reselecting the landing point, or adjusting the end trajectory; micro-corrections can be implemented through local optimization, MPC, or rule-based safety strategies. Macro-replanning refers to regenerating the sub-target map and skill sequence when the plan structure fails (e.g., the target is moved, the channel is blocked, or key skills fail consecutively) or the task objective changes; macro-replanning calls the memory module to retrieve historical recovery strategies and prioritizes trying recovery routes with lower costs. Meanwhile, to support stable and reliable long-term execution, the skill orchestrator in this embodiment introduces a failure classification unit, classifying failures into at least: perception failure, motion unreachability failure, grasping failure, stability failure, and external interference failure. Different failure types correspond to different fallback strategies: for example, perception failure prioritizes re-observation and map update; grasping failure prioritizes local retrieval and posture adjustment; stability failure prioritizes stopping, reducing speed, changing support mode, or putting down the object; external interference failure prioritizes safe retreat and replanning.

[0128] In this optional embodiment, by performing text preprocessing, semantic feature extraction, element recognition, and vectorization encoding on task instructions, the system can transform long-term task instructions described in natural language into structured semantic feature vectors. This transformation process achieves a precise mapping from human intent to a machine-understandable semantic space. Text preprocessing eliminates redundancy and noise in language expression, and the semantic feature extraction model deeply analyzes core elements such as task objectives, operation objects, action requirements, and constraint information, avoiding semantic ambiguity and element omission problems common in traditional keyword matching methods. Each element is vectorized and encoded separately and then fused to generate a comprehensive semantic feature vector, enabling the complete connotation of the task instructions to be uniformly represented in a high-dimensional semantic space. This provides an accurate query benchmark for subsequent historical experience retrieval based on semantic similarity, and also provides clear task boundaries and constraints for the language model planner to generate immediately executable hierarchical task plans, thereby significantly improving the accuracy of legged robots in understanding complex instructions and the pertinence of task planning.

[0129] Optionally, the step of constructing a structured environment representation for the legged robot based on the multi-source perception data for environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking includes:

[0130] The multi-source sensing data is subjected to time synchronization and external parameter calibration to obtain spatiotemporally consistent standardized sensing data;

[0131] Based on the standardized perception data, environmental objects are identified to generate an object attribute set for the legged robot. The object attribute set includes the category, pose, and grasping parameters of the manipulated object.

[0132] Based on the standardized perception data, terrain accessibility analysis is performed on the operating object to determine the terrain slope, step height, ground friction coefficient, and slip risk parameters within a preset area between the legged robot and the operating object.

[0133] A semantic map of the preset area is generated based on the terrain slope, the step height, the ground friction coefficient, and the slippage risk parameter;

[0134] Dynamic obstacle detection and tracking are performed based on the time series of the standardized sensing data to generate a dynamic obstacle state sequence.

[0135] The object attribute set, the semantic map, and the dynamic obstacle state sequence are fused to obtain the structured environment representation of the legged robot within the preset area.

[0136] Specifically, firstly, the multi-source perception data undergoes time synchronization and extrinsic parameter calibration. A unified clock reference is used to align the timestamps of visual perception data, laser point cloud data, inertial data, wrist perception data, and foot tactile perception data. This completes the extrinsic parameter calibration of cameras, LiDAR, IMU, force sensors, and tactile sensors, unifying the observations of each sensor to the robot's base coordinate system, eliminating spatiotemporal offsets and observation biases, and obtaining standardized perception data that is spatiotemporally consistent and directly usable for environmental modeling. Secondly, environmental object recognition is performed based on the standardized perception data. Through target detection, pose estimation, and operable area segmentation, operable entities such as doors, cabinets, tools, and transported objects are identified in the environment. An object attribute set is generated, including object category, spatial pose, graspable area, size attributes, and reachability labels, providing object-level semantic basis for subsequent grasping planning, skill invocation, and feasibility verification. Thirdly, based on the standardized perception data and the operable objects, terrain accessibility analysis is conducted within a locally preset area. This analysis relies on point cloud ground fitting, slope calculation, and height... The process involves several key steps: First, a gradient difference detection and friction coefficient estimation are used to determine the terrain slope, step height, ground friction coefficient, and slip risk parameters within the target area. This provides quantitative terrain indicators for foot movement stability constraints, gait selection, and foot placement planning. Then, based on these parameters, a semantic map of the predefined area is generated. Walkable areas, restricted areas, risk areas, and terrain accessibility layers are overlaid on a raster or sparse graph structure, encoding terrain costs and safety constraints into the map to form an environmental spatial representation compatible with language model planning, task and motion planning. Next, dynamic obstacle detection and tracking are performed based on the time-series information of standardized perception data. Continuous frame association, state prediction, and trajectory updates are performed on dynamic targets such as pedestrians and moving objects, generating a dynamic obstacle state sequence containing the location, speed, movement trend, and occupied space of the obstacle. This provides dynamic constraints for real-time obstacle avoidance and execution-level safety protection. Finally, the object attribute set, semantic map, and dynamic obstacle state sequence are fused within a unified spatial framework, with the object... The relational graph structure integrates object attributes, terrain semantics, and dynamic obstacle information to obtain a complete structured environment representation covering static environment, operable objects, and dynamic disturbances, serving as a unified environment input for the language model planner, feasibility verification module, and skill orchestrator.

[0137] In this optional embodiment, by performing time synchronization and extrinsic parameter calibration, environmental object recognition, terrain accessibility analysis, semantic map generation, dynamic obstacle tracking, and multi-dimensional information fusion on multi-source perception data, unstructured and highly redundant raw observations can be transformed into a spatiotemporally consistent, semantically clear, and physically verifiable structured environmental representation. This provides the language model planner with understandable object, relationship, and constraint information, reducing planning illusions, and provides the feasibility and safety verification module with accurate pose, terrain, obstacle, and risk parameters. This significantly improves the accessibility, stability, and safety of foot-based mobility-operation skill invocation. At the same time, it enables efficient integration of environmental information with the skill library and memory module, supporting continuous, robust, and reusable hierarchical planning and closed-loop execution in long-term tasks, effectively reducing error accumulation and replanning frequency, and improving overall task execution efficiency and reliability.

[0138] Optionally, the step of retrieving historical task experience fragments corresponding to the task instruction from a preset historical task database based on the semantic features includes:

[0139] The semantic features are normalized to obtain a standardized semantic feature vector.

[0140] Based on the standardized semantic feature vector, a retrieval vector adapted to the retrieval rules of the historical task database is generated;

[0141] Based on the retrieval vector, a cosine similarity matching retrieval is performed in a preset historical task database to obtain the historical task data;

[0142] The historical task data is structured and parsed to determine skill strategies, skill parameters, failure types, and recovery strategies.

[0143] The skill strategy, skill parameters, failure type, and recovery strategy are integrated to obtain the historical task experience fragment that matches the task instruction.

[0144] Specifically, firstly, the semantic features are normalized by scaling the semantic feature vectors corresponding to task instructions to a model-compatible standard vector space according to their magnitude and numerical range. This eliminates amplitude differences caused by different element encodings, resulting in standardized semantic feature vectors with aligned dimensions and normalized numerical values, ensuring consistency and accuracy in subsequent retrieval calculations. Secondly, a retrieval vector adapted to the historical task database retrieval rules is generated based on the standardized semantic feature vectors. The standardized semantic feature vectors are then concatenated with the summary information of the current structured environment representation, supplementing retrieval auxiliary information such as environment topology, object status, and terrain constraints to form a retrieval vector that conforms to the matching rules of the long-term memory vector index. Thirdly, a cosine similarity matching retrieval is performed in the preset historical task database based on the retrieval vectors. Using the retrieval vectors as query terms, the cosine similarity between the retrieval vectors and the embedded vectors of historical tasks, skill fragments, and failure event entries is calculated in the long-term memory vector index, and the top results are selected based on the scores. The system retrieves K most relevant historical task data points to quickly locate similar tasks and scenarios. Then, it performs structured parsing of the retrieved historical task data, extracting and determining reusable skill strategies, optimal skill parameter ranges, typical failure types, and corresponding effective recovery strategies according to the field format (state, sub-goal, skill, parameter, result, failure type, recovery strategy) in long-term memory. Finally, it filters, deduplicates, and logically integrates the parsed skill strategies, skill parameters, failure types, and recovery strategies, eliminating conflicting or low-scoring entries and retaining highly reliable and adaptable effective experience information. This forms structured historical task experience fragments that can be directly injected into the language model planner, providing transferable prior knowledge for hierarchical plan generation and anomaly rollback.

[0145] In this optional embodiment, by normalizing semantic features, generating adapted retrieval vectors, performing accurate matching of historical tasks based on cosine similarity, and structurally parsing and integrating key experiences, it is possible to quickly extract skill strategies, parameter ranges, failure types, and recovery strategies that are highly adapted to the current task from the historical task library. This forms historical task experience fragments that can be directly used for planning prompts, effectively providing reliable prior knowledge for the language model planner, reducing invalid planning and repeated trial and error, improving the feasibility and execution efficiency of long-term task planning, and at the same time, using historical failure cases and recovery strategies to avoid execution risks in advance, suppress error accumulation, enhance cross-task experience transfer capabilities, and enable the system to complete long-term movement operation tasks more stably and robustly in dynamic unstructured environments.

[0146] Optionally, generating a hierarchical task plan for the legged robot using a language model, based on the historical task experience fragments and combined with the structured environment representation, includes:

[0147] The historical task experience fragments, the structured environment representation, and the task instructions are input into the language model, wherein the language model includes a semantic planning sub-model and a parameter calculation sub-model;

[0148] The semantic planning sub-model is used to break down the task objective in the task instruction into multiple stage sub-objectives corresponding to the task objective and multiple skills corresponding to each stage sub-objective. The skills are then integrated according to the order of the stage sub-objectives to obtain multiple skill sequences.

[0149] The sub-model is calculated using the parameters. For each skill in the skill sequence, the execution parameters corresponding to each skill are calculated using a tool function based on the structured environment representation.

[0150] Each skill sequence is associated with the execution parameters corresponding to each skill to obtain the hierarchical task plan corresponding to each skill sequence.

[0151] Specifically, firstly, historical task experience fragments, structured environment representations, and task instructions are input into a language model planner composed of a semantic planning sub-model and a parameter calculation sub-model. Historical experience serves as the prior cue, the structured environment representation as the physical constraint, and the task instructions as the top-level objective, forming a complete planning input condition. Secondly, the semantic planning sub-model decomposes the overall task objective in the task instructions hierarchically from top to bottom. It breaks it down into multiple sequential stage sub-objectives according to execution logic such as cross-room movement, terrain traversal, object manipulation, and target placement. Based on the prerequisites and effect predicates of skills in the skill library, corresponding movement, manipulation, stability, or composite skills are matched to each stage sub-objective. Finally, the skills are ordered according to the execution sequence of the stage sub-objectives. The algorithm first arranges multiple discrete skill sequences that satisfy semantic logic and task flow. Then, for each skill in the skill sequence, the parameter calculation sub-model calls external tool functions such as reachability query, crawling candidate generation, terrain accessibility assessment, and stability assessment. Combined with information such as object pose, terrain parameters, passage cost, and safety constraints in the structured environment representation, numerical calculations are performed to determine the continuous execution parameters such as target pose, gait pattern, gripping force threshold, speed limit, and stability margin corresponding to each skill. Finally, each skill in each skill sequence is bound to its execution parameters calculated by the tool functions to form a hybrid hierarchical task plan that combines discrete skill call order with continuous execution parameters, serving as a direct basis for subsequent feasibility verification and skill orchestration.

[0152] In this optional embodiment, a semantic planning sub-model and a parameter calculation sub-model are used to collaboratively generate a hierarchical task plan. This approach leverages historical task experience and structured environmental constraints to decompose abstract natural language instructions into feasible, verifiable, and executable stage sub-goals and skill sequences. Simultaneously, it uses tool functions to accurately calculate skill execution parameters. This approach retains the advantages of language models in semantic understanding and task decomposition while effectively avoiding planning illusions and physical infeasibility issues. The plan possesses clear execution logic and stable constraint boundaries, significantly improving the rationality, feasibility, and interpretability of long-term task planning. This provides a well-structured and constrained plan foundation for subsequent feasibility verification, skill orchestration, and closed-loop execution, significantly reducing the number of replanning attempts and the risk of execution failure.

[0153] Optionally, the feasibility and safety verification of the hierarchical task plan to obtain the target skill sequence of the legged robot includes:

[0154] Obtain the standard skill entry corresponding to each skill in the skill sequence from the preset skill library. Each standard skill entry includes the prerequisites, effect definition, parameter interface and safety envelope of the skill.

[0155] Based on the prerequisites, effect definitions, parameter interfaces, and security envelopes, the feasibility of the execution parameters of each skill in the hierarchical task plan is verified and security constraints are checked to obtain the comprehensive cost of the hierarchical task plan.

[0156] The task plan at the level with the highest overall value is taken as the target skill sequence.

[0157] Specifically, firstly, standard skill entries corresponding to each skill in the skill sequence included in the hierarchical task plan are retrieved from the preset foot-based movement-operation skill library. These standard skill entries strictly include the skill's prerequisites, effect definitions, parameter interfaces, and safety envelope. The prerequisites and effect definitions are presented in predicate form, the parameter interfaces correspond to the continuously adjustable variables of the skill, and the safety envelope is a set of constraints on skill execution stability, collisions, energy consumption, and risks. Secondly, based on the aforementioned prerequisites, effect definitions, parameter interfaces, and safety envelope, the feasibility verification and safety constraint checks are performed on the execution parameters of each skill in the hierarchical task plan, starting with checking the execution parameters of each skill. The system checks whether the environment and robot state meet the skill prerequisites, then verifies motion accessibility and collision-free capability through the task and motion planning unit, determines stability margin based on support polygons and centroid projection through the stability constraint unit, and calculates the weighted comprehensive cost of energy consumption, time, and risk through the multi-objective cost evaluation unit. Simultaneously, it performs item-by-item verification of all skills and cost calculation of the entire plan. Finally, it determines the hierarchical task plan with the optimal comprehensive cost as the target skill sequence. This target skill sequence is the final executable skill scheme that simultaneously satisfies physical feasibility, safety constraints, and multi-objective optimization, and is used to input the skill orchestrator for subsequent scheduling and execution.

[0158] In a preferred embodiment of the present invention, the feasibility and security verification module is used to project the semantic plan onto the physical feasible space before execution to avoid plan failure. This module includes at least: (a) skill prerequisite check: verifying each skill k. Is it in the current state? (a) Semantic Map: (b) Task and Motion Planning (TAMP): Generate feasible motion trajectories for skill sequences, and check joint constraints, collisions, and accessibility; (c) Stability Constraint Check: For stages involving grasping, carrying, crossing, and interaction with external forces, evaluate stability margins based on support polygons and centroid projection; (d) Risk and Energy Consumption Assessment: Estimate energy consumption costs, time costs, and risk costs, and perform multi-objective screening. Stability constraints can be expressed in the following form: Let the centroid projection on the ground be... Supporting polygons are Define the distance function ;Require ,in This serves as a stability margin threshold. For dynamic gait, the zero-moment point (ZMP) or capture point constraint can be incorporated into the stability constraint set. .

[0159] The feasibility and safety verification module adopts a comprehensive cost ,in: Energy consumption cost Time cost Risk and cost The corresponding weighting coefficients are used to filter candidate plans. To accurately calculate each cost component in a real system, this embodiment provides the following specific definitions:

[0160] Energy consumption cost This reflects the energy consumed by a robot in executing a skill sequence. For legged locomotion-manipulation robots, energy consumption is mainly generated by the work done by the joint motors, and can be expressed as the integral of the power of each joint over time. ,in The number of skills in the skill sequence. For the first The estimated execution time for each skill The total number of joints. The first Each joint in skill Torque and angular velocity during execution. In practical applications, these can be quickly estimated using pre-stored energy consumption models in the skill library or regression models based on historical data.

[0161] Time cost This indicates the total estimated time required to complete the entire task, which can be directly summed by adding the estimated execution time of each skill: ,in Skills can be obtained from skill entries in a skill library (e.g., average skill execution time) or adjusted using an adaptive model based on the current environmental conditions (e.g., terrain complexity, obstacle density). Risk and Cost The assessment comprehensively evaluates the safety risks during task execution, including the probability of collision, fall, and object damage. This embodiment uses a weighted linear combination.

[0162] ;

[0163] in, The probability of colliding with the environment or itself can be estimated based on the collision detection results and uncertainty propagation model in motion planning; The probability of a robot becoming unstable and falling can be calculated using a statistical model of stability margin and dynamic disturbances. The probability of damage to the manipulated object or tool depends on the precision of the gripping force control and prior knowledge of the object's fragility. Weighting coefficients. It can be preset or adjusted according to the task scenario or through online learning.

[0164] Weighting coefficient Directly influencing the preference for plan selection, this embodiment proposes an adaptive adjustment strategy based on task status and resource constraints:

[0165] When the battery level is below a threshold At the same time, increase the energy consumption weight: ;

[0166] When the remaining time for a task is tight (e.g., the deadline is approaching), increase the time weight: ;

[0167] When the environmental risk level increases (e.g., entering a slippery area or approaching a dynamic obstacle), the risk weight is increased: .

[0168] in, As the benchmark weight, For adjustment coefficients, This is the current battery level. For the current time, Environmental risk level (between 0 and 1). The adjusted weights must meet normalization conditions, such as... This can be achieved by rescaling.

[0169] In this optional embodiment, by performing prerequisite verification, feasibility verification, safety constraint checks, and multi-objective cost calculations on hierarchical task plans based on standard entries in the skill library, the semantic plan can be mapped to the physical feasible space before execution. This eliminates infeasible solutions such as unreachable motion, insufficient stability, collision risks, and excessive energy consumption in advance. At the same time, it selects the target skill sequence with the optimal comprehensive cost. This not only avoids planning failure problems at the root but also explicitly incorporates stability, safety, efficiency, and energy consumption into the planning closed loop. This significantly improves the reliability and safety of long-term task execution, effectively reduces the failure rate, replanning frequency, and equipment damage risk during execution, and provides a safe, efficient, and feasible execution basis for subsequent skill orchestration and closed-loop execution.

[0170] Optionally, the step of arranging skills according to the target skill sequence to generate a skill execution scheduling strategy for the legged robot includes:

[0171] Based on the execution parameters, skill attributes, and sequential relationships of each skill in the target skill sequence, determine the execution priority and timing requirements of each skill.

[0172] Set the switching logic for each skill, which includes the triggering conditions and execution thresholds for the skill;

[0173] Based on the dynamic obstacle state and terrain constraints in the structured environment representation, an execution timing window and action adjustment margin are configured for each of the skills.

[0174] The execution priority, timing requirements, switching logic, execution timing window, and action adjustment margin are integrated to construct the basic scheduling framework for the skill, and the skill execution scheduling strategy is generated through the basic scheduling framework.

[0175] Specifically, firstly, based on the execution parameters, skill attributes, and sequential relationships of each skill in the target skill sequence, combined with the execution flow and safety priorities of long-duration tasks, the execution priorities of movement, operation, stability, and composite skills are determined, and the timing requirements for skill initiation, waiting, continuation, and mutual exclusion are clarified to ensure that skill invocation conforms to the sub-target advancement logic and overall control constraints. Secondly, a switching logic adapted to the leg-based movement-operation characteristics is set for each skill, clarifying the pre-start state triggering conditions, stability and contact judgment thresholds during execution, success criterion thresholds for execution completion, and abnormal exit thresholds, forming an automatically flowing state machine switching rule. Thirdly, based on the dynamic obstacle state sequence in the structured environment representation... The system considers topographical and dynamic constraints such as terrain slope, slip risk, and foothold areas to assign safe and feasible execution time windows to each skill, and reserves motion adjustment margins for posture correction, gait adjustment, force control compensation, and foothold offset to adapt to execution deviations caused by unstructured environments and dynamic interference. Finally, it unifies and integrates execution priority, timing connection requirements, switching logic, execution time windows, and motion adjustment margins to construct a basic scheduling framework covering skill scheduling, state switching, online correction, and anomaly reservation. Based on this framework, a skill execution scheduling strategy is formed that includes skill scheduling order, real-time triggering rules, online adjustment space, and anomaly handling boundaries to support the skill orchestrator in completing a two-layer closed-loop execution of macro replanning and micro correction.

[0176] In a preferred embodiment of the present invention, an online optimization method based on gradient descent is used to update parameters, and the specific steps are as follows:

[0177] S1: Deviation Definition: Setting Skills The expected output is The current actual output is Define the deviation vector For different skills, y represents the output quantity, which can be the end-effector pose, contact force, center of mass position, etc.

[0178] S2: Cost Function: Define the local cost function ,in, Representation skills The parameter vector, It is a positive definite weight matrix. This represents the transpose of the deviation vector. This represents the deviation vector.

[0179] S3: Parameter update law: ,in For learning rate, Represents the cost function For parameters gradient, This represents the assignment operator. , To represent the transpose of the Jacobian matrix, Represents the Jacobian matrix, Represents the weighted bias vector, Jacobian matrix It can be calculated analytically based on skill models (such as kinematics and dynamics) or approximated through numerical difference.

[0180] S4: Safety Constraint Projection: The updated parameters must satisfy the safety envelope constraint function of the skill. , Representation skills If the safety envelope constraint function violates the constraint after updating, it is mapped to the feasible region through a projection operation:

[0181] ;

[0182] in, To satisfy the set of parameters for a secure envelope, This represents the updated parameter vector. This represents the assignment operator.

[0183] S5: Termination condition: when The correction stops when the number of iterations exceeds the maximum value M; if the deviation still does not converge after N consecutive corrections, macro reprogramming is triggered (N can be 3~5).

[0184] The system calculates the execution deviation based on real-time sensor data and compares it with a preset threshold. Taking stability deviation as an example: let the desired centroid projection position be... The actual projection is Supporting polygons are Define stability margin .like If the safety threshold is reached, a stability risk is assessed, triggering micro-corrections to adjust gait parameters (such as stride length and fuselage height) to increase stability. Energy consumption deviation can be defined as the difference between the actual power integral and the expected value; contact force deviation is the difference between the measured force and the gripping force threshold. Correction is initiated when the normalized weighted sum of all deviations exceeds the threshold.

[0185] In this optional embodiment, by configuring the priority and timing of the target skill sequence, setting the switching logic, allocating the execution timing window and action adjustment margin, and integrating them into a complete scheduling framework and execution strategy, the skill execution can strictly conform to the strong coupling characteristics of foot-based movement and operation and the dynamic environmental constraints, achieving smooth connection, safe switching and online adaptive adjustment between skills. At the same time, it reserves flexible space for micro-parameter correction and macro-replanning, effectively suppressing error accumulation in long-term tasks, improving execution smoothness and robustness, and improving task execution efficiency while ensuring stability and security. It provides reliable, controllable and error-correctable scheduling support for two-layer closed-loop execution.

[0186] Optionally, the step involves driving the legged robot to execute skills according to the skill execution scheduling strategy; simultaneously, when the legged robot executes a skill, the skill execution scheduling strategy is corrected in real-time using real-time sensor data of the legged robot, including:

[0187] According to the skill execution scheduling strategy, drive the executor of each skill to work;

[0188] Simultaneously, when the legged robot performs a skill, real-time sensing data of the legged robot is acquired; the real-time sensing data includes the real-time pose, real-time joint torque, and real-time foot contact force of the legged robot.

[0189] The real-time pose, real-time joint torque, and real-time foot contact force are analyzed in real time to determine the execution deviation value of each skill.

[0190] Determine whether the execution deviation value of the skill is greater than or equal to a preset correction threshold;

[0191] If so, the skill execution scheduling strategy is dynamically corrected based on the real-time sensing data according to the preset correction rules until the execution deviation value is less than the preset correction threshold.

[0192] If not, the skill execution scheduling strategy remains unchanged.

[0193] Specifically, according to the generated skill execution scheduling strategy, control commands are output to the leg drive mechanism, end effector, and other actuators of the legged robot, driving each skill in the target skill sequence to execute in an orderly manner, ensuring that skill invocation, timing, and state switching all follow the rules set by the scheduling framework. Simultaneously, throughout the entire process of the legged robot executing skills, real-time sensor data is collected and acquired by the multi-source sensing and state estimation module. This real-time sensor data specifically includes real-time pose output by the IMU and pose estimation module, real-time joint torque feedback from the joint actuators, and real-time foot contact force collected by the foot tactile and force sensors, providing comprehensive low-level feedback for execution state monitoring. Subsequently, the acquired real-time pose, real-time joint torque, and real-time foot contact force are analyzed online and... The system calculates and compares the real-time state with the preset expected pose, torque limit, and contact force threshold in the skill execution scheduling strategy to determine the execution deviation value of the current skill in terms of position, attitude, force control, and contact state. Then, it compares the calculated execution deviation value with the system's preset micro-correction threshold to determine whether the deviation exceeds the allowable range. When the execution deviation value is greater than or equal to the preset correction threshold, the system immediately adjusts the skill execution scheduling strategy dynamically based on real-time sensor data according to the preset micro-correction rules, updating only the skill parameters without changing the high-level sub-targets and skill sequences, until the execution deviation value falls back to within the preset correction threshold. When the execution deviation value is less than the preset correction threshold, the current execution state is determined to be stable and controllable, the original skill execution scheduling strategy remains unchanged, and the task execution continues according to the original plan.

[0194] In this optional embodiment, the actuator is driven to complete the skill operation according to the scheduling strategy, and real-time sensor data such as pose, joint torque, and foot contact force are collected simultaneously. The execution deviation is analyzed in real time and compared with the preset threshold. When the deviation exceeds the limit, parameter-level dynamic correction is initiated, and the strategy is kept stable when the deviation is normal. This enables closed-loop monitoring and rapid fine-tuning of the entire skill execution process, timely suppression of small deviations such as position, force control, and posture, and avoids the continuous accumulation of errors in long-term tasks. At the same time, micro-correction is completed without changing the high-level semantics and skill structure, taking into account both execution stability and task efficiency. This significantly improves the robustness, safety, and task success rate of the legged robot in unstructured and dynamic environments.

[0195] Combination Figure 2 As shown, the legged robot task planning and control system of this invention includes:

[0196] The data acquisition unit is used to acquire task instructions and multi-source sensor data of the legged robot.

[0197] The data analysis unit is used to extract the task instructions, obtain the semantic features of the task instructions, and perform environmental object recognition, terrain accessibility analysis and dynamic obstacle tracking based on the multi-source perception data to construct a structured environment representation of the legged robot.

[0198] The retrieval unit is used to retrieve historical task experience fragments corresponding to the task instruction from a preset historical task database based on the semantic features.

[0199] The planning unit is used to generate a hierarchical task plan for the legged robot by using a language model, based on the historical task experience fragments, and in combination with the structured environment representation.

[0200] The verification unit is used to perform feasibility and safety verification on the hierarchical task plan to obtain the target skill sequence of the legged robot.

[0201] The orchestration unit is used to orchestrate skills according to the target skill sequence and generate a skill execution scheduling strategy for the legged robot.

[0202] An execution unit is used to drive the legged robot to perform skills according to the skill execution scheduling strategy; at the same time, when the legged robot performs skills, the skill execution scheduling strategy is corrected in real time in a closed loop based on the real-time sensor data of the legged robot.

[0203] This invention also provides a legged robot task planning and control system, which stores a computer program. When the computer program is executed by a processor, it implements the above-described legged robot task planning and control method.

[0204] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A task planning and control method for a legged robot, characterized in that, include: Acquire task instructions and multi-source sensor data from the legged robot; The task instructions are extracted to obtain their semantic features, and environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking are performed based on the multi-source perception data to construct a structured environment representation for the legged robot. The step of extracting the task instructions to obtain the semantic features of the task instructions includes: performing text preprocessing on the task instructions to obtain the preprocessed text of the task instructions; The preprocessed text is analyzed using a semantic feature extraction model to determine the task objectives, operation objects, action requirements, and constraint information in the task instructions. The task objective, the operation object, the action requirements, and the constraint information are vectorized and encoded to generate semantic feature vectors. Based on the semantic feature vector, the semantic features of the task instruction are obtained; The step of constructing a structured environment representation for the legged robot based on the multi-source perception data includes: environmental object recognition, terrain accessibility analysis, and dynamic obstacle tracking. The multi-source sensing data is subjected to time synchronization and external parameter calibration to obtain spatiotemporally consistent standardized sensing data; Based on the standardized perception data, environmental objects are identified to generate an object attribute set for the legged robot. The object attribute set includes the category, pose, and grasping parameters of the manipulated object. Based on the standardized perception data, terrain accessibility analysis is performed on the operating object to determine the terrain slope, step height, ground friction coefficient, and slip risk parameters within a preset area between the legged robot and the operating object. A semantic map of the preset area is generated based on the terrain slope, the step height, the ground friction coefficient, and the slippage risk parameter; Dynamic obstacle detection and tracking are performed based on the time series of the standardized sensing data to generate a dynamic obstacle state sequence. The object attribute set, the semantic map, and the dynamic obstacle state sequence are fused to obtain the structured environment representation of the legged robot within the preset area. Based on the semantic features, a search is performed in the preset historical task database to obtain the historical task experience fragments corresponding to the task instruction. Using a language model, based on the historical task experience fragments and combined with the structured environment representation, a hierarchical task plan for the legged robot is generated. The feasibility and safety of the hierarchical task plan are verified to obtain the target skill sequence of the legged robot. Skills are arranged according to the target skill sequence to generate a skill execution scheduling strategy for the legged robot; The legged robot is driven to perform skills according to the skill execution scheduling strategy; at the same time, when the legged robot performs skills, the skill execution scheduling strategy is corrected in real time in a closed loop based on the real-time sensor data of the legged robot.

2. The task planning and control method for a legged robot according to claim 1, characterized in that, The acquisition of task instructions and multi-source sensor data for the legged robot includes: The visual perception data, laser point cloud data, inertial data, wrist perception data, and foot tactile perception data of the legged robot are acquired. The visual perception data, the laser point cloud data, the inertial data, the wrist perception data, and the foot tactile perception data are used as the multi-source perception data, and the task instructions issued to the legged robot are obtained.

3. The task planning and control method for a legged robot according to claim 1, characterized in that, The step of retrieving historical task experience fragments corresponding to the task instruction from a preset historical task database based on the semantic features includes: The semantic features are normalized to obtain a standardized semantic feature vector. Based on the standardized semantic feature vector, a retrieval vector adapted to the retrieval rules of the historical task database is generated; Based on the retrieval vector, a cosine similarity matching retrieval is performed in a preset historical task database to obtain the historical task data; The historical task data is structured and parsed to determine skill strategies, skill parameters, failure types, and recovery strategies. The skill strategy, skill parameters, failure type, and recovery strategy are integrated to obtain the historical task experience fragment that matches the task instruction.

4. The task planning and control method for a legged robot according to claim 1, characterized in that, The step of generating a hierarchical task plan for the legged robot using a language model, based on historical task experience fragments and combined with the structured environment representation, includes: The historical task experience fragments, the structured environment representation, and the task instructions are input into the language model, wherein the language model includes a semantic planning sub-model and a parameter calculation sub-model; The semantic planning sub-model is used to break down the task objective in the task instruction into multiple stage sub-objectives corresponding to the task objective and multiple skills corresponding to each stage sub-objective. The skills are then integrated according to the order of the stage sub-objectives to obtain multiple skill sequences. The sub-model is calculated using the parameters. For each skill in the skill sequence, the execution parameters corresponding to each skill are calculated using a tool function based on the structured environment representation. Each skill sequence is associated with the execution parameters corresponding to each skill to obtain the hierarchical task plan corresponding to each skill sequence.

5. The legged robot task planning and control method according to claim 4, characterized in that, The feasibility and safety verification of the hierarchical task plan, to obtain the target skill sequence of the legged robot, includes: Obtain the standard skill entry corresponding to each skill in the skill sequence from the preset skill library. Each standard skill entry includes the prerequisites, effect definition, parameter interface and safety envelope of the skill. Based on the prerequisites, effect definitions, parameter interfaces, and security envelopes, the feasibility of the execution parameters of each skill in the hierarchical task plan is verified and security constraints are checked to obtain the comprehensive cost of the hierarchical task plan. The hierarchical task plan with the optimal overall cost value is taken as the target skill sequence.

6. The task planning and control method for a legged robot according to claim 5, characterized in that, The step of arranging skills according to the target skill sequence to generate a skill execution scheduling strategy for the legged robot includes: Based on the execution parameters, skill attributes, and sequential relationships of each skill in the target skill sequence, determine the execution priority and timing requirements of each skill. Set the switching logic for each skill, which includes the triggering conditions and execution thresholds for the skill; Based on the dynamic obstacle state and terrain constraints in the structured environment representation, an execution timing window and action adjustment margin are configured for each of the skills. The execution priority, timing requirements, switching logic, execution timing window, and action adjustment margin are integrated to construct the basic scheduling framework for the skill, and the skill execution scheduling strategy is generated through the basic scheduling framework.

7. The task planning and control method for a legged robot according to claim 6, characterized in that, The process involves driving the legged robot to execute skills according to the skill execution scheduling strategy; simultaneously, when the legged robot executes a skill, the skill execution scheduling strategy is adjusted in real-time based on the real-time sensor data of the legged robot, including: According to the skill execution scheduling strategy, drive the executor of each skill to work; Simultaneously, when the legged robot performs a skill, real-time sensing data of the legged robot is acquired; the real-time sensing data includes the real-time pose, real-time joint torque, and real-time foot contact force of the legged robot. The real-time pose, real-time joint torque, and real-time foot contact force are analyzed in real time to determine the execution deviation value of each skill. Determine whether the execution deviation value of the skill is greater than or equal to a preset correction threshold; If so, the skill execution scheduling strategy is dynamically corrected based on the real-time sensing data according to the preset correction rules until the execution deviation value is less than the preset correction threshold. If not, the skill execution scheduling strategy remains unchanged.

8. A task planning and control system for a legged robot, characterized in that, The legged robot task planning and control system, applied to the legged robot task planning and control method as described in claim 1, comprises: The data acquisition unit is used to acquire task instructions and multi-source sensor data of the legged robot. The data analysis unit is used to extract the task instructions, obtain the semantic features of the task instructions, and perform environmental object recognition, terrain accessibility analysis and dynamic obstacle tracking based on the multi-source perception data to construct a structured environment representation of the legged robot. The retrieval unit is used to retrieve historical task experience fragments corresponding to the task instruction from a preset historical task database based on the semantic features. The planning unit is used to generate a hierarchical task plan for the legged robot by using a language model, based on the historical task experience fragments, and in combination with the structured environment representation. The verification unit is used to perform feasibility and safety verification on the hierarchical task plan to obtain the target skill sequence of the legged robot. The orchestration unit is used to orchestrate skills according to the target skill sequence and generate a skill execution scheduling strategy for the legged robot. An execution unit is used to drive the legged robot to perform skills according to the skill execution scheduling strategy; at the same time, when the legged robot performs skills, the skill execution scheduling strategy is corrected in real time in a closed loop based on the real-time sensor data of the legged robot.

Images