A robotic task planning system, method and apparatus

Abstract

This invention discloses a robot task planning system, method, and apparatus, relating to the field of embodied intelligence, to improve the efficiency and success rate of long-term robot task planning. The system includes an expert skill library encapsulating multiple independent skill modules; a vector knowledge base maintaining historical scene data; a decision-making engine that receives and understands natural language instructions and executes instructions based on historical scene data; it determines the skill modules required to execute the instructions; during instruction execution, based on the current task progress, it activates only the currently needed skill modules, and immediately loads the necessary resources for the activated skill modules, including at least one of historical scene data, historical context data, and current scene data; the currently activated skill modules and their operation results are written into the historical context data. This invention significantly reduces the reasoning burden and improves the planning efficiency and execution success rate of long-term tasks.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a robot task planning system, method and apparatus. Background Technology

[0002] With the deep integration of Multimodal Large Language Model (MLLM) and Embodied AI technologies, robots are gradually becoming able to understand and execute human natural language commands.

[0003] Currently known technological applications typically encapsulate the robot's low-level APIs, such as perception, movement, and grasping, into atomic tools, which are executable tools at the fine-grained action level. After receiving natural language instructions, the system utilizes the powerful reasoning capabilities of the Large Language Model (LLM) (such as the ReAct framework) to determine which specific tool function to call in one or more rounds of dialogue. The model (agent) decomposes the complex instructions into a series of tool call sequences by reading the tool's name and parameter description. After calling the tool, it obtains the execution result and proceeds to the next round of action.

[0004] The above technical logic is as follows: perform global indexing based on the instruction, then generate the overall action sequence for executing the instruction, then perform overall action planning on the overall action sequence, and finally execute the planned overall action commands in sequence to control the corresponding robot hardware facilities.

[0005] While the above technical approach can achieve the understanding and execution of natural language instructions, it has the following shortcomings: (1) Redundancy in reasoning computation: When the system calls the tool, it only focuses on the function that the tool focuses on. However, when calling the tool, the model cannot grasp the global context data (i.e. the operations that have been performed and the results of the operations). In the separate tool call process, the existing context data cannot be reused, and repeated reasoning is required. The token consumption cost is high, and it will also cause reasoning delay.

[0006] (2) Context fragmentation: The overall action planning is carried out in advance. The calling of tools is independent of the overall action sequence planning of the model. The intermediate results generated by calling tools (such as sensors, actuators, APIs) cannot be fed back to the decision model in real time, causing the "perception-action" closed loop to break.

[0007] (3) Context pollution: Since the model needs to maintain the overall action plan at all times, including the tools to be called and the full amount of data; all tools need to be activated and called synchronously, which will generate a large number of unnecessary tool commands, pollute the context data, and cause instruction deviation or even target loss in the execution of long-term tasks, which can easily lead to the failure of long-term task execution.

[0008] (4) Heavy inference burden: After the overall action sequence is planned, all the tools that need to be called are activated. Regardless of the task progress, the model always needs to perceive and process scene data from the overall action planning level (i.e., load all data in real time), resulting in a heavy inference burden and high inference latency. Summary of the Invention

[0009] The purpose of this invention is to provide a robot task planning system, method, and apparatus to address all or part of the problems mentioned above, thereby improving the efficiency and success rate of long-term robot task planning.

[0010] The technical solution adopted in this invention is as follows: In a first aspect, this application provides a robot task planning system, comprising: The expert skill library encapsulates multiple independent skill modules, and each skill module is configured with a state machine that indicates whether it is active. Vector knowledge base, maintaining historical scenario data; The decision-making brain receives and understands natural language instructions, and implements the instructions based on the historical scene data; it determines the skill modules that need to be invoked to implement the instructions; during the implementation of the instructions, it activates only the currently required skill modules according to the current task progress, and loads the required resources for the activated skill modules in real time, the resources including at least one of historical scene data, historical context data, and current scene data; the currently activated skill modules and the operation results of the current skill modules are written into the historical context data.

[0011] Optionally, the expert skill base may include at least the following skill modules: The task execution bus module defines a standardized four-stage protocol from instruction reception, knowledge retrieval, hierarchical planning to closed-loop execution. The intent parsing and knowledge screening module parses the received instructions and, based on the parsing results, retrieves the corresponding memory entries from the loaded historical scene data. The memory screening module removes outdated and redundant data from memory entries based on predefined discrimination rules, and selects valid memory data. The feasibility assessment module evaluates the feasibility of effectively storing data. The step planning module, based on effective memory data, initially plans a macro-level sequence of steps with temporal logic; The motion orchestration module generates local motion commands for the current step based on the current resources. The status monitoring and self-healing module monitors the execution status of the current step in real time and triggers remedial measures when the current step fails.

[0012] Optionally, the process indicated by the standardized four-phase protocol includes: Instruction reception: Receive the instruction; Knowledge retrieval: The process involves invoking the intent parsing and knowledge coarse screening module, the memory fine screening module, and the feasibility assessment module; using the intent parsing and knowledge coarse screening module to initially filter memory entries from the loaded historical scene data; using the memory fine screening module to filter out valid memory data from the memory entries; and using the feasibility assessment module to evaluate the feasibility of the valid memory data. Hierarchical planning: Invoke the step planning module and the action orchestration module; initially plan the macro step sequence through the step planning module; extract each step sequentially from the macro step sequence, and generate the local action command for the current step through the action orchestration module when implementing each step; Closed-loop execution: Invoke the status monitoring and self-healing module; during the implementation of each step, monitor the execution status of the current step in real time through the status monitoring and self-healing module, and trigger remedial measures when the current step fails.

[0013] Optionally, the decision-making brain determines the skill modules to be invoked to implement the instructions based on the following configuration: During the execution of the instructions, the current stage of the task progress is determined; based on the description information of each skill module in the expert skill library, the skill module that needs to be invoked is determined.

[0014] Optionally, the decision-making brain implements the instructions based on the historical scenario data according to the following configuration: Based on the understood instructions, determine whether to invoke the task execution bus module; if so, execute the following process: Activate the task execution bus module and load the standardized four-stage protocol into the historical context data; According to the process indicated by the standardized four-stage protocol, the intent parsing and knowledge coarse screening module and the memory fine screening module are activated sequentially to retrieve valid memory data from the historical scene data; the feasibility assessment module is activated to assess the feasibility of the valid memory data. Furthermore, after the feasibility of valid memory data is assessed, the step planning module is activated to plan the macro-level step sequence; Furthermore, each step is extracted sequentially from the macro-step sequence. When implementing each step, the action orchestration module is activated to generate the local action command for the current step, and the status monitoring and self-healing module is activated to monitor the execution status of the current step in real time and trigger remedial measures when the current step fails to execute.

[0015] Optionally, the remedial measures include in-step local self-healing and global replanning; The status monitoring and self-healing module is equipped with a first counter, which counts the number of times local self-healing is triggered within the triggering step. If the first counter has not overflowed when the current step fails, the monitoring and self-healing module triggers local self-healing within the step; if the first counter overflows, it triggers global replanning.

[0016] Optionally, triggering local self-healing within a step includes: calling the action orchestration module to regenerate the local action commands for the current step; triggering global replanning includes: returning to the knowledge retrieval stage.

[0017] Optionally, the decision-making brain, vector knowledge base, and expert skill base are connected via the Model Context Protocol (MCP) interface.

[0018] In a second aspect, this application also provides a robot task planning method, which includes: Receive and understand natural language instructions, determine the independently encapsulated skill modules to be invoked from the expert skill library to implement the instructions, and each skill module is configured with a state machine indicating whether it is activated; The instructions are executed based on historical scene data. During the execution of the instructions, only the currently required skill modules are activated according to the current task progress, and the required resources are loaded for the activated skill modules in real time. The resources include at least one of historical scene data, historical context data, and current scene data. The currently activated skill modules and the operation results of the current skill modules are written into the historical context data.

[0019] In a third aspect, this application also provides a robot task planning apparatus, which includes a processor and a storage medium, the storage medium storing a computer program, the processor running the computer program to execute the robot task planning method described above.

[0020] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: This application independently encapsulates skill modules (i.e., expert skill tools) and configures state machines for each skill module. During reasoning, the decision-making agent can dynamically load / activate new skill modules in a "plug-and-play" manner, without needing to pre-load global tools. Furthermore, this system architecture avoids strongly binding tools to decision-making, decoupling high-level decision logic from low-level perception / execution hardware, thus improving migration efficiency and versatility across heterogeneous platforms.

[0021] When loading a skill module, the decision-making brain can activate only the currently needed skill module and load only the resources required by the current skill module, thereby shielding it from interference from other irrelevant skill modules. It focuses solely on the input data and output results of the currently activated skill module, avoiding instruction deviations or target loss caused by excessive context data during long-term task execution, and improving the success rate of long-term task execution.

[0022] The decision-making brain does not need to activate global tools in real time, nor does it need to load all data in real time, which greatly reduces the reasoning burden, improves reasoning efficiency, and reduces reasoning latency. By maintaining historical context data, it avoids repeated reasoning operations in complex physical interactions and avoids the repeated consumption of tokens. Attached Figure Description

[0023] The present invention will be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a construction diagram of a robot task planning system in one embodiment.

[0024] Figure 2 It is a standardized four-stage flowchart.

[0025] Figure 3 It is a sequence diagram for implementing the standardized four-stage process. Detailed Implementation

[0026] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0027] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0028] To address the shortcomings of current technologies that employ global planning and execution approaches in terms of inference burden, inference efficiency, and success rate, this application proposes a robot task planning system. This system aims to significantly reduce the inference burden of robot task planning, decrease inference costs and latency, and improve the success rate of long-term task execution.

[0029] like Figure 1 As shown, the robot task planning system proposed in this application includes: (1) Expert Skills Database.

[0030] The expert skill library encapsulates multiple independent skill modules. The library includes descriptive information for the skills in all modules, essentially a "semantic summary" of each module's skills. For example, a skill module might be described as "This module is responsible for handling complex cross-room grabbing and movement tasks." Independence means that the specific functional context information and local constraints contained within each skill module are encapsulated within the module itself, preventing overlapping definitions. Furthermore, each skill module is configured with a state machine that indicates its active state. Changing the state machine activates or deactivates the skill module.

[0031] The decision-making brain pre-obtains descriptive information of all skill modules, rather than the complete execution script of specific skill modules. In the preliminary reasoning and analysis stage before receiving instructions and preparing to implement them, it performs semantic alignment between the semantics of the user's instructions (such as "go to the study" or "grab") and the descriptive information. This allows it to discover the most matching skill module at the perceptual level and then decide whether to activate the skill module and which skill module to activate.

[0032] In known technical approaches, implementing instructions involves global planning of the entire action sequence, requiring the simultaneous loading of all necessary tools. Consequently, none of these tools are configured with state machines, nor is it necessary. This leads to the potential for cross-contamination of context data generated by different tools during long-term tasks; furthermore, the model needs to load all data throughout the entire process, resulting in an extremely heavy inference burden. The expert skill library constructed above differs significantly from known tool construction methods. When executing task instructions, it only loads the description information of each skill module in the skill library, autonomously selecting which skill module to load to implement the instructions. This achieves "plug and play, use and unload," realizing physical isolation between technical modules. Unloaded / closed skill modules do not load resources, effectively reducing the inference burden on the decision-making brain.

[0033] As an optional implementation, the expert skill base shall include at least the following skill modules: 1) Task Execution Bus Module. Defines a standardized four-stage protocol from instruction reception, knowledge retrieval, hierarchical planning to closed-loop execution.

[0034] 2) Intent parsing and knowledge screening module. Parses the received instructions and, based on the parsing results, retrieves the corresponding memory entries from the loaded historical scene data.

[0035] For example, for natural language instructions, semantic encoding and other methods are used to decompose fuzzy natural language instructions into structured tags, and preliminary relevant multimodal memory entries are retrieved / recalled from the vector knowledge base.

[0036] 3) Memory screening module. Based on predefined discrimination rules, outdated and redundant data are removed from memory entries, and valid memory data is selected.

[0037] The so-called discrimination rule refers to the rule used to filter the validity of recalled memory entries. As an optional implementation, this discrimination rule is a spatiotemporally aware discrimination criterion, such as the ID minimization / maximization criterion (corresponding to the numbering rule of minimum / maximum ID of the latest snapshot). For example, if multiple memory entries are recalled for the same target object, the closer the time, the smaller the ID of the memory entry. According to the ID minimization criterion, the memory entry with the smallest ID (i.e., the most recent time) is retained as valid memory data. This ensures the validity of the referenced historical memory data.

[0038] 4) Feasibility assessment module. Assess the feasibility of effectively storing data.

[0039] For feasibility assessment, evaluation can be conducted from dimensions such as the completeness, coordinates, category, and confidence level of the valid memory data. For example, if the implementation instruction explicitly mentions target objects A and B, but the selected valid memory data only contains A and lacks B, then the valid memory data is incomplete and insufficient to support the implementation of the instruction. Alternatively, if the selected valid memory data also contains B, but its identification confidence level is only 40%, lower than the evaluation threshold (e.g., 80%), then the confidence level of the valid memory data is insufficient to support the implementation of the instruction. The judgment of other evaluation indicators follows the same principle.

[0040] A feasibility assessment of effective memory data can prevent blind planning from being triggered when perceptual information is incomplete.

[0041] 5) Step planning module. Based on the effective memory data, a preliminary macro-level step sequence with temporal logic is planned.

[0042] The macro-level step sequence contains all the steps required to complete the instruction, and these steps have strict temporal logic (such as sequential order, coordination rules, etc.). Furthermore, these steps only indicate the macro-level execution path of the task and do not involve specific execution actions. For example, a step might be "pick up target object A," but it doesn't specify how to pick up target object A.

[0043] 6) Motion orchestration module. Generates local motion commands for the current step based on the current resources.

[0044] Local action commands are operation commands that instruct the specific implementation of the current step, responsible for translating macro-level steps into low-level control commands. As in the example above, a local action might be a sub-step of "picking up target object A," but it still doesn't involve how to actually pick up target object A.

[0045] For each step, the action orchestration module is activated separately, and the reasoning of local action commands is performed only in conjunction with the resources required for that step (such as instantaneous perception data).

[0046] 7) Status monitoring and self-healing module. Monitors the execution status of the current step in real time and triggers remedial measures when the current step fails.

[0047] The above definitions of each skill module, i.e., the template descriptions for each skill module, include the "contextual information" (implementation scripts, etc.) and "local constraints" (implementation logic constraints, etc.) involved in the implementation of the corresponding skill by the skill module. Furthermore, the skills possessed by each skill module are used by the decision-making brain to issue corresponding commands and execute corresponding steps during the instruction execution process, such as "picking up target object A" as mentioned above, which informs the robot of the command to be performed. How the command is specifically executed is determined by the robot itself according to its configuration; this application only involves issuing commands to the robot and perceiving the results of the robot executing the corresponding commands.

[0048] In known technical approaches, all actions are planned holistically. Scenario data from intermediate stages cannot be fed back to the model's decision-making logic in real time. Furthermore, it's impossible to update only a portion of the actions while implementing the overall plan; the entire sequence of actions must be replanned. This approach is prone to causing long-term task failures if intermediate stages deviate from expectations.

[0049] To improve the success rate of task execution, and in conjunction with the underlying architecture where the skill module can be activated independently, the step planning module does not involve planning specific actions, and the action choreography module is responsible for independently coding each step, this application, in an optional implementation, allows the status monitoring and self-healing module to perform local self-healing within the steps in the triggered remedial measures, that is, only correcting the action of a single step without re-planning the overall action.

[0050] Specifically, the remedial measures include in-step local self-healing and global replanning. The status monitoring and self-healing module is configured with a first counter, which counts the number of times in-step local self-healing is triggered. When the current step fails, if the first counter has not overflowed, the monitoring and self-healing module triggers in-step local self-healing; if the first counter overflows, it triggers global replanning.

[0051] The first counter is pre-configured with an overflow threshold, i.e., a maximum count value. If this overflow threshold is exceeded, it indicates a state overflow, which also means that the number of corrections to the local action command has reached the maximum allowed value. It is determined that further corrections may not be able to complete the step, so the local self-healing within the step is abandoned, triggering a global replanning, i.e., replanning the macroscopic step sequence.

[0052] In one optional implementation, the aforementioned triggering step involves local self-healing, including: invoking the action orchestration module to regenerate the local action commands for the current step. The aforementioned triggering of global replanning includes: returning to the knowledge retrieval stage, i.e., re-executing the knowledge retrieval, hierarchical planning, and closed-loop execution process. It should be noted that as instructions are implemented, historical scene data is updated; therefore, the retrieved valid memory data may alter the planned macro-level step sequence, thereby changing the subsequently planned local action commands and the execution result.

[0053] The aforementioned task execution bus module is responsible for instructing the decision-making brain to implement standardized procedures. The standardized four-stage protocol serves as the underlying logical framework for the decision-making brain to execute long-term tasks. Upon activation of the task execution bus module, the decision-making brain performs reasoning based on the four-stage process indicated by the standardized four-stage protocol.

[0054] In one alternative implementation, such as Figure 2 As shown, the process indicated by the standardized four-phase protocol defined by the task execution bus module includes: Phase 1: Command Reception: Receiving commands.

[0055] The instructions here can be understood as raw natural language instructions, or as instructions that have undergone preliminary reasoning by the decision-making brain. For the latter, after preliminary reasoning, the decision-making brain can better understand the type of instruction, such as whether it instructs the robot entity to perform a specific operation on a target object, thereby deciding whether to activate the standardized four-stage process. Therefore, the instructions received here can be understood as instructions that have undergone preliminary reasoning by the decision-making brain and have decided to activate the standardized four-stage process.

[0056] Phase Two: Knowledge Retrieval: This phase involves invoking the Intent Resolution and Knowledge Coarse Screening module, the Memory Fine Screening module, and the Feasibility Assessment module. The Intent Resolution and Knowledge Coarse Screening module initially filters memory entries from the loaded historical scene data. The Memory Fine Screening module then selects valid memory data from the memory entries. Finally, the Feasibility Assessment module evaluates the feasibility of the valid memory data.

[0057] By recalling historical memory entries through the intent parsing and knowledge coarse screening modules, the problem of context fragmentation can be effectively avoided. The refined screening module effectively eliminates outdated / invalid and redundant historical memory entries, locking in the latest (visual, etc.) evidence covering all key objectives. The feasibility assessment module evaluates the operability of the selected valid memory data, preventing the blind triggering of hierarchical planning operations when perceptual information is incomplete, further improving the success rate of long-term task execution.

[0058] Phase 3: Hierarchical Planning: Invoking the step planning module and the action orchestration module. The step planning module initially plans the macro-level step sequence. Each step is extracted sequentially from this macro-level step sequence. When implementing each step, the action orchestration module generates the local action command for the current step.

[0059] In other words, in this embodiment, instead of planning the overall action level of the long-term task in advance, it first plans only the sequence of macro steps (without involving specific actions), and then activates the corresponding skill modules to choreograph / plan the specific actions when implementing each step. This effectively avoids contextual pollution, reduces the reasoning burden on the decision-making brain, and improves reasoning efficiency.

[0060] Phase 4: Closed-Loop Execution: Invoking the Status Monitoring and Self-Healing Module. During the implementation of each step, the status monitoring and self-healing module monitors the execution status of the current step in real time, and triggers remedial measures if the current step fails.

[0061] (2) Vector knowledge base.

[0062] The vector knowledge base is used to maintain historical scene data. The so-called historical scene data is the background data for implementing instructions, which includes historical scene images, semantic maps, and object semantic feature vectors (coordinates, categories, confidence levels, etc.).

[0063] Specifically, a vector knowledge base is a non-relational database specifically designed for storing, indexing, and retrieving high-dimensional vector (embedded) data. Unlike traditional key-value databases, its core capability is to transform unstructured historical scene data into numerical vectors using deep learning models (such as pre-trained models like BERT). By calculating cosine similarity or Euclidean distance in the vector space, the decision-making system can achieve "fuzzy matching" and rapid semantic retrieval of historical scene data, thereby providing the most relevant background support for the execution of current instructions.

[0064] 1. Semantic Map.

[0065] Definition: Semantic maps are enhanced maps that assign "attribute labels" to spatial regions or objects, based on traditional geometric topological maps.

[0066] Its role in the vector knowledge base: It abstracts the physical environment in which the robot is located, enabling the decision-making brain to understand spatially constrained instructions such as "perform the task near the sofa".

[0067] 2. Object Semantic Features.

[0068] Definition: A data set that provides multi-dimensional attribute descriptions of specific entity objects identified in a scene. It typically includes the following four core elements: Spatial coordinates: The specific position (x, y, z) and orientation information of an entity in the world coordinate system or local coordinate system.

[0069] Category: The target classification after recognition by visual algorithms (such as "wrench", "robotic arm", "person").

[0070] Confidence: A numerical assessment of the accuracy of the recognition result by the perception system (usually a floating-point number between 0 and 1).

[0071] Feature Vector: A deep, high-dimensional vector representing the visual or geometric features of an object, extracted by a neural network (any neural network that can be vectorized, such as the BERT model used in the previous examples), used for accurate comparison in the database.

[0072] Vector knowledge bases are used to provide descriptive information about target objects for implementing instructions, such as the target object's location and state. Historical scene images in the vector knowledge base can be updated periodically to improve the effectiveness of descriptions of historical scene images, semantic maps, and object semantic features, thereby supporting the success rate of long-term task execution.

[0073] (3) Decision-making brain.

[0074] The decision-making brain is an intelligent agent object, encapsulated using LLM (Limited Language Model). As the core module of the system, the decision-making brain is responsible for receiving and understanding natural language instructions, implementing these instructions based on historical scenario data, and determining the skill modules that need to be invoked to implement the instruction.

[0075] It is particularly important to note that during the execution of instructions, the decision-making brain activates only the currently required skill modules based on the current task progress, and loads the necessary resources for the activated skill modules in real time. These resources include at least one of the following: historical scene data, historical context data, and current scene data, determined according to the data required by the skill module.

[0076] Because a state machine is configured separately for each skill module in this application, each skill module can be activated / deactivated independently. The decision-making brain calls and activates skill modules on demand, and loads the necessary data resources only for the currently activated skill module. This on-demand activation and data loading method greatly reduces the reasoning burden of the decision-making brain, improves reasoning efficiency, reduces reasoning latency, avoids context pollution, and improves the success rate of cross-scenario long-term task execution.

[0077] Furthermore, in the embodiments of this application, the decision-making brain does not simultaneously decide which skill module to invoke, but rather, after invoking and executing the previous skill module, it determines the next skill module to invoke based on historical context data (the loaded standardized four-stage protocol process).

[0078] Specifically, the decision-making brain determines the skill modules that need to be invoked to implement instructions based on the following configuration: During the execution of instructions, determine the current stage of the task progress; based on the description information of each skill module in the expert skill library, determine the skill module that needs to be invoked.

[0079] Based on the above design, as an optional implementation method, such as Figure 3 As shown, the decision-making brain executes instructions based on historical scenario data according to the following configuration: Based on the understood instructions, decide whether to invoke the task execution bus module. If so, execute the following process: (1) Activate the task execution bus module and load the standardized four-stage protocol into the historical context data.

[0080] (2) According to the process indicated by the standardized four-stage protocol, the intent parsing and knowledge coarse screening module and the memory fine screening module are activated in sequence to retrieve effective memory data from historical scene data; the feasibility assessment module is activated to assess the feasibility of effective memory data.

[0081] (3) After the feasibility of valid memory data is assessed, the step planning module is activated to plan the macro-step sequence.

[0082] (4) Extract each step sequentially from the macro step sequence. When implementing each step, activate the action orchestration module to generate the local action command for the current step, activate the status monitoring and self-healing module to monitor the execution status of the current step in real time, and trigger remedial measures when the current step fails to execute.

[0083] In addition, the currently activated skill module (the operation performed) and the operation result of the current skill module are written into the historical context data to provide effective supporting data for the reasoning of subsequent skill modules and to avoid repeated reasoning of duplicate content.

[0084] As another key design feature in this application embodiment, the decision-making brain, vector knowledge base, and expert skill base are connected through the Model Context Protocol (MCP) interface.

[0085] The MCP interface, serving as the underlying cross-modal data bus and standardized communication protocol stack, is built upon the Model Context Protocol. By defining a unified data interaction schema, the MCP interface achieves deep decoupling between the expert skill base and underlying resources (including historical scene data, historical context data, current scene data, and other data resources related to storage, perception, and execution).

[0086] Specifically, the MCP interface is responsible for the following functions in this application system: (1) Resource Encapsulation: Using the MCP interface protocol, dispersed backend resources (including but not limited to vector knowledge base (storing historical scene images, object semantic features, etc.), real-time sensor streams (visual, tactile, SLAM map and other current scene data) and external hardware communication interfaces) are encapsulated into standardized indexable resources.

[0087] (2) Cross-module Communication Bridge: Serves as a standard communication gateway between the decision-making brain and external modules (such as ROS (Robot Operating System) nodes for navigation, control, and voice) as well as third-party perception modules. The MCP interface converts the logical instructions generated by the upper-level skill modules into requests that conform to the MCP standard, and captures the execution status signals (Feedback) fed back from the underlying robot hardware in real time, and sends them back to the decision-making brain.

[0088] (3) Data Standardization and Alignment: Standardize the format of data from different sources. For example, align image data from different perspectives and timestamps on the timeline to ensure that the information provided to the expert skill base has logical consistency and spatiotemporal relevance.

[0089] Based on the underlying design of the MCP interface described above, the core workflow of the expert skill base working in conjunction with the MCP interface includes: Step 1: Skill module activation and on-demand handshake.

[0090] When the decision-making brain decides to activate a certain skill module (such as the memory screening module) based on the task progress, the system does not directly load all the full data.

[0091] The function of the MCP interface: The decision-making brain issues instructions through the MCP interface to load only the resource schema related to the (activated) skill module.

[0092] Function of the skill module: When the skill module is activated, its internal professional prompts and local constraints are loaded.

[0093] Step 2: Targeted Context Routing.

[0094] The role of the MCP interface is as a data bus. Based on the current skill requirements, MCP accurately retrieves specific fragments (such as the latest image with the target object label) from the underlying perception layer (vector knowledge base).

[0095] The function of the skill module is to process data. Using its internal spatiotemporal discrimination criteria, the skill module performs secondary reasoning and discrimination on the data (memory entries) transmitted from the MCP interface, retaining only information crucial to the current plan (i.e., effective memory data).

[0096] Step 3: Progressive Disclosure.

[0097] The function of the skill module: The action choreography module only choreographs the actions to be done "next". It does not load the details of the global plan (i.e., the macro step sequence), but only requests resources such as real-time point cloud or visual stream at the current position through the MCP interface.

[0098] The function of the MCP interface is to send the generated local action commands to the robot hardware (i.e., physical actuators) through a standardized message format, and to listen for and monitor the execution status feedback in a timely manner.

[0099] Step 4: Status Feedback and Self-Healing Loop.

[0100] The function of the MCP interface is to capture the success / fail signals (i.e., execution status) of the robot hardware in real time and convert them into a standardized feedback structure to be sent back.

[0101] The function of the skill module is to read this feedback signal and determine, through the internal first counter, whether to retry in place (local self-healing within the step - internal self-healing) or to request a reset of the global context for replanning through the MCP interface (global replanning - external self-healing).

[0102] Example 1 To facilitate the interpretation of the technical solution, this application provides the following specific embodiments to introduce the operation process of the robot task planning system.

[0103] 1. Initial intent triggering and intent parsing.

[0104] The user gave a natural language command: "Go to the study and bring the red teddy bear to the sofa." Skill Activation: The intelligent agent configured by the decision brain performs reasoning analysis on the user's input instructions. For example, by reasoning analysis on the user instruction "go to the study and take the red teddy bear to the sofa," the model reasoning result is: "The user instruction instructs the robot entity to complete the operation of the target object. The instruction contains both 'spatial displacement (study, sofa)' and 'entity operation (take the red teddy bear),' which exceeds the scope of simple question and answer or a single API call, triggering the semantic requirement of 'long-range task flow.' Therefore, based on the description information of all skill modules in the expert skill library, 'task execution bus module activation' should be selected, and the complete standardized four-stage protocol of the entity robot's specific operation task flow should be loaded into the context data of the decision brain." The decision brain loads the skill module into the context data, and then performs reasoning again, calling the script configured in the expert skill library to load the standardized four-stage protocol information into the long-term memory of the decision brain.

[0105] After the previous skill module execution is completed, the decision-making brain continues to load all context data, skill module description information, and current execution status for reasoning and analysis. At this time, the context data contains standardized four-stage protocol information for the robot to complete the user's natural language commands. Therefore, the decision-making brain will choose to activate the "intent parsing and knowledge coarse screening module" according to the process. By combining the encapsulation information of the skill module, the decision-making brain analyzes the key information items (tags) that need to be retrieved from the vector knowledge base, calls the script that can be called as specified by the "intent parsing and knowledge coarse screening module", and requests historical scene data from the MCP interface module.

[0106] MCP Interface Operation: The Decision Brain module retrieves memory entries related to key information entries through the MCP interface module. For example, this module requests the vector knowledge base to retrieve memory entries with tags ["red_toy_bear","study_room","sofa"].

[0107] The MCP interface module outputs the following: Suppose five historical visual memories (i.e., memory entries) are retrieved from the vector knowledge base, including related memory images of a teddy bear on the desk, a teddy bear next to the bookshelf, and a sofa in the living room. The MCP interface module also adds the retrieved related memory information to the decision-making brain's long-term memory (i.e., contextual data).

[0108] 2. Precise memory filtering and spatiotemporal alignment.

[0109] Skill Activation: Based on the current task execution status (i.e., current scene data), historical context data, and description information of all skill modules, the decision brain selects and loads all logical information from the "Memory Screening Module" and adds the operation and results of the previous skill module to the historical context data of the decision brain at this time. The decision brain then analyzes and filters based on the historical context data.

[0110] Example of core logic: This skill module compares and selects five memory entries from the historical context data. It finds ID:001 (taken 2 minutes ago) showing a teddy bear on the desk, while ID:045 (taken yesterday) shows a teddy bear on the bookshelf. ID:003 shows a sofa memory image, directly related to the placement of the red teddy bear in the instruction. Based on the spatiotemporal awareness criterion, it selects the memory images [001, 003] related to the instruction, returns these two IDs, and calls the corresponding script to add the finally selected effective memory data that can assist in generating the global plan to the historical context data.

[0111] 3. Feasibility verification and overall planning.

[0112] The decision-making brain continues to combine the historical context data (operations and results) of previous skill modules with all descriptive information in the expert skill library to make decisions. It selects to load the "Feasibility Assessment Module," analyzes image ID:001, confirms that the teddy bear's bounding box is clear, and also determines that image ID:003 contains a sofa. By activating this skill module, the decision-making brain performs reasoning analysis on the filtered valid memory data, determining that these two memory images can support the completion of the corresponding instructions.

[0113] Next, the decision-making brain continues to select skill modules, activates the "step planning module," loads the operation information of the skill module into the decision-making brain's historical context data, generates a macro-step sequence for the selected memory image, for example: ["access(red_bear_toy)", "pickup(red_bear_toy)", "access(sofa)", "release(red_bear_toy)"], and calls the corresponding script to add the generated macro-step sequence to the historical context data.

[0114] 4. Atomic loop execution.

[0115] After generating the macro-level step sequence, the decision-making brain obtains the global plan. At this point, based on the current task execution status and historical context data, the decision-making brain selects to activate the "action orchestration module" and issues each step command in stages. For each unique step command, the decision-making brain obtains the scene data currently faced by the robot through the MCP interface module. The decision-making brain combines the current step command to generate detailed local action commands and issues the generated local action commands to the robot hardware such as navigation and control through the MCP interface module, while listening for feedback information during task execution.

[0116] 5. Implement monitoring and closed-loop self-healing.

[0117] For example, when attempting to grasp an object, the robotic arm failed to do so due to changes in lighting.

[0118] MCP Interface Feedback: The MCP interface module captures the "grab_failure" signal returned by the underlying robot hardware, encapsulates it as a standard feedback, and sends it back to the decision-making brain. The decision-making brain analyzes the current scene data and feedback information to make one of the following two decisions: Self-healing decision: The decision-making brain loads the "status monitoring and self-healing module" and the current scene data indicating the current task execution status to make a decision, and chooses one of the following two methods for instruction execution self-healing: Inner loop self-healing: The current count value of the first counter is 0, which has not reached the upper limit (let's assume it's 2). The decision-making brain continues to issue step instructions to the action choreography module, such as: "Keep the plan, fine-tune the posture and try again."

[0119] Outer loop trigger: If two retries (adjustable) still fail (e.g., the teddy bear falls to the ground), the counter overflows (exceeding the upper limit of 2 times), and the decision brain will call the "intent parsing and knowledge screening module" again to rescan the historical scene data, so as to start the global hierarchical planning, perform memory retrieval on the latest vector knowledge base of the current object, and regenerate the macro step continuation and action commands according to the user's instructions.

[0120] Designed using the above method, this application has the following characteristics: (1) Extremely compresses inference cost. Through the dynamic context injection mechanism, it significantly reduces the token consumption and inference latency of the agent in complex physical interactions.

[0121] (2) Solving “cognitive drift” and “logical illusion”: Through modular expert skills, physical isolation of reasoning logic is achieved, effectively avoiding instruction deviation or target loss caused by excessive context data in long-term tasks.

[0122] (3) "Plug and play" of skill modules in heterogeneous environments: decouples high-level decision-making logic from low-level perception / execution hardware, improving the system's migration efficiency and versatility across heterogeneous platforms.

[0123] Based on the ideas of this application, embodiments of this application also provide a robot task planning method, which includes: Receive and understand natural language instructions, determine the independently encapsulated skill modules to be invoked from the expert skill library to implement the instructions, and each skill module is configured with a state machine indicating whether it is activated; The instructions are executed based on historical scene data. During the execution of the instructions, only the currently required skill modules are activated according to the current task progress, and the required resources are loaded for the activated skill modules in real time. The resources include at least one of historical scene data, historical context data, and current scene data. The currently activated skill modules and the operation results of the current skill modules are written into the historical context data.

[0124] Furthermore, this application also provides a robot task planning device, which includes a processor and a storage medium storing a computer program. The processor runs the computer program to execute the robot task planning method described in the above embodiments.

[0125] It should be noted that the steps designed in the above method can be optimized by referring to the relevant features in the system embodiments described above.

[0126] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A robot task planning system, characterized in that, include: The expert skill library encapsulates multiple independent skill modules, and each skill module is configured with a state machine that indicates whether it is active. The expert skills base shall contain at least the following skills modules: The task execution bus module defines a standardized four-stage protocol from instruction reception, knowledge retrieval, hierarchical planning to closed-loop execution. The intent parsing and knowledge screening module parses the received instructions and, based on the parsing results, retrieves the corresponding memory entries from the loaded historical scene data. The memory screening module removes outdated and redundant data from memory entries based on predefined discrimination rules, and selects valid memory data. The feasibility assessment module evaluates the feasibility of effectively storing data. The step planning module, based on effective memory data, initially plans a macro-level sequence of steps with temporal logic; The motion orchestration module generates local motion commands for the current step based on the current resources. The status monitoring and self-healing module monitors the execution status of the current step in real time and triggers remedial measures when the current step fails. The remedial measures include local self-healing within the step and global replanning. The status monitoring and self-healing module is equipped with a first counter, which counts the number of times local self-healing within the step is triggered. If the monitoring and self-healing module fails to execute the current step, and the first counter has not overflowed, it triggers local self-healing within the step, including: calling the action orchestration module to regenerate the local action command for the current step; if the first counter overflows, it triggers global replanning, including: returning to the knowledge retrieval stage. Vector knowledge base, maintaining historical scenario data; The decision-making brain receives and understands natural language instructions, and implements the instructions based on the historical scene data; it determines the skill modules that need to be invoked to implement the instructions; during the implementation of the instructions, it activates only the currently required skill modules according to the current task progress, and loads the required resources for the activated skill modules in real time, the resources including at least one of historical scene data, historical context data, and current scene data; the currently activated skill modules and the operation results of the current skill modules are written into the historical context data.

2. The robot task planning system as described in claim 1, characterized in that, The standardized four-phase protocol indicates the following process: Instruction reception: Receive the instruction; Knowledge retrieval: The process involves invoking the intent parsing and knowledge coarse screening module, the memory fine screening module, and the feasibility assessment module; using the intent parsing and knowledge coarse screening module to initially filter memory entries from the loaded historical scene data; using the memory fine screening module to filter out valid memory data from the memory entries; and using the feasibility assessment module to evaluate the feasibility of the valid memory data. Hierarchical planning: Invoke the step planning module and the action orchestration module; initially plan the macro step sequence through the step planning module; extract each step sequentially from the macro step sequence, and generate the local action command for the current step through the action orchestration module when implementing each step; Closed-loop execution: Invoke the status monitoring and self-healing module; during the implementation of each step, monitor the execution status of the current step in real time through the status monitoring and self-healing module, and trigger remedial measures when the current step fails.

3. The robot task planning system as described in claim 2, characterized in that, The decision-making brain determines the skill modules that need to be invoked to implement the instructions based on the following configuration: During the execution of the instructions, the current stage of the task progress is determined; based on the description information of each skill module in the expert skill library, the skill module that needs to be invoked is determined.

4. The robot task planning system as described in claim 3, characterized in that, The decision-making brain implements the instructions based on the historical scenario data according to the following configuration: Based on the understood instructions, determine whether to invoke the task execution bus module; if so, execute the following process: Activate the task execution bus module and load the standardized four-stage protocol into the historical context data; According to the process indicated by the standardized four-stage protocol, the intent parsing and knowledge coarse screening module and the memory fine screening module are activated sequentially to retrieve valid memory data from the historical scene data; Activate the feasibility assessment module to assess the feasibility of the valid memory data; Furthermore, after the feasibility of valid memory data is assessed, the step planning module is activated to plan the macro-level step sequence; Furthermore, each step is extracted sequentially from the macro-step sequence. When implementing each step, the action orchestration module is activated to generate the local action command for the current step, and the status monitoring and self-healing module is activated to monitor the execution status of the current step in real time and trigger remedial measures when the current step fails to execute.

5. The robot task planning system as described in claim 1, characterized in that, The decision-making brain, vector knowledge base, and expert skill base are connected via the Model Context Protocol (MCP) interface.

6. A robot task planning method, characterized in that, include: Receive and understand natural language instructions, determine the independently encapsulated skill modules to be invoked from the expert skill library to implement the instructions, and each skill module is configured with a state machine indicating whether it is activated; The expert skills base shall contain at least the following skills modules: The task execution bus module defines a standardized four-stage protocol from instruction reception, knowledge retrieval, hierarchical planning to closed-loop execution. The intent parsing and knowledge screening module parses the received instructions and, based on the parsing results, retrieves the corresponding memory entries from the loaded historical scene data. The memory screening module removes outdated and redundant data from memory entries based on predefined discrimination rules, and selects valid memory data. The feasibility assessment module evaluates the feasibility of effectively storing data. The step planning module, based on effective memory data, initially plans a macro-level sequence of steps with temporal logic; The motion orchestration module generates local motion commands for the current step based on the current resources. The status monitoring and self-healing module monitors the execution status of the current step in real time and triggers remedial measures when the current step fails. The remedial measures include local self-healing within the step and global replanning. The status monitoring and self-healing module is equipped with a first counter, which counts the number of times local self-healing within the step is triggered. If the monitoring and self-healing module fails to execute the current step, and the first counter has not overflowed, it triggers local self-healing within the step, including: calling the action orchestration module to regenerate the local action command for the current step; if the first counter overflows, it triggers global replanning, including: returning to the knowledge retrieval stage. The instructions are executed based on historical scene data. During the execution of the instructions, only the currently required skill modules are activated according to the current task progress, and the required resources are loaded for the activated skill modules in real time. The resources include at least one of historical scene data, historical context data, and current scene data. The currently activated skill modules and the operation results of the current skill modules are written into the historical context data.

7. A robot task planning device, characterized in that, It includes a processor and a storage medium, the storage medium storing a computer program, the processor running the computer program to perform the robot task planning method as described in claim 6.

Images