Method, system and machine readable information storage medium for constructing a task planning framework

Abstract

The present disclosure relates to methods, systems, and machine-readable information storage media for constructing a task planning framework. The disclosed methods and systems overcome the obstacles of a robot task planner for DSU by introducing a task planning framework. The core of the framework is a robot system ontology that organizes the components of a robot system in a coherent and systematic manner. The ontology enables the planning framework to efficiently capture the contextual representation of DSU using LLMs. Additionally, the research introduces a LLM tuning scheme called the Coherent Hierarchical Thought (CoHT) chain, specifically designed to complement the robot system ontology. The integration of these two components enables the enhancement of the accuracy, robustness, and throughput of the robot in a cost-effective manner. Furthermore, an empirical approach is provided for generating a guaranteed performance LLM tuning dataset size using a heuristic-based approach.

Description

[0001] Cross-reference and priority of related applications

[0002] This application claims priority to Indian Application No. 202421096220, filed on 5 December 2024. Technical Field

[0003] The implementations described herein generally relate to the field of robotic systems, and more specifically, to methods and systems for constructing task planning frameworks for robots based on Large Language Models (LLM) to execute Domain-Specific Use Cases (DSUs). Background Technology

[0004] Within the robotics research community, task planning and reasoning using large language models (LLMs) has become a focus of attention. However, directly applying LLMs, even with very large word size cues, cannot achieve the task planning performance required for domain-specific industry use cases (DSUs).

[0005] Recent research, particularly Microsoft's Chat-GPT Robot Manipulation (MCRM), provides a structured and scalable cueing approach for Domain-Specific Tasks (DSUs). However, several domain-specific technical challenges remain unresolved. These include the lack of consideration for failure scenarios when updating the robot environment, the need for multiple user feedback calls to obtain accurate task plans, and the lack of example-sequence-based cueing (considered a crucial aspect of contextualizing LLMs for Domain-Specific Tasks (DSTs)). Several approaches exist for contextualizing LLMs to human instructions. Some of these include pre-training, fine-tuning, and retrieval augmentation methods; however, these either require high computational costs or large datasets, raising questions about their use and applicability. On the other hand, various standard cueing techniques exist that are readily applicable to any readily available LLM and contextualize it to any custom DSU. This requires a small dataset or no LLM tuning at all. Meanwhile, it has been observed that easily contextualizing any off-the-shelf LLM to a DSU is quite challenging. LLMs repeatedly fail to reason about and fully understand the domain rules and nuances of the DSU. Therefore, structurally efficient knowledge representations centered on LLMs are essential for robotic systems and DSUs. These representations must be used to achieve better contextualization. In similar directions, recent advances have significantly improved existing prompting capabilities, including Chain of Thought (CoT), ToT, thought algorithms, context enhancement, etc. Currently, CoT and ToT are primarily used to improve the reasoning capabilities of LLMs. However, it should be noted that CoT is effective at the level of abstract contextual knowledge, sometimes missing any low-level domain rules in the DSU. CoT is significantly dependent on the size of the LLM and generally performs poorly on smaller LLMs. Furthermore, the scope of validated intermediate reasoning / thoughts is limited, thus increasing the likelihood of arriving at incorrect solutions. On the other hand, ToT effectively addresses most of these limitations. Considering this, implementing ToT in practical DSUs seems advantageous, but it is limited by its complexity. The cost is frequent output lexical exhaustion and computational complexity, which impacts the goal of achieving low latency and high throughput in DSUs. Therefore, ToT may not be necessary for tasks that can be performed well through intermediate prompting. Thus, a new prompting technique suitable for DSUs is needed. Summary of the Invention

[0006] The embodiments disclosed herein offer technical improvements as technical solutions to one or more of the aforementioned technical problems recognized by the inventors in conventional systems.

[0007] For example, in one implementation, a method is provided for constructing a task planning framework for a robot. The method includes: providing a robot system ontology having multiple modules including: (a) use cases, (b) embodiments, (c) workspaces, (d) objectives, (e) relationships, (f) experiences, and (g) plans, for capturing a contextual representation of the domain-specific unit (DSU), thereby generating a task plan for a domain-specific task using a large language model (LLM).

[0008] Furthermore, the method includes: initializing cue training via a hierarchical thought chain (CoHT) framework enhanced with robot system ontology, deriving a CoHT cue structure for LLM, wherein the training dataset used is constrained by multiple dataset features including unit size, diversity, and uncertainty, wherein the multiple dataset features satisfy a minimum number of mathematical equations acceptable for constrained instruction trials.

[0009] Furthermore, the method includes: employing a user query as a command to query the LLM for the robot to perform a domain-specific task, wherein the user query is supported by static data from the robot agent's workspace, including system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past input-output, and wherein the combination of static data constructed according to the CoHT cue structure generates cue for the LLM.

[0010] In addition, the method includes classifying user queries as valid user queries based on whether the LLM response to the prompt falls into one of a plurality of actions defined for the robot to perform a domain-specific task while complying with a set of domain rules.

[0011] In addition, the method includes generating a task plan for a domain-specific task from a prompt with a valid user query by an LLM.

[0012] In addition, the method includes: verifying the task plan to check for the presence of one or more predefined unsafe operations, thereby generating a verified task plan.

[0013] In addition, the method includes having a robot perform a domain-specific task according to a validated task plan.

[0014] In another aspect, a system for constructing a task planning framework for a robot is provided. The system includes: a memory storing instructions; one or more input / output (I / O) interfaces; and one or more hardware processors coupled to the memory via the one or more I / O interfaces, wherein the one or more hardware processors are instructed to provide a robot system ontology having multiple modules including: (a) use cases, (b) embodied entities, (c) workspace, (d) objectives, (e) relationships, (f) experiences, and (g) plans for capturing a contextual representation of the DSU, thereby generating a task plan for a domain-specific task using a Large Language Model (LLM).

[0015] Furthermore, one or more hardware processors are configured to initialize cue training via a hierarchical thought chain (CoHT) framework enhanced with robot system ontology, deriving a CoHT cue structure for LLM, wherein the training dataset used is constrained by multiple dataset features including unit size, diversity, and uncertainty, wherein these multiple dataset features satisfy a minimum acceptable number of mathematical equations for constrained instruction trials.

[0016] Furthermore, one or more hardware processors are configured to use user queries as commands to query the LLM for the robot to perform domain-specific tasks, wherein the user queries are supported by static data from the robot agent's workspace, including system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past input-output, and wherein the combination of static data constructed according to the CoHT cue structure generates cue for the LLM.

[0017] In addition, one or more hardware processors are configured to classify user queries as valid user queries based on whether the LLM response to the prompt falls into one of a plurality of actions defined for the robot to perform a domain-specific task while complying with a set of domain rules.

[0018] In addition, one or more hardware processors are configured to generate task plans for domain-specific tasks by the LLM in response to prompts with valid user queries.

[0019] In addition, one or more hardware processors are configured to validate the task plan to check for the presence of one or more predefined unsafe operations, thereby generating a validated task plan.

[0020] In addition, one or more hardware processors are configured to allow the robot to perform domain-specific tasks according to a validated task plan.

[0021] In another aspect, one or more non-transitory machine-readable information storage media are provided, including one or more instructions that, when executed by one or more hardware processors, cause the execution of a method for constructing a task planning framework for a robot.

[0022] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device provides a robot system ontology having multiple modules including: (a) use cases, (b) embodied entities, (c) workspace, (d) objectives, (e) relationships, (f) experience, and (g) plans for capturing a contextual representation of the DSU, thereby generating a task plan for a domain-specific task using a large language model (LLM).

[0023] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device initializes cue training via a hierarchical thought chain (CoHT) framework enhanced with robot system ontology, deriving a CoHT cue structure for LLM, wherein the training dataset used is defined by multiple dataset features including unit size, diversity, and uncertainty, wherein these multiple dataset features satisfy a minimum number of mathematical equations acceptable for the defined instruction trials.

[0024] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device uses a user query as a command to query the LLM so that the robot can perform a domain-specific task. The user query is supported by static data from the robot agent's workspace, including system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past input-output. The combination of static data based on the CoHT cue structure generates cue for the LLM.

[0025] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device classifies a user query as a valid user query based on whether the LLM response to the prompt falls into one of a plurality of actions defined for the robot to perform a domain-specific task while complying with a set of domain rules.

[0026] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device generates a task plan for a domain-specific task based on prompts with valid user queries from the LLM.

[0027] In addition, when one or more instructions are executed by one or more hardware processors, the computing device verifies the task plan to check for the presence of one or more predefined unsafe operations, thereby generating a verified task plan.

[0028] Furthermore, when one or more instructions are executed by one or more hardware processors, they enable the computing device to perform domain-specific tasks by the robot according to a validated task plan.

[0029] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention as claimed in the claims. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the specification, serve to explain the principles disclosed: Figure 1 This is a functional block diagram of a system (robot) with a large language model (LLM)-based task planning framework that performs domain-specific use cases (DSUs) according to some embodiments of this disclosure.

[0031] Figures 2A to 2C The overall architecture of a task planning framework for performing DSU according to some embodiments of this disclosure is described.

[0032] Figure 3A and Figure 3B This is a flowchart illustrating a method for constructing a task planning framework for performing domain-specific use cases (DSUs) through system 100 (robot) according to some embodiments of the present disclosure.

[0033] Figure 4A and Figure 4B The components of a task planning framework for a system according to some embodiments of the present disclosure are described.

[0034] Figure 5 A test bench for a DSU to be performed by a system (robot) is depicted according to some embodiments of the present disclosure.

[0035] Figure 6A and Figure 6B A fragment depicts a method of incorporating a hierarchical thought chain (CoHT) into a few-sample prompt according to some embodiments of the present disclosure.

[0036] Those skilled in the art will understand that any block diagram herein represents a conceptual view of an illustrative system and apparatus embodying the principles of the subject matter. Similarly, it should be understood that any flowchart, flow diagram, etc., represents various processes that can be substantially represented in a computer-readable medium and therefore executed by a computer or processor, whether or not such computer or processor is explicitly shown. Detailed Implementation

[0037] Exemplary embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numerals identify the figure where the numeral first appears. Where convenient, the same numerals are used throughout the drawings to refer to the same or similar parts. Although examples and features of the disclosed principles have been described herein, modifications, adaptations, and other implementations are possible without departing from the scope of the disclosed embodiments.

[0038] This disclosure provides a method and system (robot) for a task planning framework based on a Large Language Model (LLM) for performing Domain-Specific Use Cases (DSUs). Applying a Large Language Model (LLM), even with very large lexical size cues, fails to achieve the task planning performance required for Domain-Specific Use Cases (DSUs). The disclosed method and system overcome this obstacle for robot task planners by introducing a task planning framework. At the heart of this framework is a robot system ontology that organizes the components of the robot system in a coherent and systematic manner. This ontology enables the planning framework to efficiently capture the contextual representation of the DSU using LLM. Furthermore, the research introduces an LLM tuning scheme called Hierarchical Thinking Chain (CoHT), specifically designed to complement the robot system ontology. Integrating these two components allows for cost-effective enhancement of the robot's accuracy, robustness, and throughput. Additionally, an empirical approach is provided for generating performance-guaranteed LLM tuning datasets using heuristic-based methods.

[0039] Now refer to the attached diagram, and more specifically to... Figures 1 to 6B Similar reference numerals are found throughout the appendix. Figure 1 The corresponding features are indicated accordingly, and preferred embodiments are shown, which are described in the context of the following exemplary systems and / or methods.

[0040] Figure 1 This is a functional block diagram of a system 100 (robot) with a task planning framework for performing domain-specific use cases (DSUs) according to some embodiments of this disclosure.

[0041] In an implementation, system 100 (also referred to as a robot or robotic system) includes: a processor 104; a communication interface device, alternatively referred to as an input / output (I / O) interface 106; and one or more data storage devices or memories 102 operatively coupled to the processor 104. System 100, having one or more hardware processors, is configured to perform the functions of one or more functional blocks of system 100.

[0042] Referring to the components of system 100, in an embodiment, processor 104 may be one or more hardware processors 104. In an embodiment, the one or more hardware processors 104 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, the one or more hardware processors 104 are configured to fetch and execute computer-readable instructions stored in memory 102. In an embodiment, system 100 may be implemented in various computing systems, including laptop computers, notebook computers, handheld devices such as mobile phones, workstations, host computers, servers, etc.

[0043] I / O interface 106 may include various software and hardware interfaces, such as network interfaces, graphical user interfaces, etc., and can facilitate various communications within multiple network N / W and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (such as WLAN, cellular, etc.). In an implementation, I / O interface 106 may include one or more ports for connecting to multiple external devices or another server or device. Multimodal format user query instructions are received via I / O interface 106 to instruct the LLM to perform tasks of interest in a specific domain.

[0044] Memory 102 may include any computer-readable medium known in the art, including volatile memory such as static random access memory (SRAM) and dynamic random access memory (DRAM), and / or non-volatile memory such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk and magnetic tape.

[0045] In an implementation, the memory 102 includes multiple modules 110, such as LLM, robot body system modules, etc.

[0046] Multiple modules 110 include programs or coded instructions that complement applications or functions executed by system 100 to perform different steps involved in building an LLM-based task planning framework (also known as an LLM Robotics System Planning Framework (LLM-RSPF)) for execution domain-specific use cases (DSUs) executed by system 100. Multiple modules 110 may include routines, programs, objects, components, and data structures, etc., that perform specific tasks or implement specific abstract data types. Multiple modules 110 may also be used as signal processors, node machines, logic circuits, and / or any other devices or components that manipulate signals based on operation instructions. Furthermore, multiple modules 110 may be used via hardware, via computer-readable instructions executed by one or more hardware processors 104, or via a combination thereof. Multiple modules 110 may include various submodules (not shown).

[0047] In addition, memory 102 may include information relating to each step executed by processor 104 of system 100 and the inputs / outputs of the methods of this disclosure.

[0048] In addition, memory 102 includes database 108. Database 108 (or repository) may include multiple abstract code snippets for optimization and data that is processed, received, or generated as the execution result of multiple modules in module 110.

[0049] Although database 108 is shown as being internal to system 100, it should be noted that in alternative embodiments, database 108 may also be implemented externally to system 100 and communicatively coupled to system 100. Data contained within such an external database can be updated periodically. For example, new data can be added to the database (…). Figure 1 (Not shown in the image) and / or existing data can be modified and / or useless data can be deleted from the database. In one example, the data may be stored on an external system, such as a Lightweight Directory Access Protocol (LDAP) directory and a Relational Database Management System (RDBMS). Now combined Figures 2A to 5 The steps in the flowchart are used to explain the function of the components of system 100.

[0050] Figures 2A to 2C The overall architecture of a task planning framework according to some embodiments of this disclosure is described. Combined with Figure 3A and Figure 3B To explain Figures 2A to 2C .

[0051] Figure 3A and Figure 3BThis is a flowchart illustrating a method for constructing a task planning framework (LLM-RSPF) for performing domain-specific use cases (DSUs) through a system 100 (robot) according to some embodiments of the present disclosure.

[0052] In one implementation, system 100 includes one or more data storage devices or memories 102 operatively coupled to processor 104 and configured to store instructions for execution of the steps of method 200 by the processor or one or more hardware processors 104. Reference will now be made to... Figure 1 , Figures 2A to 2C The components or blocks of the system 100 depicted herein, and such as Figures 3A to 3B The steps of the method 200 described herein are explained by the flowcharts depicted herein. While process steps, method steps, techniques, etc., may be described sequentially, these processes, methods, and techniques may be configured to operate in an alternating order. In other words, any order or sequence of steps that can be described does not necessarily indicate that the steps must be performed in that order. The steps of the process described herein may be performed in any actual order. Furthermore, some steps may be performed simultaneously.

[0053] The reference describes the process flow of the task planning framework (LLM-RSPF). Figure 2B In step 302 of method 300, one or more hardware processors 104 are instructed to provide a robot system body having multiple modules. The multiple modules of the robot system body are... Figure 4A The description includes: (a) use cases, (b) embodied entities, (c) workspaces, (d) goals, (e) relationships, (f) experiences, and (g) plans, for capturing the contextual representation of the DSU, thereby generating task plans for domain-specific tasks using a large language model (LLM).

[0054] (a) Use cases enable the definition of the problem statement and the detailed use case description.

[0055] (b) Embodied entities enable the contraction of multiple agents to collaborate in order to accomplish domain-specific tasks.

[0056] (c) The workspace enables the test bench for DSU to be defined by multiple fields including mobility space, pick location, placement location, arrangement and objects.

[0057] (d) The objective enables the definition of domain rules, performance metrics, and operations.

[0058] (e) Relations enable the constraint of mapping and classification.

[0059] (f) Experience enables the definition of enhancement, failure, and success.

[0060] (g) The plan enables the provision of task plan constraints and generation.

[0061] like Figure 4B The depicted embodied entities are different intelligent agents that collaborate to complete a specific task, namely (a) physical robots, (b) behaviors, (c) end-of-arm tools (EOAT), and (d) sensors, wherein the intelligent agents are also uniquely identified based on capabilities, interfaces, states, and experiences.

[0062] The Use Case module is the parent module, where the problem statement and detailed use case description are provided. It sets up an abstract representation of the use cases for any LLM. The statement and description are the two components responsible for this provision.

[0063] The embodied entity module defines the different agents that collaborate to complete a specific task. An agent is uniquely identified by four components: (a) physical robot, (b) behavior, (c) end-of-arm tool (EOAT), and (d) sensors. The embodied entity module is further elaborated based on its capabilities, interfaces, states, and experiences, such as... Figure 4B As shown. The abilities of embodied entities are categorized in the form of sensing and movement, which originates from the title " React: Synergizing reasoning and acting in Language models (React: Collaborative reasoning and action in language models) The known ReAct method is found in the literature (2023).

[0064] Sensing endows embodied entities with general perception, vision, and tactile perception capabilities. General perception components are typically open-vocabulary object detection based on a visual foundation model, providing real-time workspace information. Next, actions serve as "action" capabilities based on the agent's behavior. This "action" capability here represents robotic skills. Each robotic skill is essentially a combination of multiple atomic skills, such as seeing, moving, picking up, placing, etc., and possesses its own mature perception, manipulation, and movement capabilities. The reason behind designating skills as composite skills is to simplify the complexity of planning and to achieve system scalability by integrating new mature skills. A skill for an agent has its own limitations on properties derived from physical workspace components. The Kth skill of agent a is limited to the title "..." Do what I can, not what I say: Grounding language in robotic affordances (Doing what I can do, not what I say: making language...) (Based on the functions of robots) In the literature work (2022).

[0065] Continuing, the state component represents the current state of the agent. This can be robot joint angles, attached grippers, etc. Finally, experience captures successful and failed experiences at the agent level.

[0066] (1)

[0067] The workspace module defines the test bench for any DSU. It inherently consists of five configurable components. The mobility space defines the detailed mobility space comprised of all location identifiers and movement indicators (if any). An object is a target item that should be handled by the embodied entity when performing any task. Pick-up and place locations refer to the locations within the workspace specifically allocated for picking up or placing any target object by the embodied entity. Finally, the arrangement determines the physical layout of the different agents within the workspace relative to their respective location identifiers.

[0068] The target module defines the mission objectives for the DSU. It sets all domain rules related to the DSU, which must be satisfied during any mission planning. In short, the generated plan must adhere to the domain rules at all costs. Performance metrics are a list of metrics used to evaluate the performance of the embodied system relative to the generated plan. The operational components actually determine the overall operational objectives to be met, which may or may not be functions of performance metrics.

[0069] Relationship modules identify any relationship mappings between components, whether intra-module or inter-module. For example, an agent might be associated with picking up from a specific location in the workspace. Mappings adapt to these relationship mappings. Next, classification provides the option to categorize user instructions based on their nature, ensuring they are appropriately mapped relative to different DSU scenarios. This component is crucial in the classification of instructions and dialogue responses generated by LLM. This simplifies the UI's decision-making process.

[0070] The experience module (memory) maintains various intermediate states related to the system or task and stores overall success and failure scenarios. This module helps LLMs recall past experiences and incorporate them into future task planning. Failure experiences are particularly important here, as they significantly reduce system latency and limit any re-failure scenarios. Furthermore, chat history can be used to store experiences via truncation or RAG methods.

[0071] The planning module consists of task plan constraints and any template-specific conditions that must be followed when generating the task plan. The dictionary format used for robot skill constraints and their sequence in the agent's plan is as follows: Web Ontology Language: OWL, pp. 91-110. Springer Berlin, Heidelberg. Berlin, Heidelberg, 2009 As stated in "", and the final task plan template is in " Taxonomy of educational Objectives: The classification of educational goals. (Classification). Longman Publishing, 1956. The term "" is represented by P. The generation component defines the different types of inputs required to generate the final task plan.

[0072] These inputs can come from state, sensing, targets, etc.

[0073] (2) (3) At step 304 of method 300, and referring to Figure 2C One or more hardware processors 104 are instructed to initialize cue training via a Layered Thinking Chain (CoHT) framework enhanced with robot system ontology, deriving a CoHT cue structure for LLM. The training dataset for the CoHT framework initial cue training is defined by multiple dataset features including unit size, diversity, and uncertainty. These multiple dataset features satisfy the mathematical equation provided in Equation 5, which specifies the minimum acceptable number of instruction trials.

[0074] LLM-driven task planning scheme: Dataset Preparation: To begin planning, it is important to ensure that the task plan representation is appropriately constrained in the planning module. For good contextualization of LLM, the DSU dataset is essential. The complete dataset used for contextualization is divided into three parts in a 1:1:2 ratio: training, validation, and testing. Note that the terminology used here refers to LLM tips and does not refer to training or fine-tuning an LLM from scratch. Regarding dataset creation, three types of estimation are performed, as described below.

[0075] Unit Size: The dataset creation and its unit size are estimated based on the task plan representation and the classifications defined in the planning and relational modules, respectively. Here, the term unit size refers to the smallest standard dataset size that can be used to estimate the size of the training, validation, and test datasets. Let us assume that the sum of all instruction classifications is m, and mˆ represents the unit vector corresponding to each classification label. The total number of robot skills derived from equation (2) is S. Considering the Mth classification as the only classification that outputs a valid task plan for the total agent A, the total combination of instructions corresponding to these two inputs becomes .choose Factors that allow each category to span a sufficient number of user instructions.

[0076] Diversity: Recognizing the need to ensure sufficient diversity and fuzzy learning during contextualization to adapt to different use case scenarios, [the following was introduced] 'δ' refers to the total number of such instructions required. Note that... It must ideally remain greater than or equal to This is because it ensures that datasets related to diverse scenarios are also given equal (if not more) importance compared to similar scenarios that are only action-specific.

[0077] Uncertainty: Uncertainty is analogous to the number of repeated trials chosen for each category to ensure the repeatability of the generated plan. It is denoted by g(ω) and empirically calculated using Equation (4). The empirical permutation in Equation (4) follows typical range mapping logic for the category priority set ω. ω provides the option to assign weights to different categories based on their importance and probability of occurrence. High-weighted categories produce high-confidence and robust instruction sets belonging to that category. Combining size and diversity estimates, the final dataset unit size is |D|, while the minimum number of instruction trials / experiments required is given by |T| in Equation (5). Note that choosing an estimate larger than the result of Equation (5) is a computational guarantee to ensure the contextualization required for LLM.

[0078] (4) (5) in, This refers to the size of the dataset. , This refers to domain rule injection. This refers to CoHT tuning, and It introduces uncertainty.

[0079] Layered thinking chain tips ( Figure 2CRegarding few-shot examples, this paper discloses a novel prompting technique that integrates input context descriptions into few-shot examples, forming a hierarchical chain of thought (CoHT). CoHT prompts are essentially inspired by the hierarchical structure of Bloom's taxonomy of thought, as known in the literature. CoHT builds upon CoT, leveraging its strengths and achieving the desired goals of Domain-Specific Tasks (DSUs). Two key introductions are made in CoHT compared to CoT. First, hierarchical prompts are used on top of an LLM-based task planning framework (LLM-RSPF), where hierarchical intermediate reasoning is performed instead of linear intermediate reasoning steps similar to CoT. Regarding complex reasoning involved in Domain-Specific Tasks (DSTs), the hierarchical thinking process achieves enhanced modular reasoning by shifting from abstract thinking to narrower thinking, enabling LLM to accept fuzzy or indirect user instructions. This lack of low-level understanding is found in CoT whenever complex DSU reasoning is involved. In other words, CoHT reasoning involves first abstracting the task into high-level reasoning and then extending to low-level reasoning, allowing LLM to understand reasonable fuzziness at the object or behavioral level. In addition to mapping, any hard rules are explicitly highlighted in the low-level reasoning itself to avoid any learning stagnation. Furthermore, CoHT integrates the framework's modules in a linear hierarchy to bridge multiple inference levels. Secondly, recursive critique and improvement have been implemented, enabling the LLM to perform self-evaluation and optimization of the plan at the generation level itself. Finally, the few-shot examples implemented with CoHT ideally consist of both successful and failed scenarios, preventing any performance-driven bias.

[0080] CoHT-based hint tuning using the instruction dataset: Three subsets of the dataset are reserved for training, validation, and testing in a ratio of 1:0.5:2.

[0081] Cue Training: The training dataset used for cue training was created using annotations based on human experts. Cue training primarily targets the first part of the three-part formula given in Equation (5). It focuses on the injection of domain rules in LLM and the establishment of abstract representations of DSU. During training, a significant amount of work is dedicated to improving LLM’s understanding of descriptive fragments with few-shot cues by performing repeated plan generation on the training dataset. In addition, basic critical understanding is constructed through examples. Tuning during training is essentially done by evaluating the planning and relational modules of the robot system ontology. The evaluation metrics considered during training are: (a) accuracy, (b) recall, and (c) F1 score.

[0082] Hint Validation: For hint validation, the validation dataset is equal to half the unit size of the dataset. It is important to note that the unit size should minimize the impact on the distribution of instructions in the dataset. Hint validation targets the second part of Equation (5), which aims to further improve the intermediate inference of the CoHT construction to address diverse and ambiguous DSU scenarios. Hint validation is primarily about efficient low-level tuning of examples in CoHT, since high-level tuning of domain rules is likely to be done during hint training. The evaluation metrics considered during hint validation are: (a) accuracy, (b) recall, (c) F1 score, and (d) human-in-the-loop (HIL). To further fine-tune intermediate inference capabilities during validation, human-level effort is required; therefore, HIL is introduced.

[0083] Hint testing: The test size is twice the size of the dataset unit, providing ample scope for comprehensive hint testing. The third part of Equation (5) ensures repeatability testing and model confidence, which is crucial to check before finalizing and deploying any contextualized LLM. The evaluation metrics considered during testing are: (a) accuracy, (b) recall, (c) F1 score, and (d) human-in-the-loop (HIL).

[0084] At step 306 of method 300 and refer to Figure 2B One or more hardware processors 104 are instructed to use user queries as commands to query the LLM for domain-specific tasks to be performed by the robot. For example... Figure 2A and Figure 2B As described, user queries are supported by static data from the robot's defined workspace for performing domain-specific tasks. This static data includes system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot from past input-output pairs. The combination of static data, constructed according to the CoHT cue structure, generates cues to instruct the LLM (Local Level Management).

[0085] At step 308 of method 300 and refer to Figure 2BOne or more hardware processors 104 are instructed to classify user queries as valid user queries based on whether the LLM response to a query falls into one of a set of actions defined for the robot to perform a domain-specific task while adhering to a set of domain rules. Classification ensures that the robot can operate interactively and efficiently. Relevant queries are accepted. The task planning framework of system 100 (the robot) through classification can instruct the user to provide only relevant commands. In one case, if a user query to the LLM is generally responsive but the user query is irrelevant to industry operations—for example, the user queries "who won the cricket match" or some trivial question—system 100 can refuse to respond (to a response that the LLM would typically generate) and instead instruct the user to use the commands the robot should execute. Furthermore, system 100's task planning framework uses the LLM's common-sense capabilities to generate a plan, thereby minimizing the number of steps required to complete the task, making the system efficient.

[0086] At step 310 of method 300 and refer to Figure 2B One or more hardware processors 104 are instructed to generate task plans for domain-specific tasks when prompted by a valid user query for task plan generation.

[0087] At step 312 of method 300 and refer to Figure 2B One or more hardware processors 104 are instructed to verify the task plan to check for one or more predefined unsafe actions before clearing the verified task plan. This verification ensures that the task plan is validated for safety checks before execution. An LLM might generate a plan based on an incorrect instruction or some error, and the robot might be able to perform such unsafe actions, but the plan verifier enables the termination of such unsafe plans, thus providing intelligent and safe actions through the robot. Furthermore, for any user query that is very close to a valid executable action but lacks some data, the robot can assist the user by requesting that additional data, thus achieving a highly interactive and efficient operation. Examples include instructions lacking an object name, a pick-up location, a place location, or multiple items that can be identified as cups from the instructions (e.g., not specifying a color in the case of multiple-colored cups), or a combination of all these.

[0088] At step 314 of method 300 and refer to Figure 2B One or more hardware processors 104 are instructed to perform domain-specific tasks via a robot according to a validated task plan. All actions at the classification and execution steps are stored as experience or memory for future user queries of task planning.

[0089] experiment: This section implements LLM-RSPF on a real DSU ( Figure 1 and Figure 2A The system 100 and planning scheme are described, and the generated results are discussed. LLM-RSPF is used in a retail order fulfillment system, which has the following defined robot system ontology, and the experiments employ the following... Figure 5 The robot and workspace depicted in the text are explained.

[0090] Use Case: Problem Statement: The goal of the robot is to receive and understand any natural language query / instruction from humans, take action in accordance with DSU domain rules, and ultimately successfully execute the query / instruction within an acceptable level of fault tolerance, while achieving high throughput, high success rate, and low latency.

[0091] Description: The use case involves order fulfillment in the retail industry, where a robot picks objects from multiple boxes based on user instructions and follows a good-to-picker model.

[0092] Embodied Entity: The system's embodied entity has two fixed-base industrial arms (main <Robot_1> and auxiliary <Robot_2>) as intelligent agents <Agent_1>. The user instructs the main robot, which has the following capabilities, to perform tasks. System 100 (the robot) is equipped with general perception capabilities based on open-vocabulary object detection. For actions, <Agent_1> possesses the following robot manipulation skills: 1. <dips>Domain-independent picking skills (DIPS) are picking and placing skills with a perception based on unsupervised learning. Skill attributes are object count, pick position, and place position.

[0093] 2. <iris>The Instance Retrieval Picking Skill (IRIS) uses supervised learning-based perception to perform picking based on the domain-dependent environment. Skill attributes include object count, object class name, pick location, and placement location.

[0094] 3. <atcs>Automatic Tool Change Skill (ATCS) is a customized manipulation skill used to move a manipulator arm between two points based on specific actions (speed and force). ATCS is responsible for changing the EOAT. The skill attributes are the current EOAT and the next EOAT.

[0095] Since each of the aforementioned pickup skills uses a different EOAT, the robot's EOAT is merged as a state.

[0096] Workspace: DSU workspace such as Figure 5 As shown. The workspace has two types of pickup locations: homogeneous object bins (bins 2 and 3) for frequently ordered items and heterogeneous object bins (bin 1) for less frequently ordered items. The workspace has three placement locations: a conveyor (for delivering items to the assist robot), a carton (for order packaging), and a location for user pickup. (a) Items available in bin 1: red cup, brown cup, Dove soap, Cintosh soap, jasmine soap, and coconut oil; (b) Items available in bin 2: black soft drink bottle (ThumsUp); (c) Items available in bin 3: aseptically packaged beverage (Frooti).

[0097] Target Domain rules: DIPS uses a two-finger gripper (Robotiq 2f-85) as the EOAT to grasp objects, while IRIS uses a vacuum gripper (Robotiq Epick). However, this is not a mandatory standard for skills to be performed in normal scenarios. Objects in homogeneous bins are frequently ordered items. The order of these bins can change during operation, and general awareness is responsible for ensuring the properties of the bins before executing any user queries.

[0098] Relationship: For this DSU, five types of instructions are considered: valid, invalid (unachievable tasks), general (general queries), system information instructions (SII, system-related queries), and additional data requests (ADR, instructions that require additional information for achievable tasks).

[0099] Plan generation is a three-step method. First, instruction classification is performed to categorize instructions. Then, plan generation is performed. Finally, plan verification is performed to self-optimize the generated plan according to domain rules.

[0100] The plan is a sequential combination of the three skills mentioned above.

[0101] Additional implementation details for the retail order fulfillment use case: The following provides a detailed description of some modules in the LLM-RSPF ontology corresponding to the aforementioned retail order fulfillment system: State: The state of <Agent_1> considered here is the robot's EOAT, i.e., <current_eoat>. Since each pick-up skill defined above uses a different EOAT, the robot's EOAT is merged as a state.

[0102] Interface: This represents commentary communication within an agent, between agents, or between an agent and a user. Here, commentary includes intra-agent communication such as object transfer messages between two robots, as well as task-related dialogue between agents and users.

[0103] Experience: For this DSU, only consider failure scenarios to avoid re-failure scenarios.

[0104] Picking locations: Box 1 contains a heterogeneous collection of objects and is designed for objects from high-mixing, low-batch categories. Box 2 contains a homogeneous collection of objects. Box 3 contains a homogeneous collection of objects, however, different from those available in Box 2.

[0105] Placement Locations: Placement 1 (Conveyor) is a flat belt conveyor used to transfer objects from <Robot_1> to <Robot_2>. Placement 2 (Box) is a carton or retail box used to send objects for packaging or order fulfillment. Placement 3 (User) is a convenient fixed location designed for user pickup.

[0106] Arrangement: <Robot_1> and <Robot_2> are located at the front and rear ends of the conveyor, respectively. There are specific reasons behind the choice of Domain-Independent Picking Skills (DIPS) and Instance Retrieval Picking Skills (IRIS) for the manipulation skills of the "embodied entities" robots. These skills are inspired by the goods-to-picker model in retail order fulfillment. DIPS is advantageous when the system needs to quickly adapt to new objects without any deep learning-based training. DIPS is useful for homogeneous items because these items can be completely replaced and the system can quickly adapt to such replacements. On the other hand, IRIS is designed for heterogeneous items, which are intended for object classes in high-mixing, low-batch categories—a very popular concept in the order fulfillment industry. This helps achieve high space utilization.

[0107] Mapping: Box 1 <iris>Box 2 <dips>Box 3 <dips> ； <iris> <Suction cup>; <dips> Two-finger gripper.

[0108] Classification: – Effective: It produces effective plans because all information related to the plan is available in the instructions.

[0109] - Invalid: Given the capabilities of the robotic system, it has requested an unreasonable task.

[0110] – General: It seeks irrelevant queries about robotic systems.

[0111] – System Information Instruction (SII): SII seeks system-related information, such as workspace details, embodied entity details, etc.

[0112] – Additional Data Request (ADR): ADR is relevant to the robot system and is generally effective; however, it is missing any information.

[0113] The planning scheme proposed by the system Dataset: Estimation of dataset unit size: Considering five categories of instructions, three robot skills, single agent, and... With ' set to 5 and five instructions per category, the dataset unit size becomes 40 according to the formula in equation (5). Based on the prescribed training, validation, and testing ratio of 1:0.5:2, the total dataset sizes are 40, 20, and 80 respectively. These 140 instructions were now created using 14 human experts who had a brief understanding of the framework, purpose, and instruction classification.

[0114] Tip: In the few-sample hint, the CoHT used as the example sequence can be based on... Figure 6A and Figure 6B The illustration provided is for comprehension. It should be noted that this depiction is not a complete hint, but a fragment illustrating how CoHT is incorporated in few-shot scenarios to generate better results. Initially, CoHT essentially performs few-shot hints. Any DSU is interpreted and transformed into a basic hint. Next, standard hint styles using placeholders, contextual separation using symbolic cues, the use of tone or style, contextual priming, etc., are used to construct the hint and make it clear. This also helps to set the optimal positioning of the textual domain rules. Continuing, the next part of the hint is the qualifying of few-shot examples. It is observed that in DSUs, the complex reasoning ability of LLM is significantly dominated by domain examples compared to abstract representations of domain rules in the form of context-separated paragraphs. Therefore, few-shot examples must be carefully designed with sufficient diversity so that LLM achieves contextual understanding in appropriate planning. One of the key advantages of CoHT is its range of hard rule proofs for each hierarchical thought in linear reasoning steps, which significantly helps improve LLM's reasoning for specific events, which is quite difficult to achieve otherwise. While increasing few-shot examples improves LLM performance, it is observed that maintaining a single highly complex example instead of multiple simple examples essentially improves LLM performance. This is also used for another purpose, namely, to reduce lexical length by replacing multiple examples with a single example.

[0115] result: Consider the cost comparison of different LLMs used for LLM-RSPF (System 100): Table 1 lists the cost analysis of five proprietary LLMs and two open source LLMs.

[0116] Table 1

[0117] The cheapest and most expensive models provided by GPT are GPT3.5 and GPT4, respectively. The different input lexical units specified for each LLM are the result of adapting and fine-tuning the prompts over time. In terms of performance, GPT4 performs best among all models and consumes the fewest lexical units simultaneously, while GPT3.5 performs worst. GPT4-Enhanced and Gemini-Pro are comparable; however, Gemini-Pro falls short in terms of frequent output lexical exhaustion for longer and highly complex plans and inefficient lexical unit usage. Both of these open-source models easily outperform GPT3.5 and share their performance with fine-tuned GPT3.5. It is important to note that while fine-tuned GPT3.5 may be a favorable choice, it requires high computational cost or a large dataset and can become quite expensive. In contrast, the method and system 100 (LLM-RSPF) disclosed in this paper achieves similar goals using only few-sample LLM prompts, thus reducing costs. Several key hyperparameters need to be considered when configuring LLM outputs, such as temperature, kernel sampling (top p), frequency penalty, and presence penalty. Considering cost and performance, GPT4-Enhanced is a fairly good evaluation choice.

[0118] The comparison results of LLM-RSPF (System 100) with other LLM-based planning frameworks (such as MCRM and programming hints (ProgPrompt: PgmPmt)) are generated, and the evaluation results generated on the test dataset are given in Table 2.

[0119] Table 2

[0120] A comparison of LLM-RSPF (System 100) with two other closest task planning methods is presented. These two results include MCRM and programming hints: generating contextualized robot task plans using a large language model. The effectiveness of LLM-RSPF is also concisely expressed in Table 3. It significantly outperforms both methods in terms of planning and classification accuracy in DST. It is observed that MCRM performs well for explicit and more reasonable plans and instructions of simple to moderate complexity. Note that some improvements were made in the implementation of MCRM to avoid unnecessary environment modifications and frequent user-level feedback. On the other hand, programming hints perform well for simple plans but poorly in highly domain-specific scenarios. Despite LLM-RSPF having good evaluation metric scores, there are some instructions in which all three methods failed to classify the instructions correctly. In example 17 ("Invalid"), LLM failed to recheck the count of an aseptically packaged beverage after picking it up and misclassified it, while in example 34 ("ADR"), LLM assumed that the pick-up location of the jasmine soap was Place 3.

[0121] Comparison of CoHT with CoT and ToT: Table 4 shows the effectiveness of CoHT, as disclosed in System 100, in using exemplary sequences with few-sample prompts relative to CoT and ToT techniques. A poor-performing LLM and a good-performing LLM, namely GPT3.5 and GPT4-Enhanced, respectively, were considered for generating results. The "effective" instructions in Table 2 were evaluated. It is evident that CoHT, as the exemplary sequence, outperforms both CoT and ToT in terms of planning accuracy for both LLMs. However, the profit using GPT-4-Enhanced is significantly higher. The average output token length (OTL) of CoHT is almost equal to that of ToT, but slightly higher than that of CoT.

[0122] The scalability of LLM-RSPF was tested by extending the DSU described in Part 4. To demonstrate its scalability, it was extended by adding (a) agents, (b) capabilities (sensing and motion), and (c) workspace. First, an agent <Agent_2> was introduced as a mobile manipulator, and two manipulator robots under <Agent_1> were added. Two new robot skills were added, corresponding to the two new robots under <Agent_1>, using different sensing methods for tactile perception. The workspace was modified by adding mobility space and pick-up and place-down positions. Compared to the manipulator under <Agent_1> (which is fixed and has static pick-up and place-down positions), the mobile manipulator under <Agent_1> is able to have flexible pick-up and place-down positions. Due to space limitations, the detailed framework of the extended use cases is not explained here; however, it follows a similar approach to that explained previously. With GPT4-Enhanced, the planned accuracy achieved by the extended DSU is approximately 0.91. The results confirm that a modular framework with composite robotic skills, such as LLM-RSPF, is sufficient to demonstrate an scalable LLM-driven task planner.

[0123] Table 3

[0124] Table 4

[0125] This method and system 100 provide a task planning framework, also known as the Large Language Model-Based Robot System Planning Framework (LLM-RSPF) specifically designed for Domain-Specific Use Cases (DSUs). This framework comprises two key components: a dedicated robot system ontology designed for DSUs, and an LLM tuning scheme called Hierarchical Mind Chain (CoHT), which complements the proposed ontology to achieve cost-effective LLM contextualization. Additionally, an empirical quantization of the cue dataset is proposed to optimize LLM tuning. To evaluate the effectiveness of LLM-RSPF, it is applied to real-world DSUs in the retail and packaging industries. Comparative experiments are conducted using popular proprietary and open-source LLM models, accompanied by cost-benefit analyses. Furthermore, the disclosed LLM-RSPF is benchmarked against notable achievements in the literature, such as Microsoft+ChatGPT and programming hints, demonstrating superior accuracy in plan generation and query classification tasks. The disclosed LLM tuning scheme, CoHT, is compared with CoT and ToT methods, highlighting the robustness of CoHT. Finally, scalability tests were performed by adding agents, incorporating additional sensing modalities (such as haptic feedback), and expanding workspace entities, in which LLM-RSPF consistently achieved 91% planning accuracy.

[0126] The written description describes the subject matter of this document to enable those skilled in the art to make and use embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that may be conceived by those skilled in the art. Such other modifications are intended to fall within the scope of the claims if they have similar elements that are not different from the wording of the claims, or if they include equivalent elements that are not substantially different from the wording of the claims.

[0127] It should be understood that the scope of protection is extended to programs containing messages, and beyond computer-readable storage devices, such computer-readable storage devices contain program code means for implementing one or more steps of the method when the program is run on a server or mobile device or any suitable programmable device. The hardware device can be any kind of programmable device, including, for example, any kind of computer, such as a server or personal computer, or any combination thereof. The device may also include means, which may be, for example, hardware means such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, such as an ASIC and an FPGA, or at least one microprocessor and at least one memory having software processing components located therein. Thus, the means may include both hardware and software means. The method implementations described herein can be implemented in both hardware and software. The device may also include software means. Alternatively, implementations may be implemented on different hardware devices (e.g., using multiple CPUs).

[0128] The embodiments described herein may include hardware and software elements. Software implementations include, but are not limited to, firmware, resident software, microcode, etc. The functions performed by the various components described herein may be implemented in other components or combinations of other components. For the purposes of this specification, a computer-usable or computer-readable medium may be any device that may include, store, transmit, propagate, or transport programs for use by or in connection with an instruction execution system, apparatus, or device.

[0129] The steps shown are provided to explain the exemplary embodiments illustrated, and it should be expected that ongoing technological development will change the way particular functions are performed. These examples are given herein for illustrative purposes and not for limitation. Furthermore, for ease of description, the boundaries of functional building blocks are arbitrarily defined herein. Alternative boundaries can be defined as long as the specified functions and their relationships are properly performed. Based on the teachings contained herein, alternatives (including equivalents, extensions, variations, deviations, etc., of what is described herein) will be apparent to those skilled in the art. Such alternatives fall within the scope of the disclosed embodiments. Furthermore, the words "comprising," "having," "including," and "comprising," and other similar forms, are intended to be equivalent in meaning and are open-ended, as one or more items following any of these words do not imply an exhaustive list of such items, nor do they imply limitation to the listed items. It must also be noted that, as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly specifies otherwise.

[0130] Furthermore, one or more computer-readable storage media can be used to implement embodiments consistent with this disclosure. A computer-readable storage medium refers to any type of physical memory on which processor-readable information or data can be stored. Therefore, a computer-readable storage medium can store instructions for execution by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disk drives, CD-ROMs, DVDs, flash drives, magnetic disks, and any other known physical storage media.

[0131] This disclosure and examples are intended to be illustrative only, and the true scope of the disclosed implementations is indicated by the following claims.< / dips> < / iris> < / dips> < / dips> < / iris> < / atcs> < / iris> < / dips>

Claims

1. A method (300) for constructing a processor implementation of a task planning framework for robots for domain-specific use cases, the method comprising: (302) A robot system ontology is provided via one or more hardware processors, the robot system ontology having multiple modules including: a) use cases; b) embodied entities; c) workspace; d) objectives; e) relationships; f) experience; and g) plans for capturing contextual representations of domain-specific use cases, thereby generating task plans for domain-specific tasks using a large language model executed by the one or more hardware processors; By means of the one or more hardware processors, and by means of a hierarchical thought chain framework with said robot system ontology enhancement, a hierarchical thought chain prompt structure for said large language model is derived (304), wherein the training dataset used is defined by a plurality of dataset features including unit size, diversity and uncertainty, wherein said plurality of dataset features satisfy a minimum number of mathematical equations acceptable for the limited instruction trial. The large language model is queried by the robot to perform the domain-specific task by means of a user query as a command via the one or more hardware processors, wherein the user query is supported by static data from the workspace of the robot agent, the static data including system state, multiple current robot parameter states, general perception output of the scene and experience gained by the robot agent from past input-output, and wherein the combination of the static data according to the hierarchical thinking chain prompt structure generates prompts for the large language model; The user query is classified (308) as a valid user query based on whether the large language model response to the prompt falls into one of a plurality of actions defined for the robot to perform a domain-specific task while complying with a set of domain rules, via the one or more hardware processors. The task plan for the domain-specific task is generated (310) by the large language model executed by the one or more hardware processors for the prompts with the valid user query. The task plan is verified (312) via the one or more hardware processors to check for the presence of one or more predefined unsafe operations, thereby generating a verified task plan; and The domain-specific task (314) is performed by the robot according to the verified task plan via the one or more hardware processors.

2. The processor-implemented method according to claim 1, wherein, a) The use cases described enable the definition of the problem statement and the detailed use case description. (b) The embodied entity enables the contraction of multiple intelligent agents to collaborate in order to complete the domain-specific task. c) The workspace enables a test bench to be defined for domain-specific use cases via multiple fields including mobility space, pick-up location, placement location, layout, and objects. d) The stated objective enables the definition of domain rules, performance metrics, and operations. e) The aforementioned relationship enables the constraint of mapping and classification. f) The experience described enables the definition of enhancement, failure, and success, and g) The plan enables the provision of the constraints and generation of the task plan.

3. The processor-implemented method according to claim 2, wherein, The plurality of intelligent agents defined in the embodied entity includes a) physical robots, b) behaviors, c) end-effector tools, and d) sensors that cooperate to complete a specific task, wherein the plurality of intelligent agents are also uniquely identified based on capabilities, interfaces, states, and experience.

4. The processor-implemented method according to claim 1, wherein, The task planning framework: The minimum amount of dataset required to train and validate the hierarchical thought chain technique for the reliable performance of the robot is limited. The robot system body enables the robot to quickly adapt to new use cases to achieve the scalability of the framework. The classification ensures the interactive and efficient operation of the robot, and The verification ensures that the task plan is verified for safety checks before execution.

5. A system (100) for constructing a task planning framework for robots for domain-specific use cases, the system (100) comprising: Memory (102), store instructions; One or more input / output interfaces (106); as well as One or more hardware processors (104) are coupled to the memory (102) via the one or more input / output interfaces (106), wherein the one or more hardware processors (104) are configured by the instructions to: A robot system ontology is provided, the robot system ontology having multiple modules, the multiple modules including: a) use cases; b) embodied entities; c) workspace; d) objectives; e) relationships; f) experience; and g) plans for capturing contextual representations of domain-specific use cases, thereby generating task plans for domain-specific tasks using a large language model executed by the one or more hardware processors; By initializing the cue training with a hierarchical thought chain framework enhanced by the robot system ontology, a hierarchical thought chain cue structure for the large language model is derived, wherein the training dataset used is defined by multiple dataset features including unit size, diversity and uncertainty, wherein the multiple dataset features satisfy the minimum acceptable number of mathematical equations for the limited instruction trials. User queries are used as commands to query the large language model so that the robot can perform the domain-specific task. The user queries are supported by static data from the robot agent's workspace, including system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past input-output. The combination of the static data, constructed according to a hierarchical thought chain prompting structure, generates prompts for the large language model. The user query is classified as a valid user query based on whether the response to the prompt falls into one of a plurality of actions defined for the robot to perform a domain-specific task while adhering to a set of domain rules. The large language model generates a task plan for a domain-specific task based on the prompts given the valid user query. The task plan is validated to check for the presence of one or more predefined unsafe operations, thereby generating a validated task plan; and The robot performs the domain-specific task according to the validated task plan.

6. The system according to claim 5, wherein, a) The use cases described enable the definition of the problem statement and the detailed use case description. (b) The embodied entity enables the contraction of multiple intelligent agents to collaborate in order to complete the domain-specific task. c) The workspace enables a test bench to be defined for domain-specific use cases via multiple fields including mobility space, pick-up location, placement location, layout, and objects. d) The stated objective enables the definition of domain rules, performance metrics, and operations. e) The aforementioned relationship enables the constraint of mapping and classification. f) The experience described enables the definition of enhancement, failure, and success, and g) The plan enables the provision of the constraints and generation of the task plan.

7. The system according to claim 6, wherein, The plurality of intelligent agents defined in the embodied entity includes a) physical robots, b) behaviors, c) end-effector tools, and d) sensors that cooperate to complete a specific task, wherein the plurality of intelligent agents are also uniquely identified based on capabilities, interfaces, states, and experience.

8. The system according to claim 5, wherein, The task planning framework: The minimum amount of dataset required to train and validate the hierarchical thought chain technique for the reliable performance of the robot is limited. The robot system body enables the robot to quickly adapt to new use cases to achieve the scalability of the framework. The classification ensures the interactive and efficient operation of the robot, and The verification ensures that the task plan is verified for safety checks before execution.

9. One or more non-transitory machine-readable information storage media, comprising one or more instructions, which, when executed by one or more hardware processors, cause: A robot system body is provided, the robot system body having multiple modules, the multiple modules including: a) Use cases; b) Embodied entity; c) Workspace; d) Goal; e) Relationship; f) Experience; and g) a plan for capturing contextual representations of domain-specific use cases, thereby generating a task plan for a domain-specific task using a large language model executed by the one or more hardware processors; By initializing the cue training with a hierarchical thought chain framework enhanced by the robot system ontology, a hierarchical thought chain cue structure for the large language model is derived, wherein the training dataset used is defined by multiple dataset features including unit size, diversity and uncertainty, wherein the multiple dataset features satisfy the minimum acceptable number of mathematical equations for the limited instruction trials. User queries are used as commands to query the large language model so that the robot can perform the domain-specific task. The user queries are supported by static data from the robot agent's workspace, including system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past input-output. The combination of the static data, constructed according to a hierarchical thought chain prompting structure, generates prompts for the large language model. The user query is classified as a valid user query based on whether the response to the prompt falls into one of a plurality of actions defined for the robot to perform a domain-specific task while adhering to a set of domain rules. The executed large language model generates a task plan for a domain-specific task based on the prompts given the valid user query. The task plan is validated to check for the presence of one or more predefined unsafe operations, thereby generating a validated task plan; and The robot performs the domain-specific task according to the validated task plan.

10. One or more non-transitory machine-readable information storage media according to claim 9, in, a) The use cases described enable the definition of the problem statement and the detailed use case description. (b) The embodied entity enables the contraction of multiple intelligent agents to collaborate in order to complete the domain-specific task. c) The workspace enables a test bench to be defined for domain-specific use cases via multiple fields including mobility space, pick-up location, placement location, layout, and objects. d) The stated objective enables the definition of domain rules, performance metrics, and operations. e) The aforementioned relationship enables the constraint of mapping and classification. f) The experience described enables the definition of enhancement, failure, and success, and g) The plan enables the provision of the constraints and generation of the task plan; The plurality of intelligent agents defined in the embodied entity includes a) physical robots, b) behaviors, c) end-effector tools, and d) sensors that cooperate to complete a specific task, wherein the plurality of intelligent agents are also uniquely identified based on capabilities, interfaces, states, and experience; and The task planning framework includes: The minimum amount of dataset required to train and validate the hierarchical thought chain technique for the reliable performance of the robot is limited. The robot system body enables the robot to quickly adapt to new use cases to achieve the scalability of the framework. The classification ensures the interactive and efficient operation of the robot, and The verification ensures that the task plan is verified for safety checks before execution.

Images