An LLM-based task planning framework for robots performing domain-specific use cases (DSUs).

Abstract

This provides a novel prompting technique suitable for domain-specific industrial use cases (DSUs) and an empirical methodology for generating LLM tuning dataset sizes to ensure performance. [Solution] A method and system for overcoming the obstacles of robot task planners for DSUs by introducing a task planning framework, the core of which is a robot system ontology that organizes the components of the robot system in a consistent and systematic manner. This ontology enhances the planning framework, which efficiently incorporates the contextual representation of the DSU using LLM. Furthermore, it introduces an LLM tuning method called Hierarchical Chain of Thought (CoHT), which has been specially created to complement the robot system ontology. By integrating these two components, the accuracy, robustness, and throughput of the robot are improved in a cost-effective manner.

Description

Technical Field

[0001] (Cross - reference to related applications and priority) This application claims priority to Indian Patent Application No. 202421096220, filed on December 5, 2024.

[0002] (Technical Field) Embodiments herein generally relate to the field of robotic systems, and more particularly, to methods and systems for constructing a large - language - model (LLM) - based task planning framework for robots that execute domain - specific use cases (DSUs).

Background Art

[0003] The use of large - language models (LLMs) for task planning and reasoning has emerged as an area of interest within the robotics research community. However, when directly applying an LLM, the task - planning performance required for domain - specific industrial use cases (DSUs) is not achieved, even for prompts with extremely large token sizes.

[0004] Recent research results in Microsoft Chat-GPT Robot Manipulation (MCRM) provide a structured and extensible prompting approach usable in DSUs. However, several technical challenges remain unaddressed. For example, failure scenarios are not considered during robot environment updates, multiple user feedback requests are required for accurate task planning, and there is no exemplary sequence-based prompting that is considered a crucial facet for contexturing LLMs in domain-specific tasks (DSTs). Several approaches are used to contextualize LLMs in response to human commands. Some of these include pre-training, fine-tuning, and search augmentation, but these approaches require high computer computation or massive training datasets, raising questions about their usability and application conditions. On the other hand, there are various standard prompting techniques that are readily usable to contextualize any LLM on any custom DSU. These require small datasets or no LLM tuning at all. At the same time, it has been acknowledged and pointed out that it is extremely difficult to easily contextualize any off-the-shelf LLM on a DSU. LLMs cannot repeatedly infer and fully understand the nuances of domain rules and DSUs. Therefore, a structurally robust knowledge representation of the robotic system and DSUs is needed at the LLM's center. This representation must help achieve better contextualization. In a similar vein, recent developments have significantly improved existing prompting capabilities, including Chain of Thought (CoT), Tree of Thought (ToT), and contextualization augmentation of thinking algorithms. Currently, CoT and ToT are prominently used to improve the reasoning capabilities of LLMs. However, CoT is effective for abstract-level contextual knowledge and may miss low-level domain rules in some cases, even in DSTs. CoT is significantly dependent on the size of the LLM, and typically underperforms in smaller LLMs. Also, because the scope for validating generated intermediate reasoning / thoughts is limited, it is more likely to arrive at incorrect solutions.On the other hand, ToT effectively addresses most of these limitations. Considering this, implementing ToT in a practical DSU seems advantageous, but its complexity has limitations. This comes at the cost of frequent output token depletion and computational complexity, impacting the goal of achieving low latency and high throughput in DSTs. Consequently, ToT may not be necessary for tasks where intermediate prompt approaches excel. Therefore, a new prompting technique suitable for DSUs is needed. [Overview of the project]

[0005] Embodiments of this disclosure present technical improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems.

[0006] For example, in one embodiment, a method is provided for constructing a task planning framework for a robot. The method includes providing a robot system ontology having multiple modules, including a plan for capturing (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a contextual representation of a DSU, to generate a task plan for a domain-specific task using a Large Language Model (LLM).

[0007] Furthermore, this method includes a step of deriving a CoHT prompt structure for LLM via a Hierarchical Chain of Thought (CoHT) framework-initialization-prompt training enhanced with a robot system ontology, wherein the training dataset used is defined by multiple dataset features including unit size, diversity, and uncertainty, and these multiple dataset features satisfy a mathematical equation that defines the minimum acceptable number of instruction attempts.

[0008] Furthermore, the method includes a step that allows the robot to query the LLM with user queries as commands to execute domain-specific tasks, the user queries being supported by static data from the robot agent's workspace, the static data including system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past inputs and outputs, and a combination of static data structured according to the CoHT prompt structure generates prompts for the LLM.

[0009] Furthermore, the method includes a step of classifying a user query as a valid user query based on whether the LLM response to a prompt corresponds to an action among several actions defined for the robot to perform a domain-specific task while adhering to a set of domain rules.

[0010] Furthermore, the method includes the step of using LLM to generate a task plan for a domain-specific task for a prompt with a valid user query.

[0011] Furthermore, the method includes a step of validating the task plan to check for the presence of one or more predefined unsafe behaviors in order to generate a valid task plan.

[0012] Furthermore, the method includes the step of having a robot perform domain-specific tasks according to a validated task plan.

[0013] In another embodiment, a system is provided for constructing a task planning framework for a robot. The system comprises a memory for storing instructions, one or more input / output (I / O) interfaces, and one or more hardware processors coupled to the memory via the one or more I / O interfaces, wherein the one or more hardware processors are configured to provide a robot system ontology having multiple modules, including a plan for capturing (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a contextual representation of a DSU, to generate a task plan for a domain-specific task using a Large Language Model (LLM) by the instructions.

[0014] Furthermore, one or more hardware processors are configured to derive CoHT prompt structures for the LLM via a layered chain of thought (CoHT) framework-initialization-prompt-training enhanced with the robot system ontology, and the training dataset used is defined by multiple dataset features including unit size, diversity, and uncertainty, and these multiple dataset features satisfy a mathematical equation that defines the minimum acceptable number of instruction attempts.

[0015] Furthermore, one or more hardware processors are configured to allow the LLM to query user queries as commands for the robot to perform domain-specific tasks. These user queries are supported by static data from the robot agent's workspace, which includes system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past inputs and outputs. Combinations of this static data, structured according to the CoHT prompt structure, generate prompts for the LLM.

[0016] Furthermore, one or more hardware processors are configured to classify user queries as valid user queries based on whether the LLM response to a prompt corresponds to an action among several actions defined for the robot to perform domain-specific tasks while adhering to a set of domain rules.

[0017] Furthermore, one or more hardware processors are configured by the LLM to generate task plans for domain-specific tasks in response to prompts with valid user queries.

[0018] Furthermore, one or more hardware processors are configured to validate the task plan to check for the presence of one or more predefined unsafe behaviors in order to generate a valid task plan.

[0019] Furthermore, one or more hardware processors are configured to allow the robot to perform domain-specific tasks according to a verified task plan.

[0020] In yet another embodiment, one or more non-temporary machine-readable information storage media containing one or more instructions are provided, which, when executed by one or more hardware processors, cause a method for constructing a robot task planning framework.

[0021] Furthermore, one or more instructions, when executed by one or more hardware processors, cause a computing device to provide a robotic system ontology having multiple modules, including (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a plan for capturing a contextual representation of the DSU for generating a task plan for a domain-specific task using a large language model (LLM).

[0022] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device is prompted to derive a CoHT prompt structure for the LLM through a Hierarchical Chain of Thought (CoHT) framework-initialization-prompt training that enhances the robot system ontology, and the training dataset used is defined by multiple dataset features including unit size, diversity, and uncertainty, and the multiple dataset features satisfy a mathematical equation that defines the minimum acceptable number of instruction attempts.

[0023] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device can query the LLM with user queries as commands for the robot to perform domain-specific tasks. User queries are supported by static data from the robot agent's workspace, which includes system state, multiple current robot parameter states, general perception outputs of the scene, and experience gained by the robot agent from past inputs and outputs. A combination of static data structured according to the CoHT prompt structure generates prompts in the LLM.

[0024] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device classifies a user query as a valid user query based on whether the LLM response to a prompt corresponds to an action among several actions defined for the robot to perform a domain-specific task while conforming to a set of domain rules.

[0025] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device causes the LLM to generate a task plan for a domain-specific task in response to a prompt with a valid user query.

[0026] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device is caused to verify a task plan to check for the presence of one or more predefined unsafe operations in order to generate a valid task plan.

[0027] Furthermore, when one or more instructions are executed by one or more hardware processors, the computing device is caused to execute domain-specific tasks by a robot according to a valid task plan.

[0028] It should be understood that both the foregoing summary and the following detailed description are illustrative and explanatory only and are intended to further explain the embodiments of the invention described in the claims.

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments and together with the specification serve to explain these embodiments.

Brief Description of the Drawings

[0030] [Figure 1] FIG. is a functional block diagram of a system (robot) having a large language model (LLM)-based task planning framework for the execution of domain-specific use cases (DSUs) according to some embodiments of the present disclosure.

[0031] [Figure 2A] FIG. is a diagram showing the overall architecture of a task planning framework for the execution of DSUs according to some embodiments of the present disclosure. [Figure 2B] FIG. is a diagram showing the overall architecture of a task planning framework for the execution of DSUs according to some embodiments of the present disclosure. [Figure 2C] FIG. is a diagram showing the overall architecture of a task planning framework for the execution of DSUs according to some embodiments of the present disclosure.

[0032] [Figure 3A] This flowchart illustrates how to construct a task planning framework for executing domain-specific use cases (DSUs) by System 100 (Robot) according to several embodiments of the present disclosure. [Figure 3B] This flowchart illustrates how to construct a task planning framework for executing domain-specific use cases (DSUs) by System 100 (Robot) according to several embodiments of the present disclosure.

[0033] [Figure 4A] This figure shows the components of a system task planning framework according to some embodiments of the present disclosure. [Figure 4B] This figure shows the components of a system task planning framework according to some embodiments of the present disclosure.

[0034] [Figure 5] This figure shows a testbed of a DSU executed by a system (robot) according to some embodiments of the present disclosure.

[0035] [Figure 6A] This figure shows snippets of how a chain of hierarchical thoughts (CoHT) is incorporated into a few-shot prompt, according to some embodiments of the present disclosure. [Figure 6B] This figure shows snippets of how a chain of hierarchical thoughts (CoHT) is incorporated into a few-shot prompt, according to some embodiments of the present disclosure. [Modes for carrying out the invention]

[0036] Those skilled in the art will understand that any block diagram in this specification represents a conceptual diagram of an exemplary system and device embodying the principles of the subject matter of the present invention. Similarly, any flowchart, flow chart, etc., whether or not such a computer or processor is explicitly shown, will be understood to represent various processes that can be performed by such a computer or processor, substantially represented in a computer-readable medium.

[0037] Exemplary embodiments will be described with reference to the accompanying drawings. In the drawings, the leftmost digit of the reference number identifies the drawing in which the reference number first appears. For convenience, the same reference number is used throughout the drawings to refer to the same or similar parts. While embodiments and features of the disclosed principle are described herein, modifications, alterations, and other embodiments are possible without departing from the scope of the disclosed embodiments.

[0038] Embodiments of this disclosure provide a method and system (robot) for a Large-Scale Language Model (LLM)-based task planning framework for robots performing domain-specific use cases (DSUs). Applying Large-Scale Language Models (LLMs) fails to achieve the task planning performance required for domain-specific industrial use cases (DSUs), even with extremely large token-size prompts. The disclosed method and system overcomes the limitations of robot task planners for DSUs by introducing a task planning framework. At the heart of the framework is a robot system ontology that organizes the components of the robot system in a consistent and systematic manner. This ontology enhances the planning framework, which efficiently incorporates the contextual representation of the DSU using LLMs. Furthermore, this study introduces an LLM tuning scheme called Hierarchical Chain of Thought (CoHT), specifically designed to complement the robot system ontology. Integrating these two components enables cost-effective improvements in the robot's accuracy, robustness, and throughput. Additionally, an empirical methodology is provided that leverages heuristic-based methods to generate LLM tuning dataset sizes to guarantee performance.

[0039] Herein, preferred embodiments are shown with reference to the drawings, more specifically to Figures 1 to 6B, where the same reference numerals indicate features corresponding throughout the drawings, and these embodiments are described in relation to the following exemplary systems and / or methods.

[0040] Figure 1 is a functional block diagram of a system 100 (robot) with a task planning framework for executing domain-specific use cases (DSUs) according to several embodiments of the present disclosure.

[0041] In one embodiment, the system 100, also called a robot or robotic system, includes a processor 104, a communication interface device alternatively referred to as an input / output (I / O) interface 106, and one or more data storage devices or memories 102 operably coupled to the processor 104. The system 100 having one or more hardware processors is configured to perform the functions of one or more functional blocks of the system 100.

[0042] Referring to the components of system 100, in one embodiment, the processor 104 may be one or more hardware processors 104. In one embodiment, the one or more hardware processors 104 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or devices that process signals based on operation instructions. Among other capabilities, the one or more hardware processors 104 may be configured to fetch and execute computer-readable instructions stored in memory 102. In one embodiment, system 100 may be implemented in a variety of computing systems, including laptop computers, notebooks, handheld devices such as mobile phones, workstations, mainframe computers, servers, and the like.

[0043] The I / O interface 106 can include various software and hardware interfaces, such as a web interface and a graphical user interface, and can facilitate multiple communications within a wide variety of network and protocol types, such as wired networks like LANs and cables, and wireless networks like WLANs and cellular networks. In one embodiment, the I / O interface 106 can include one or more ports for connecting to multiple external devices or other servers or devices. Multimodal format user query commands for instructing the LLM to perform tasks of interest in a particular domain are received via the I / O interface 106.

[0044] The memory 102 may include any computer-readable medium known in the art, such as volatile memory like static random-access memory (SRAM) and dynamic random-access memory (DRAM), and / or non-volatile memory like read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and magnetic tape.

[0045] In one embodiment, the memory 102 includes a plurality of modules 110, such as an LLM, a robot ontology system module, etc.

[0046] Multiple modules 110 include programs or coded instructions that complement applications or functions performed by system 100 to perform various steps in the process of building an LLM-based task planning framework, also known as the LLM-RSPF, for the execution of domain-specific use cases (DSUs) performed by system 100. Multiple modules 110 may include, among other things, routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. Multiple modules 110 may also be used as signal processors, node machines, logic circuits, and / or any other devices or components that manipulate signals based on operation instructions. Furthermore, multiple modules 110 can be used by hardware, by computer-readable instructions executed by one or more hardware processors 104, or by a combination thereof. Multiple modules 110 may include various submodules (not shown).

[0047] Furthermore, the memory 102 may include information relating to the processor 104 of the system 100 and the inputs / outputs of each step performed by the method of this disclosure.

[0048] Furthermore, memory 102 includes a database 108. The database (or repository) 108 may include multiple abstracted code elements for refinement and data processed, received, or generated as a result of the execution of multiple modules in module 110.

[0049] Although database 108 is shown as being internal to system 100, it should be noted that in an alternative embodiment, database 108 may also be implemented externally to system 100 and communicately coupled to system 100. Data contained in such an external database can be updated periodically. For example, new data can be added to the database (not shown in Figure 1), and / or existing data can be modified, and / or useless data can be deleted from the database. In one example, data may be stored in an external system such as a Lightweight Directory Access Protocol (LDAP) directory and a relational database management system (RDBMS). The functions of the components of system 100 will now be described with reference to the steps in the flowcharts of Figures 2A to 5.

[0050] Figures 2A to 2C illustrate the overall architecture of a task planning framework according to several embodiments of this disclosure. Figures 2A to 2C are described in conjunction with Figures 3A and 3B.

[0051] Figures 3A and 3B are flowcharts illustrating how to construct a task planning framework (LLM-RSPF) for executing domain-specific use cases (DSUs) by System 100 (Robot) according to several embodiments of this disclosure.

[0052] In one embodiment, system 100 comprises one or more data storage devices or memories 102 operably coupled to a processor 104, configured to store instructions for the execution of steps of method 200 by the processor or one or more hardware processors 104. Next, the steps of method 200 of the present disclosure will be described with reference to the components or blocks of system 100 as shown in Figures 1 and 2 and the steps in the flowchart as shown in Figure 3. Process steps, method steps, and techniques, etc., may be described in a sequential order, but such processes, methods, and techniques may be configured to operate in an alternative order. In other words, the order or sequence of steps that can be described does not necessarily indicate a requirement that the steps be executed in that order. The steps of the processes described herein may be executed in any executable order. Furthermore, some steps may be executed simultaneously.

[0053] Referring to Figure 2B, which illustrates the process flow and steps of Method 300 of the Task Planning Framework (LLM-RSPF), in step 302, one or more hardware processors 104 are configured by instruction to provide a robot system ontology having multiple modules. The multiple modules of the robot system ontology are depicted in Figure 4A and include a plan for capturing (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a contextual representation of the DSU, in order to generate a task plan for a domain-specific task using a Large-Scale Language Model (LLM). (a) Use cases allow for the definition of problem statements and detailed use case descriptions. (b) The embodiment makes it possible to define multiple agents that cooperate to accomplish domain-specific tasks. (c) The workspace allows you to define a DSU testbed through multiple fields including movement space, pick location, drop location, placement, and objects. (d) The purpose is to enable the definition of domain rules, performance metrics, and behavior; (e) Relationships enable the definition of mappings and classifications. (f) Experience makes it possible to define augmentation, failure, and success. (g) The plan enables the definition and generation of the task plan.

[0054] As shown in Figure 4B, different agents of the embodiment, namely (a) a physical robot, (b) behavior, (c) an end-eb arm tool (EOAT), and (d) a sensor, cooperate to accomplish a specific task, and the agents are further uniquely identified based on their capability, interaction, state, and experience.

[0055] The Use Case module is the parent module, providing the problem statement and a detailed use case description. This establishes an abstract representation of the use case for any LLM. The statement and description are the two components responsible for providing this.

[0056] The modules of the embodiment define various agents that cooperate and participate to accomplish a specific task. Each agent is uniquely identified by four components: (a) a physical robot, (b) behavior, (c) an end-eb arm tool (EOAT), and (d) sensors. The modules of the embodiment are further refined based on their capabilities, interfaces, states, and experiences, as shown in Figure 4B. The capabilities of the embodiment are categorized into forms of sensing and action derived from the ReAct approach, known from the literature entitled "React: Synergizing reasoning and acting in language models, 2023."

[0057] Sensing enhances embodiments with general perceptual, visual, and tactile capabilities. General perceptual components are typically visually-based, open-vocabulary object detection, providing real-time workspace information. Next, Action provides “Act” capabilities depending on the agent’s actions. This “Act” capability here refers to the robot’s skills. Each robot skill is essentially a combination of multiple minimal unit skills such as seeing, moving, picking up, and placing, possessing mature perceptual, manipulative, and locomotion capabilities. The background reason for assigning skills as composite skills is to alleviate planning complexity and to allow for system scalability through the integration of new mature skills. Agent skills have their own attribute definitions from physical workspace components. Agent a’s Kth skill is defined in a study of the literature titled “Do as i can, not as i say: Grounding language in robotic affordances, 2022”.

[0058] The forward movement and state components represent the current agent state. This can include robot joint angles, gripper attachment status, etc. Finally, experience represents agent-level success and failure experiences.

number

[0059] The Workspace module defines the testbed for the DSU. Essentially, it consists of five configurable components. The Movement Space defines the detailed movement space, comprising all location identifiers and movement indicators (if any). Objects are the target items that the embodiment is expected to handle when performing any task. Pick-and-Drop Locations refer to locations specifically assigned within the workspace for any target objects to be picked up or dropped by the embodiment. Finally, Placement determines the physical placement of different agents within the workspace for different location identifiers.

[0060] The Objective module defines the task's objectives for the DSU. This module sets all the domain rules related to the DSU that must be met during any task plan. In short, the generated plan must, in any way, steadily execute the domain rules. Performance metrics are a list of metrics used to evaluate the performance of the embodied system against the generated plan. Operational components determine the overall operational objectives that must be met, although it is unclear whether they are actually functions of the performance metrics.

[0061] The relation module identifies relational mappings between component definitions, either within or between modules. For example, an agent can be associated with picking from a specific location within the workspace. Mappings correspond to these relational mappings. Next, classification categorizes user commands based on their nature and provides options for appropriately mapping them to different DSU scenarios. This component is crucial in command classification and conversational responses generated by the LLM. This facilitates the UI's decision-making job.

[0062] The experience module (memory) maintains various system or task-related intermediate states and stores overall success and failure scenarios. This module helps the LLM recall past experiences and incorporate them into future task planning. Failure experiences are crucial here because they help significantly reduce system latency and limit recurring failure (Re-Failure) scenarios. Furthermore, chat history can be used to store experiences using either truncation or the RAG method.

[0063] The planning module consists of task plan definitions and template-specific conditions that must be followed when generating the task plan. As described in "Web Ontology Language: OWL, pages 91-110. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009," a dictionary format is used for the robot skill definitions and their order in the agent's plan, while the template for the final task plan is denoted as P in "Taxonomy of educational objectives: The classification of educational goals. Longmans, Green, 1956." The generation components define the various types of inputs required to generate the final task plan.

[0064] These inputs can be things like State, Sensing, and Objective.

number

number

[0065] In step 304 of Method 300, referring to Figure 2C, one or more hardware processors 104 are configured to derive a Hierarchical Chain of Thought (CoHT) prompt structure for the LLM by instruction via CoHT framework-initialization-prompt training augmented with the robot system ontology. The training dataset by CoHT framework-initialization-prompt training is defined by a plurality of dataset features including unit size, diversity, and uncertainty, where the plurality of dataset features satisfy the following mathematical equation 5, which indicates the minimum acceptable number of instruction attempts.

[0066] LLM-based task planning method: • Dataset Preparation: To begin the planning process, it is crucial to ensure that the task plan representation is properly defined in the planning module. The DSU dataset is a prerequisite for achieving good contextualization of the LLM. The complete dataset for contextualization is divided into three parts: training, validation, and testing, each used in a 1:1:2 ratio. Note that the term "training" here is used in relation to the LLM prompt and does not refer to training or fine-tuning the LLM from scratch. The following three types of estimations are performed regarding the creation of the dataset. • Unit Size: Based on the task plan representation and classification defined in the planning and relation modules, the dataset creation and its unit size are estimated. Here, the term "unit size" refers to the smallest standard dataset size that can be used to estimate the dataset sizes for training, validation, and testing. Here, the sum of all instruction classifications is m, and m^ is the unit vector corresponding to each classification label. The sum of robot skills derived from equation (2) is S. If the Mth classification is the only classification that outputs a valid task plan for all agents A, then the sum of instruction combinations corresponding to these two inputs is:

number

number

number

number

number

number

number

number

[0067] In step 306 of Method 300, referring to Figure 2B, one or more hardware processors 104 are configured with instructions that allow the robot to query the LLM with user queries as commands to perform domain-specific tasks. As depicted in Figures 2A and 2B, the user queries are supported by static data from a workspace defined by the robot to perform domain-specific tasks. The static data includes system states, current states of multiple robot parameters, general perception outputs of the scene, and experiences gained by the robot from past inputs and outputs. A combination of static data structured according to the CoHT prompt structure generates prompts for instructing the LLM.

[0068] In step 308 of Method 300, referring to Figure 2B, one or more hardware processors 104 are configured, by instruction, to classify user queries as valid user queries based on whether the LLM response to the query corresponds to an action among several actions defined for the robot to perform a domain-specific task while conforming to a set of domain rules. This classification ensures the interactive and efficient operation of the robot. Relevant queries are accepted. Through classification, the task planning framework of System 100 (the robot) can notify the user to provide only relevant commands. In some situations, a user query to the LLM is something that the LLM can generally respond to, but the user query is not related to industrial operation, for example, a user query such as "Who won the cricket match?" or something trivial. In such situations, System 100 can refuse the response that the LLM can normally generate and require the user to use a command that the robot should execute. Furthermore, the LLM's common sense capabilities are used when the task planning framework of System 100 generates a plan, so that the number of steps to perform to complete the task is minimized and the system can be made more efficient.

[0069] In step 310 of method 300, referring to Figure 2B, one or more hardware processors 104 are configured by instruction to generate task plans for domain-specific tasks when the prompt is classified as a valid user query for task plan generation.

[0070] In step 312 of Method 300, referring to Figure 2B, one or more hardware processors 104 are configured to validate the task plan by instruction to check for the presence of one or more predefined unsafe actions before clearing a valid task plan. Validation ensures that the task plan is validated before execution for a safe check. The LLM may generate an inappropriate plan due to an incorrect instruction or some error, and furthermore, the robot may perform such an unsafe action; however, the plan validator allows such unsafe plans to be aborted, providing intelligent safe operation by the robot. Furthermore, if there is a user query that is very close to a valid executable action but lacks some data, the robot can assist the user by prompting for this additional data, leading to interactive and efficient operation. For example, an instruction that lacks an object name, or a pick position, or a drop position, or an instruction with multiple items that can be identified as a cup (e.g., not specifying a color if multiple colored cups exist), or a combination of all of these.

[0071] In step 314 of method 300, referring to Figure 2B, one or more hardware processors 104 are configured, by instruction, to execute domain-specific tasks via a robot according to a valid task plan. All operations in the classification and execution steps are stored as experience or memory for task planning of future user queries. 〔experiment〕

[0072] This section discusses the execution of the LLM-RSPF (System 100 in Figures 1 and 2A) and planning scheme on an actual DSU, and the resulting outcomes. The LLM-RSPF is used in a retail order fulfillment system with a robotic system ontology defined below, and the experiment is described using a robot and workspace as shown in Figure 5.

[0073] Use cases Problem statement: The robot's objective is to receive and understand natural language queries and commands from humans, act according to DSU domain rules, and ultimately successfully execute within an acceptable tolerance level that provides high throughput, success rate, and low latency. • Description: The use case is order processing in retail, where a robot picks objects from multiple bins based on user instructions, following a Good-to-Picker model.

[0074] Embodiment: An embodiment of the system is an agent<agent_1> Two fixed base industrial arms (primary)<robot_1> and secondary<robot_2> ) has. The user commands the primary robot, which has the following capabilities, to perform a task. System 100 (robot) has open vocabulary object detection-based general perception capabilities. In the action, Agent 1 has the following robotic capabilities: 1. <DIPS>: Domain-Independent Picking Skills (DIPS) are unsupervised learning-based perceptual pick-and-place skills. Skill attributes include the number of objects, pick location, and drop location. 2. <iris>The Instance retrieval picking skill (IRIS) uses supervised learning-based perceptual abilities to perform picking in a domain-dependent environment. Skill attributes include the number of objects, object class name, pick location, and drop location. 3. <atcs>The Automatic Tool Changer Skill (ATCS) is a custom motion skill that moves a manipulator arm between two points according to specific movements (speed and force). The ATCS is responsible for changing the EOAT. The skill attributes are the current EOAT and the next EOAT. Since each picking skill defined above uses a different EOAT, the robot's EOAT is incorporated as a State.

[0075] The workspace of the DSU is shown in Figure 3. It has two categories of pick locations for high-order and low-order items: homogeneous bins (Bin2 and Bin3) and heterogeneous bins (Bin1). It has three drop locations, a conveyor (to send items to the secondary robot), a carton / box (to pack the order), and a location for user search. (a) Items available in Bin1: red cups, brown cups, Dove soap, Cinthol soap, Mogra soap, and coconut oil (b) Bin2: ThumsUp (black soft drink bottles) (c) Items available in Bin3: Frooti (aseptically packaged beverages).

[0076] the purpose • Domain Rules: DIPS skills use the Robotiq 2f-85 (two-finger gripper) as the EOAT for gripping objects, while IRIS uses the Robotiq Epick (vacuum gripper). However, this is not a mandatory criterion for running the skill in a typical scenario. Objects in homogeneous bins are frequently ordered items. The order of these bins can change at runtime, and it is the responsibility of general awareness to check the nature of the bins before executing user queries.

[0077] Relationship: In this DSU, five classes of instructions are possible: valid, invalid (unfeasible tasks), general (general queries), system information instructions (SII, system-related queries), and additional data requests (ADR, instructions that request additional information for feasible tasks).

[0078] Planning: Plan generation is a three-stage generation method: first, instruction classification is performed to classify the instructions; then, plan generation is performed; and finally, plan validation is performed to refine the generated plan according to domain rules. Planning is a sequential combination of the three skills described above.

[0079] Details of the additional implementation of the retail order processing use case:

[0080] Below are some detailed descriptions of the LLM-RSPF ontology modules that correspond to the retail order processing system described earlier. • Status: Considered here<agent_1> This state is the robot's EOAT, i.e.<current_eoat> Therefore, since each picking skill defined above uses a different EOAT, a robot EOAT is incorporated as a state. • Interface: This represents comment communication within an agent, between agents, or between an agent and a user. This includes inter-agent communication of object transfer messages between two robots and task-related conversations between agents and users. • Experience: In this DSU, only failure scenarios are considered to avoid ReFailure scenarios. • Pick location: Bin1 contains heterogeneous objects, targeting objects from a high-mix, low-volume category. Bin2 contains a uniform set of objects. Bin3 contains a uniform set of objects, but different from those available in Bin2. • Drop location: Drop1 (conveyor) is,<robot_1> from<robot_2> Drop2 (box) is a flat belt conveyor for transferring objects. Drop2 (box) is a carton or retail packaging box for sending objects for packing or order processing. Drop3 (user) is a convenient fixed position for the user to retrieve the items. ·Placement:<robot_1> and<robot_2> These are located at the front and rear ends of the conveyor, respectively. Regarding the robotic motion skills of the "Embodiment," there are specific reasons behind the choice to employ Domain Independent Picking Skills (DIPS) and Instance Search Picking Skills (IRIS). These skills are inspired by the Good-to-Picker model in retail order processing. DIPS is advantageous when the system must quickly adapt to new objects without training based on deep learning. DIPS is effective for uniform items, these items can change completely, and the system can quickly adapt to them. IRIS, on the other hand, targets heterogeneous items and is aimed at object classes from high-mix low-volume categories, a fairly popular concept in the order fulfillment industry. This helps achieve high space utilization. JPEG2026098919000013.jpg17170 · Classification: -Valid: The plan is valid because all information related to the plan is available in the instructions. -Invalid: The robot system is being asked to perform a task that is beyond its capabilities. General: Please direct unrelated inquiries regarding robot systems. System Information Instruction (SII): The SII requests system-related information, such as workspace details and embodiment details. Additional Data Request (ADR): The ADR is related to the robot system and is mostly valid, but some information is missing.

[0081] Proposal for a planning method for the system • Dataset: Estimation of dataset unit size: Considering five instruction classifications, three robot skills, a single agent, ε' = 5, and five instructions per class, the dataset unit size is 40 according to equation (5). The total dataset sizes for training, effectiveness, and test, with a specified ratio of 1:0.5:2, are 40, 20, and 80, respectively. These 140 instructions are created using 14 human oracles that summarize the framework, purpose, and instruction classification. • Prompt: CoHT, used as an exemplary sequence of multi-shot prompts, can be understood from the examples shown in Figures 6A and 6B. Note that these figures are not complete prompts, but snippets illustrating how CoHT is incorporated into multi-shots to produce better results. First, CoHT essentially exercises a multi-shot prompt. Any DSU is interpreted and converted into a base prompt. Next, standard prompt styles using placeholders, context separation using symbolic cues, use of tone or style, context priming, etc., are used to structure and clarify the prompt. This also helps in setting the optimal positioning of text domain rules. Subsequently, the next part of the prompt defines the multi-shot example. It is found that the complex reasoning capabilities of the LLM are significantly controlled by domain examples in DSUs compared to abstractly representing domain rules in the form of contextually separated paragraphs. As a result, it becomes necessary to carefully craft multi-shots with sufficient diversity for the LLM to gain contextual understanding in proper plan generation. One of the main advantages of CoHT is the extent to which it can prove hard-bound rules in each hierarchical thinking step within a linear inference process, which greatly helps improve the inference of LLMs to any particular event that would otherwise be extremely difficult to achieve. However, while increasing the number of small-shot examples improves the performance of LLMs, it has been observed that maintaining a single extremely complex example instead of multiple simple examples significantly improves the performance of LLMs. This also serves another purpose by shortening the token length by replacing multiple examples with a single example.

[0082] (result) Cost comparison of different LLMs considered in LLM-RSPF (System 100): Table 1 shows a cost analysis of five proprietary LLMs and two open-source LLMs. [Table 1]

[0083] The cheapest and most expensive GPT models available are GPT3.5 and GPT4. The different input tokens specified for each LLM are the result of prompt adaptation and fine-tuning over time. In terms of performance, GPT4 is the most efficient of all and consumes the fewest tokens, while GPT3.5 is the worst. GPT4-Turbo and Gemini-pro are comparable, but Gemini-pro is inferior in frequent exhaustion of output tokens and inefficient use of tokens in long and complex plans. Both open-source models easily outperform GPT3.5 and share performance with a fine-tuned GPT3.5. While a fine-tuned GPT3.5 may be a favorable choice, it is important to note that it requires high computational power or a massive dataset and is considerably more expensive. In contrast, the method and system 100 (LLM-RSPF) disclosed herein achieves a similar objective using only a few shots of LLM prompts for cost reduction. Key hyperparameters to consider when configuring the LLM output include temperature, top p, frequency penalty, and presence penalty. Considering both cost and performance, GPT4-Turbo is a reasonable and good choice for evaluation.

[0084] A comparison of LLM-RSPF (System 100) with other LLM-based planning frameworks such as MCRM and ProgPrompt (PgmPmt) is generated, and the evaluation results generated on the test dataset are shown in Table 2. [Table 2] JPEG2026098919000016.jpg238161 JPEG2026098919000017.jpg224161 JPEG2026098919000018.jpg233160 JPEG2026098919000019.jpg193162 JPEG2026098919000020.jpg102162

[0085] A comparison of LLM-RSPF (System 100) with two other closely related task planning methods is presented. These two studies include MCRM and ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. The effectiveness of LLM-RSPF is also concisely expressed in Table 3. LLM-RSPF significantly outperforms both methods in both DST planning accuracy and classification accuracy. MCRM is found to be suitable for simple to moderately complex plans and instructions that are unambiguous and easy to reason. Several improvements were made to the implementation of MCRM to avoid unnecessary environment changes and frequent user-level feedback. On the other hand, ProgPrompt works well only for simple plans and does not work well in highly domain-specific scenarios. Despite LLM-RSPF's good evaluation metric scores, there are instructions that all three methods failed to correctly classify. In the 17th case ("Invalid"), LLM fails to recheck the number of fruity after picking one, resulting in an incorrect classification, while in the 34th case ("ADR"), it assumes the Mogra soap's pick position is Drop 3.

[0086] Comparison of CoHT with CoT and ToT: The effectiveness of CoHT disclosed by System 100 is shown in Table 4 compared to CoT and ToT techniques using exemplary sequences in a few-shot prompt. Poor and good LLMs, namely GPT3.5 and GPT4-Turbo, are considered for generating results. The evaluation was performed against the “effective” instructions in Table 2. It can be seen that CoHT, an exemplary sequencing, is superior in terms of planning accuracy to both CoT and ToT LLMs. However, GPT-4 Turbo has a significantly higher margin. The average output token length (OTL) of CoHT is almost the same as ToT, but slightly higher than CoT.

[0087] The extensibility of LLM-RSPF will be tested by extending the DSU as described in Section 4. This extension will be done by increasing (a) agent capabilities (b) sensing and behavior and (c) workspace. First, as a mobile manipulator<agent_2> We will introduce,<agent_1> Add two manipulation robots below it. They will have different sensing properties for touch.<agent_1> Two new robot skills will be added to correspond to the two new robots below. The workspace will be modified by adding a movement space and a pick-and-drop location.<agent_1> The mobile manipulator under Agent 2 is capable of flexible pick-and-drop, while the manipulator under Agent 2 is capable of fixed pick-and-drop. Due to space limitations, a detailed framework for extended use cases will not be described here, but it follows a similar approach to that described earlier. The planning accuracy of the extended DSU using GPT4-Turbo was approximately 0.91. This result confirms that the adoption of a modular framework like LLM-RSPF and complex robotic skills is sufficient to create a case for an extensible LLM-based task planner. [Table 3] [Table 4]

[0088] This method and system 100 provide a task planning framework, also known as a Large-Scale Language Model-Based Robot System Planning Framework (LLM-RSPF), tailored for domain-specific use cases (DSUs). This framework includes two main components: a specialized robot system ontology designed for DSUs, and an LLM tuning method called CoHT (Chain of Hierarchical Thought) that complements the proposed ontology to achieve a cost-effective LLM context. Furthermore, an empirical quantification of the prompt dataset is proposed to optimize LLM tuning. To evaluate the effectiveness of LLM-RSPF, it was applied to real-world DSUs in the retail and packaging industry. Comparative experiments with common proprietary and open-source LLM models were conducted, and a cost-benefit analysis was performed. Furthermore, LLM-RSPF demonstrated superior accuracy in plan generation and query classification tasks compared to prominent literature such as Microsoft+ChatGPT and ProgPrompt. The LLM tuning method CoHT disclosed here is compared to the CoT and ToT methods, highlighting the robustness of CoHT. Finally, by increasing the number of agents, adding sensing modalities such as haptic feedback, and expanding the entities in the workspace, we conducted scalability tests, and LLM-RSPF consistently achieved a planning accuracy of 91%.

[0089] This specification describes the subject matter of the present invention in such a way that those skilled in the art can carry out and utilize the embodiments. The scope of the embodiments of the subject matter is defined by the claims and may include other modifications that will be conjured up by those skilled in the art. Such other modifications shall be within the scope of the claims if they have similar elements that do not differ from the language of the claims, or if they include equivalent elements that differ only slightly from the language of the claims.

[0090] The scope of protection extends beyond such programs to computer-readable means having messages internally, and such computer-readable storage means includes program code means for performing one or more steps of the method when the program is executed on a server or mobile device or any suitable programmable device. The hardware device can be any type of programmable device, including any type of computer, such as a server or personal computer, or the same, or any combination thereof. The device may also include means that may be hardware means such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, such as an ASIC and an FPGA, or at least one memory having at least one microprocessor and internally located software processing components. Thus, the means may include both hardware and software means. Embodiments of the method described herein can be implemented in hardware and software. The device may also include software means. Alternatively, embodiments may be implemented using different hardware devices, such as multiple CPUs.

[0091] Embodiments of this specification may include hardware and software elements. Software-implemented embodiments include, but are not limited to, firmware, resident software, and microcode. Functions performed by the various components described herein may be implemented by other components or combinations thereof. In this specification, a computer-usable medium or computer-readable medium may be any device capable of storing, communicating, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0092] The illustrated steps are set up to illustrate the illustrated exemplary embodiments, and it should be expected that the way in which specific functions are performed may change due to ongoing technological development. These examples are presented herein for illustrative purposes only and are not limiting. Furthermore, boundaries of functional units are arbitrarily defined herein for the convenience of explanation. Alternative boundaries can be defined as long as the specified functions and their relationships are adequately performed. Alternatives (including equivalents, extensions, modifications, deviations, etc., of those described herein) will be apparent to those skilled in the art based on the teachings contained herein. Such alternatives are within the scope of the disclosed embodiments. Also, the terms “equipment,” “possess,” “encompass,” and “include,” and other similar forms, are intended to be semantically equivalent and open-ended in that the one or more items following any one of these terms do not mean an exhaustive list of such one or more items, nor do they mean that the list is limited to only the one or more items. Furthermore, it should be noted that, as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural anaphora unless the context clearly indicates otherwise.

[0093] Furthermore, one or more computer-readable storage media may be used when carrying out embodiments consistent with the Disclosure. Computer-readable storage media refers to any type of physical memory capable of storing information or data that is readable by a processor. Thus, computer-readable storage media can store instructions for execution by one or more processors, including instructions for causing a processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable media" should be understood to include tangible objects and exclude carrier waves and transient signals, i.e., non-transient. Examples of such media include random-access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, and any other known physical storage media.

[0094] This disclosure and examples are merely illustrative, and the true scope of the disclosed embodiments shall be indicated by the appended claims.< / atcs> < / iris>

Claims

1. A processor implementation method (300) for constructing a robot task planning framework for domain-specific use cases (DSUs), (302) Steps include providing a robot system ontology having multiple modules, including (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a plan for capturing a contextual representation of a DSU, via one or more hardware processors, using a Large Language Model (LLM) executed by the one or more hardware processors, (304) A step of deriving a CoHT prompt structure for the LLM via one or more hardware processors through a Hierarchical Chain of Thought (CoHT) framework initialization-prompt-training enhanced in the robot system ontology, wherein the training dataset used is defined by a plurality of dataset features including unit size, diversity and uncertainty, and the plurality of dataset features satisfy a mathematical equation that defines the minimum acceptable number of instruction attempts. (306) A step that enables querying the LLM with a user query as a command to perform a domain-specific task by the robot via the one or more hardware processors, wherein the user query is supported by static data from the robot agent's workspace, the static data includes system state, a plurality of current robot parameter states, general perceptual outputs of the scene, and experience obtained by the robot agent from past inputs and outputs, and a combination of the static data structured according to a CoHT prompt structure generates a prompt for the LLM. The step of classifying the user query as a valid user query via the one or more hardware processors, based on whether the LLM response to the prompt corresponds to an action among a set of actions defined for the robot to perform the domain-specific task in accordance with a set of domain rules (308), (310) A step in which the LLM, which is executed by the one or more hardware processors, generates the task plan for the domain-specific task for the prompt having the valid user query, The step of verifying the task plan to check for the presence of one or more predefined unsafe operations via the one or more hardware processors in order to generate a valid task plan (312), The steps include (314) having a robot execute the domain-specific task in accordance with a verified task plan via one or more hardware processors, Methods that include...

2. (a) The use case enables the definition of a problem statement and a detailed use case description, (b) The embodiments enable the definition of multiple agents that cooperate to accomplish the domain-specific task, (c) The workspace allows defining the testbed of the DSU through multiple fields including movement space, pick position, drop position, placement, and objects, (d) The objective is to enable the definition of domain rules, performance metrics, and behavior, (e) The above relationship makes it possible to define mapping and classification, (f) The above experience makes it possible to define enhancement, failure, and success, (g) The plan enables the definition and generation of the task plan, The method according to claim 1.

3. The method according to claim 2, wherein the plurality of agents defined in the above embodiment consist of (a) physical robots, (b) behaviors, (c) end-of-arm tools (EOATs), and (d) sensors, which cooperate to accomplish a specific task, and the plurality of agents are further uniquely identified based on their capabilities, interfaces, states, and experiences.

4. The aforementioned task planning framework, To ensure the reliable performance of the robot, define the minimum number of datasets required to train and validate the CoHT technology. The robot system ontology enables the robot to quickly adapt to new use cases for framework extensibility, The aforementioned classification ensures the interactive and efficient operation of the robot. The aforementioned verification ensures that the task plan is verified before execution for safety checks. The method according to claim 1.

5. A system (100) for building a robot task planning framework for domain-specific use cases (DSUs), A memory (102) for storing instructions, One or more input / output (I / O) interfaces (106), One or more hardware processors (104) connected to the memory (102) via the one or more I / O interfaces (106), Equipped with, The one or more hardware processors (104) described above, in accordance with the instructions, The steps include providing a robot system ontology having multiple modules, including (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a plan for capturing a contextual representation of a DSU, using a large-scale language model (LLM) executed by one or more hardware processors, A step of deriving a CoHT prompt structure for the LLM via a Hierarchical Chain of Thought (CoHT) framework—initialization—prompt—training—enhanced by the robot system ontology, wherein the training dataset used is defined by a plurality of dataset features including unit size, diversity, and uncertainty, and the plurality of dataset features satisfy a mathematical equation that defines the minimum acceptable number of instruction attempts. A step that enables querying the LLM with a user query as a command to execute a domain-specific task by the robot, wherein the user query is supported by static data from the robot agent's workspace, the static data includes system state, multiple current robot parameter states, general perception outputs of the scene, and experience obtained by the robot agent from past inputs and outputs, and a combination of the static data structured according to a CoHT prompt structure generates a prompt for the LLM. The steps include classifying the user query as a valid user query based on whether the LLM response to the prompt corresponds to an action among several actions defined for the robot to perform the domain-specific task in accordance with a set of domain rules, The LLM comprises the steps of generating the task plan for the domain-specific task for the prompt having the valid user query, The steps include verifying the task plan to check for the presence of one or more predefined unsafe actions in order to generate a valid task plan, The steps include: having a robot perform the domain-specific task according to a verified task plan; A system configured to perform the following actions.

6. (a) The use case enables the definition of a problem statement and a detailed use case description, (b) The embodiments enable the definition of multiple agents that cooperate to accomplish the domain-specific task, (c) The workspace allows defining the testbed of the DSU through multiple fields including movement space, pick position, drop position, placement, and objects, (d) The objective is to enable the definition of domain rules, performance metrics, and behavior, (e) The above relationship makes it possible to define mapping and classification, (f) The above experience makes it possible to define enhancement, failure, and success, (g) The plan enables the definition and generation of the task plan, The system according to claim 5.

7. The plurality of agents defined in the above embodiment consist of (a) physical robots, (b) behaviors, (c) end-of-arm tools (EOATs), and (d) sensors, which cooperate to accomplish a specific task, and the plurality of agents are further uniquely identified based on their capabilities, interfaces, states, and experiences. The system according to claim 6.

8. The aforementioned task planning framework, To ensure the reliable performance of the robot, define the minimum number of datasets required to train and validate the CoHT technology. The robot system ontology enables the robot to quickly adapt to new use cases for framework extensibility, The aforementioned classification ensures the interactive and efficient operation of the robot. The aforementioned verification ensures that the task plan is verified before execution for safety checks. The system according to claim 5.

9. One or more non-temporary machine-readable information storage media containing one or more instructions, wherein the instructions, when executed by one or more hardware processors, The steps include providing a robot system ontology having multiple modules, including (a) a use case, (b) an embodiment, (c) a workspace, (d) an objective, (e) a relationship, (f) an experience, and (g) a plan for capturing a contextual representation of a DSU, using a large-scale language model (LLM) executed by one or more hardware processors, A step of deriving a CoHT prompt structure for the LLM through a Hierarchical Chain of Thought (CoHT) framework initialization-prompt-training enhanced by the robot system ontology, wherein the training dataset used is defined by a plurality of dataset features including unit size, diversity, and uncertainty, and the plurality of dataset features satisfy a mathematical equation that defines the minimum acceptable number of instruction attempts. A step that enables querying the LLM with a user query as a command to execute a domain-specific task by the robot, wherein the user query is supported by static data from the robot agent's workspace, the static data includes system state, multiple current robot parameter states, general perception outputs of the scene, and experience obtained by the robot agent from past inputs and outputs, and a combination of the static data structured according to a CoHT prompt structure generates a prompt for the LLM. The steps include classifying the user query as a valid user query based on whether the LLM response to the prompt corresponds to an action among several actions defined for the robot to perform the domain-specific task in accordance with a set of domain rules, The LLM comprises the steps of generating the task plan for the domain-specific task for the prompt having the valid user query, The steps include verifying the task plan to check for the presence of one or more predefined unsafe actions in order to generate a valid task plan, The steps include: having a robot perform the domain-specific task according to a verified task plan; One or more non-temporary machine-readable information storage media that perform the following actions.

Images