Robotic systems, methods, control modules, and computer program products utilizing a

Abstract

Robot control systems, methods, control modules, and computer program products are described that utilize one or more large language models (LLMs) in order to achieve at least some degree of autonomy. The robot control parameters and / or instructions may advantageously be specified in a natural language (NL) and communicate with the LLM via NL prompts or queries. The LLM module provides a task plan in the NL, which may be evaluated for at least one fault or error. If at least one fault or error is identified, the LLM module may be queried to provide a solution.

Description

[0001] Previous application data

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 441,897, filed on January 30, 2023, entitled “Robot Control Systems, Methods, and Computer Program Products That Leverage Large Language Models,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The systems, methods, control modules, and computer program products of the present application generally relate to robotic control, and particularly to deploying, utilizing, and / or generally using large language models in robotic control.

[0004] background

[0005] Description of Related Technology

[0006] A robot is a machine that can be deployed to perform a task. Robots can come in a variety of form factors (including humanoid form factors). Humanoid robots can be operated by teleoperations, which allow the robot to simulate the physical movements of a human operator or driver. Specialized robots can be designed to perform a specific task, while general-purpose robots can be designed to perform multiple tasks.

[0007] Humans perform many tasks in their personal and work lives. Examples include making beds, washing dishes, loading the dishwasher, mowing the lawn, taking inventory, checking out customers, stocking shelves, painting, styling hair, preparing meals, cleaning, measuring, performing calculations, recording data, performing analysis, creating art / music, performing art / music, building, manufacturing, assembling, destroying, disassembling, moving objects, picking and placing, navigating, and so on. In many cases, there is a strong desire and ongoing need to automate various tasks so that humans can redirect their time and / or attention to other things.

[0008] Large Language Models (LLMs) are a form of artificial intelligence that is trained on a large corpus of text data to produce human-like text responses to natural language (NL) inputs. Popular examples in the field today include OpenAI TMVarious incarnations of the Generative Pre-Trained Transformer (GPT) such as text-davinci-003, text-curie-001, text-babbage-001, and text-ada-001. LLM can be accessed through or deployed in text-based user interfaces to allow chat-like interaction between users and computers, such as in OpenAI TM LLM-based GPT-3 TM ChatGPT built in series TM application.

[0009] Brief Overview

[0010] One approach can be summarized as using at least one large language model (LLM) in the operation of a robot as described herein. The LLM can be utilized to automate at least one process selected from the group consisting of: task planning, motion planning, human interaction, and logical reasoning.

[0011] The robotic system can be generalized to use at least one large language model as described herein.The LLM can be utilized to automate at least one process selected from the group consisting of: task planning, motion planning, human interaction, and logical reasoning.

[0012] The computer program product may be summarized as a non-transitory processor-readable storage medium storing data and / or processor-executable instructions that, when executed by at least one processor of a robot, causes the robot to utilize at least one large language model as described herein. The LLM may be utilized to automate at least one process selected from the group consisting of: task planning, motion planning, human interaction, and logical reasoning.

[0013] According to a broad aspect, the present disclosure describes a method of operating a robotic system including a robotic agent, the method comprising: capturing, by at least one sensor of the robotic system, sensor data representing information about an environment of the robotic agent; generating, by at least one processor of the robotic system, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing, to a large language model (LLM) module, a first NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receiving, from the LLM module, a task plan expressed in NL and comprising a plurality of steps; determining, by the at least one processor, whether the task plan includes at least one fault; if the task plan includes at least one fault: generating, by the at least one processor, an NL description of the at least one fault; providing, to the LLM module, a second NL query comprising at least an NL description of the at least one fault and an NL request for a resolution plan; receiving, from the LLM module, a resolution plan expressed in NL; and executing, by the robotic system, an updated task plan based at least in part on the resolution plan; and if the task plan does not include at least one fault: executing the task plan to completion by the robotic system.

[0014] The method may further include: if the mission plan includes at least one fault, generating an NL description of any steps in the mission plan that do not include the fault; and the second NL query may further include the NL description of any steps in the mission plan that do not include the fault.

[0015] The solution plan may include an NL description of at least one solution selected from the group of solutions consisting of: adjustment to the order of steps in the task plan; adjustment to the success criteria for any step in the task plan; adjustment to any action expressed in any step of the task plan; adjustment to any interaction goal for any step of the task plan; replacement of any step of the task plan with at least one new step; removal of at least one step of the task plan; and updating the task plan to replace all steps of the task plan.

[0016] Determining whether the mission plan includes at least one fault may include: executing at least one step of the mission plan; and, for each of the at least one step, identifying whether execution of the corresponding step produces a corresponding fault scenario. Each corresponding fault scenario may include at least one scenario selected from the group consisting of: execution of the step produces a result inconsistent with a work objective; execution of the step prevents execution of at least one other step in the mission plan; and execution of the step produces at least one unacceptable effect in the environment.

[0017] Determining whether the mission plan includes at least one fault may include determining whether the mission plan violates any rule in a set of rules specified in at least a portion of the inference engine.

[0018] Determining whether the mission plan includes at least one fault may include: accessing an environment model representing the environment; simulating, by at least one processor, at least one step of the mission plan in the environment model; and identifying, by at least one processor, whether each of the at least one step of the mission plan produces a corresponding fault scenario when executed in the simulation. Determining whether the mission plan includes at least one fault may also include: simulating, by at least one processor, at least one additional step of the mission plan in the environment model if no corresponding fault scenario is identified in each of the at least one step of the simulated mission plan; and identifying, by at least one processor, whether each of the at least one additional step of the mission plan produces a corresponding fault scenario when executed in the simulation. The corresponding fault scenario may include at least one scenario selected from the group consisting of: simulated execution of the step produces a simulated result that is inconsistent with the work objective; simulated execution of the step prevents execution of at least one other step in the mission plan; and simulated execution of the step produces at least one unacceptable effect in the environment model.

[0019] According to another broad aspect, the present disclosure describes a robot control module that includes at least one non-transitory processor-readable storage medium storing processor-executable instructions or data that, when executed by at least one processor of a processor-based system, causes the processor-based system to: capture sensor data representing information about an environment of the robot body by at least one sensor carried by a robot body of the processor-based system; generate, by the at least one processor, a natural language (NL) description of at least one aspect of the environment based on the sensor data; and provide a first NL query to a large language model (LLM) module, the first NL query including the NL description of at least one aspect of the environment, the NL description of a work goal, and a set of instructions executable by the processor-based system. a NL description of the at least one fault and a NL request for a solution plan; receiving a mission plan from an LLM module, the mission plan being expressed in NL and comprising a plurality of steps; determining, by at least one processor, whether the mission plan includes at least one fault; if the mission plan includes at least one fault: generating, by at least one processor, an NL description of the at least one fault; providing a second NL query to the LLM module, the second NL query comprising at least an NL description of the at least one fault and a NL request for a solution plan; receiving a solution plan from the LLM module, the solution plan being expressed in NL; and executing, by a processor-based system, an updated mission plan, the updated mission plan being based at least in part on the solution plan; and if the mission plan does not include at least one fault: executing, by the processor-based system, the mission plan to completion.

[0020] The processor-executable instructions or data may also cause the at least one processor to: if the mission plan includes at least one fault, generate an NL description of any steps in the mission plan that do not include the fault; and the second NL query may also include an NL description of any steps in the mission plan that do not include the fault.

[0021] The solution plan may include an NL description of at least one solution selected from the group of solutions consisting of: an adjustment to the order of steps in the task plan; an adjustment to the success criteria for any step in the task plan; an adjustment to any action expressed in any step of the task plan; an adjustment to any interaction goal for any step of the task plan; replacement of any step of the task plan with at least one new step; removal of at least one step of the task plan; and an updated task plan for replacing all steps of the task plan.

[0022] Processor-executable instructions or data that cause at least one processor to determine whether a mission plan includes at least one fault may cause the processor-based system to: execute at least one step of the mission plan; and, for each of the at least one step, identify, by the at least one processor, whether execution of the corresponding step produces a corresponding fault scenario. Each corresponding fault scenario may include at least one scenario selected from the group consisting of: execution of the step produces a result inconsistent with a work objective; execution of the step prevents execution of at least one other step in the mission plan; and execution of the step produces at least one unacceptable effect in an environment.

[0023] The processor-executable instructions or data that cause the at least one processor to determine whether the mission plan includes at least one fault may cause the at least one processor to determine whether the mission plan violates any rule in a set of rules specified in at least a portion of the inference engine.

[0024] The processor-executable instructions or data that cause at least one processor to determine whether a mission plan includes at least one fault may cause the at least one processor to: access an environmental model representing an environment; simulate, by the at least one processor, at least one step of the mission plan in the environmental model; and identify, by the at least one processor, whether each of the at least one step of the mission plan, when executed in the simulation, produces a corresponding fault scenario. The processor-executable instructions or data that cause the at least one processor to determine whether a mission plan includes at least one fault may further cause the at least one processor to: simulate, by the at least one processor, at least one additional step of the mission plan in the environmental model if no corresponding fault scenario is identified in each of the at least one step of the simulated mission plan; and identify, by the at least one processor, whether each of the at least one additional step of the mission plan, when executed in the simulation, produces a corresponding fault scenario. The corresponding fault scenario may include at least one scenario selected from the group consisting of: simulated execution of the step produces a simulated result that is inconsistent with a work objective; simulated execution of the step prevents execution of at least one other step in the mission plan; and simulated execution of the step produces at least one unacceptable effect in the environmental model.

[0025] According to yet another broad aspect, the present disclosure describes a robotic system comprising: a robotic body; at least one sensor; a robotic controller comprising at least one processor and at least one non-transitory processor-readable storage medium, the at least one non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the at least one processor, cause the robotic system to: capture sensor data representing information about an environment of the robotic body by the at least one sensor; generate, by the at least one processor, a natural language (NL) description of at least one aspect of the environment based on the sensor data; provide a first NL query to a large language model (LLM) module, the first NL query comprising the NL description of at least one aspect of the environment, the NL description of a work goal, a machine learning algorithm, and a plurality of other algorithms. an NL description of an instruction set executable by a human system and an NL request for a mission plan; receiving the mission plan from an LLM module, the mission plan expressed in NL and comprising a plurality of steps; determining, by at least one processor, whether the mission plan includes at least one fault; if the mission plan includes at least one fault: generating, by at least one processor, an NL description of the at least one fault; providing a second NL query to the LLM module, the second NL query comprising at least an NL description of the at least one fault and an NL request for a solution plan; receiving a solution plan from the LLM module, the solution plan expressed in NL; and executing, by the robotic system, an updated mission plan, the updated mission plan being based at least in part on the solution plan; and if the mission plan does not include at least one fault: executing the mission plan to completion by the robotic system.

[0026] The task plan may indicate at least one action executable by the robotic system expressed in a natural language; and the processor-executable instructions may further cause the robotic system, by at least one processor, to generate a robotic language task plan based on the task plan expressed in NL, the robotic language task plan including a robotic control instruction set that, when executed by at least one processor, causes the robotic system to perform at least one of the at least one action indicated in the task plan.

[0027] Brief Description of Several Drawings

[0028] The various elements and actions depicted in the accompanying drawings are provided for illustrative purposes to support the detailed description. Unless the specific context requires otherwise, the size, shape, and relative positions of the elements and actions shown are not necessarily shown to scale and are not necessarily intended to convey any information or limitations. Generally, the same reference numerals are used to identify similar elements or actions.

[0029] Figure 1is a flow chart illustrating an exemplary embodiment of an automated mission planner utilizing an LLM according to the systems, control modules, methods, and computer program products of the present application.

[0030] Figure 2 is a flow chart illustrating another exemplary embodiment of the operation of a robotic system utilizing an LLM according to the systems, control modules, methods, and computer program products of the present application.

[0031] Figure 3 is a flow chart illustrating yet another exemplary embodiment of the operation of a robotic system utilizing an LLM according to the systems, control modules, methods, and computer program products of the present application.

[0032] Figure 4A is an illustrative diagram of an exemplary embodiment of a robot with access to an LLM according to the systems, control modules, methods, and computer program products of the present application, wherein the LLM is stored and executed externally to the robot (e.g., in the cloud) and is called or accessed by the robot control system.

[0033] Figure 4B is an illustrative diagram of an exemplary embodiment of a robot having access to an LLM according to the systems, control modules, methods, and computer program products of the present application, wherein the LLM is stored and executed locally on the robot as an integral part of the robot's control system.

[0034] Figure 5 is an illustrative diagram of an exemplary robotic system that includes various features and components described throughout the systems, control modules, methods, and computer program products of this application.

[0035] Detailed description

[0036] The following description sets forth specific details in order to illustrate and provide an understanding of various embodiments and examples of the systems, methods, control modules, and computer program products of the present application. Those skilled in the art will appreciate that some of the specific details described herein may be omitted or modified in alternative embodiments and examples, and that the various embodiments and examples described herein may be combined with each other and / or with other methods, components, materials, etc. to produce additional embodiments and examples.

[0037] In some instances, well-known structures and / or processes associated with computer systems and data processing have not been shown or provided in detail to avoid unnecessarily complicating or obscuring the description of the implementations and examples.

[0038] Unless the specific context requires otherwise, throughout this specification and the appended claims, the term "comprise" and variations thereof (e.g., "comprises" and "comprising") are used in an open, inclusive sense to mean "including, but not limited to."

[0039] Throughout this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the specific context requires otherwise. For example, reference to "an embodiment" and "the embodiment" include "embodiments" and "the embodiments," respectively, and reference to "an implementation" and "the implementation" include "implementations" and "the implementations," respectively. Similarly, the term "or" is generally employed in its broadest sense to mean "and / or" unless the specific context clearly dictates otherwise.

[0040] The title and abstract of the present disclosure are provided for convenience only and are not intended to and should not be construed as interpreting the scope or meaning of the systems, methods, control modules, and computer program products of the present application.

[0041] Various embodiments described herein provide systems, methods, control modules, and computer program products for enhancing, facilitating, augmenting, or enabling control of one or more robotic systems using one or more LLMs. Exemplary robotic systems that may employ the teachings of the systems, methods, control modules, and computer program products of the present application include, but are not limited to, a general-purpose humanoid robot developed by Sanctuary Cognitive Systems, Inc., various aspects of which are described in the following documents: U.S. patent application serial number 18 / 375,943, U.S. patent application serial number 18 / 513,440, U.S. patent application serial number 16 / 940,566 (publication number US2021-0031383 A1), U.S. patent application serial number 17 / 023,929 (publication number US2021-0090201 A1), U.S. patent application serial number 17 / 061,187 (publication number US2021-0122035 A1), U.S. patent application serial number 17 / 098,716 (publication number US2021-0146553 A1), U.S. patent application serial number 17 / 111,789 (publication number US2021-0170607 A1), U.S. Patent Application Serial No. 17 / 158,244 (Publication No. US2021-0234997A1), U.S. Provisional Patent Application Serial No. 63 / 001,755 (Publication No. US2021-0307170 No. 63 / 293,968, U.S. Provisional Patent Application Serial No. 63 / 293,973 and / or U.S. Provisional Patent Application Serial No. 63 / 278,817, each of which is incorporated herein by reference in its entirety.

[0042] In some embodiments, a robotic system or control module may employ a finite instruction set comprising general reusable work primitives that can be combined (in various combinations and / or arrangements) to perform tasks. For example, a robotic control system may store a library of reusable work primitives, each work primitive corresponding to a corresponding basic subtask or sub-action (hereinafter referred to as an instruction set) that the robot is operable to perform autonomously. The work objective may be analyzed to determine a sequence (i.e., combination and / or arrangement) of reusable work primitives that, when executed by the robot, will complete the work objective. The robot may execute a sequence of reusable work primitives to complete the work objective. In this way, a finite instruction set may be used to perform a wide range of different types of tasks and work objectives across a wide range of industries. The method is described in U.S. Patent Publication No. 2022-0258340, based on U.S. patent application serial number 17 / 566,589, which is incorporated herein by reference in its entirety.

[0043] To expand on the above, a general-purpose robot can accomplish a plurality of different work objectives. As used throughout this specification and the appended claims, the term "work objective" refers to a specific task, job, assignment, or application with a specified goal and a determinable result, typically (although not necessarily) to facilitate some economically valuable work. Work objectives exist in many aspects of business, research and development, commercial operations, and personal activities. Exemplary work objectives include, but are not limited to: cleaning locations (e.g., bathrooms) or objects (e.g., bathroom mirrors), preparing meals, loading / unloading storage containers (e.g., trucks), taking inventory, collecting one or more samples, taking one or more measurements, building or assembling objects, destroying or disassembling objects, transmitting articles, receiving objects and / or data, and the like. Various embodiments described herein provide robots, systems, control modules, computer program products, and methods for operating a robotic system to at least semi-autonomously accomplish tasks or work objectives.

[0044] According to the robots, systems, control modules, computer program products, and methods of the present application, a work goal can be deconstructed or broken down into a "workflow" comprising a set of one or more "work primitives," wherein the successful completion of the work goal involves executing each work primitive in the workflow. Depending on the specific implementation, completion of the work goal can be achieved by (i.e., the workflow can include the following): i) executing the corresponding set of work primitives sequentially or continuously; ii) executing the corresponding set of work primitives in parallel; or iii) executing the corresponding set of work primitives in any combination of sequential and parallel execution (e.g., sequentially overlapping) appropriate to the work goal and / or the robots executing the work goal. Thus, in some implementations, work primitives can be interpreted as lower-level activities, steps, or subtasks that are implemented or executed as a workflow to complete a higher-level work goal.

[0045] Advantageously, according to the robots, systems, control modules, computer program products and methods of the present application, a catalog of "reusable" work primitives can be defined. A work primitive is reusable if it can be universally called, executed, adopted or applied in completing multiple different work objectives. For example, a reusable work primitive is a work primitive that is common to the corresponding workflows of multiple different work objectives. In some embodiments, a reusable work primitive may include at least one variable defined when calling the work primitive or before calling the work primitive. For example, "picking up an *object*" can be a reusable work primitive, where the process of "picking up" can generally be performed at least semi-autonomously to facilitate multiple different work objectives, and the *object* to be picked up can be defined based on the specific work objective being pursued.

[0046] As previously described, various embodiments described herein provide robots, systems, control modules, computer program products, and methods in which the robots are capable of performing tasks or completing work objectives at least semi-autonomously. Unless the specific context requires otherwise, the term "autonomously" is used throughout this specification and the appended claims to mean "not under the control of another party," while the term "semi-autonomously" is used to mean "at least partially autonomously." In other words, throughout this specification and the appended claims, the term "semi-autonomously" means "under limited control by another party," unless the specific context requires otherwise. An example of a semi-autonomous robot is one that can independently and / or automatically perform and control some of its own low-level functions (such as its mobility and grasping functions), but relies on some external control for high-level instructions (such as what to do and / or how to do it).

[0047] According to the robots, systems, control modules, computer program products and methods of the present application, a catalog of reusable work primitives can be defined, identified, developed or constructed so that any given work objective across multiple different work objectives can be completed by executing a corresponding workflow that includes a specific combination and / or arrangement of reusable work primitives selected from the catalog of reusable work primitives. Once such a catalog of reusable work primitives has been established, one or more robots can be trained to autonomously or automatically execute each individual reusable work primitive in the catalog of reusable work primitives without having to include the following context: i) the specific workflow of which the specific reusable work primitive being trained is a part, and / or ii) any other reusable work primitives that may precede or follow the specific reusable work primitive being trained in the specific workflow. In this way, a semi-autonomous robot can operate to autonomously or automatically execute each individual reusable work primitive in a catalog of reusable work primitives, and only requires instructions, directions, or guidance from another party (e.g., an operator, user, or driver) when deciding which reusable work primitive to execute and / or in what order. In other words, an operator, user, driver, or LLM module can provide a workflow consisting of reusable work primitives to a semi-autonomous robotic system, and the semi-autonomous robotic system can autonomously or automatically execute the reusable work primitives according to the workflow to complete a work objective. For example, a semi-autonomous humanoid robot can operate to autonomously look left when instructed to look left, autonomously open its right end effector when instructed to open its right end effector, and so on, without relying on a third party for detailed low-level control of such functions. Once given instructions regarding a workflow that details which reusable work primitives it must execute and in what order in order to complete the work objective, such a semi-autonomous humanoid robot can autonomously complete the work objective. In addition, according to the robots, systems, methods, control modules and computer program products of the present application, if the robotic system is trained or otherwise configured (for example, via consultation with an LLM module, which can be included in the robotic system) to analyze work objectives and independently define the corresponding workflow itself by deconstructing the work objectives into a set of reusable work primitives from a library of reusable work primitives that the robotic system can operate to autonomously execute.

[0048] In the context of a robotic system, a reusable work primitive may correspond to a basic low-level function that the robotic system is operable to perform (e.g., autonomously or automatically) and that the robotic system can call or execute in order to achieve something. Examples of reusable work primitives for a humanoid robot include, but are not limited to: look up, look down, look left, look right, move right arm, move left arm, close right end effector, open right end effector, close left end effector, open left end effector, move forward, turn left, turn right, move backward, etc., as well as cognitive functions such as analysis, calculation, planning, determination, reasoning, etc.; however, those skilled in the art will recognize that: i) the foregoing list of exemplary reusable work primitives for a humanoid robot is by no means exhaustive; ii) in the robots, systems, control modules, computer program products, and methods of the present application, the high-level functions that the robot is operable to perform are deconstructed or broken down into a set of basic components or constituent parts, referred to throughout this specification and the appended claims as "work primitives." Unless the specific context requires otherwise, work primitives may be interpreted as building blocks for constructing higher-level robotic functions.

[0049] In some embodiments, training a robotic system to autonomously execute reusable work primitives can be accomplished in a real-world environment or a simulated environment. Once the robot has been trained to autonomously execute a catalog of reusable work primitives, the robot's operations can be abstracted to the level of reusable work primitives; for example, an LLM module that prepares a task plan for the robot can do so by determining which reusable work primitives to execute and, in some embodiments, in what order to execute them, and the robot can have sufficient autonomy or automation to execute a complete work objective based on this limited control instruction.

[0050] As mentioned previously, "clean the bathroom mirror" is an illustrative example of a work goal that can be deconstructed into a set of work primitives to achieve the goal, and whose results are determinable. In this case, the goal is a clean bathroom mirror, and an exemplary set of work primitives (or workflows) to accomplish the work goal are as follows:

[0051]

[0052] Those skilled in the art will appreciate that the exemplary workflow described above, comprising nine work primitives, is used as an illustrative example of a workflow that can be deployed to accomplish the work objective of cleaning a bathroom mirror; however, the precise definition and composition of each work primitive, as well as the specific combination and / or arrangement of work primitives selected / executed to accomplish the work objective (i.e., the specific configuration of the workflow), can vary in different embodiments according to the present robots, systems, control modules, computer program products, and methods. For example, in some embodiments, work primitives 3, 4, and 5 described above (i.e., positioning the mirror, directing the cleaning solution toward the mirror, and dispensing the cleaning solution onto the mirror) can all be combined into one higher-level work primitive, such as "spraying cleaning solution onto the mirror," while in other embodiments, these same work primitives can be decomposed into additional lower-level work primitives, such as, for example:

[0053] Positioning mirror

[0054] Identifying the mirror's boundaries

[0055] Aim the cleaning solution at the first location within the boundaries of the mirror

[0056] Squeeze cleaning solution

[0057] Aim the cleaning solution at a second location within the boundaries of the mirror

[0058] Squeeze cleaning solution

[0059] etc.

[0060] Based on the above examples and description, those skilled in the art will recognize that the granularity of work primitives can vary across different implementations of the robots, systems, control modules, computer program products, and methods of the present application. Furthermore, according to the robots, systems, control modules, computer program products, and methods of the present application, work primitives are advantageously "reusable" in the sense that each work primitive can be employed, invoked, applied, or "reused" in the execution of more than one overall work objective. For example, while cleaning a bathroom mirror may involve the work primitive "grab cleaning solution," other work objectives may also utilize the "grab cleaning solution" work primitive, such as, for example, "clean toilet," "clean windows," and / or "clean floor." In some implementations, work primitives can be abstracted to become more general. For example, "grab cleaning solution" can be abstracted to become "grab spray bottle" or "grab *object1*," where the *object1* variable is defined as "*object1* = spray bottle," while "locate mirror" can be abstracted to become "locate the object to be sprayed" or simply "locate *object2*," where "*object2* = mirror." In such a case, the "grab spray bottle" work primitive can be used for tasks that do not involve cleaning, such as "paint walls" (where spray bottle = spray paint), "style hair" (where spray bottle = hair spray), or "stir-fry meal" (where spray bottle = cooking oil spray).

[0061] Unless the specific context requires otherwise, throughout this specification and the appended claims, references to "LLM" or "LLM module" should be interpreted as including one or more LLMs or one or more LLM modules. In addition, references to interacting with an LLM or LLM module (e.g., providing input to the LLM, receiving output from the LLM, making queries to the LLM, querying the LLM, etc.) can be made through an application or interface that uses the LLM module (e.g., a chat application that accesses the LLM to interpret input and formulate output).

[0062] In some embodiments of the systems, methods, control modules, and computer program products of the present application, an LLM is used to help determine a sequence of reusable work primitives (hereinafter referred to as "instructions") selected from a limited library of reusable work primitives (hereinafter referred to as an "instruction set") that, when executed by a robot, will enable the robot to complete a task or enable the robot to complete a task. For example, a robot control system may take as input a natural language (NL) command and return a task plan formed by a sequence of permissible instructions extracted from the instruction set, the completion of which realizes the intent of the NL input. An example application is to "assemble" a chess set consisting of sixteen white chess pieces and sixteen black chess pieces. A person can say or type to the robot, for example, "Put all the white chess pieces in the right box and all the black chess pieces in the left box," and the LLM can support a fully autonomous system that converts this input into a sequence of permissible instructions that successfully executes the task. In this case, the LLM can help enable the robot to perform a general task specified in the NL. General tasks include, but are not limited to, all jobs in the current economy.

[0063] Throughout the systems, methods, control modules and computer program products of the present application, the term "natural language" refers to any language that has evolved naturally in humans, and includes but is not limited to: English, French, Spanish, Chinese (Mandarin, Cantonese, Wu, etc.), Portuguese, Japanese, Russian, Korean, Arabic, Hebrew, German, Polish, Hindi, Bengali, Italian, Punjabi, Vietnamese, Hausa, Swedish, Finnish, etc.

[0064] Figure 1 is a flowchart illustrating an exemplary embodiment of an automated task planner as method 100. LLM101 (such as, but not limited to, OpenAI TM The text-davinci-03) is provided or otherwise combined with: i) a description of the current scene 102 generated using the perception system of the robotic system, ii) a description of an instruction set 103 formed by parameterized reusable work primitives or instructions that can be executed by the robotic system, and iii) a wrapper that incorporates NL phrases into prompts 104 to automatically generate a task plan 105, which can then be executed by the robotic system to realize the intent of the NL input.

[0065] Although Figure 1The scene description 102 and instruction set 103 are shown as being contained “inside” the robot (i.e., within the dashed box representing the robot), and the NL prompts 104, LLM 101, and mission plan 105 (and the conversion to robot control signals) are shown as being “outside” the robot (i.e., outside the dashed box representing the robot), but this is not intended to be limiting and is done for illustrative purposes only. While it is true that in some embodiments, any or all of the NL prompts 104, LLM 101, and / or mission plan 105 (and the conversion to robot control signals) may be stored, generated, accessed, or otherwise present outside (or outside or remote from) the robot, in other embodiments, any or all of the NL prompts 104, LLM 101, and / or mission plan 105 (and the conversion to robot control signals) may be stored, generated, accessed, or otherwise present inside (or on) the robot.

[0066] The following is conversation 1, which includes example queries and responses, obtained by running the python script:

[0067] Dialogue 1

[0068] Query 1:

[0069] On a table are White Chess Pawn 1, White Chess Pawn 2, Black Chess Pawn 1, Black Chess Pawn 2, a White Chess Rook, a Black Chess Bishop, and two blue boxes. Across from me stands a person. My goal is to do what this person says. This person says, "Put all the White pieces in the right box and all the Black pieces in the left box." I can only choose from the following options:

[0070] 1.) Grab the object

[0071] 2.) Place the object on the object

[0072] 3.) Look at the object

[0073] 4.) Slice Object

[0074] 5.) Say hello to that person

[0075] 6.) Put objects into objects

[0076] Where objects should be replaced with the appropriate objects on the table. Here is a list of the above commands I should execute to accomplish my goal:

[0077] Response 1:

[0078] Step 1. Capture White Chess Pawn 1

[0079] Step 2. Place White Chess Pawn 1 in the box on the right

[0080] Step 3. Capture White Chess Pawn 2

[0081] Step 4. Place White Chess Pawn 2 in the box on the right

[0082] Step 5. Capture Black Chess Pawn 1

[0083] Step 6. Place Black chess pawn number 1 in the box on the left

[0084] Step 7. Capture Black Chess Pawn No. 2

[0085] Step 8. Place Black Chess Pawn 2 in the box on the left

[0086] Step 9. Capture the White Chess Rook

[0087] Step 10. Place the White chess rook in the box on the right

[0088] Step 11. Capture Black's Chess Bishop

[0089] Step 12. Place the Black chess bishop in the box on the left

[0090] Finish.

[0091] In the above example, the response provided by the LLM corresponds to the mission plan. If the robot system executes the sequence of instructions specified in the mission plan, the task specified in the NL via the prompt will be successfully completed by the robot system. Throughout this disclosure, the term "motion plan" may be used instead of "task plan." In this regard, the sequence of instructions specified in the mission plan (motion plan) may include instructions that cause the robot to undergo a series of motions or movements.

[0092] In some embodiments, LLM can be used to autonomously troubleshoot mission plans, such as Figure 2As shown in the flowchart of FIG. For example, if a given mission plan 201 fails to execute at 202 (i.e., fails to proceed to completion and / or fails to complete the intended task) or encounters an error or failure as shown at 204, an NL prompt can be sent (returned) to the LLM at 205. The NL prompt includes all successful parts of the executed mission plan, as well as additional wording describing the failures and asking the LLM what to do next. This NL prompt can be generated and sent autonomously. In addition, an external checker can review the proposed plan and reject it for some reason. In some embodiments, the external checker can virtually simulate the robot's execution of the mission plan (i.e., it can simulate 202 in FIG. 202). In other embodiments, the external checker can be a logic-based system, such as Cycorp's CYC Machine Reasoning AI platform. For example, the reason for rejection can be a safety violation related to the robot's safety or the safety of any human or other living being. In the event of rejection, the NL prompt or query can be sent back to the LLM and modified to prevent the plan from failing the external check. If no errors or failures are found and / or the mission plan executes correctly, success is achieved as shown at 203.

[0093] Throughout this disclosure, references to an error or failure in a mission plan refer to at least one problem in the mission plan such that executing the mission plan will produce an undesirable result or undesirable scenario. Figure 3 A detailed example is discussed in action 310.

[0094] Figure 3 is a flow chart illustrating an exemplary method 300 of operation of a robotic system. Figure 3 The method 300 in is similar in at least some respects to Figure 1 Method 100 and Figure 2 Method 200 in. Generally speaking, Figure 3 The method 300 in the embodiment of the present invention describes a method that can be implemented by Figure 1 Method 100 and Figure 2 Method 300 is a detailed implementation of the method 200 in the embodiment of the present invention. Figure 5In general, throughout this specification and the appended claims, a method of operating a robotic system is a method in which at least some, if not all, of the various actions are performed by the robotic system. For example, certain actions of the method of operating a robotic system may be performed by at least one processor or processing unit (hereinafter referred to as a "processor") of the robotic system communicatively coupled to a non-transitory processor-readable storage medium of the robotic system (collectively, a robotic controller of the robotic system), and in some embodiments, certain actions of the method of operating a robotic system may be performed by peripheral components of the robotic system communicatively coupled to the at least one processor, such as one or more physically actuatable components (e.g., arms, legs, end effectors, grippers, hands), one or more sensors (e.g., optical sensors, audio sensors, tactile sensors, haptics sensors), mobility systems (e.g., wheels, legs), communication and networking hardware (e.g., receivers, transmitters, transceivers), and the like. The non-transitory processor-readable storage medium of the robotic system can store data (including, for example, at least one library of reusable work primitives and at least one associated perception library) and / or processor-executable instructions that, when executed by at least one processor, cause the robotic system to perform a method and / or cause the at least one processor to perform the actions of the method performed by the at least one processor. The robotic system can communicate with a remote system and / or a remote non-transitory processor-readable storage medium via communication and networking hardware communicatively coupled to the at least one processor of the robotic system. Therefore, unless the specific context requires otherwise, references to the non-transitory processor-readable storage medium of the robotic system and the data and / or processor-executable instructions stored in the non-transitory processor-readable storage medium are not intended to be limiting with respect to the physical location of the non-transitory processor-readable storage medium relative to the at least one processor of the robotic system and the rest of the robotic hardware. In other words, unless the specific context requires otherwise, the non-transitory processor-readable storage medium of the robotic system can include non-transitory processor-readable storage medium located on the robot body of the robotic system and / or non-transitory processor-readable storage medium located remotely from the robot body. Furthermore, a method of operating a robotic system, such as method 300 (or any other method discussed herein), can be implemented as a robotic control module or computer program product. Such a control module or computer program product includes processor-executable instructions or data that, when stored on a non-transitory processor-readable storage medium of the robotic system and executed by at least one processor of the robotic system, causes the robotic system to perform the actions of the method.

[0095] return Figure 3 , the method 300 shown includes actions 302, 304, 306, 308, and 310, although those skilled in the art will appreciate that in alternative implementations, certain actions may be omitted and / or additional actions may be added. Those skilled in the art will also appreciate that the order of actions shown is shown for exemplary purposes only and may be changed in alternative implementations.

[0096] At 302, sensor data representing information about the environment of a robotic body of a robotic system is captured. To this end, the robotic body may carry at least one exemplary sensor that captures sensor data, as described later with reference to FIG. Figure 5 As discussed. In some embodiments, the captured sensor data can be comprehensive, providing a detailed and relatively complete representation of the robot's environment (e.g., a full field of view around the robot's body within a certain distance, with detailed representations of objects or features in the environment surrounding the robot's body). However, such detailed sensor data is not required. In other embodiments, the sensor data may represent only a portion of the environment surrounding the robot's body (e.g., a limited field of view visible to the robot's body's image sensor). In still other embodiments, the sensor data may be even more limited, such as representing only a single object or feature of the environment. How detailed the sensor data representing information about the environment should be can be determined with appropriate precision for a given application.

[0097] At 304, at least one processor of the robotic system generates a natural language (NL) description of at least one aspect of the environment based on the sensor data. Such NL description is Figure 1 The NL description of at least one aspect of the environment need not describe every feature or object present in the sensor data, but may focus on one or more features or objects of particular relevance.

[0098] In an exemplary embodiment, at least one processor executes an object or feature detection model (e.g., a classification module such as a YOLO model or any other suitable model) that identifies objects or features in the environment (as represented in the sensor data) and assigns text labels to such features or objects. Such text labels may be "robot language." Throughout this disclosure, the term "robot language" or the like refers to a language that is used as a result of or intended for use within a robot or program context, as opposed to the natural human language that humans use to communicate with each other. Referring to the previous chess set example, a particular chess pawn may be identified in robot language as "chess_pawn_54677." This is an example of robot language because underscores are used instead of spaces and the numerical identifiers for pawns are much higher than those used by humans in normal contexts.

[0099] Regardless, there are useful commonalities (particularly common vocabulary) between robot language and human language. In the example of "chess_pawn_54677", the terms "chess" and "pawn" are also used in human natural language. To generate an NL description of at least one aspect of the environment, at least one processor can execute a text string matching module that matches text in the robot language text label with NL vocabulary. For example, the NL description of "chess_pawn_54677" can be generated as "chess pawn number 1". In addition, the objects or features identified in the environment can also be associated with metadata, which can be used to generate an NL description of the environment. For example, the tag "chess_pawn_54677" can be associated with metadata indicating the color of the chess pawn (typically "white" or "black"). For example, at least one processor can use this metadata to generate "chess_pawn_54677" as an NL description of "white chess pawn number 1". However, the inclusion of metadata is not required. For example, a tag may also indicate such information (eg, "white_chess_pawn_54677").

[0100] Additional NL descriptions of other aspects of the environment can also be generated. Referring to the exemplary prompts discussed above, NL descriptions are generated for several different chess pieces, boxes, people, and tables. Such NL descriptions can be generated in a similar manner to that discussed above.

[0101] Furthermore, generating an NL description of an environment is not necessarily limited to generating an NL description of objects or features in the environment. In some embodiments, the positioning or placement of such objects or features can also be described. Referring to the exemplary prompt discussed above, the sentence "On the table are White Chess Pawn 1, White Chess Pawn 2, Black Chess Pawn 1, Black Chess Pawn 2, White Chess Rook, Black Chess Bishop, and 2 blue boxes, and a person is standing across from me." describes several objects in the environment and their locations. Such an NL description can be generated by at least one processor by, for example, populating a template description with a list of objects based on where the objects fit within the template.

[0102] At 306, the first NL query is provided to a large language model (LLM) module. In some embodiments, the LLM module is a software or data module stored on at least one non-transitory processor-readable storage medium of the system (at the robot body or at a robot controller remote from the robot body). In such embodiments, the first NL query can be prepared by at least one processor of the robot system and provided as input to the LLM module. In other embodiments, the LLM module can be a software or data module stored on a non-transitory processor-readable storage medium of a device separate from the robot system. In still other embodiments, the LLM module can refer to a hardware module that receives input prompts and executes the LLM module on the inputs. In such other embodiments, the first NL query can be prepared by at least one processor of the robot system and provided to the device where the LLM module is located via a communication interface of the robot system. As a specific example, the LLM module can be stored on one or more servers remote from the robot body and can receive prompts as input via a website, form, or appropriate API. At least one processor of the robot system can prepare the first NL query in an appropriate format, and the robot system can send the first NL query via the communication interface of the robot system.

[0103] The first NL query provided to the LLM module includes an NL description of at least one aspect of the environment, as generated at 304. Additionally, the first NL query includes an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan. Each of these NL descriptions is described in detail below.

[0104] As previously discussed, a work objective generally refers to a specific task, job, assignment, or application with a specified purpose and a determinable outcome. The NL description of such a work objective is an expression of such a work objective in a natural human form. Referring to the example of assembling a chess set discussed previously, the prompt includes the phrase "My purpose is to do what the person says." This in itself can be considered an NL description of a work objective, but further input provides more details about the specific purpose that the robotic system should accomplish. In this example, the prompt also includes the phrase "The person said 'Put all the white chess pieces in the box on the right, and put all the black chess pieces in the box on the left.'" This can also be considered an NL description of a work objective and provides specific instructions about what the robotic system is expected to do. In some embodiments, the NL description of the work objective includes the entirety of both phrases "My purpose is to do what the person said. The person said 'Put all the white chess pieces in the box on the right, and put all the black chess pieces in the box on the left."

[0105] The NL description of the work goal can be based on a variety of information or data. In some embodiments, an indication of the work goal can be explicitly available to the robot system (e.g., sent to the robot system from a management device or server, or stored in at least one non-transitory processor-readable storage medium of the robot system). The indication of the work goal can be available to the robot system in NL format, so that at least one processor only needs to access the indication of the work goal and provide it to the LLM module. In this sense, at least one processor of the robot system does not need to generate an NL description of the work goal, but can provide an existing NL description of the work goal to the LLM module. Alternatively, the indication of the work goal can be not in NL format (e.g., it can be a robot language), and at least one processor can generate an NL description of the work goal based on the indication of the work goal (e.g., by executing a robot language conversion module, such as a text string matching module similar to the previously discussed). In other embodiments, at least one processor of the robot system can generate an NL description of the work goal based on other information (such as the role in which the robot is deployed). In this context, "role" generally refers to the category of purpose that the robot can serve within the relevant environment. For example, a cleaning robot can act as a role to clean a specific area or facility. In this case, referring to the previous example of assembling a chess set, at least one processor may generate an NL description of the work objective as "clean up the scattered chess pieces and put them into the appropriate boxes."

[0106] The NL description of the work goal can be generated based on any appropriate additional information. In another embodiment, the capabilities of the robot system can be considered when generating the NL description of the work goal. For example, a robot body lacking a motion element can only successfully complete the work goal in the area near the robot body.

[0107] As previously discussed, the set of instructions executable by the robotic system can be a library of reusable working primitives, such as "grasp object", "place object on object", or any other suitable action. These examples are presented here in natural language, but can be stored and accessed in robotic language form, such as "grasp(object)" or "place(object1, object2)" (as non-limiting examples). In the example of assembling a chess set discussed previously, "options" 1, 2, 3, 4, 5, and 6 represent NL descriptions of the set of instructions executable by the robotic system. The exemplary prompt also includes qualifying statements "I can only strictly choose from the following options:" and "the position where the object should be replaced by the appropriate object on the table", which provide the LLM module with additional information about how the NL description of the instruction set should be interpreted, used, or applied. For a given application and a given instruction set, such qualifying statements can be added, removed, or changed as appropriate. Such qualifying statements can be further included in the first NL query, for example by being included in a template on which the first NL query is based, as discussed in more detail later.

[0108] In some embodiments, an NL description of an instruction set may be pre-generated and loaded into the robotic system such that the robotic system may provide the pre-generated NL description of the instruction set to the LLM module at 306. For example, a management device, server, or configuration device may generate an NL description of an instruction set that may be stored at a non-transitory processor-readable storage medium of the robotic system for subsequent access (e.g., during configuration or deployment of the robotic system). As a specific example, a reusable work primitive "place(object1, object2)" may be stored as "place object on object" along with metadata of the NL description of the reusable work primitive. Such an NL description of an instruction may be provided manually by a human, or may be generated by at least one processor (and potentially reviewed and / or modified by a human to ensure accuracy).

[0109] In some embodiments, an NL description of an instruction set can be generated by the robotic system. Regardless of where the generation of the NL description of the instruction set is performed (in examples where the NL description is generated by at least one processor), the at least one processor performing the generation can execute a robotic language conversion module, which generates an NL description for each instruction in the instruction set based on the corresponding instruction expressed in the robotic language. Similar to the previously discussed content, such a robotic language conversion module can include a text string matching module operable to compare robotic language instructions in the instruction set with natural language vocabulary representing actions executable by the robotic system. Matching text strings can be identified for inclusion in the NL description of the instruction set. As an example, for the instruction "grasp(object)", the text string matching module can identify "grasp" and "object" as corresponding to the NL vocabulary. Furthermore, the at least one processor can infer, based on the general structure of the programming function, that the instruction is intended to cause the robot to "grasp" an input "object". To this end, an NL description "grasp object" can be generated for the instruction. As another example, for the instruction "place(object1, object2)", the text string matching module can identify "place" and "object" as corresponding to the NL vocabulary. Furthermore, at least one processor can infer, based on the general structure of the programming function, that the intention of the instruction is to cause the robot to "place" the input "object 1" on the input "object 2." To this end, an NL description "place object on object" can be generated for the instruction.

[0110] An NL request for a task plan generally refers to a statement or phrase intended to tell the LLM module how to process the other information in the NL query. Referring to the example of assembling a chess set discussed above, the phrase "Here is a list of the above commands that I should execute in order to accomplish my purpose:". In this example, the phrase is intended to inform the LLM module that it will generate a list of commands selected from an instruction set in order to accomplish the stated purpose (such as the work objective described above). For example, an NL request for a task plan can be generated by at least one processor of a robotic system based on a request template. As another example, the NL request can be included in an NL query template that is used to construct or format a first NL query, as described below.

[0111] In some embodiments, at least one non-transitory processor-readable storage medium of the robotic system stores at least one pre-generated NL template. Such an NL template may include any or all of the corresponding template aspects of an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set, and / or an NL request of a task plan. Exemplary templates are discussed below with reference to the generation of example prompts for assembling a chess set discussed previously. However, other exemplary templates may be used to generate different NL queries in different scenarios. The non-limiting exemplary NL templates discussed may be:

[0112] "There exists [object_array_1] [(,)(and)] [object_array_2] at [position_1] ... [and [object_array_j] at [position_j]. My purpose is [work_objective], and I can only strictly choose from the following options:

[0113] 1.)[reusable_work_primitive_1(variable)]

[0114] …

[0115] k.)[reusable_work primitive_k(variable)]

[0116] Where [variable] should be replaced with the appropriate object at [position_1]…[position_j]. Here is a list of the above actions I should perform to accomplish my purpose:".

[0117] In the above template, the elements in square brackets can be filled in by at least one processor inserting appropriate NL descriptions. In particular, object_array_1 represents at least one array of objects at position_1 (e.g., an array of chess pieces on a table). In this example, at least one processor can replace the text [object_array_1] with the NL description of the chess pieces and the text [position_1] with the NL description of the table. Furthermore, in this example, object_array_2 represents a person standing at position_2 relative to the robot body. In this example, at least one processor can replace the text [object_array_2] with the NL description of the person and the text [position_2] with the NL description of the person's position. Furthermore, from the text [(,)(and)], at least one processor can select either "," or "and" to connect the text about object_array_1 and object_array_2 in a natural way (based on whether object_array_3 exists). In this example, there are no additional object arrays (no object_array_3 or object_array_j), so at least one processor chooses to connect the text "and". Additionally, because there is no additional object array, at least one processor deletes or ignores (eg, replaces with no text) the text "[object_array_j]" at [position_j].

[0118] As a result of the above steps, a first sentence of a first NL query generated based on the template may be "There are white chess pawn 1, white chess pawn 2, black chess pawn 1, black chess pawn 2, white chess rook, black chess bishop, and 2 blue boxes at the table, and a person is standing opposite me." This is similar to the exemplary prompt described above, but the chess pieces are described as "at the table" instead of "on the table." To improve the generation of the first NL query, the template may include options for transition words or positioning words (such as "on..." or "at...") so that at least one processor can select the most natural words for a given scenario.

[0119] Returning to the template described above, at least one processor can replace the text [work_objective] with an NL description of the work objective of the robotic system. In this example, at least one processor can replace the text [work_objective] so that the second statement of the first NL query is "My objective is to do what this person says. This person said 'Put all the white pieces in the right box and all the black pieces in the left box'" similar to the prompt in the example discussed previously.

[0120] In addition, at least one processor may replace the text of the available instructions "1.) [reusable_work_primitive_1 (variable)] ... k.) [reusable_work_primitive_k (variable)]" with the NL description of each available reusable work primitive. In an example scenario, the text for the available instructions may be replaced with "1.) Grab object," "2.) Place object on object," "3.) Look at object," "4.) Slice object," "5.) Say hello to that person," and "6.) Place object on object," as shown in the exemplary prompts discussed previously.

[0121] In the example, the word "variable" for each reusable work primitive is replaced with the appropriate text for "object" or "person," depending on what the given reusable work primitive applies to. Furthermore, the text [variable] and [position_1]...[position_j] in the penultimate statement of the template are also replaced with the appropriate text for "object," along with the associated position of the object. In this regard, the penultimate statement of the generated NL query reads "the object should be replaced with the appropriate object at the table's location," as in the example prompt presented previously.

[0122] In view of the above, by replacing or importing the selection elements in the pre-generated template, an NL query suitable for providing to the LLM module is generated.

[0123] While the NL templates are described above as text in which certain elements are "replaced" or "entered," this is not strictly necessary in terms of implementation. For example, instead of literally "replacing" text, the NL template can also be implemented as a set of instructions or functions (e.g., a program or script) that stitches together the basic statement with the relevant elements in a segmented manner. In such an example, the NL query is "assembled" into segments, rather than elements that are literally "replaced." In this sense, the NL templates presented are intended to be a logical representation of how elements can be stitched together, rather than a strict process for how text generation actually occurs.

[0124] Return to Figure 3In the method 300, at 308, the robotic system receives the mission plan from the LLM module. That is, after providing the first NL query to the large LLM module at 306, the LLM module generates a mission plan (expressed in NL) and provides the generated mission plan to the robotic system.

[0125] As described above, the task plan provided by the LLM module is expressed in NL. For example, the task plan may indicate at least one action expressed in NL that the robotic system can perform. In order for the robotic system to execute or verify the action plan, at least one processor may optionally first generate a robotic language task plan based on the task plan expressed in NL. The robotic language task plan may include a set of robotic control instructions that, when executed by at least one processor, cause the robotic system to perform at least one action indicated in the task plan. For example, the set of robotic control instructions may include a library or collection of at least one reusable work primitives that can be executed by the robotic system. In addition, at least one action indicated in the task plan expressed in NL may include an NL description of a specific reusable work primitive (e.g., grab chess pawn 1), while the robotic control instructions in the robotic language task plan may include the actions of the NL task plan, but specified in a language format usable by the robotic system (e.g., grab(chess_pawn_54677)).

[0126] Similar to the above description, generating a robot language task plan can include executing a robot language conversion module that converts at least one action expressed in NL and executable by the robot system into at least one reusable work primitive in an instruction set executable by the robot system. Referring to the example in which the NL task plan includes an action expressed in NL as "grab chess pawn 1", the robot language conversion module can match the text string in the action expressed in NL with a text string available in a reusable work primitive usable by the robot system (e.g., grasp(object)) or an object in the environment with which the robot system can interact (e.g., chess_pawn_54677). Thus, at least one processor can generate a robot language action such as grab(chess_pawn_54677).

[0127] In method 300, troubleshooting of the mission plan is performed. For example, if a given mission plan fails to execute (i.e., fails to be verified, fails to proceed to completion, and / or fails to complete the intended task) or encounters an error or failure, a second NL query can be sent (returned) to the LLM module, which optionally includes all successful parts of the executed or verified mission plan, with additional wording describing the failure and asking the LLM module what to do next. This is discussed below with reference to actions 310, 330, 332, 334, 336, and 338.

[0128] At least one processor of the robotic system determines whether the mission plan includes at least one fault at 310. This determination can be performed in many different ways, some non-limiting exemplary embodiments of which are discussed below.

[0129] In a first embodiment, action 310 requires executing at least one step of the mission plan. For each step of the mission plan being executed, at least one processor identifies whether the execution of the corresponding step generates a fault scenario.

[0130] As an example, a failure scenario can include a scenario where the execution of a step produces a result that is inconsistent with the work objective. In the exemplary use case, where the work objective is for the robotic system to win at chess, the robotic system's execution of a step that moves the king pawn into checkmate is a failure scenario where a result that is inconsistent with the work objective is produced. This is because moving the king pawn into checkmate results in an immediate loss of the chess game.

[0131] As another example, a failure scenario may include a scenario in which execution of a step prevents execution of at least one further step in the mission plan. In an exemplary use case, execution of a step that results in destruction of an object required by at least one further step in the mission plan (e.g., a component of the robot or an object with which the robot needs to interact) is a failure scenario in which execution of at least one further step in the mission plan is prevented.

[0132] As another example, a failure scenario can include a scenario in which execution of a step produces at least one unacceptable effect in the environment. In an exemplary use case, execution of a step that results in damage to the environment (even if the damage is not inconsistent with the work objective and does not prevent further steps in the mission plan) is a failure scenario in which at least one unacceptable effect is produced in the environment. For example, a robot body may strike a wall, leaving an unacceptable hole, dent, or other damage, even if the integrity of the wall is not relevant to achieving the work objective of the robotic system or executing further steps in the mission plan.

[0133] In a second embodiment, action 310 requires the execution of at least one step of the simulated mission plan. In an example, an environment model representing the environment of the robot body is accessed. Such an environment model can be stored on at least one non-transitory processor-readable medium of the robot body, and / or generated or updated based on sensor data captured by at least one sensor of the robot body (e.g., data captured at 302). For example, the environment model can be populated by collecting data (e.g., tactile data or visual data) of each object. Based on the collected data, each object can be identified, and the corresponding profiles of the identified objects can be accessed in a database. Then, based on the corresponding profiles accessed for the corresponding objects, the environment model is populated with visual and / or tactile representations of each individual object. Detailed embodiments for generating and / or populating environment models are discussed in U.S. Patent No. 11,717,974, which is incorporated herein by reference in its entirety.

[0134] The at least one processor then simulates at least one step of the mission plan in the environment model. For each step of the mission plan in which execution is simulated, the at least one processor identifies whether each of the at least one step of the mission plan generates a failure scenario when execution of the corresponding individual step is simulated.

[0135] Furthermore, if no corresponding failure scenario is identified in the simulation of each of the at least one step of the mission plan, the at least one processor may simulate at least one additional step of the mission plan in the environment model. For each of the at least one additional step of the mission plan, when the execution of the corresponding step is simulated, the at least one processor may identify whether each of the at least one step of the mission plan generates the corresponding failure scenario. In this manner, the mission plan may be evaluated in stages, wherein selected steps of the mission plan are individually simulated.

[0136] As an example, a failure scenario may include a scenario where the simulated execution of a step produces a simulation result that is inconsistent with the work objective. In an exemplary use case, where the work objective is for the robotic system to win at chess, the simulated execution of a step in which the robotic system moves a king pawn into checkmate is a failure scenario that produces a simulation result that is inconsistent with the work objective. This is because moving the king pawn into checkmate results in an immediate loss of the chess game.

[0137] As another example, a failure scenario may include a scenario in which the simulated execution of a step prevents the execution of at least one further step in the mission plan. In an exemplary use case, the simulated execution of a step resulting in the simulated destruction of an object required by the at least one further step in the mission plan (e.g., a component of the robot or an object with which the robot needs to interact) is a failure scenario that prevents the execution of the at least one further step in the mission plan.

[0138] As another example, a failure scenario can include a scenario in which the simulated execution of a step produces at least one unacceptable simulated effect in the environment. In an exemplary use case, the simulated execution of a step that results in simulated damage to the environment (even if the damage is not inconsistent with the work objective and does not prevent other steps of the task plan) is a failure scenario that produces at least one unacceptable effect in the environment. For example, a robot body can be simulated to strike a wall, leaving an unacceptable hole, dent, or other damage, even if the integrity of the wall is not relevant to achieving the work objective of the robotic system or executing other steps in the task plan.

[0139] By simulating the execution of at least one step of a mission plan, efficiency can be improved by avoiding wasting energy or time by actually executing an erroneous mission plan. Furthermore, actual damage to an object can be avoided where the mission plan would otherwise result in such damage.

[0140] In a third embodiment, act 310 can include determining whether the mission plan violates at least one rule in a set of rules specified in at least a portion of an inference engine. In some examples, at least one processor of the robotic system validates the mission plan using the inference engine (based on inference engine data stored on at least one non-transitory processor-readable storage medium of the robotic system). In some embodiments, the inference engine can be external, and the robotic system can send the mission plan to a device (e.g., a server or a peripheral device) that stores and executes the inference engine for validation.

[0141] Based on the inference engine or other logic-based system, the mission plan can be reviewed or verified and approved or rejected for one reason or multiple reasons. As a non-limiting example, the logic-based system or inference engine can be Cycorp's Machine reasoning AI platform. The reasoning engine (sometimes referred to as an inference engine) can utilize a library of logical rules, statements, terms, knowledge fragments, or similar content, and can draw logical conclusions based on these contents. In this way, the mission plan mentioned in method 300 can be verified by the reasoning engine by comparing the mission plan with a set of rules (or similar rules) at least partially specified by the reasoning engine. That is, at least a portion of the logic of the reasoning engine can be applied to the mission plan to verify whether the mission plan makes logical sense and / or identify any logical inconsistencies or impossibilities in the mission plan. For example, the reason for rejection may be a security violation related to the safety of the robot or the safety of any human or other creature. In the event of rejection, the second NL query can be sent back to the LLM module in action 334 discussed later to prevent the mission plan from failing external inspection.

[0142] If the mission plan is determined not to include at least one fault at 310, method 300 proceeds to 320. At 320, the mission plan is executed by the robotic system to completion. In embodiments where at least one step is executed at 310, act 320 includes executing the remaining steps of the mission plan. In some embodiments, act 320 may be a passive result of executing each step of the mission plan to determine whether the mission plan includes at least one fault. That is, if each step of the mission plan is executed to determine whether the mission plan includes any faults, act 320 includes completing execution of the mission plan without any faults.

[0143] In embodiments where execution of the mission plan is simulated or identification of faults is performed by a logic or reasoning engine (rather than actually executing the actions of the mission plan), action 320 includes executing the mission plan (which has been determined not to include any faults).

[0144] If, at 310 , the mission plan is determined to include at least one fault, method 300 proceeds to 330 .

[0145] At 330, at least one processor generates an NL description of at least one fault. Optionally, at 332, at least one processor generates an NL description of any steps of the mission plan that do not include the fault. Examples of actions 330 and 332 are discussed later with reference to dialogues 1 through 6.

[0146] At 334, a second NL query is provided to the LLM module. The second NL query includes at least an NL description of the at least one fault and an NL request for a solution plan. At 336, the solution plan is received from the LLM module. The solution plan is expressed in NL and includes information and / or steps for resolving the at least one fault in the mission plan. Examples of actions 334 and 336 are discussed later with reference to dialogues 1 through 6.

[0147] Next, we discuss Dialogue 2. Dialogue 2 follows Dialogue 1, discussed previously, and is an example dialogue in which a fault is identified in the mission plan provided in Response 1 of Dialogue 1. Dialogue 2 illustrates an exemplary second query, Query 2, provided to the LLM in accordance with action 334. Dialogue 2 also includes a response, Response 2, which includes a solution plan in accordance with action 336. In the example of Dialogue 2, the steps of the mission plan are actually executed by the robotic system until the fault is identified.

[0148] Dialogue 2

[0149] Query 2:

[0150] Step 1 was successfully executed.

[0151] While doing step 2, the box on the right hits the floor.

[0152] Steps 3 to 12 are not performed.

[0153] Please provide a solution plan.

[0154] Response 2:

[0155] Step A: Place White chess pawn number 1 on the table

[0156] Step B: Look at the floor to identify the box to the right and any chess pieces on the floor

[0157] Step C: Grab the box on the right with one hand

[0158] Step D: For each chess piece on the floor:

[0159] Step D1: Grab the chess piece with your other hand

[0160] Step D2: Place the chess pieces in the box on the right

[0161] Step E: Place the right box on the table

[0162] Step F: Resume the task plan from step 2. In each of steps 2, 4, and 10, use your free hand to grab the box on the right. In each of steps 6, 8, and 12, use your free hand to grab the box on the left.

[0163] In the example of Dialog 2, the NL description for at least one fault generated in action 330 is "While executing step 2, the box on the right crashed into the floor." This fault not only represents an undesirable situation but also indicates an inability to complete the task plan originally generated in Dialog 1. Furthermore, the NL description for any step in the task plan that does not include the fault generated in action 332 is "Step 1 executed successfully." This description is optional, as the execution of step 2 implies the successful completion of step 1.

[0164] In Query 2, the NL descriptions generated at 330 and 332 are combined into a second query in action 334, where the statement "Please provide a solution plan." is used as an NL request for a solution plan. Response 2 is generated by the LLM module as a solution plan and received from the LLM module.

[0165] In the example of Dialogue 2, because the mission plan was actually executed and a failure occurred (knocking the right box to the floor), the solution plan provided in Response 2 not only modifies the original mission plan but also includes steps to address the consequences of the failure (in this case, cleaning up the right box and any chess pieces that fell to the floor in steps A, B, C, D, D1, D2, and E).

[0166] In the example shown, the solution plan is an additional step to the original task plan, and the original task plan is mentioned in the solution plan in step F. That is, the solution plan itself is not a complete task plan (in the example shown), but rather includes steps to correct faults in the task plan. In particular, after cleaning up the consequences of the right box being knocked to the floor, the solution plan instructs the robot system to resume the task plan with the statement "Resume task plan from step 2." In addition, the solution plan provides additional instructions for avoiding the fault (or similar faults) seen during the execution of step 2. In particular, the solution plan states "During each of steps 2, 4, and 10, grab the right box with your free hand. During each of steps 6, 8, and 12, grab the left box with your free hand." In this way, the solution plan modifies the original task plan so that the robot system stabilizes the right box and the left box when placing the chess pieces into them to reduce the risk of the right box and the left box hitting the floor.

[0167] In an alternative embodiment, the solution plan may include an entirely new mission plan as illustrated below in Dialogue 3. Dialogue 3 follows Dialogue 1, discussed previously, and is an example dialogue in which a fault is identified in the mission plan provided in Response 1 of Dialogue 1. Dialogue 3 illustrates an exemplary second query, Query 3, provided to the LLM in accordance with action 334. Dialogue 3 also includes a response, Response 3, which includes the solution plan in accordance with action 336. In the example of Dialogue 3, the steps of the mission plan are actually performed by the robotic system until the fault is identified.

[0168] Dialogue 3

[0169] Query 3:

[0170] Step 1 was successfully executed.

[0171] While doing step 2, the box on the right hits the floor.

[0172] Steps 3 to 12 are not performed.

[0173] Please provide a solution plan.

[0174] Response 3:

[0175] Step 1: Place White chess pawn number 1 on the table

[0176] Step 2: Look at the floor to identify the box on the right and any chess pieces on the floor

[0177] Step 3: Grab the box on the right with one hand

[0178] Step 4: For each chess piece on the floor:

[0179] Step 4A: Grab the Chess Piece with Your Other Hand

[0180] Step 4B: Place the chess pieces in the box on the right

[0181] Step 5: Place the right box on the table

[0182] Step 6: Capture White Chess Pawn 1

[0183] Step 7: Use your free hand to grab the box on the right

[0184] Step 8: Place White Chess Pawn 1 in the box on the right

[0185] Step 9: Capture White Chess Pawn 2

[0186] Step 10: Place White Chess Pawn 2 in the box on the right

[0187] Step 11: Stop Grabbing the Box on the Right

[0188] Step 12: Capture Black Chess Pawn 1

[0189] Step 13: Use your free hand to grab the box on the left

[0190] Step 14: Place Black Chess Pawn 1 in the Box on the Left

[0191] Step 15: Capture Black Chess Pawn 2

[0192] Step 16: Place Black Chess Pawn 2 in the Box on the Left

[0193] Step 17: Stop Grabbing the Box on the Left

[0194] Step 18: Grab the White Chess Rook

[0195] Step 19: Use your free hand to grab the box on the right

[0196] Step 20: Place the White Chess Rook in the Right Box

[0197] Step 21: Stop Grabbing the Box on the Right

[0198] Step 22: Capture Black's Chess Bishop

[0199] Step 23: Use your free hand to grab the box on the left

[0200] Step 24: Place the Black Chess Bishop in the Box on the Left

[0201] Step 25: Stop grabbing the box on the left.

[0202] Query 3 in conversation 3 is similar to Query 2 in conversation 2. The description of Query 2 is fully applicable to Query 3 and is not repeated for the sake of brevity.

[0203] In the example of Dialogue 3, because the mission plan was actually executed and a failure occurred (the right box hit the floor), the solution plan provided in Response 3 includes steps for addressing the consequences of the failure (in this case, cleaning up the right box and any chess pieces that fell to the floor in steps 1, 2, 3, 4, 4A, 4B, and 5).

[0204] In the example shown in Dialog 3, the solution plan replaces the original mission plan. That is, the robotic system can complete the mission by executing the solution plan without continuing to execute the original mission plan. The solution plan provides additional instructions for avoiding the failure (or similar failures) seen during the execution of original step 2 in the original mission plan. Specifically, in steps 7 and 19, the right box is grasped to prevent it from hitting the floor during steps 8, 9, 10, and 20. Similarly, in steps 13 and 23, the left box is grasped to prevent it from hitting the floor during steps 14, 15, 16, and 24.

[0205] Next, we discuss Dialogue 4. Dialogue 4 follows Dialogue 1, discussed previously, and is an example dialogue in which a fault is identified in the mission plan provided in Response 1 of Dialogue 1. Dialogue 4 illustrates an exemplary second query, Query 4, provided to the LLM in accordance with action 334. Dialogue 4 also includes a response, Response 4, which includes a solution plan in accordance with action 336. In the example of Dialogue 4, steps of the mission plan are simulated by the robotic system (rather than actually being executed) to identify any faults in the mission plan.

[0206] Dialogue 4

[0207] Query 4:

[0208] The simulation indicates that execution of steps 1, 3, 5, 6, 7, 8, 9, 11, and 12 will be successful.

[0209] The simulation indicates that in step 2, the box on the right will hit the floor.

[0210] The simulation indicates that steps 4 and 10 cannot be performed because the box on the right is shifted.

[0211] Please provide a solution plan.

[0212] Response 4:

[0213] During each of steps 2, 4, and 10, use your free hand to grab the box on your right.

[0214] In the example of dialogue 4, the NL description for at least one of the faults generated in action 330 is "The simulation indicates that in step 2, the box on the right will hit the floor. The simulation indicates that steps 4 and 10 cannot be executed because the box on the right is displaced." These faults not only represent undesirable situations, but also represent situations where the mission plan originally generated in dialogue 1 cannot be completed. In addition, the NL description for any step of the mission plan that does not include a fault generated in action 332 is "The simulation indicates that execution of steps 1, 3, 5, 6, 7, 8, 9, 11, and 12 will be successful." This description is optional because successful completion of steps 1, 3, 5, 6, 7, 8, 9, 11, and 12 means that no faults are reported for those steps.

[0215] In query 4, the NL descriptions generated at 330 and 332 are combined into a second query in action 334, where the statement "Please provide a solution plan" is used as an NL request for a solution plan. Response 4 is generated by the LLM module as a solution plan and received from the LLM module.

[0216] In the example of Dialogue 4, the solution plan is an additional step or information to the original task plan, and the original task plan is referenced in the solution plan. That is, the solution plan itself is not a complete task plan (in the example shown), but rather includes steps to correct faults in the task plan. Specifically, the solution plan states, "During each of steps 2, 4, and 10, grab the right box with your free hand." In this way, the solution plan modifies the original task plan so that the robotic system stabilizes the right box when placing the chess piece in it, reducing the risk of the right box hitting the floor in step 2 and shifting in steps 4 and 10. Since no fault is reported for the step of placing the chess piece in the left box in Query 4, grabbing the left box during steps 6, 8, and 12 is unnecessary. However, under extreme caution, if necessary, the solution plan could instruct the robotic system to grab the left box during steps 6, 8, and 12.

[0217] In an alternative embodiment, the solution plan may include an entirely new mission plan, similar to that discussed above with respect to Dialogue 3, and for the sake of brevity, will not be repeated.

[0218] Exemplary dialogues 5 and 6 are discussed below, in which act 310 of method 300 is performed by a verification tool such as a logic-based system (e.g., an inference engine as previously described). For dialogues 5 and 6, corresponding exemplary mission plans α and β are shown and can be generated and received as described in acts 306 and 308 discussed previously, and are not repeated for the sake of brevity (e.g., mission plans α and β can be generated and received in response to query 1 in dialogue 1 discussed previously, or any other similar query).

[0219] Mission plan α is related to the example of dialogue 5 and is as follows.

[0220] Mission Plan α:

[0221] Step 1: Capture White Chess Pawn 1

[0222] Step 2: Place White Chess Pawn 1 in White Chess Rook

[0223] Step 3: Capture White Chess Pawn 2

[0224] Step 4: Place White Chess Pawn 2 in White Chess Rook

[0225] Step 5: Capture White Chess Pawn 1, White Chess Pawn 2, and White Chess Rook

[0226] Step 6: Place White Chess Pawn 1, White Chess Pawn 2, and White Chess Rook in the box on the right

[0227] Step 7: Capture Black Chess Pawn 1

[0228] Step 8: Place Black Chess Pawn 1 inside Black Chess Bishop

[0229] Step 9: Capture Black Chess Pawn 2

[0230] Step 10: Place Black Chess Pawn 2 in Black Chess Bishop

[0231] Step 11: Capture Black Chess Pawn 1, Black Chess Pawn 2, and Black Chess Bishop

[0232] Step 12: Place Black Pawn 1, Black Pawn 2, and Black Bishop in the box on the left.

[0233] Summarizing task plan α, the robot is expected to merge all white chess pieces together (by placing white pawns 1 and 2 in the white rook) and then place the merged white chess piece in the right box. Similarly, the robot is expected to merge all black chess pieces together (by placing black pawns 1 and 2 in the black bishop) and then place the merged black chess piece in the left box. In action 310, the verification tool can identify this strategy as a failure in action 310 because chess pieces are generally solid and cannot be simply placed inside each other. This can be handled as shown in dialogue 5 below.

[0234] Dialogue 5:

[0235] Query 5A:

[0236] Steps 2, 4, 8, and 10 cannot be performed. Chess pieces are solid objects and cannot be placed inside them. Please provide a solution plan.

[0237] Response 5A:

[0238] Step A. Slice the White Chess Rook

[0239] Step B. Slice the Black Chess Bishop

[0240] Step C. Continue with the original mission plan steps 1 to 12.

[0241] Query 5B:

[0242] Slicing chess pieces destroys them. Please provide a solution plan that does not include slicing chess pieces.

[0243] Response 5B:

[0244] Execute the following updated task schedule:

[0245] Step 1: Capture White Chess Pawn 1

[0246] Step 2: Place White Chess Pawn 1 in the box on the right

[0247] Step 3: Capture White Chess Pawn 2

[0248] Step 4: Place White Chess Pawn 2 in the box on the right

[0249] Step 5: Capture Black Chess Pawn 1

[0250] Step 6: Place Black Chess Pawn 1 in the box on the left

[0251] Step 7: Capture Black Chess Pawn 2

[0252] Step 8: Place Black Chess Pawn 2 in the box on the left

[0253] Step 9: Capture the White Chess Rook

[0254] Step 10: Place the White Chess Rook in the Box on the Right

[0255] Step 11: Capture Black's Chess Bishop

[0256] Step 12: Place the Black Chess Bishop in the Box on the Left

[0257] For query 5A, at least one NL description of the fault is generated according to act 330 , which in this example is the statement “Steps 2, 4, 8, and 10 cannot be performed. The chess piece is a solid object and other chess pieces cannot be placed inside it.” This is parsed according to act 334 of method 300 , where the NL request for a solution plan is “Please provide a solution plan” and provided to the LLM module as a second NL query.

[0258] Response 5A corresponds to the solution plan received at 336. In this example, the LLM module has generated a solution plan (an updated mission plan) in which the White rook and the Black bishop are sliced. In this manner, the LLM module may intend to create space within the White rook and the Black bishop to accommodate other chess pieces. In action 310, the verification tool may again identify that the updated mission plan includes at least one fault.

[0259] According to act 330 of method 300 , an NL description of at least one fault may be generated (in this example, the statement “slicing the chess pieces destroys them”). According to act 334 of method 300 , this is parsed, wherein a request for a solution plan is made (in this example, the statement “please provide a solution plan that does not include slicing the chess pieces”) and provided to the LLM module as another second NL query (query 5B).

[0260] Response 5B is received from the LLM module and includes a solution plan, which is a completely new mission plan. This mission plan is substantially similar to the mission plan provided in response 1 of dialog 1, and for the sake of brevity, its details are not repeated. In this example, the verification tool can determine at 310 that the updated mission plan provided in response 5B does not have faults, and method 300 proceeds to act 320, where the updated mission plan is executed to completion.

[0261] The example of dialogue 5 illustrates an important concept that is optionally applicable to any appropriate implementation of method 300 (and any examples discussed herein): method 300 can be iterative or recursive such that a received solution plan (or an updated task plan based thereon) can be tested for a fault according to action 310.

[0262] The task plan β is related to the example of dialogue 6 and is shown below.

[0263] Mission Plan β:

[0264] Step 1: Capture White Chess Pawn 1

[0265] Step 2: Place White Chess Pawn 1 on White Chess Rook

[0266] Step 3: Capture White Chess Pawn 2

[0267] Step 4: Place White Chess Pawn 2 on White Chess Rook

[0268] Step 5: Grab the box on the right

[0269] Step 6: Place the right box on White Pawn 1, White Pawn 2, and White Rook

[0270] Step 7: Capture Black Chess Pawn 1

[0271] Step 8: Place Black Chess Pawn 1 on Black Chess Bishop

[0272] Step 9: Capture Black Chess Pawn 2

[0273] Step 10: Place Black Chess Pawn 2 on Black Chess Bishop

[0274] Step 11: Grab the box on the left

[0275] Step 12: Place the box on the left on Black Pawn 1, Black Pawn 2, and Black Bishop.

[0276] Summarizing task plan β, the robot is expected to gather all of the White chess pieces together (by placing White pawn 1 and pawn 2 on the White rook), and then place the box on the right side on the gathered White chess pieces. Similarly, the robot is expected to gather all of the Black chess pieces together (by placing Black pawn 1 and pawn 2 on the Black bishop), and then place the box on the left side on the gathered Black chess pieces. In action 310, the verification tool can identify this strategy as a failure in action 310 because placing a box "on" the chess pieces is not equivalent to placing the chess pieces in the box (because the box is intended to hold the chess pieces together so that they can easily move as a group). This can be handled as shown in dialogue 6 below.

[0277] Dialogue 6

[0278] Query 6:

[0279] Placing a box on a chess piece does not result in the chess piece being in the box in a useful capacity. Please provide a solution plan in which the chess piece is in the box such that movement of the box moves the chess piece within it.

[0280] Response 6:

[0281] Execute the following updated task schedule:

[0282] Step 1: Capture White Chess Pawn 1

[0283] Step 2: Place White Chess Pawn 1 in the box on the right

[0284] Step 3: Capture White Chess Pawn 2

[0285] Step 4: Place White Chess Pawn 2 in the box on the right

[0286] Step 5: Capture Black Chess Pawn 1

[0287] Step 6: Place Black Chess Pawn 1 in the box on the left

[0288] Step 7: Capture Black Chess Pawn 2

[0289] Step 8: Place Black Chess Pawn 2 in the box on the left

[0290] Step 9: Capture the White Chess Rook

[0291] Step 10: Place the White Chess Rook in the Box on the Right

[0292] Step 11: Capture Black's Chess Bishop

[0293] Step 12: Place the Black chess bishop in the box on the left.

[0294] For query 6, at least one NL description of the fault is generated according to act 330 . In this example, the NL description is the statement “Placing a box on the chess piece does not result in the chess piece being in the box in a useful capacity.” This is parsed according to act 334 of method 300 , where the NL request for a solution plan is “Please provide a solution plan where the chess piece is in the box such that moving the box moves the chess piece therein.” This is provided to the LLM module as a second NL query.

[0295] Response 6 is received from the LLM module and includes a solution plan, which is a completely new mission plan. This mission plan is substantially similar to the mission plan provided in response 1 of dialog 1, and for the sake of brevity, its details are not repeated. In this example, the verification tool can determine at 310 that the updated mission plan provided in response 5 is free of faults, and method 300 proceeds to act 320, where the updated mission plan is executed to completion.

[0296] The solution plan discussed herein may include any appropriate modification or replacement of the NL description of the mission plan. As an example, the solution plan may include an adjustment to the order of the steps in the mission plan (e.g., if a step places the robotic system in a configuration that prevents other unrelated future steps, the order of the steps may be changed). As another example, the solution plan may include an adjustment to the success criterion of any step in the mission plan (e.g., in the example of assembling a set of chess, if the box is full, the success criterion may be changed to placing the chess piece on top of the other chess pieces in the box, or placing the chess piece in another box (if any)). As another example, the solution plan may include an adjustment to any action expressed in any step of the mission plan (such as the adjustment discussed with reference to dialogues 2, 4, and 5). As another example, the solution plan may include an adjustment to any interactive target of any step of the mission plan (e.g., if a step will cause the robotic system to drop a large object, the solution plan may cause the robotic system to attempt to grab an object, such as a handle, at different locations). As another example, the solution plan may include replacing any step of the mission plan with at least one new step. As another example, the solution plan may include removing at least one step of the mission plan. As yet another example, the solution plan may include an updated mission plan that replaces all steps of the mission plan (as discussed above with respect to conversations 3, 5, and 6).

[0297] Return to Figure 3In method 300 , the robotic system executes the updated mission plan at 338 . The updated mission plan is based at least in part on the solution plan.

[0298] In some embodiments, the LLM can help autonomously assign parameters or definitions to generalized and / or parameterized objects in a robot control system. For example, parameterized work primitives or "instructions" can be assigned by the LLM, as in the case of the chess set example above. As another example, if a task plan is successfully executed, the successful task plan can be stored and then re-parameterized to become generalized. When the robot encounters a future instance of a similar task, it can call the stored successful task plan and ask the LLM module (e.g., via a simple NL prompt) to replace the parameterized objects of the previous successful instance of the task plan with new objects specific to the current instance of the task plan. For example, if a plan was generated to successfully sort two types of specific objects, the robot can reuse the plan by asking the LLM to replace these objects with different objects.

[0299] Various embodiments of the systems, methods, control modules, and computer program products of the present application relate to using NL expressions (descriptions) (e.g., via NL prompts, which can be entered directly in text form by a user, or can be spoken by a user and converted to text by an intervening speech-to-text system) to control the functions and operations of a robot, wherein an LLM module can provide an interface between the NL expressions and the robot control system. This framework can be particularly advantageous when certain elements of the robot control architecture employ programming and / or instructions that can be expressed in NL. A suitable, but non-limiting, example is the instruction set described above. For example, as previously described, the task plan output of the LLM module can be parsed by looking for words that match the instruction set commands (e.g., parsed autonomously by the robot control system), and the actual parameters of the instruction set can be found by string matching within the input NL prompt (e.g., by a text string matching module as previously described). In some embodiments, a 1-1 mapping can be generated between the actual parameters and the NL variables used in the robot control system to increase the chances of the LLM module processing the text correctly. For example, even if an object is represented in the robot control system (e.g., in the world model environment portion of the robot control system) as chess_pawn_54677, it may also be referred to as "chessspawn1" in the NL prompt. In this case, if the returned mission plan contains the phrase "grasp chesspawn1," this may match the instruction set "grasp" and the object "chess pawn 1," and thus the phrase may be mapped to grab(chess_pawn_54677). This type of parsing and / or word matching (e.g., a text string matching module) may be used in any of the situations discussed herein where a robot language is being converted to a natural language, or vice versa.

[0300] In some embodiments, the robot control system can generate and / or employ a scene graph that describes the robot's environment, and can apply a function to act on the scene graph and create an NL hint or description that describes the scene from the robot's perspective (e.g., in the context of action 304 of method 300). This automatically generated NL hint or description can then be used as input to the LLM module to facilitate various operations, such as reasoning, fact checking, and mission planning.

[0301] In some embodiments, the quality of the mission plan may depend at least in part on the robot's understanding of its environment, so the robot control system may periodically check and compare its scene graph and internal world model in the background. According to the systems, methods, control modules, and computer program products of the present application, such checking and comparing the scene graph (e.g., actual data from the robot's external environment) and the internal world model (e.g., the robot's simulation of its external environment) may be accomplished by automatically generating NL hints or descriptions for each and feeding these NL hints or descriptions through the LLM module.

[0302] In some embodiments, an LLM module acting as a mission planner may be frequently invoked by the robot control system to answer the question "What can I (the robot) do here / now?" For example, the robot control may automatically generate an NL description of at least one aspect of its environment (scene graph) and capabilities (instruction set), and feed these NL descriptions into the LLM module along with the following query: What can I do?: or "What should I do?" (or similar variations, such as "What is the most useful thing for me to do?", "What are the productive things I can do?", etc.). A set of answers to this or similar questions may then each be generated by generating a mission plan (e.g., as described above with reference to Figure 3 Each of these mission plans can be checked for faults (e.g., verified by an inference engine) as in action 310 of method 300, and the tasks that are passed are now a set of things that the robotic system can do autonomously without being asked. This type of behavior is a form of autonomous agency because the embodied general artificial intelligence (AGI) utilized by the robotic system creates its own grounded task plans and then executes them. In some embodiments, the robotic system presents an output in NL that describes what it plans to do to request permission from a person (or driver / supervisor) to execute the plan. A separate inference system, such as to explain each step in the plan. Plans can be provided at any desired level of abstraction, where the highest level is likely to be the most useful descriptor for a human supervisor to determine whether the plan should be executed. The LLM module can also be used to check whether a plan passes a certain criterion, for example by issuing an NL hint asking whether the top-level plan is compatible with a set of desired constraints.

[0303] Some mission plans may contain steps that cannot be parsed into instruction set elements and are computational in nature. For example, a mission plan may require the calculation of an integral or some other computational process that may not be possible given a particular instruction set. In these cases, the robotic system can send these mission plan steps to an LLM-based system or LLM module that requests the generation of a piece of code, such as a python script, that generates the functions used to perform the task. In some embodiments, this script can then reside in a "code library" where a human engineer can review all the automatically generated scripts generated by the background "what can I do here?" process and check that they do what is expected. Such scripts generated by an LLM-based device or module can provide new instruction set elements that can be called to "unlock" a mission plan that is blocked due to a lack of access to the appropriate instructions, or can be otherwise accessed by the robotic system for incorporation and use in a mission plan.

[0304] In some embodiments, the LLM module can be stored and executed outside the robot (e.g., in the cloud) and called or accessed by the robot system (e.g., Figure 4A Specifically, Figure 4A is a schematic diagram of a robot body 400 accessing an LLM module 420 via a cloud 410 (such that the LLM module is separate from the robot system including the robot body 400). In other embodiments, the LLM module can be stored and executed locally on the robot system as part of the robot's control system (e.g., Figure 4B Specifically, Figure 4B is a schematic diagram of a robot body 450 having an LLM module 420 locally at a non-transitory processor-readable storage medium of the robot body 450. Both of these embodiments are referred to herein as "LLM-accessible robots."

[0305] Various embodiments described herein include systems, methods, control modules, and computer program products for utilizing one or more LLMs in a robotic control system, including, for example, establishing an NL interface between the LLM and the robotic control system and invoking the LLM to help autonomously instruct the robot what to do. Example applications of this approach include task planning, motion planning, reasoning about the robot's environment (e.g., "What can I do now?"), and the like. Such embodiments are particularly well-suited for robotic control systems in which at least some control parameters and / or instructions (e.g., the instruction sets previously described) are specified in an NL. Thus, some embodiments may include converting or translating the robot control instructions and / or parameters into NL for such communication with the LLM via the NL interface.

[0306] Figure 5 is an illustrative diagram of an exemplary robotic system 500 that includes various features and components described throughout the systems, robots, methods, control modules, and computer program products of this application. For example, the robotic system 500 can perform method 300 and associated actions as previously described (and not repeated for the sake of brevity). The robotic system 500 includes a robotic body 501 having a first physically actuatable component 502a and a second physically actuatable component 502b mechanically coupled to the body 501. In the illustrated embodiment, the first physically actuatable component 502a and the second physically actuatable component 502b each correspond to a respective robotic hand, although those skilled in the art will appreciate that in alternative embodiments, the physically actuatable components can take other forms (e.g., arms or legs, non-hand-like end effectors (e.g., cutters or suction tubes), or any other form useful for the particular application the robot is intended to perform). The robotic hand 502a mimics a human hand and includes a plurality of fingers 521a, 522a, 523a, and 524a and an opposable thumb 525a. Robotic hand 502b is similar to a mirror image of robotic hand 502a, and corresponding details are not labeled for robotic hand 502b to reduce clutter. Robotic hands 502a and 502b can be physically actuated in a variety of different ways (including electromechanical actuation, cable-driven actuation, magnetorheological fluid-based actuation, and / or hydraulic actuation). Some exemplary details of actuation techniques that can be used to physically actuate robotic hands 502a and 502b are described in U.S. patent application serial number 17 / 491,577 and U.S. patent application serial number 17 / 749,536, which are incorporated herein by reference in their entireties.

[0307] The robot body 501 also includes at least one sensor 503 that detects and / or collects data about the environment and / or objects in the environment of the robot system 500 (e.g., including people, such as customers). In the illustrated embodiment, the sensor 503 corresponds to a sensor system including a camera, a microphone, and an initial measurement unit, which itself includes three orthogonal accelerometers, a magnetometer, and a compass. However, any suitable sensor can be included or excluded from the at least one sensor 503 as appropriate for a given application. For example, sensor data such as that captured in act 302 of method 300 can be captured by the sensor 503.

[0308] For illustrative purposes, Figure 5Details of certain exemplary components carried by or within a robot body 501, including systems, robots, methods, control modules, and computer program products according to the present application. Such components include at least one processor 530 and at least one non-transitory processor-readable storage medium or "memory" 540 communicatively coupled to the processor 530. The memory 540 stores data 541 and processor-executable instructions 542 (e.g., collectively referred to as a robot control module or computer program product) that, when executed by the processor 530, cause the robot body 501 (including applicable actuatable components, such as either or both of the robot hands 502a and / or 502b) to perform actions and / or functions associated with the systems, methods, control modules, and computer program products of the present application. The at least one processor 530 and the at least one non-transitory processor-readable storage medium 540 together can be considered a robot controller.

[0309] In some embodiments, the actions or processes can be performed entirely locally at the robot body 501. For example, in some embodiments, the entire method 300 can be performed locally at the robot body 501. In such an embodiment, at least one sensor 503 captures sensor data in action 302, and at least one processor 530 generates an NL description of at least one aspect of the environment in action 304. As previously discussed, the at least one processor 530 can further generate any other NL descriptions included in the first NL query. In addition, in such an embodiment, the memory 540 also stores an LLM module, and the first NL query is provided to the LLM in action 306. Providing the first NL query in this case can refer to the at least one processor 530 executing the LLM module with the first NL query as input. In addition, receiving the mission plan from the LLM module as in action 308 of method 300 can include the at least one processor 530 receiving the mission plan output by the LLM module. Furthermore, further, determining whether the mission plan includes at least one fault as in act 310 is performed by at least one processor 530 (e.g., as previously described, by causing the robot system to execute steps in the mission plan, simulating the execution of the mission plan, or running a verification tool (such as an inference engine)). Executing the mission plan until completion as in act 320 includes at least one processor 530 executing instructions that cause the robot agent 501 to perform actions specified in the mission plan. Generating the NL description as in acts 330 and 332 is performed by at least one processor 530. In this case, providing the second NL query to the LLM module as in act 334 can refer to at least one processor 530 executing the LLM module with the second NL query as input. Furthermore, receiving the solution plan from the LLM module as in act 336 of method 300 can include at least one processor 530 receiving the solution plan output by the LLM module. Executing the updated mission plan as in act 338 includes at least one processor 530 executing instructions that cause the robot agent 501 to perform actions specified in the updated mission plan, or at least one processor 530 updates the mission plan based on the solution plan and then executes the updated mission plan.

[0310] In some embodiments, the actions or processes may be performed locally at the robot body 501 or separately by a device separate from the robot body 501. In this regard, the at least one processor 530 is also communicatively coupled to a wireless transceiver 550 via which the robot body 501 sends and receives wireless communication signals 570.

[0311] Figure 5Also included is a standalone device 580, which is part of the robot system 500 but physically separate from the robot body 501. As non-limiting examples, the standalone device 580 can be a processing unit located in close proximity (e.g., in the same room) to the robot body 501, or the standalone device 580 can be a remote server located remote from the robot body 501. The standalone device 580 includes a wireless transceiver 581 via which the standalone device 580 transmits and receives wireless communication signals 570. The wireless transceiver 550 and the wireless transceiver 581 can be referred to as communication interfaces (together or individually) through which the robot body 501 and the standalone device 580 communicate. Furthermore, the transceivers 550 and 581 do not need to communicate directly (although they can); for example, the transceivers 550 and 581 can communicate with each other via a network or the Internet. Furthermore, the transceivers 550 and 581 do not need to be wireless; in some embodiments, either transceiver can be replaced with a wired communication interface. In some embodiments where the LLM module is not stored in the memory 540 , the robot body 501 can access the LLM module via the wireless signal 570 through the transceivers 550 and 581 .

[0312] In particular, the standalone device 580 is further shown as including at least one processor 582 communicatively coupled to the wireless transceiver 581, and at least one non-transitory processor-readable storage medium 590 (or "memory" 590) communicatively coupled to the at least one processor 582. The memory 590 stores data 591 and processor-executable instructions 592 (e.g., together as a robot control module or computer program product) that, when executed by the processor 582, cause the standalone device 580 (or components thereof) to perform actions and / or functions associated with the systems, robots, methods, robot control modules, and computer program products of the present application. The memory 590 may also store an LLM module. Alternatively, the standalone device 580 may access an LLM module stored at another device (e.g., a cloud-based or internet-based LLM module).

[0313] The methods or processes discussed herein (e.g., Figure 3The method 300 in ( ) can be performed by a combination of a robot body 501 and a stand-alone device 580. In an exemplary embodiment, at least one sensor 503 captures sensor data in action 302, and at least one processor 530 generates an NL description of at least one aspect of the environment in action 304. As previously described, the at least one processor 530 may further generate any other NL descriptions included in the first NL query. In this exemplary embodiment, the memory 590 of the stand-alone device 580 stores an LLM module, and in action 306, the first NL query is provided to the LLM module. In this example, providing the first NL query refers to the robot body 501 transmitting the first NL query to the stand-alone device via the transceivers 550 and 581 (communication interface). Then, the at least one processor 582 executes the LLM module with the first NL query as input.

[0314] In some cases, as in act 308 of method 300, receiving a mission plan from the NL includes the robot body 501 receiving a mission plan output by the LLM module, which is transmitted from the standalone device 580 by the transceivers 550 and 581 to be received by the robot controller (or at least one processor 530). As in act 310, determining whether the mission plan includes at least one fault is performed by the at least one processor 530 (for example, as described above, by causing the robot system to execute steps in the mission plan, simulating the execution of the mission plan, or running a verification tool (such as an inference engine)). As in act 320, executing the mission plan until completion includes at least one processor 530 executing instructions that cause the robot body 501 to execute the actions specified in the mission plan. As in acts 330 and 332, generating the NL description is performed by the at least one processor 530. In this example, providing a second NL query refers to the robot body 501 transmitting the second NL query to the standalone device via the transceivers 550 and 581 (communication interface). Then, the at least one processor 582 executes the LLM module with the second NL query as input. Furthermore, receiving the solution plan from the LLM module in act 336 of method 300 may include the robot body 501 receiving the solution plan output by the LLM module, the solution plan being transmitted from the standalone device 580 by the transceivers 550 and 581 for receipt by the robot controller (or the at least one processor 530). Executing the updated mission plan in act 338 may include the at least one processor 530 executing instructions that cause the robot body 501 to perform actions specified in the updated mission plan, or the at least one processor 530 updating the mission plan based on the solution plan and then executing the updated mission plan.

[0315] In other cases, receiving the mission plan from the NL as in act 308 of method 300 includes at least one processor 582 of the standalone device 580 receiving the mission plan output by the LLM module. Determining whether the mission plan includes at least one fault as in act 310 is performed by the at least one processor 582 (e.g., as previously described, by simulating the execution of the mission plan, or running a verification tool (such as an inference engine)). If there is no fault in the mission plan, the mission plan is sent from the standalone device 580 to the robot body 501 via transceivers 550 and 581 (communication interfaces). In such a scenario, executing the mission plan until completion as in act 320 includes at least one processor 530 executing instructions that cause the robot body 501 to perform the actions specified in the mission plan. On the other hand, if there is at least one fault in the mission plan, the at least one processor 582 generates the NL description as in acts 330 and 332. In this scenario, providing the second NL query to the LLM module means that the at least one processor 582 executes the LLM module with the second NL query as input. Furthermore, receiving the solution plan from the LLM module in act 336 of method 300 may include the robot body 501 receiving the solution plan output by the LLM module, the solution plan being transmitted from the standalone device 580 by the transceivers 550 and 581 for receipt by the robot controller (or the at least one processor 530). Executing the updated mission plan in act 338 may include the at least one processor 530 executing instructions that cause the robot body 501 to perform actions specified in the updated mission plan, or the at least one processor 530 updating the mission plan based on the solution plan and then executing the updated mission plan.

[0316] Alternatively, receiving the solution plan from the LLM module as in act 336 of method 300 may include receiving the solution plan output by the LLM module by the at least one processor 582. The at least one processor 582 may then generate an updated mission plan based on the solution plan, which is transmitted from the standalone device 580 by the transceivers 550 and 581 for receipt by the robot controller (or at least one processor 530). Executing the updated mission plan as in act 338 may include the at least one processor 530 executing instructions that cause the robot body 501 to perform the actions specified in the updated mission plan.

[0317] Throughout this specification and the appended claims, the term "communicative" (as in "communicative coupling" and in variations such as "communicatively coupled") is generally used to refer to any engineering arrangement for transmitting and / or exchanging information. For example, communicative coupling can be achieved through a variety of different media and / or forms of communication paths, including but not limited to: conductive paths (e.g., conductive wires, conductive traces), magnetic paths (e.g., magnetic media), wireless signal transmission (e.g., radio frequency antennas), and / or optical paths (e.g., optical fibers). Exemplary communicative couplings include but are not limited to: electrical coupling, magnetic coupling, radio frequency coupling, and / or optical coupling.

[0318] Throughout this specification and the appended claims, infinitive verb forms are frequently used. Examples include, but are not limited to, "encode," "provide," "store," and the like. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, i.e., as in "to at least encode," "to at least provide," "to at least store," and the like.

[0319] This specification, including the drawings and abstract, is not intended to be an exhaustive or limiting description of all implementations and embodiments of the systems, methods, control modules, and computer program products of the present application. Those skilled in the art will appreciate that the various descriptions and drawings provided may be modified without departing from the spirit and scope of the present disclosure. In particular, the teachings herein are not intended to be limited to or restricted by the illustrative examples of computer systems and computing environments provided.

[0320] This specification provides various implementations and examples in the form of block diagrams, schematic diagrams, flow charts and examples. It will be understood by those skilled in the art that any function and / or operation in such block diagrams, schematic diagrams, flow charts or examples can be implemented individually and / or collectively by a wide range of hardware, software and / or firmware. For example, the various embodiments disclosed herein in whole or in part can be equivalently implemented in one or more of the following: an application-specific integrated circuit (i.e., an ASIC); a standard integrated circuit; a computer program executed by any number of computers (e.g., a program running on any number of computer systems); a program executed by any number of controllers (e.g., a microcontroller); and / or a program executed by any number of processors (e.g., a microprocessor, a central processing unit, a graphics processing unit), and any combination of firmware and the foregoing.

[0321] Throughout this specification and the appended claims, a “memory” or “storage medium” is a processor-readable medium, which is an electronic, magnetic, optical, electromagnetic, infrared, semiconductor, or other physical device or apparatus that contains or stores processor data, data objects, logic, instructions, and / or programs. When data, data objects, logic, instructions, and / or programs are implemented as software and stored in a memory or storage medium, they may be stored in any suitable processor-readable medium for use by any suitable processor-related instruction execution system, apparatus, or device, such as a computer-based system, a system containing a processor, or other system capable of retrieving data, data objects, logic, instructions, and / or programs from a memory or storage medium and performing various actions or operations (i.e., processing steps) thereon and / or in response thereto. Thus, a “non-transitory processor-readable storage medium” may be any element that stores data, data objects, logic, instructions, and / or programs for use by or in conjunction with an instruction execution system, apparatus, and / or device. As specific non-limiting examples, the processor readable medium may be: a portable computer floppy disk (magnetic, compact flash card, secure digital device, etc.), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, EEPROM or flash memory), a portable compact disk read-only memory (CDROM), digital magnetic tape, and / or any other non-transitory medium.

[0322] The claims of the present disclosure are appended hereto. This disclosure is intended to support, enable, and illustrate the claims, but is not intended to limit the scope of the claims to any specific implementation or example. In general, the claims should be interpreted to include all possible implementations and examples, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A method of operating a robotic system including a robotic body, the method comprising: capturing, by at least one sensor of the robotic system, sensor data representing information about an environment of the robotic body; generating, by at least one processor of the robotic system, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing a first NL query to a large language model (LLM) module, the first NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receiving the mission plan from the LLM module, wherein the mission plan is expressed in NL and includes a plurality of steps; determining, by the at least one processor, whether the mission plan includes at least one fault; If the mission plan includes at least one failure: generating, by the at least one processor, an NL description of the at least one fault; providing a second NL query to the LLM module, the second NL query including at least an NL description of the at least one fault and an NL request for a solution plan; receiving the solution plan from the LLM module, the solution plan being expressed in NL; as well as executing, by the robotic system, an updated mission plan based at least in part on the solution plan; as well as If the mission plan does not include at least one failure: The mission plan is executed by the robotic system to completion.

2. The method according to claim 1, wherein: The method further includes: if the mission plan includes at least one fault, generating an NL description of any steps in the mission plan that do not include the fault; and The second NL query also includes an NL description of any step in the mission plan that does not include a failure.

3. The method according to claim 1, wherein The solution plan includes an NL description of at least one solution selected from the group of solutions consisting of: Adjustments to the sequence of steps in the mission plan; Adjustments to the success criteria for any step in the mission plan; adjustments to any action expressed in any step of the mission plan; Adjustment of any interaction objectives for any step of the mission plan; replacement of any step of said mission plan with at least one new step; removing at least one step of the mission plan; as well as The updated mission plan is used to replace all steps of the mission plan.

4. The method according to claim 1, wherein Determining whether the mission plan includes at least one fault may include: executing at least one step of the mission plan; and For each of the at least one step, it is identified whether execution of the corresponding step generates a corresponding fault scenario.

5. The method according to claim 4, wherein Each respective failure scenario includes at least one scenario selected from the group of scenarios consisting of: The execution of the steps produces results inconsistent with the stated work objectives; Execution of a step prevents execution of at least one other step in the task plan; as well as Execution of the steps produces at least one unacceptable effect in the environment.

6. The method according to claim 1, wherein Determining whether the mission plan includes at least one fault includes determining whether the mission plan violates any rule in a set of rules specified in at least a portion of an inference engine.

7. The method according to claim 1, wherein Determining whether the mission plan includes at least one fault includes: accessing an environment model representing the environment; simulating, by the at least one processor, at least one step of the mission plan in the environment model; and The at least one processor identifies whether each of the at least one step of the mission plan generates a corresponding failure scenario when the simulated execution is performed.

8. The method according to claim 7, wherein: Determining whether the mission plan includes at least one fault further comprises: simulating, by the at least one processor, at least one additional step of the mission plan in the environment model if a corresponding failure scenario is not identified in each of the at least one step of simulating the mission plan; and The at least one processor identifies whether each of the at least one additional step of the mission plan generates a corresponding failure scenario when the simulated execution is performed.

9. The method according to claim 7, wherein: The corresponding failure scenario includes at least one scenario selected from the following scenario group: The simulated execution of the steps produces simulation results that are inconsistent with the work objectives; The simulated execution of a step prevents execution of at least one other step in the mission plan; as well as The simulated performance of the steps produces at least one unacceptable effect in the environment model.

10. A robotic control module comprising at least one non-transitory processor-readable storage medium storing processor-executable instructions or data that, when executed by at least one processor of a processor-based system, cause the processor-based system to: capturing, via at least one sensor carried by a robotic body of the processor-based system, sensor data representing information about an environment of the robotic body; generating, by the at least one processor, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing a first NL query to a large language model (LLM) module, the first NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the processor-based system, and an NL request for a task plan; receiving the mission plan from the LLM module, wherein the mission plan is expressed in NL and includes a plurality of steps; determining, by the at least one processor, whether the mission plan includes at least one fault; If the mission plan includes at least one failure: generating, by the at least one processor, an NL description of the at least one fault; providing a second NL query to the LLM module, the second NL query including at least an NL description of the at least one fault and an NL request for a solution plan; receiving the solution plan from the LLM module, the solution plan being expressed in NL; as well as executing, by the processor-based system, an updated mission plan based at least in part on the solution plan; as well as If the mission plan does not include at least one failure: The mission plan is executed to completion by the processor-based system.

11. The robot control module according to claim 10, wherein: The processor-executable instructions or data further cause the at least one processor to: if the mission plan includes at least one fault, generate an NL description for any steps in the mission plan that do not include a fault; and The second NL query also includes an NL description of any step in the mission plan that does not include a failure.

12. The robot control module according to claim 10, wherein: The solution plan includes an NL description of at least one solution selected from the group of solutions consisting of: Adjustments to the sequence of steps in the mission plan; Adjustments to the success criteria for any step in the mission plan; adjustments to any action expressed in any step of the mission plan; Adjustment of any interaction objectives for any step of the mission plan; replacement of any step of said mission plan with at least one new step; removing at least one step of the mission plan; as well as The updated mission plan is used to replace all steps of the mission plan.

13. The robot control module according to claim 10, wherein: The processor-executable instructions or data that cause the at least one processor to determine whether the mission plan includes at least one fault cause the processor-based system to: executing at least one step of the mission plan; as well as For each of the at least one step, the at least one processor identifies whether execution of the corresponding step generates a corresponding fault scenario.

14. The robot control module according to claim 13, wherein: Each respective failure scenario includes at least one scenario selected from the group of scenarios consisting of: The execution of the steps produces results inconsistent with the stated work objectives; Execution of a step prevents execution of at least one other step in the task plan; as well as Execution of the steps produces at least one unacceptable effect in the environment.

15. The robot control module according to claim 10, wherein: The processor-executable instructions or data that cause the at least one processor to determine whether the mission plan includes at least one fault cause the at least one processor to determine whether the mission plan violates any rule in a rule set specified in at least a portion of an inference engine.

16. The robot control module according to claim 10, wherein: The processor-executable instructions or data that cause the at least one processor to determine whether the mission plan includes at least one fault cause the at least one processor to: accessing an environment model representing the environment; simulating, by the at least one processor, at least one step of the mission plan in the environment model; as well as The at least one processor identifies whether each of the at least one step of the mission plan generates a corresponding failure scenario when the simulated execution is performed.

17. The robot control module according to claim 16, wherein: The processor-executable instructions or data that cause the at least one processor to determine whether the mission plan includes at least one fault further cause the at least one processor to: simulating, by the at least one processor, at least one additional step of the mission plan in the environment model if a corresponding failure scenario is not identified in each of the at least one step of simulating the mission plan; as well as The at least one processor identifies whether each of the at least one additional step of the mission plan generates a corresponding failure scenario when the simulated execution is performed.

18. The robot control module according to claim 16, wherein: The corresponding failure scenario includes at least one scenario selected from the following scenario group: The simulated execution of the steps produces simulation results that are inconsistent with the work objectives; The simulated execution of a step prevents execution of at least one other step in the mission plan; as well as The simulated performance of the steps produces at least one unacceptable effect in the environment model.

19. A robotic system comprising: Robot body; at least one sensor; a robotic controller comprising at least one processor and at least one non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the at least one processor, cause the robotic system to: capturing, by the at least one sensor, sensor data representing information about an environment of the robotic body; generating, by the at least one processor, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing a first NL query to a large language model (LLM) module, the first NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receiving the mission plan from the LLM module, wherein the mission plan is expressed in NL and includes a plurality of steps; determining, by the at least one processor, whether the mission plan includes at least one fault; If the mission plan includes at least one failure: generating, by the at least one processor, an NL description of the at least one fault; providing a second NL query to the LLM module, the second NL query including at least an NL description of the at least one fault and an NL request for a solution plan; receiving the solution plan from the LLM module, the solution plan being expressed in NL; as well as executing, by the robotic system, an updated mission plan based at least in part on the solution plan; as well as If the mission plan does not include at least one failure: The mission plan is executed by the robotic system to completion.

20. The robotic system of claim 19, wherein: The mission plan indicates at least one action, expressed in a natural language, that can be performed by the robotic system; and The processor-executable instructions further cause the robotic system to generate, by the at least one processor, a robotic language task plan based on the task plan expressed in NL, the robotic language task plan comprising a set of robotic control instructions that, when executed by the at least one processor, cause the robotic system to perform at least one of the at least one action indicated in the task plan.

Images