Robotic systems, methods, control modules, and computer program products utilizing a

Abstract

Robot control systems, methods, control modules, and computer program products are described that utilize one or more large language models (LLMs) in order to achieve at least some degree of autonomy. The robot control parameters and / or instructions may advantageously be specified in a natural language (NL) and communicate with the LLM via NL prompts or queries. The NL response from the LLM may then be converted into robot control parameters and / or instructions. In this manner, the robot control system may utilize LLM to enhance autonomy of various operations and / or functions, including but not limited to task planning, motion planning, human interaction, and / or reasoning about the environment.

Description

[0001] Previous application data

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 441,897, filed on January 30, 2023, entitled “Robot Control Systems, Methods, and Computer Program Products That Leverage Large Language Models,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The systems, methods, control modules, and computer program products of the present application generally relate to robotic control, and particularly to deploying, utilizing, and / or generally using large language models in robotic control.

[0004] background

[0005] Description of Related Technology

[0006] A robot is a machine that can be employed to perform a task. Robots can come in a variety of form factors, including humanoid form factors. Humanoid robots can be operated by teleoperations, which allow the robot to mimic the physical movements of a human operator or driver. Specialized robots can be designed to perform a specific task, while general-purpose robots can be designed to perform multiple tasks.

[0007] Humans perform many tasks in their personal and work lives. Examples include making beds, washing dishes, loading the dishwasher, mowing the lawn, taking inventory, checking out customers, stocking shelves, painting, styling hair, preparing meals, cleaning, measuring, performing calculations, recording data, performing analysis, creating art / music, performing art / music, building, manufacturing, assembling, destroying, disassembling, moving objects, picking and placing, navigating, and so on. In many cases, there is a strong desire and ongoing need to automate various tasks so that humans can redirect their time and / or attention to other things.

[0008] Large Language Models (LLMs) are a form of artificial intelligence that is trained on a large corpus of text data to produce human-like text responses to natural language (NL) inputs. Popular examples in the field today include OpenAI TMVarious incarnations of the Generative Pre-Trained Transformer (GPT) such as text-davinci-003, text-curie-001, text-babbage-001, and text-ada-001. LLM can be accessed through or deployed in text-based user interfaces to allow chat-like interaction between users and computers, such as in OpenAI TM LLM-based GPT-3 TM ChatGPT built in series TM application.

[0009] Brief Overview

[0010] One approach can be summarized as using at least one large language model (LLM) in the operation of a robot as described herein. The LLM can be utilized to automate at least one process selected from the group consisting of: task planning, motion planning, human interaction, and logical reasoning.

[0011] The robotic system can be generalized to use at least one large language model as described herein.The LLM can be utilized to automate at least one process selected from the group consisting of: task planning, motion planning, human interaction, and logical reasoning.

[0012] The computer program product may be summarized as a non-transitory processor-readable storage medium storing data and / or processor-executable instructions that, when executed by at least one processor of a robot, causes the robot to utilize at least one large language model as described herein. The LLM may be utilized to automate at least one process selected from the group consisting of: task planning, motion planning, human interaction, and logical reasoning.

[0013] According to a broad aspect, the present disclosure describes a method of operating a robotic system including a robotic body, the method comprising: capturing sensor data representing information about an environment of the robotic body by at least one sensor of the robotic system; generating a natural language (NL) description of at least one aspect of the environment based on the sensor data by at least one processor of the robotic system; providing an NL query to a large language model (LLM) module, the NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receiving a task plan from the LLM module, the task plan being expressed in NL; and executing the task plan by the robotic system.

[0014] The task plan may indicate at least one action executable by the robotic system expressed in a natural language; and the method may further include generating, by at least one processor, a robotic language task plan based on the task plan expressed in a natural language, the robotic language task plan including a set of robotic control instructions that, when executed by the at least one processor, cause the robotic system to perform the at least one action indicated in the task plan. Generating the robotic language task plan may include executing a robotic language conversion module that converts the at least one action executable by the robotic system expressed in the natural language into at least one reusable work primitive in an instruction set executable by the robotic system.

[0015] Generating the NL description of the environment may include: identifying objects or features in the environment using robot language text labels; and executing a text string matching module that matches text in the robot language text labels with natural language vocabulary.

[0016] The method may further include generating, by the at least one processor, an NL description of the instruction set, and generating the NL description of the instruction set may include executing a robot language conversion module that generates an NL description of each instruction in the instruction set expressed in a robot language that is executable by the robotic system. The robot language conversion module may include a text string matching module that is operable to compare the robot language instructions in the instruction set with a natural language vocabulary representing actions executable by the robotic system and identify matching text strings for inclusion in the NL description of the instruction set.

[0017] The method may also include generating, by at least one processor, a NL request for the mission plan based on the request template.

[0018] The method may further include accessing a pre-generated NL template from at least one non-transitory processor-readable storage medium, the pre-generated NL template including at least one of an NL description of a work objective, an NL description of an instruction set, or an NL request for a mission plan.

[0019] The method may further include validating the mission plan prior to executing the mission plan by comparing the mission plan to a set of rules specified in at least a portion of the inference engine.

[0020] According to another broad aspect, the present disclosure describes a robot control module comprising at least one non-transitory processor-readable storage medium storing processor-executable instructions or data that, when executed by at least one processor of a processor-based system, causes the processor-based system to: capture sensor data representing information about an environment of the robot body by at least one sensor carried by a robot body of the processor-based system; generate a natural language (NL) description of at least one aspect of the environment based on the sensor data by the at least one processor; provide an NL query to a large language model (LLM) module, the NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the processor-based system, and an NL request for a task plan; receive a task plan from the LLM module, the task plan being expressed in NL; and execute the task plan by the processor-based system.

[0021] The task plan may indicate at least one action expressed in a natural language to be performed by the processor-based system; the processor-executable instructions or data may also cause the at least one processor to generate a robot language task plan based on the task plan expressed in NL, the robot language task plan including a set of robot control instructions that, when executed by the at least one processor, cause the processor-based system to perform at least one action indicated in the task plan; and the processor-executable instructions or data that cause the at least one processor to generate the robot language task plan may cause the at least one processor to execute a robot language conversion module that converts the at least one action expressed in a natural language to be performed by the processor-based system into at least one reusable work primitive in an instruction set executable by the processor-based system.

[0022] Processor-executable instructions or data causing at least one processor to generate an NL description of an environment may cause the at least one processor to: identify objects or features in the environment using robot language text labels; and execute a text string matching module that matches text in the robot language text labels to natural language vocabulary.

[0023] The processor-executable instructions or data may further cause the processor-based system to generate an NL description of the instruction set by at least one processor; the processor-executable instructions or data causing the processor-based system to generate the NL description of the instruction set may cause the at least one processor to execute a robot language conversion module that generates an NL description of each instruction in the instruction set expressed in a robot language that is executable by the processor-based system; and the robot language conversion module may include a text string matching module that is operable to compare the robot language instructions in the instruction set with a natural language vocabulary representing actions executable by the processor-based system, and identify matching text strings to include in the NL description of the instruction set.

[0024] The processor-executable instructions or data may further cause the at least one processor to generate a NL request for the mission plan based on the request template.

[0025] The processor-executable instructions or data may also enable the processor-based system to access a pre-generated NL template from at least one non-transitory processor-readable storage medium, the pre-generated NL template comprising at least one of an NL description of a work objective, an NL description of an instruction set, or an NL request for a task plan.

[0026] The processor-executable instructions or data may also cause the processor-based system to validate the mission plan before executing the mission plan by comparing the mission plan to a set of rules specified in at least a portion of the inference engine.

[0027] According to another broad aspect, the present disclosure describes a robotic system comprising: a robotic body; at least one sensor; a robotic controller comprising at least one processor and at least one non-transitory processor-readable storage medium, the at least one non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the at least one processor, cause the robotic system to: capture sensor data representing information about an environment of the robotic body by the at least one sensor; generate a natural language (NL) description of at least one aspect of the environment based on the sensor data by the at least one processor; provide an NL query to a large language model (LLM) module, the NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receive a task plan from the LLM module, the task plan being expressed in NL; and execute the task plan.

[0028] The LLM module may be stored at at least one non-transitory processor-readable storage medium of the robotic controller; and the processor-executable instructions that cause the robotic system to provide the NL query to the LLM module may cause the robotic system to execute the LLM module with the NL query as input to the LLM module.

[0029] The LLM module can be stored at a separate device from the robotic system; the robotic system can include at least one communication interface; the processor-executable instructions that cause the robotic system to provide an NL query to the LLM module can cause the at least one communication interface to transmit the NL query to the separate device; and the processor-executable instructions that cause the robotic system to receive a mission plan from the LLM module can cause the at least one communication interface to receive the mission plan from the separate device and provide the mission plan to the robotic controller.

[0030] The task plan may indicate at least one action executable by the robotic system expressed in a natural language; and the processor-executable instructions may further cause the robotic system, by at least one processor, to generate a robotic language task plan based on the task plan expressed in NL, the robotic language task plan including a set of robotic control instructions that, when executed by at least one processor, cause the robotic system to perform at least one action indicated in the task plan.

[0031] Brief Description of Several Drawings

[0032] The various elements and actions depicted in the accompanying drawings are provided for illustrative purposes to support the detailed description. Unless the specific context requires otherwise, the size, shape, and relative positions of the elements and actions shown are not necessarily shown to scale and are not necessarily intended to convey any information or limitations. Generally, the same reference numerals are used to identify similar elements or actions.

[0033] Figure 1 is a flow chart illustrating an exemplary embodiment of an automated mission planner utilizing an LLM according to the systems, control modules, methods, and computer program products of the present application.

[0034] Figure 2 is a flow chart illustrating an exemplary embodiment of the operation of a robotic system utilizing an LLM according to the systems, control modules, methods, and computer program products of the present application.

[0035] Figure 3 is an illustrative diagram of an exemplary embodiment of a robot with access to an LLM according to the systems, control modules, methods, and computer program products of the present application, wherein the LLM is stored and executed externally to the robot (e.g., in the cloud) and is called or accessed by the robot control system.

[0036] Figure 4is an illustrative diagram of an exemplary embodiment of a robot having access to an LLM according to the systems, control modules, methods, and computer program products of the present application, wherein the LLM is stored and executed locally on the robot as an integral part of the robot's control system.

[0037] Figure 5 is an illustrative diagram of an exemplary robotic system that includes various features and components described throughout the systems, control modules, methods, and computer program products of this application.

[0038] Detailed description

[0039] The following description sets forth specific details in order to illustrate and provide an understanding of various embodiments and examples of the systems, methods, control modules, and computer program products of the present application. Those skilled in the art will appreciate that some of the specific details described herein may be omitted or modified in alternative embodiments and examples, and that the various embodiments and examples described herein may be combined with each other and / or with other methods, components, materials, etc. to produce additional embodiments and examples.

[0040] In some instances, well-known structures and / or processes associated with computer systems and data processing have not been shown or provided in detail to avoid unnecessarily complicating or obscuring the description of the implementations and examples.

[0041] Unless the specific context requires otherwise, throughout this specification and the appended claims, the term "comprise" and variations thereof (e.g., "comprises" and "comprising") are used in an open, inclusive sense to mean "including, but not limited to."

[0042] Throughout this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the specific context requires otherwise. For example, reference to "an embodiment" and "the embodiment" include "embodiments" and "the embodiments," respectively, and reference to "an implementation" and "the implementation" include "implementations" and "the implementations," respectively. Similarly, the term "or" is generally employed in its broadest sense to mean "and / or" unless the specific context clearly dictates otherwise.

[0043] The title and abstract of the present disclosure are provided for convenience only and are not intended to and should not be construed as interpreting the scope or meaning of the systems, methods, control modules, and computer program products of the present application.

[0044] Various embodiments described herein provide systems, methods, control modules, and computer program products for enhancing, facilitating, augmenting, or enabling control of one or more robotic systems using one or more LLMs. Exemplary robotic systems that may employ the teachings of the systems, methods, control modules, and computer program products of the present application include, but are not limited to, a general-purpose humanoid robot developed by Sanctuary Cognitive Systems, Inc., various aspects of which are described in the following documents: U.S. patent application serial number 16 / 940,566 (publication number US2021-0031383 A1), U.S. patent application serial number 17 / 023,929 (publication number US2021-0090201 A1), U.S. patent application serial number 17 / 061,187 (publication number US2021-0122035 A1), U.S. patent application serial number 17 / 098,716 (publication number US2021-0146553 A1), U.S. patent application serial number 17 / 111,789 (publication number US2021-0170607 A1), U.S. patent application serial number 17 / 158,244 (publication number US2021-0158,244). 2021-0234997A1), U.S. Provisional Patent Application Serial No. 63 / 001,755 (Publication No. US2021-0307170 A1) and / or U.S. Provisional Patent Application Serial No. 63 / 057,461 and U.S. Provisional Patent Application Serial No. 63 / 151,044, U.S. Provisional Patent Application Serial No. 63 / 173,670, U.S. Provisional Patent Application Serial No. 63 / 184,268, U.S. Provisional Patent Application Serial No. 63 / 213,385, U.S. Provisional Patent Application Serial No. 63 / 232,694, U.S. Provisional Patent Application Serial No. 63 / 316,693, U.S. Provisional Patent Application Serial No. 63 / 253,591, U.S. Provisional Patent Application Serial No. 63 / 293,968, U.S. Provisional Patent Application Serial No. 63 / 293,973 and / or U.S. Provisional Patent Application Serial No. 63 / 278,817, each of which is incorporated herein by reference in its entirety.

[0045] In some embodiments, a robotic system or control module may employ a finite instruction set comprising general reusable work primitives that can be combined (in various combinations and / or permutations) to perform tasks. For example, a robotic control system may store a library of reusable work primitives, each work primitive corresponding to a corresponding basic subtask or sub-action (hereinafter referred to as an instruction set) that the robot is operable to perform autonomously. The work objective may be analyzed to determine a sequence (i.e., combination and / or permutation) of reusable work primitives that, when executed by the robot, will complete the work objective. The robot may execute a sequence of reusable work primitives to complete the work objective. In this way, a finite instruction set may be used to perform a wide range of different types of tasks and work objectives across a wide range of industries. The method is described in U.S. Patent Publication No. 2022-0258340, based on U.S. patent application serial number 17 / 566,589, which is incorporated herein by reference in its entirety.

[0046] To expand on the above, a general-purpose robot can accomplish a plurality of different work objectives. As used throughout this specification and the appended claims, the term "work objective" refers to a specific task, job, assignment, or application with a specified goal and a determinable result, typically (although not necessarily) to facilitate some economically valuable work. Work objectives exist in many aspects of business, research and development, commercial operations, and personal activities. Exemplary work objectives include, but are not limited to: cleaning locations (e.g., bathrooms) or objects (e.g., bathroom mirrors), preparing meals, loading / unloading storage containers (e.g., trucks), taking inventory, collecting one or more samples, taking one or more measurements, building or assembling objects, destroying or disassembling objects, transmitting articles, receiving objects and / or data, and the like. Various embodiments described herein provide robots, systems, control modules, computer program products, and methods for operating a robotic system to at least semi-autonomously accomplish tasks or work objectives.

[0047] According to the robots, systems, control modules, computer program products, and methods of the present application, a work goal can be deconstructed or broken down into a "workflow" comprising a set of one or more "work primitives," wherein the successful completion of the work goal involves executing each work primitive in the workflow. Depending on the specific implementation, completion of the work goal can be achieved by (i.e., the workflow can include the following): i) executing the corresponding set of work primitives sequentially or continuously; ii) executing the corresponding set of work primitives in parallel; or iii) executing the corresponding set of work primitives in any combination of sequential and parallel execution (e.g., sequentially overlapping) appropriate to the work goal and / or the robots executing the work goal. Thus, in some implementations, work primitives can be interpreted as lower-level activities, steps, or subtasks that are implemented or executed as a workflow to complete a higher-level work goal.

[0048] Advantageously, according to the robots, systems, control modules, computer program products and methods of the present application, a catalog of "reusable" work primitives can be defined. A work primitive is reusable if it can be universally called, executed, adopted or applied in completing multiple different work objectives. For example, a reusable work primitive is a work primitive that is common to the corresponding workflows of multiple different work objectives. In some embodiments, a reusable work primitive may include at least one variable defined when calling the work primitive or before calling the work primitive. For example, "picking up an *object*" can be a reusable work primitive, where the process of "picking up" can generally be performed at least semi-autonomously to facilitate multiple different work objectives, and the *object* to be picked up can be defined based on the specific work objective being pursued.

[0049] As previously described, various embodiments described herein provide robots, systems, control modules, computer program products, and methods in which the robots are capable of performing tasks or completing work objectives at least semi-autonomously. Unless the specific context requires otherwise, the term "autonomously" is used throughout this specification and the appended claims to mean "not under the control of another party," while the term "semi-autonomously" is used to mean "at least partially autonomously." In other words, throughout this specification and the appended claims, the term "semi-autonomously" means "under limited control by another party," unless the specific context requires otherwise. An example of a semi-autonomous robot is one that can independently and / or automatically perform and control some of its own low-level functions (such as its mobility and grasping functions), but relies on some external control for high-level instructions (such as what to do and / or how to do it).

[0050] According to the robots, systems, control modules, computer program products and methods of the present application, a catalog of reusable work primitives can be defined, identified, developed or constructed so that any given work objective across multiple different work objectives can be completed by executing a corresponding workflow that includes a specific combination and / or arrangement of reusable work primitives selected from the catalog of reusable work primitives. Once such a catalog of reusable work primitives has been established, one or more robots can be trained to autonomously or automatically execute each individual reusable work primitive in the catalog of reusable work primitives without having to include the following context: i) the specific workflow of which the specific reusable work primitive being trained is a part, and / or ii) any other reusable work primitives that may precede or follow the specific reusable work primitive being trained in the specific workflow. In this way, a semi-autonomous robot can operate to autonomously or automatically execute each individual reusable work primitive in a catalog of reusable work primitives, and only requires instructions, directions, or guidance from another party (e.g., an operator, user, or driver) when deciding which reusable work primitive to execute and / or in what order. In other words, an operator, user, driver, or LLM module can provide a workflow consisting of reusable work primitives to a semi-autonomous robotic system, and the semi-autonomous robotic system can autonomously or automatically execute the reusable work primitives according to the workflow to complete a work objective. For example, a semi-autonomous humanoid robot can operate to autonomously look left when instructed to look left, autonomously open its right end effector when instructed to open its right end effector, and so on, without relying on a third party for detailed low-level control of such functions. Once given instructions regarding a workflow that details which reusable work primitives it must execute and in what order in order to complete the work objective, such a semi-autonomous humanoid robot can autonomously complete the work objective. In addition, according to the robots, systems, methods, control modules and computer program products of the present application, if the robotic system is trained or otherwise configured (for example, via consultation with an LLM module, which can be included in the robotic system) to analyze work objectives and independently define the corresponding workflow itself by deconstructing the work objectives into a set of reusable work primitives from a library of reusable work primitives that the robotic system can operate to autonomously execute.

[0051] In the context of a robotic system, a reusable work primitive may correspond to a basic low-level function that the robotic system is operable to perform (e.g., autonomously or automatically) and that the robotic system can call or execute in order to achieve something. Examples of reusable work primitives for a humanoid robot include, but are not limited to: look up, look down, look left, look right, move right arm, move left arm, close right end effector, open right end effector, close left end effector, open left end effector, move forward, turn left, turn right, move backward, etc., as well as cognitive functions such as analysis, calculation, planning, determination, reasoning, etc.; however, those skilled in the art will recognize that: i) the foregoing list of exemplary reusable work primitives for a humanoid robot is by no means exhaustive; ii) in the robots, systems, control modules, computer program products, and methods of the present application, the high-level functions that the robot is operable to perform are deconstructed or broken down into a set of basic components or constituent parts, referred to throughout this specification and the appended claims as "work primitives." Unless the specific context requires otherwise, work primitives may be interpreted as building blocks for constructing higher-level robotic functions.

[0052] In some embodiments, training a robotic system to autonomously execute reusable work primitives can be accomplished in a real-world environment or a simulated environment. Once the robot has been trained to autonomously execute a catalog of reusable work primitives, the robot's operations can be abstracted to the level of reusable work primitives; for example, an LLM module that prepares a mission plan for the robot can do so by determining which reusable work primitives to execute and, in some embodiments, in what order to execute them, and the robot can have sufficient autonomy or automation to execute a complete work objective based on such limited control instructions.

[0053] As mentioned previously, "clean the bathroom mirror" is an illustrative example of a work goal that can be deconstructed into a set of work primitives to achieve the goal, and whose results are determinable. In this case, the goal is a clean bathroom mirror, and an exemplary set of work primitives (or workflows) to accomplish the work goal are as follows:

[0054]

[0055]

[0056] Those skilled in the art will appreciate that the exemplary workflow described above, comprising nine work primitives, is used as an illustrative example of a workflow that can be deployed to accomplish the work objective of cleaning a bathroom mirror; however, the precise definition and composition of each work primitive, as well as the specific combination and / or arrangement of work primitives selected / executed to accomplish the work objective (i.e., the specific configuration of the workflow), can vary in different embodiments according to the present robots, systems, control modules, computer program products, and methods. For example, in some embodiments, work primitives 3, 4, and 5 described above (i.e., positioning the mirror, directing the cleaning solution toward the mirror, and dispensing the cleaning solution onto the mirror) can all be combined into one higher-level work primitive, such as "spraying cleaning solution onto the mirror," while in other embodiments, these same work primitives can be decomposed into additional lower-level work primitives, such as, for example:

[0057] Positioning mirror

[0058] Identifying the mirror's boundaries

[0059] Aim the cleaning solution at the first location within the boundaries of the mirror

[0060] Squeeze cleaning solution

[0061] Aim the cleaning solution at a second location within the boundaries of the mirror

[0062] Squeeze cleaning solution

[0063] etc.

[0064] Based on the above examples and description, those skilled in the art will recognize that the granularity of work primitives can vary across different implementations of the robots, systems, control modules, computer program products, and methods of the present application. Furthermore, according to the robots, systems, control modules, computer program products, and methods of the present application, work primitives are advantageously "reusable," in the sense that each work primitive can be employed, invoked, applied, or "reused" in the execution of more than one overall work objective. For example, while cleaning a bathroom mirror may involve the work primitive "grab cleaning solution," other work objectives may also utilize the "grab cleaning solution" work primitive, such as, for example, "clean toilet," "clean windows," and / or "clean floor." In some implementations, work primitives can be abstracted to become more general. For example, "grab cleaning solution" can be abstracted to become "grab spray bottle" or "grab *object1*," where the *object1* variable is defined as "*object1* = spray bottle," while "locate mirror" can be abstracted to become "locate the object to be sprayed" or simply "locate *object2*," where "*object2* = mirror." In such a case, the "grab spray bottle" work primitive can be used for tasks that do not involve cleaning, such as "paint walls" (where spray bottle = spray paint), "style hair" (where spray bottle = hair spray), or "stir-fry meal" (where spray bottle = cooking oil spray).

[0065] Unless the specific context requires otherwise, throughout this specification and the appended claims, references to "LLM" or "LLM module" should be interpreted to include one or more LLMs or one or more LLM modules, and / or one or more applications or programs that run, access, use, or otherwise utilize at least one LLM. For example, references to interacting with an LLM or LLM module (e.g., providing input to the LLM, receiving output from the LLM, interrogating the LLM, querying the LLM, etc.) can be understood by using an application or interface of the LLM module (e.g., a chat application such as OpenAI that accesses the LLM to interpret input and formulate output). TM GPT-3 in LLM TM ChatGPT built on series TM ) to carry out.

[0066] In some embodiments of the systems, methods, control modules, and computer program products of the present application, an LLM is used to help determine a sequence of reusable work primitives (hereinafter referred to as "instructions") selected from a limited library of reusable work primitives (hereinafter referred to as "instruction sets") that, when executed by a robot, will enable the robot to complete a task or enable the robot to complete a task. In some embodiments, an LLM is used to help determine a "workflow." For example, a robot control system may take a natural language (NL) command as input and return a task plan formed by a series of allowed instructions extracted from the instruction set, the completion of which realizes the intention of the NL input. Throughout this specification and the appended claims, unless otherwise required by a specific context, a task plan may include or consist of a workflow depending on a specific embodiment. An exemplary application is to "assemble" a set of chess pieces comprising sixteen white chess pieces and sixteen black chess pieces. A person can say or type to a robot something like, "Put all the white pieces in the right-hand box, and all the black pieces in the left-hand box," and the LLM can support a fully autonomous system that converts that input into a permissible sequence of instructions to successfully perform the task. In this case, the LLM can help allow the robot to perform the general tasks specified in the NL. General tasks include, but are not limited to, all jobs in the current economy.

[0067] Throughout the systems, methods, control modules and computer program products of the present application, the term "natural language" refers to any language that has evolved naturally in humans, and includes but is not limited to: English, French, Spanish, Chinese (Mandarin, Cantonese, Wu, etc.), Portuguese, Japanese, Russian, Korean, Arabic, Hebrew, German, Polish, Hindi, Bengali, Italian, Punjabi, Vietnamese, Hausa, Swedish, Finnish, etc.

[0068] Figure 1 is a flowchart illustrating an exemplary embodiment of an automated task planner as method 100. LLM101 (such as, but not limited to, OpenAI TM The text-davinci-03) is provided with or otherwise combined with: i) a description of a current scene 102 generated using the robotic system's perception system, ii) a description of an instruction set 103 formed of parameterized reusable work primitives or instructions that can be executed by the robotic system, and iii) a wrapper that incorporates NL phrases into prompts 104 to automatically generate a mission plan 105 that can then be executed by the robotic system to implement the intent of the NL input.

[0069] Although Figure 1The scene description 102 and instruction set 103 are shown as being contained “inside” the robot (i.e., within the dashed box representing the robot), and the NL prompts 104, LLM 101, and mission plan 105 (and the conversion to robot control signals) are shown as being “outside” the robot (i.e., outside the dashed box representing the robot), but this is not intended to be limiting and is done for illustrative purposes only. While it is true that in some embodiments, any or all of the NL prompts 104, LLM 101, and / or mission plan 105 (and the conversion to robot control signals) may be stored, generated, accessed, or otherwise present outside (or outside or remote from) the robot, in other embodiments, any or all of the NL prompts 104, LLM 101, and / or mission plan 105 (and the conversion to robot control signals) may be stored, generated, accessed, or otherwise present inside (or on) the robot.

[0070] Here is a concrete example of a prompt and response pair, obtained by running a Python script:

[0071] hint:

[0072] On a table are White Chess Pawn 1, White Chess Pawn 2, Black Chess Pawn 1, Black Chess Pawn 2, a White Chess Rook, a Black Chess Bishop, and two blue boxes. Across from me stands a person. My goal is to do what this person says. This person says, "Put all the White pieces in the right-hand box, and all the Black pieces in the left-hand box." I can only choose from the following options:

[0073] 1.) Grab the object

[0074] 2.) Place the object on the object

[0075] 3.) Look at the object

[0076] 4.) Slice Object

[0077] 5.) Say hello to that person

[0078] 6.) Put objects into objects

[0079] Where objects should be replaced with the appropriate objects on the table. Here is a list of the above commands I should execute to accomplish my goal:

[0080] response:

[0081] 1. Capture White Chess Pawn 1

[0082] 2. Place White Chess Pawn 1 in the box on the right

[0083] 3. Capture White Chess Pawn No. 2

[0084] 4. Place White Chess Pawn 2 in the box on the right

[0085] 5. Capture Black Chess Pawn No. 1

[0086] 6. Place Black Chess Pawn 1 in the box on the left

[0087] 7. Capture Black Chess Pawn No. 2

[0088] 8. Place Black Chess Pawn 2 in the box on the left

[0089] 9. Capture the White Chess Rook

[0090] 10. Place the White chess rook in the box on the right

[0091] 11. Capture the Black Chess Bishop

[0092] 12. Place the Black chess bishop in the left-hand box

[0093] Finish.

[0094] In the above example, the response provided by the LLM corresponds to the mission plan. If the robot system executes the sequence of instructions specified in the mission plan, the task specified in the NL via the prompt will be successfully completed by the robot system. Throughout this disclosure, the term "motion plan" may be used instead of "task plan." In this regard, the sequence of instructions specified in the mission plan (motion plan) may include instructions that cause the robot to undergo a series of motions or movements.

[0095] Figure 2 is a flow chart illustrating an exemplary method 200 of operation of a robotic system. Figure 2 The method 200 in is similar in at least some respects to Figure 1 Method 100. Generally speaking, Figure 2 The method 200 in the embodiment of the present invention describes a method that can be implemented by Figure 1 Method 200 is a detailed implementation of the method 100 in the embodiment of the present invention. Figure 5In general, throughout this specification and the appended claims, a method of operating a robotic system is a method in which at least some, if not all, of the various actions are performed by the robotic system. For example, certain actions of the method of operating a robotic system may be performed by at least one processor or processing unit (hereinafter referred to as a "processor") of the robotic system communicatively coupled to a non-transitory processor-readable storage medium of the robotic system (collectively, a robotic controller of the robotic system), and in some embodiments, certain actions of the method of operating a robotic system may be performed by peripheral components of the robotic system communicatively coupled to the at least one processor, such as one or more physically actuatable components (e.g., arms, legs, end effectors, grippers, hands), one or more sensors (e.g., optical sensors, audio sensors, tactile sensors, haptic sensors), mobility systems (e.g., wheels, legs), communication and networking hardware (e.g., receivers, transmitters, transceivers), and the like. The non-transitory processor-readable storage medium of the robotic system can store data (including, for example, at least one library of reusable work primitives and at least one associated perception library) and / or processor-executable instructions that, when executed by at least one processor, cause the robotic system to perform a method and / or cause the at least one processor to perform the actions of the method performed by the at least one processor. The robotic system can communicate with a remote system and / or a remote non-transitory processor-readable storage medium via communication and networking hardware communicatively coupled to the at least one processor of the robotic system. Therefore, unless the specific context requires otherwise, references to the non-transitory processor-readable storage medium of the robotic system and the data and / or processor-executable instructions stored in the non-transitory processor-readable storage medium are not intended to be limiting with respect to the physical location of the non-transitory processor-readable storage medium relative to the at least one processor of the robotic system and the rest of the robotic hardware. In other words, unless the specific context requires otherwise, the non-transitory processor-readable storage medium of the robotic system can include non-transitory processor-readable storage medium located on the robot body of the robotic system and / or non-transitory processor-readable storage medium located remotely from the robot body. Furthermore, a method of operating a robotic system, such as method 200 (or any other method discussed herein), may be implemented as a robotic control module or computer program product.Such a control module or computer program product includes processor-executable instructions or data that, when the control module or computer program product is stored on a non-transitory processor-readable storage medium of the robotic system and the control module or computer program product is executed by at least one processor of the robotic system, cause the robotic system to perform the actions of the method.

[0096] return Figure 2 , the method 200 shown includes actions 202, 204, 206, 208, and 210, although those skilled in the art will appreciate that in alternative implementations, certain actions may be omitted and / or additional actions may be added. Those skilled in the art will also appreciate that the order of actions shown is shown for exemplary purposes only and may be changed in alternative implementations.

[0097] At 202, sensor data representing information about the environment of a robot body of a robotic system is captured. To this end, the robot body may carry at least one exemplary sensor that captures sensor data, as described later with reference to Figure 5 As discussed. In some embodiments, the captured sensor data can be comprehensive, providing a detailed and relatively complete representation of the robot's environment (e.g., a full field of view around the robot's body within a certain distance, with detailed representations of objects or features in the environment surrounding the robot's body). However, such detailed sensor data is not required. In other embodiments, the sensor data may represent only a portion of the environment surrounding the robot's body (e.g., a limited field of view visible to the robot's body's image sensor). In still other embodiments, the sensor data may be even more limited, such as representing only a single object or feature of the environment. How detailed the sensor data representing information about the environment should be can be determined with appropriate precision for a given application.

[0098] At 204, at least one processor of the robotic system generates a natural language (NL) description of at least one aspect of the environment based on the sensor data. Such NL description is Figure 1 The NL description of at least one aspect of the environment need not describe every feature or object present in the sensor data, but may focus on one or more features or objects of particular relevance.

[0099] In an exemplary embodiment, at least one processor executes an object or feature detection model (e.g., a classification module such as a YOLO model or any other suitable model) that identifies objects or features in the environment (as represented in the sensor data) and assigns text labels to such features or objects. Such text labels may be "robot language." Throughout this disclosure, the term "robot language" or the like refers to a language that is used as a result of or intended for use within a robot or program context, as opposed to the natural human language that humans use to communicate with each other. Referring to the previous chess set example, a particular chess pawn may be identified in robot language as "chess_pawn_54677." This is an example of robot language because underscores are used instead of spaces and the numerical identifiers for pawns are much higher than those used by humans in normal contexts.

[0100] In any case, there are useful commonalities (particularly common vocabulary) between robot language and human language. In the example of "chess_pawn_54677", the terms "chess" and "pawn" are also used in human natural language. In order to generate an NL description of at least one aspect of the environment, at least one processor can execute a text string matching module that matches text in the robot language text label with NL vocabulary. For example, the NL description of "chess_pawn_54677" can be generated as "chess pawn number 1". In addition, the objects or features identified in the environment can also be associated with metadata, which can be used to generate an NL description of the environment. For example, the tag "chess_pawn_54677" can be associated with metadata indicating the color of the chess pawn (typically "white" or "black"). For example, at least one processor can use this metadata to generate "chess_pawn_54677" as an NL description of "white chess pawn number 1". However, the inclusion of metadata is not required. For example, a tag may also indicate such information (eg, "white_chess_pawn_54677").

[0101] Additional NL descriptions of other aspects of the environment can also be generated. Referring to the exemplary prompts discussed above, NL descriptions are generated for several different chess pieces, boxes, people, and tables. Such NL descriptions can be generated in a similar manner to that discussed above.

[0102] Furthermore, generating an NL description of an environment is not necessarily limited to generating an NL description of objects or features in the environment. In some embodiments, the positioning or placement of such objects or features can also be described. Referring to the exemplary prompt discussed above, the sentence "On the table are White Chess Pawn 1, White Chess Pawn 2, Black Chess Pawn 1, Black Chess Pawn 2, White Chess Rook, Black Chess Bishop, and two blue boxes, and a person is standing across from me" describes several objects in the environment and their locations. Such an NL description can be generated by at least one processor by, for example, populating a template description with a list of objects based on where the objects fit within the template.

[0103] At 206, the NL query is provided to a large language model (LLM) module. In some embodiments, the LLM module is a software or data module stored on at least one non-transitory processor-readable storage medium of the system (either at the robot body or at a robot controller remote from the robot body). In such embodiments, the NL query can be prepared by at least one processor of the robot system and provided as input to the LLM module. In other embodiments, the LLM module can be a software or data module stored on a non-transitory processor-readable storage medium of a device separate from the robot system. In still other embodiments, the LLM module can refer to a hardware module that receives input prompts and executes the LLM module on the inputs. In such other embodiments, the NL query can be prepared by at least one processor of the robot system and provided to the device where the LLM module is located via a communication interface of the robot system. As a specific example, the LLM module can be stored on one or more servers remote from the robot body and can receive prompts as input via a website, form, or appropriate API. At least one processor of the robot system can prepare the NL query in an appropriate format, and the robot system can send the NL query via the communication interface of the robot system.

[0104] The NL query provided to the LLM module includes an NL description of at least one aspect of the environment as generated at 204. Additionally, the NL query includes an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a mission plan. Each of these NL descriptions is described in detail below.

[0105] As previously discussed, a work objective generally refers to a specific task, job, assignment, or application with a specified purpose and a determinable outcome. The NL description of such a work objective is an expression of such a work objective in a natural human form. Referring to the example of assembling a chess set discussed previously, the prompt includes the phrase "My purpose is to do what the person says." This in itself can be considered an NL description of a work objective, but further input provides more details about the specific purpose that the robotic system should accomplish. In this example, the prompt also includes the phrase "The person said 'Put all the white pieces in the right-hand box, and all the black pieces in the left-hand box.'" This can also be considered an NL description of a work objective and provides specific instructions about what the robotic system is expected to do. In some embodiments, the NL description of the work objective includes the entirety of both phrases "My purpose is to do what the person said. The person said 'Put all the white pieces in the right-hand box, and all the black pieces in the left-hand box.'"

[0106] The NL description of the work goal can be based on a variety of information or data. In some embodiments, an indication of the work goal can be explicitly available to the robot system (e.g., sent to the robot system from a management device or server, or stored in at least one non-transitory processor-readable storage medium of the robot system). The indication of the work goal can be available to the robot system in NL format, so that at least one processor only needs to access the indication of the work goal and provide it to the LLM module. In this sense, at least one processor of the robot system does not need to generate an NL description of the work goal, but can provide an existing NL description of the work goal to the LLM module. Alternatively, the indication of the work goal can be not in NL format (e.g., it can be a robot language), and at least one processor can generate an NL description of the work goal based on the indication of the work goal (e.g., by executing a robot language conversion module, such as a text string matching module similar to the previously discussed). In other embodiments, at least one processor of the robot system can generate an NL description of the work goal based on other information (such as the role in which the robot is deployed). In this context, "role" generally refers to the category of purpose that the robot can serve within the relevant environment. For example, a cleaning robot can act as a role to clean a specific area or facility. In this case, referring to the previous example of assembling a chess set, at least one processor may generate an NL description of the work objective as "clean up the scattered chess pieces and put them into the appropriate boxes."

[0107] The NL description of the work goal can be generated based on any appropriate additional information. In another embodiment, the capabilities of the robot system can be considered when generating the NL description of the work goal. For example, a robot body lacking a motion element can only successfully complete the work goal in the area near the robot body.

[0108] As previously discussed, the set of instructions executable by the robotic system can be a library of reusable working primitives, such as "grasp object", "place object on object", or any other suitable action. These examples are presented here in natural language, but can be stored and accessed in robotic language form, such as "grasp(object)" or "place(object1, object2)" (as non-limiting examples). In the example of assembling a chess set discussed previously, "options" 1, 2, 3, 4, 5, and 6 represent NL descriptions of the set of instructions executable by the robotic system. The exemplary prompt also includes qualifying statements "I can only strictly choose from the following options:" and "the position where the object should be replaced by the appropriate object on the table", which provide the LLM module with additional information about how the NL description of the instruction set should be interpreted, used, or applied. For a given application and a given instruction set, such qualifying statements can be added, removed, or changed as appropriate. Such qualifying statements can also be included in the NL query, for example by including them in a template on which the NL query is based, as discussed in more detail later.

[0109] In some embodiments, an NL description of an instruction set may be pre-generated and loaded into the robotic system such that the robotic system may provide the pre-generated NL description of the instruction set to the LLM module at 206. For example, a management device, server, or configuration device may generate an NL description of an instruction set that may be stored at a non-transitory processor-readable storage medium of the robotic system for subsequent access (e.g., during configuration or deployment of the robotic system). As a specific example, a reusable work primitive "place(object1, object2)" may be stored as "place object on object" along with metadata of the NL description of the reusable work primitive. Such an NL description of an instruction may be provided manually by a human, or may be generated by at least one processor (and potentially reviewed and / or modified by a human to ensure accuracy).

[0110] In some embodiments, an NL description of an instruction set can be generated by the robotic system. Regardless of where the generation of the NL description of the instruction set is performed (in examples where the NL description is generated by at least one processor), the at least one processor performing the generation can execute a robotic language conversion module that generates an NL description for each instruction in the instruction set based on the corresponding instruction expressed in the robotic language. Similar to the previously discussed example, such a robotic language conversion module can include a text string matching module that is operable to compare robotic language instructions in the instruction set with natural language vocabulary representing actions executable by the robotic system. Matching text strings can be identified for inclusion in the NL description of the instruction set. As an example, for the instruction "grasp(object)", the text string matching module can identify "grasp" and "object" as corresponding NL vocabulary. Furthermore, the at least one processor can infer, based on the general structure of the programming function, that the instruction is intended to cause the robot to "grasp" an input "object". To this end, an NL description "grasp object" can be generated for the instruction. As another example, for the instruction "place(object1, object2)", the text string matching module may identify "place" and "object" as corresponding to NL vocabulary. Furthermore, the at least one processor may infer, based on the general structure of the programming function, that the instruction is intended to cause the robot to "place" the input "object1" on the input "object2". To this end, an NL description "place object on object" may be generated for the instruction.

[0111] An NL request for a task plan generally refers to a statement or phrase intended to tell the LLM module how to process the other information in the NL query. Referring to the example of assembling a chess set discussed above, the phrase "Here is a list of the above commands that I should execute in order to accomplish my purpose:". In this example, the phrase is intended to inform the LLM module that it will generate a list of commands selected from an instruction set in order to accomplish the stated purpose (such as the work objective described above). For example, an NL request for a task plan can be generated by at least one processor of a robotic system based on a request template. As another example, an NL request can be included in an NL query template, which is used to construct or format an NL query, as described below.

[0112] In some embodiments, at least one non-transitory processor-readable storage medium of the robotic system stores at least one pre-generated NL template. Such an NL template may include any or all of the corresponding template aspects of an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set, and / or an NL request of a task plan. Exemplary templates are discussed below with reference to the generation of example prompts for assembling a chess set discussed previously. However, other exemplary templates may be used to generate different NL queries in different scenarios. The non-limiting exemplary NL templates discussed may be:

[0113] "There exists [object_array_1] [(,)(and)] [object_array_2] at [position_1] ... [and [object_array_j] at [position_j]. My purpose is [work_objective], and I can only strictly choose from the following options:

[0114] 1.)[reusable_work_primitive_1(variable)]

[0115] …

[0116] k.)[reusable_work primitive_k(variable)]

[0117] Where [variable] should be replaced with the appropriate object at [position_1]…[position_j]. Here is a list of the above actions I should perform to accomplish my purpose:".

[0118] In the above template, the elements in square brackets can be filled in by at least one processor inserting appropriate NL descriptions. In particular, object_array_1 represents at least one array of objects at position_1 (e.g., an array of chess pieces on a table). In this example, at least one processor can replace the text [object_array_1] with the NL description of the chess pieces and the text [position_1] with the NL description of the table. Furthermore, in this example, object_array_2 represents a person standing at position_2 relative to the robot body. In this example, at least one processor can replace the text [object_array_2] with the NL description of the person and the text [position_2] with the NL description of the person's position. Furthermore, from the text [(,)(and)], at least one processor can select either "," or "and" to connect the text about object_array_1 and object_array_2 in a natural way (based on whether object_array_3 exists). In this example, there are no additional object arrays (no object_array_3 or object_array_j), so at least one processor chooses to connect the text "and". Additionally, because there is no additional object array, at least one processor deletes or ignores (eg, replaces with no text) the text "[object_array_j]" at [position_j].

[0119] As a result of the above steps, a first sentence of an NL query generated based on the template may be "There are white chess pawn 1, white chess pawn 2, black chess pawn 1, black chess pawn 2, white chess rook, black chess bishop, and 2 blue boxes at the table, and a person is standing opposite me." This is similar to the exemplary prompt described above, but the chess pieces are described as "at the table" instead of "on the table." To improve the generation of NL queries, the template may include options for transition words or positioning words (such as "on..." or "at...") so that at least one processor can select the most natural words for a given scenario.

[0120] Returning to the template described above, at least one processor can replace the text [work_objective] with an NL description of the work objective of the robotic system. In this example, at least one processor can replace the text [work_objective] so that the second statement of the NL query is "My objective is to do what this person says. This person says 'Put all the white pieces in the right box and all the black pieces in the left box'" similar to the prompt in the example discussed previously.

[0121] In addition, at least one processor may replace the text of the available instructions "1.) [reusable_work_primitive_1 (variable)] ... k.) [reusable_work_primitive_k (variable)]" with the NL description of each available reusable work primitive. In an example scenario, the text for the available instructions may be replaced with "1.) Grab object," "2.) Place object on object," "3.) Look at object," "4.) Slice object," "5.) Say hello to that person," and "6.) Place object in object," as shown in the exemplary prompts discussed previously.

[0122] In the example, the word "variable" for each reusable work primitive is replaced with the appropriate text for "object" or "person," depending on what the given reusable work primitive applies to. Furthermore, the text [variable] and [position_1]...[position_j] in the penultimate statement of the template are also replaced with the appropriate text for "object," along with the object's associated position. In this regard, the penultimate statement of the generated NL query reads "the position where the object should be replaced by the appropriate object on the table," as in the example prompt presented previously.

[0123] In view of the above, by replacing or importing the selection elements in the pre-generated template, an NL query suitable for providing to the LLM module is generated.

[0124] While the NL templates are described above as text in which certain elements are "replaced" or "entered," this is not strictly necessary in terms of implementation. For example, instead of literally "replacing" text, the NL template can also be implemented as a set of instructions or functions (e.g., a program or script) that stitches together the basic statement with the relevant elements in a segmented manner. In such an example, the NL query is "assembled" into segments, rather than elements that are literally "replaced." In this sense, the NL templates presented are intended to be a logical representation of how elements can be stitched together, rather than a strict process for how text generation actually occurs.

[0125] Return to Figure 2 In the method 200, at 208, the robotic system receives the mission plan from the LLM module. That is, after providing the NL query to the large LLM module at 206, the LLM module generates a mission plan (expressed in NL) and provides the generated mission plan to the robotic system.

[0126] The robotic system executes the mission plan at 210. For example, at least one processor of the robotic controller may cause at least one element (eg, an actuatable element) to perform any actions specified in the mission plan.

[0127] As described above, the task plan provided by the LLM module is expressed in NL. For example, the task plan may indicate at least one action expressed in NL that the robotic system can perform. In order for the robotic system to execute the action plan, at least one processor may first generate a robotic language task plan based on the task plan expressed in NL. The robotic language task plan may include a set of robotic control instructions that, when executed by at least one processor, cause the robotic system to perform at least one action indicated in the task plan. For example, the set of robotic control instructions may include a library or collection of at least one reusable work primitives that can be executed by the robotic system. In addition, at least one action indicated in the task plan expressed in NL may include an NL description of a specific reusable work primitive (e.g., grab chess pawn 1), while the robotic control instructions in the robotic language task plan may include the actions of the NL task plan, but specified in a language format usable by the robotic system (e.g., grab(chess_pawn_54677)).

[0128] Similar to the above description, generating a robot language task plan can include executing a robot language conversion module that converts at least one action expressed in NL and executable by the robot system into at least one reusable work primitive in an instruction set executable by the robot system. Referring to the example in which the NL task plan includes an action expressed in NL as "grab chess pawn 1", the robot language conversion module can match the text string in the action expressed in NL with a text string available in a reusable work primitive usable by the robot system (e.g., grasp(object)) or an object in the environment with which the robot system can interact (e.g., chess_pawn_54677). Thus, at least one processor can generate a robot language action such as grab(chess_pawn_54677).

[0129] In some embodiments, the LLM module can be used to autonomously troubleshoot mission plans. For example, if a given mission plan fails to execute (i.e., fails to be verified, fails to proceed to completion, and / or fails to complete the intended task) or encounters an error, an NL prompt can be sent (returned) to the LLM module, which includes all the successful parts of the mission plan that have been executed or verified, describes the failed parts with additional wording, and asks the LLM module what to do next. In addition, an external checker can review or verify the proposed plan and reject it for some reason. As a non-limiting example, the external checker can be a logic-based system or reasoning engine, such as the one from Cycorp, Inc. Machine reasoning AI platform. The reasoning engine (sometimes referred to as an inference engine) can utilize logical rules, statements, terms, knowledge fragments or similar libraries, and can draw logical conclusions based on these contents. In this way, the mission plan mentioned in method 200 can be verified by the reasoning engine by comparing the mission plan with a set of rules (or similar rules) at least partially specified by the reasoning engine. That is, at least a portion of the logic of the reasoning engine can be applied to the mission plan to verify whether the mission plan makes logical sense and / or identify any logical inconsistencies or impossibilities in the mission plan. For example, the reason for rejection may be a safety violation related to the safety of the robot or the safety of any human or other creature. In the event of rejection, the NL prompt can be sent back to the LLM module and modified to prevent the plan from failing external inspection.

[0130] In some embodiments, the LLM can help autonomously assign parameters or definitions to generalized and / or parameterized objects in a robot control system. For example, parameterized work primitives or "instructions" can be assigned by the LLM, as in the case of the chess set example above. As another example, if a task plan is successfully executed, the successful task plan can be stored and then re-parameterized to become generalized. When the robot encounters a future instance of a similar task, it can call the stored successful task plan and ask the LLM module (e.g., via a simple NL prompt) to replace the parameterized objects of the previous successful instance of the task plan with new objects specific to the current instance of the task plan. For example, if a plan was generated to successfully sort two types of specific objects, the robot can reuse the plan by asking the LLM to replace these objects with different objects.

[0131] Various embodiments of the systems, methods, control modules, and computer program products of the present application relate to controlling the functions and operations of a robot using NL expressions (descriptions) (e.g., via NL prompts, which may be entered directly in text form by a user, or may be spoken by a user and converted to text by an intervening speech-to-text system), wherein an LLM module may provide an interface between the NL expressions and the robot control system. This framework may be particularly advantageous when certain elements of the robot control architecture employ programming and / or instructions that may be expressed in NL. A suitable, but non-limiting, example is the instruction set described above. For example, as previously described, the task plan output of the LLM module may be parsed by looking for words that match the instruction set commands (e.g., autonomously by the robot control system), and the actual parameters of the instruction set may be found by string matching within the input NL prompt (e.g., by a text string matching module as previously described). In some embodiments, a 1-1 mapping may be generated between the actual parameters and the NL variants used in the robot control system to increase the chances of the LLM module processing the text correctly. For example, even if an object is represented in the robot control system (e.g., in the world model environment portion of the robot control system) as chess_pawn_54677, it may also be referred to as "chessspawn1" in the NL prompt. In this case, if the returned task plan contains the phrase "grasp chesspawn1," this may match the instruction set "grasp" and the object "chess pawn 1," and thus the phrase may be mapped to grab(chess_pawn_54677). This type of parsing and / or word matching (e.g., a text string matching module) may be used in any of the situations discussed herein where a robot language is being converted to a natural language, or vice versa.

[0132] In some embodiments, the robot control system can generate and / or employ a scene graph that describes the robot's environment, and can apply a function to act on the scene graph and create an NL hint or description that describes the scene from the robot's perspective (e.g., in the context of action 204 of method 200). This automatically generated NL hint or description can then be used as input to the LLM module to facilitate various operations, such as reasoning, fact checking, and task planning.

[0133] In some embodiments, the quality of the mission plan may depend at least in part on the robot's understanding of its environment, so the robot control system may periodically check and compare its scene graph and internal world model in the background. According to the systems, methods, control modules, and computer program products of the present application, such checking and comparing the scene graph (e.g., actual data from the robot's external environment) and the internal world model (e.g., the robot's simulation of its external environment) may be accomplished by automatically generating NL hints or descriptions for each and feeding these NL hints or descriptions through the LLM module.

[0134] In some embodiments, an LLM module acting as a mission planner may be frequently invoked by the robot control system to answer the question "What can I (the robot) do here / now?" For example, the robot control may automatically generate an NL description of at least one aspect of its environment (scene graph) and capabilities (instruction set), and feed these NL descriptions into the LLM module along with the following query: What can I do?: or "What should I do?" (or similar variations, such as "What is the most useful thing for me to do?", "What are the productive things I can do?", etc.). A set of answers to this or similar questions may then each be generated by generating a mission plan (e.g., as described above with reference to Figure 2 Each of these mission plans can be checked against some constraints or principles (e.g., validated by an inference engine), and the tasks that are passed are now a set of things that the robotic system can do autonomously without being asked. This type of behavior is a form of autonomous agency because the embodied artificial general intelligence (AGI) utilized by the robotic system creates its own grounded task plans and then executes them. In some embodiments, the robotic system presents an output in NL that describes what it plans to do to request permission from a person (or driver / supervisor) to execute the plan. A separate inference system, such as to explain each step in the plan. Plans can be provided at any desired level of abstraction, where the highest level is likely to be the most useful descriptor for a human supervisor to determine whether the plan should be executed. The LLM module can also be used to check whether a plan passes a certain criterion, for example by issuing an NL hint asking whether the top-level plan is compatible with a set of desired constraints.

[0135] Some mission plans may contain steps that cannot be parsed into instruction set elements and are computational in nature. For example, a mission plan may require the calculation of an integral or some other computational process that may not be possible given a particular instruction set. In these cases, the robotic system can send these mission plan steps to an LLM-based system or LLM module, which requests the generation of a piece of code, such as a python script, that generates the functions used to perform the task. In some embodiments, this script can then reside in a "code library" where a human engineer can review all the automatically generated scripts generated by the background "what can I do here?" process and check that they do what is expected. Such scripts generated by an LLM-based device or module can provide new instruction set elements that can be called to "unlock" a mission plan that is blocked due to a lack of access to the appropriate instructions, or can otherwise be accessed by the robotic system for incorporation and use in a mission plan.

[0136] In some embodiments, the LLM module can be stored and executed outside the robot (e.g., in the cloud) and called or accessed by the robot system (e.g., Figure 3 Specifically, Figure 3 is a schematic diagram of a robot body 300 accessing an LLM module 320 via a cloud 310 (such that the LLM module is separate from the robot system including the robot body 300). In other embodiments, the LLM module can be stored and executed locally on the robot system as an integral part of the robot's control system (e.g., Figure 4 Specifically, Figure 4 is a schematic diagram of a robot body 400 having an LLM module 420 locally at a non-transitory processor-readable storage medium of the robot body 400. Both embodiments are referred to herein as "LLM-accessible robots."

[0137] Various embodiments described herein include systems, methods, control modules, and computer program products for utilizing one or more LLMs in a robotic control system, including, for example, establishing an NL interface between the LLM and the robotic control system and invoking the LLM to help autonomously instruct the robot what to do. Example applications of this approach include task planning, motion planning, reasoning about the robot's environment (e.g., "What can I do now?"), and the like. Such embodiments are particularly well-suited for robotic control systems in which at least some control parameters and / or instructions (e.g., the instruction sets previously described) are specified in an NL. Thus, some embodiments may include converting or translating the robot control instructions and / or parameters into NL for such communication with the LLM via the NL interface.

[0138] Figure 5 is an illustrative diagram of an exemplary robotic system 500 that includes various features and components described throughout the systems, robots, methods, control modules, and computer program products of this application. For example, the robotic system 501 can perform method 200 and associated actions as previously described (and not repeated for the sake of brevity). The robotic system 500 includes a robotic body 501 having a first physically actuatable component 502a and a second physically actuatable component 502b mechanically coupled to the body 1001. In the illustrated embodiment, the first physically actuatable component 502a and the second physically actuatable component 502b each correspond to a respective robotic hand, although those skilled in the art will appreciate that in alternative embodiments, the physically actuatable components can take other forms (e.g., arms or legs, non-hand-like end effectors (e.g., cutters or suction tubes), or any other form useful for the particular application the robot is intended to perform). The robotic hand 502a mimics a human hand and includes a plurality of fingers 521a, 522a, 523a, and 524a and an opposable thumb 525a. Robotic hand 502b is similar to a mirror image of robotic hand 502a, and corresponding details are not labeled for robotic hand 502b to reduce clutter. Robotic hands 502a and 502b can be physically actuated in a variety of different ways (including electromechanical actuation, cable-driven actuation, magnetorheological fluid-based actuation, and / or hydraulic actuation). Some exemplary details of actuation techniques that can be used to physically actuate robotic hands 502a and 502b are described in U.S. patent application serial number 17 / 491,577 and U.S. patent application serial number 17 / 749,536, which are incorporated herein by reference in their entireties.

[0139] The robot body 501 also includes at least one sensor 503 that detects and / or collects data about the environment and / or objects in the environment of the robot system 500 (e.g., including people, such as customers). In the illustrated embodiment, the sensor 503 corresponds to a sensor system including a camera, a microphone, and an initial measurement unit, which itself includes three orthogonal accelerometers, a magnetometer, and a compass. However, any suitable sensor may be included or excluded from the at least one sensor 503 as appropriate for a given application. For example, sensor data such as that captured in act 202 of method 200 may be captured by the sensor 503.

[0140] For illustrative purposes, Figure 5Details of certain exemplary components carried by or within a robot body 501, including systems, robots, methods, control modules, and computer program products according to the present application. Such components include at least one processor 530 and at least one non-transitory processor-readable storage medium or "memory" 540 communicatively coupled to the processor 530. The memory 540 stores data 541 and processor-executable instructions 542 (e.g., collectively referred to as a robot control module or computer program product) that, when executed by the processor 530, cause the robot body 501 (including applicable actuatable components, such as either or both of the robot hands 502a and / or 502b) to perform actions and / or functions associated with the systems, methods, control modules, and computer program products of the present application. The at least one processor 530 and the at least one non-transitory processor-readable storage medium 540 together can be considered a robot controller.

[0141] In some embodiments, an action or process can be performed entirely locally at the robot body 501. For example, in some embodiments, the entire method 200 can be performed locally at the robot body 501. In such an embodiment, at least one sensor 503 captures sensor data in action 202, and at least one processor 530 generates an NL description of at least one aspect of the environment in action 204. As previously discussed, the at least one processor 530 can further generate any other NL descriptions included in the NL query. In addition, in such an embodiment, the memory 540 also stores an LLM module to which the NL query is provided in action 206. Providing the NL query in this case can refer to the at least one processor 530 executing the LLM module with the NL query as input. Furthermore, receiving the task plan from the NL as in action 208 of method 200 can include the at least one processor 530 receiving the task plan output by the LLM module. Executing the task plan as in action 210 can include the at least one processor 530 executing instructions that cause the robot body 501 to perform the actions specified in the task plan.

[0142] In some embodiments, the actions or processes may be performed locally at the robot body 501 or separately by a device separate from the robot body 501. In this regard, the at least one processor 530 is also communicatively coupled to a wireless transceiver 550 via which the robot body 501 sends and receives wireless communication signals 570.

[0143] Figure 5Also included is a standalone device 580, which is part of the robot system 500 but physically separate from the robot body 501. As non-limiting examples, the standalone device 580 can be a processing unit located in close proximity (e.g., in the same room) to the robot body 501, or the standalone device 580 can be a remote server located remote from the robot body 501. The standalone device 580 includes a wireless transceiver 581 via which the standalone device 580 transmits and receives wireless communication signals 570. The wireless transceiver 550 and the wireless transceiver 581 can be referred to as communication interfaces (together or individually) through which the robot body 501 and the standalone device 580 communicate. Furthermore, the transceivers 550 and 581 do not need to communicate directly (although they can); for example, the transceivers 550 and 581 can communicate with each other via a network or the Internet. Furthermore, the transceivers 550 and 581 do not need to be wireless; in some embodiments, either transceiver can be replaced with a wired communication interface. In some embodiments where the LLM module is not stored in the memory 540 , the robot body 501 can access the LLM module via the wireless signal 570 through the transceivers 550 and 581 .

[0144] In particular, the standalone device 580 is further shown as including at least one processor 582 communicatively coupled to the wireless transceiver 581, and at least one non-transitory processor-readable storage medium 590 (or "memory" 590) communicatively coupled to the at least one processor 582. The memory 590 stores data 591 and processor-executable instructions 592 (e.g., together as a robot control module or computer program product) that, when executed by the processor 582, cause the standalone device 580 (or components thereof) to perform actions and / or functions associated with the systems, robots, methods, robot control modules, and computer program products of the present application. The memory 590 may also store an LLM module. Alternatively, the standalone device 580 may access an LLM module stored at another device (e.g., a cloud-based or internet-based LLM module).

[0145] The methods or processes discussed herein (e.g., Figure 2The method 200 in FIG200 can be performed by a combination of a robot body 501 and a stand-alone device 580. In an exemplary embodiment, at least one sensor 503 captures sensor data in action 202, and at least one processor 530 generates an NL description of at least one aspect of the environment in action 204. As previously discussed, the at least one processor 530 can further generate any other NL descriptions included in the NL query. In this exemplary embodiment, the memory 590 of the stand-alone device 580 stores an LLM module, to which the NL query is provided in action 206. In this example, providing the NL query refers to the robot body 501 transmitting the NL query to the stand-alone device via transceivers 550 and 581 (communication interface). The at least one processor 582 then executes the LLM module with the NL query as input. In addition, receiving the task plan from the NL in action 208 of method 200 includes the robot body 501 receiving the task plan output by the LLM module, which is transmitted from the stand-alone device 580 by the transceivers 550 and 581 to be received by the robot controller (or at least one processor 530). Executing the mission plan as in act 210 includes at least one processor 530 executing instructions that cause the robotic agent 501 to perform the actions specified in the mission plan.

[0146] Throughout this specification and the appended claims, the term "communicative" (as in "communicative coupling" and in variations such as "communicatively coupled") is generally used to refer to any engineering arrangement for transmitting and / or exchanging information. For example, communicative coupling can be achieved through a variety of different media and / or forms of communication paths, including but not limited to: conductive paths (e.g., conductive wires, conductive traces), magnetic paths (e.g., magnetic media), wireless signal transmission (e.g., radio frequency antennas), and / or optical paths (e.g., optical fibers). Exemplary communicative couplings include but are not limited to: electrical coupling, magnetic coupling, radio frequency coupling, and / or optical coupling.

[0147] Throughout this specification and the appended claims, infinitive verb forms are frequently used. Examples include, but are not limited to, "encode," "provide," "store," and the like. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, i.e., as in "to at least encode," "to at least provide," "to at least store," and the like.

[0148] This specification, including the drawings and abstract, is not intended to be an exhaustive or limiting description of all implementations and embodiments of the systems, methods, control modules, and computer program products of the present application. Those skilled in the art will appreciate that the various descriptions and drawings provided may be modified without departing from the spirit and scope of the present disclosure. In particular, the teachings herein are not intended to be limited to or restricted by the illustrative examples of computer systems and computing environments provided.

[0149] This specification provides various implementations and examples in the form of block diagrams, schematic diagrams, flow charts and examples. It will be understood by those skilled in the art that any function and / or operation in such block diagrams, schematic diagrams, flow charts or examples can be implemented individually and / or collectively by a wide range of hardware, software and / or firmware. For example, the various embodiments disclosed herein in whole or in part can be equivalently implemented in one or more of the following: an application-specific integrated circuit (i.e., an ASIC); a standard integrated circuit; a computer program executed by any number of computers (e.g., a program running on any number of computer systems); a program executed by any number of controllers (e.g., a microcontroller); and / or a program executed by any number of processors (e.g., a microprocessor, a central processing unit, a graphics processing unit), and any combination of firmware and the foregoing.

[0150] Throughout this specification and the appended claims, a “memory” or “storage medium” is a processor-readable medium, which is an electronic, magnetic, optical, electromagnetic, infrared, semiconductor, or other physical device or apparatus that contains or stores processor data, data objects, logic, instructions, and / or programs. When data, data objects, logic, instructions, and / or programs are implemented as software and stored in a memory or storage medium, they may be stored in any suitable processor-readable medium for use by any suitable processor-related instruction execution system, apparatus, or device, such as a computer-based system, a system containing a processor, or other system capable of retrieving data, data objects, logic, instructions, and / or programs from a memory or storage medium and performing various actions or operations (i.e., processing steps) thereon and / or in response thereto. Thus, a “non-transitory processor-readable storage medium” may be any element that stores data, data objects, logic, instructions, and / or programs for use by or in conjunction with an instruction execution system, apparatus, and / or device. As specific non-limiting examples, the processor readable medium may be: a portable computer floppy disk (magnetic, compact flash card, secure digital device, etc.), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, EEPROM or flash memory), a portable compact disk read-only memory (CDROM), digital magnetic tape, and / or any other non-transitory medium.

[0151] The claims of the present disclosure are appended hereto. This disclosure is intended to support, enable, and illustrate the claims, but is not intended to limit the scope of the claims to any specific implementation or example. In general, the claims should be interpreted to include all possible implementations and examples, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A method of operating a robotic system including a robotic body, the method comprising: capturing, by at least one sensor of the robotic system, sensor data representing information about an environment of the robotic body; generating, by at least one processor of the robotic system, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing an NL query to a large language model (LLM) module, the NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receiving the mission plan from the LLM module, wherein the mission plan is expressed in NL; as well as The mission plan is executed by the robotic system.

2. The method according to claim 1, wherein: The mission plan indicates at least one action that the robotic system is capable of performing, expressed in a natural language; and The method also includes generating, by the at least one processor, a robot language task plan based on the task plan expressed in NL, the robot language task plan including a set of robot control instructions that, when executed by the at least one processor, cause the robot system to perform the at least one action indicated in the task plan.

3. The method according to claim 2, wherein: Generating the robot language task plan includes executing a robot language conversion module, which converts the at least one action executable by the robot system expressed in a natural language into at least one reusable work primitive in the instruction set executable by the robot system.

4. The method according to claim 1, wherein Generating the NL description of the environment includes: identifying objects or features in the environment using robot language text labels; and A text string matching module is executed, which matches text in the robot language text tag with natural language vocabulary.

5. The method of claim 1 , further comprising generating, by the at least one processor, an NL description of the instruction set, wherein generating the NL description of the instruction set comprises executing a robot language conversion module that generates an NL description of each instruction in the instruction set expressed in a robot language that can be executed by the robot system.

6. The method according to claim 5, wherein: The robot language conversion module includes a text string matching module operable to compare robot language instructions in the instruction set with a natural language vocabulary representing actions executable by the robotic system and identify matching text strings for inclusion in the NL description of the instruction set. 7 . The method of claim 1 , further comprising generating, by the at least one processor, a NL request for a mission plan based on a request template.

8. The method of claim 1 , further comprising accessing a pre-generated NL template from at least one non-transitory processor-readable storage medium, the pre-generated NL template comprising at least one of an NL description of a work objective, an NL description of the instruction set, or the NL request for the task plan.

9. The method of claim 1, further comprising validating the mission plan prior to executing the mission plan by comparing the mission plan to a set of rules specified in at least a portion of an inference engine.

10. A robotic control module comprising at least one non-transitory processor-readable storage medium storing processor-executable instructions or data that, when executed by at least one processor of a processor-based system, cause the processor-based system to: capturing, by at least one sensor carried by a robotic body of the processor-based system, sensor data representing information about an environment of the robotic body; generating, by the at least one processor, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing an NL query to a large language model (LLM) module, the NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the processor-based system, and an NL request for a task plan; receiving the mission plan from the LLM module, wherein the mission plan is expressed in NL; and The mission plan is executed by the processor-based system.

11. The robot control module according to claim 10, wherein: The mission plan indicates at least one action, expressed in a natural language, that can be performed by the processor-based system; The processor-executable instructions or data further cause the at least one processor to generate a robot language task plan based on the task plan expressed in NL, the robot language task plan comprising a set of robot control instructions that, when executed by the at least one processor, cause the processor-based system to perform the at least one action indicated in the task plan; and The processor-executable instructions or data that cause the at least one processor to generate the robot language task plan cause the at least one processor to execute a robot language conversion module that converts the at least one action executable by the processor-based system expressed in a natural language into at least one reusable work primitive in an instruction set executable by the processor-based system.

12. The robot control module according to claim 10, wherein: The processor-executable instructions or data causing the at least one processor to generate the NL description of the environment cause the at least one processor to: identifying objects or features in the environment using robot language text labels; and A text string matching module is executed, which matches text in the robot language text tag with natural language vocabulary.

13. The robot control module according to claim 10, wherein: The processor-executable instructions or data further cause the processor-based system to generate, by the at least one processor, the NL description of the instruction set; the processor-executable instructions or data causing the processor-based system to generate the NL description of the instruction set causes the at least one processor to execute a robot language conversion module that generates an NL description of each instruction in the instruction set expressed in a robot language that is executable by the processor-based system; and The robot language conversion module includes a text string matching module operable to compare robot language instructions in the instruction set with a natural language vocabulary representing actions executable by the processor-based system and identify matching text strings for inclusion in the NL description of the instruction set.

14. The robot control module according to claim 10, wherein: The processor-executable instructions or data further cause the at least one processor to generate the NL request for a mission plan based on a request template.

15. The robot control module according to claim 10, wherein: The processor-executable instructions or data also cause the processor-based system to access a pre-generated NL template from at least one non-transitory processor-readable storage medium, wherein the pre-generated NL template includes at least one of an NL description of a work objective, an NL description of the instruction set, or the NL request for the task plan.

16. The robot control module according to claim 10, wherein: The processor-executable instructions or data further cause the processor-based system to validate the mission plan prior to executing the mission plan by comparing the mission plan to a set of rules specified in at least a portion of an inference engine.

17. A robotic system comprising: Robot body; at least one sensor; a robotic controller comprising at least one processor and at least one non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the at least one processor, cause the robotic system to: capturing, by the at least one sensor, sensor data representing information about an environment of the robotic body; generating, by the at least one processor, a natural language (NL) description of at least one aspect of the environment based on the sensor data; providing an NL query to a large language model (LLM) module, the NL query comprising an NL description of at least one aspect of the environment, an NL description of a work goal, an NL description of an instruction set executable by the robotic system, and an NL request for a task plan; receiving the mission plan from the LLM module, wherein the mission plan is expressed in NL; as well as Execute the task plan.

18. The robotic system of claim 17, wherein: The LLM module is stored at the at least one non-transitory processor-readable storage medium of the robotic controller; and The processor-executable instructions that cause the robotic system to provide the NL query to the LLM module cause the robotic system to execute the LLM module with the NL query as input to the LLM module.

19. The robotic system of claim 17, wherein: The LLM module is stored at a separate device from the robotic system; The robotic system includes at least one communication interface; the processor-executable instructions that cause the robotic system to provide the NL query to the LLM module cause the at least one communication interface to transmit the NL query to the standalone device; and The processor-executable instructions that cause the robotic system to receive the mission plan from the LLM module cause the at least one communication interface to receive the mission plan from the standalone device and provide the mission plan to the robotic controller.

20. The robotic system of claim 17, wherein: The mission plan indicates at least one action that the robotic system can perform, expressed in a natural language; and The processor-executable instructions further cause the robotic system to generate, by the at least one processor, a robotic language task plan based on the task plan expressed in NL, the robotic language task plan comprising a set of robotic control instructions that, when executed by the at least one processor, cause the robotic system to perform the at least one action indicated in the task plan.

Images