Robotic system, method, control module and computer program product utilizing large language models

Abstract

Robotic control systems, methods, control modules, and computer program products are described that utilize one or more large language models (LLMs) in order to enable at least some degree of autonomy. Robotic control parameters, environmental details, and / or instructions can advantageously be specified in natural language (NL) and communicated with the LLM via NL prompts or queries. NL queries can include requests for one or more work objectives from the LLM, such as "What can I do here?", thereby establishing a form of agency through which the robotic system can identify activities to perform without operator intervention. The LLM can also be queried to convert each work objective into a task plan that provides a series of steps that the robotic system can perform to complete the work objective. Optionally, the robotic system can communicate with an operator to determine whether to perform the task plan.

Description

[0001] Previous application data This application claims priority to U.S. Provisional Patent Application No. 63 / 441,897, filed January 30, 2023, entitled “Robot Control Systems, Methods, and Computer Program Products That Leverage Large Language Models,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The systems, methods, control modules, and computer program products of this application generally relate to robot control, and more specifically to the deployment, utilization, and / or general use of large language models under the control of robots.

[0003] background Description of related technologies A robot is a machine that can be used to perform tasks. Robots can have various different shape factors (including humanoid shape factors). Humanoid robots can be operated by a teleoperation system, which enables the robot to simulate the physical actions of a human operator or driver. Specialized robots can be designed to perform specific tasks, while general-purpose robots can be designed to perform multiple tasks.

[0004] Humans perform numerous tasks in their personal and professional lives. Examples of these tasks include making beds, washing dishes, loading dishes into a dishwasher, mowing lawns, taking inventory, checking out customers, stocking shelves, painting, styling hair, preparing meals, cleaning, measuring, performing calculations, recording data, performing analyses, creating art / music, performing art / music, building, manufacturing, assembling, destroying, disassembling, moving, picking and placing, navigating, and many more. In many cases, there is a strong desire and a persistent need to automate various tasks so that humans can divert their time and / or attention to other things.

[0005] Large Language Models (LLMs) are a form of artificial intelligence that are trained on large corpora of text data to produce human-like textual responses to natural language (NL) input. Popular examples in the field today include various incarnations of OpenAI™'s Generative Pre-trained Transformers (GPTs), such as text-davinci-003, text-curie-001, text-babbage-001, and text-ada-001. LLMs can be accessed through or deployed within text-based user interfaces to allow chat-like interactions between users and computers, such as the ChatGPT™ application built on OpenAI™'s LLM-based GPT-3™ series.

[0006] Brief Overview An operating method for a robot system including a robot body can be summarized as including: capturing sensor data representing information about the environment of the robot body via at least one sensor of the robot system; generating a natural language (NL) description of at least one aspect of the environment based on the sensor data via at least one processor of the robot system; providing a first NL query to a large language model (LLM) module, the first NL query including the NL description of at least one aspect of the environment, the NL description of a set of instructions executable by the robot system, and the NL request for at least one work objective; and receiving at least a first work objective from the LLM module, the first work objective being expressed in NL.

[0007] The method may also include completing a first work objective through the robot system, regardless of whether a request is sent to the operator of the robot system to confirm that the robot system should complete the first work objective.

[0008] The method may further include providing a second NL query to the LLM module, the second NL query including an NL description of at least one aspect of the environment, a first work objective expressed in NL, an NL description of a set of instructions executable by the robot system, and an NL request for a first task plan to complete the first work objective; and receiving a first task plan from the LLM module, the first task plan expressed in NL. The method may further include executing the first task plan by the robot system, regardless of whether a request is sent to the operator of the robot system to confirm that the robot system should execute the first task plan. The method may include verifying the first task plan to determine whether the first task plan violates any constraints in a set of constraints.

[0009] Receiving at least a first work objective from the LLM module may include receiving multiple work objectives from the LLM module, and the method may further include sending a request to the operator of the robot system to confirm which of the multiple work objectives the robot system should complete.

[0010] Receiving at least a first work objective from the LLM module may include receiving a plurality of work objectives from the LLM module, each corresponding work objective being expressed in NL, and the method may further include: for each of at least two of the plurality of work objectives: providing a corresponding NL query to the LLM module, the corresponding NL query including an NL description of at least one aspect of the environment, the work objective expressed in NL, an NL description of a set of instructions executable by the robot system, and a corresponding NL request for a corresponding task plan for completing the work objective; and receiving a corresponding task plan from the LLM module, the corresponding task plan being expressed in NL. In some embodiments, the method may further include: selecting which corresponding task plan to execute first; and executing the selected task plan. In some embodiments, the method may further include: sending a request to the operator of the robot system to determine which corresponding task plan the robot system should execute first; receiving a selection of the corresponding task plan to be executed first by the robot system; and executing the selected task plan.

[0011] A robot control module can be summarized as including at least one non-transitory processor-readable storage medium storing processor-executable instructions or data, which, when executed by at least one processor of a robot system, cause the robot system to: capture sensor data representing information about the robot body's environment via at least one sensor carried by the robot body; generate a natural language (NL) description of at least one aspect of the environment based on the sensor data by the at least one processor; provide a first NL query to a large language model (LLM) module, the first NL query including the NL description of at least one aspect of the environment, the NL description of a set of instructions executable by the robot system, and the NL request for at least one work objective; and receive at least a first work objective from the LLM module, the first work objective being expressed in NL. The processor-executable instructions or data, when executed by at least one processor of the robot system, can also cause the robot system to complete the first work objective. The processor-executable instructions or data, when executed by at least one processor of the robot system, can also cause the robot system to send a request to the operator of the robot system to confirm that the robot system should complete the first work objective.

[0012] When executed by at least one processor of the robot system, the processor-executable instructions or data may also cause the robot system to: provide a second NL query to the LLM module, the second NL query including an NL description of at least one aspect of the environment, a first work objective expressed in NL, an NL description of a set of instructions executable by the robot system, and an NL request for a first task plan to complete the first work objective; and receive a first task plan from the LLM module, the first task plan expressed in NL. The processor-executable instructions or data, when executed by at least one processor of the robot system, may also cause the robot system to execute the first task plan.

[0013] When executed by at least one processor of the robot system, the processor-executable instructions or data can also cause the robot system to send a request to the operator of the robot system to confirm that the robot system should perform a first planned task. When executed by at least one processor of the robot system, the processor-executable instructions or data can also cause the robot system to receive at least a first work objective from the LLM module, and further cause the robot system to receive multiple work objectives from the LLM module, each corresponding work objective being expressed in NL. When executed by at least one processor of the robot system, the processor-executable instructions and / or data can also cause the robot system to: provide a corresponding NL query to the LLM module for each of at least two of the multiple work objectives, the corresponding NL query including an NL description of at least one aspect of the environment, the work objective expressed in NL, an NL description of the set of instructions executable by the robot system, and a corresponding NL request for a corresponding task plan for completing the work objective; and receive a corresponding task plan from the LLM module, the corresponding task plan being expressed in NL. When executed by at least one processor of the robot system, the processor-executable instructions or data can also cause the robot system to: select which corresponding task plan to execute first; and execute the selected task plan. When executed by at least one processor of the robot system, the processor-executable instructions or data can also enable the robot system to: send a request to the operator of the robot system to determine which corresponding task plan the robot system should execute first; receive a selection of the corresponding task plan to be executed first by the robot system; and execute the selected task plan.

[0014] Brief description of several attached views The various elements and actions depicted in the accompanying drawings are provided for illustrative purposes to support detailed description. Unless the specific context requires otherwise, the size, shape, and relative position of the elements and actions shown are not necessarily displayed to scale and are not necessarily intended to convey any information or limitation. Generally, the same reference numerals are used to identify similar elements or actions.

[0015] Figure 1This is a flowchart illustrating an exemplary implementation of an automated task planner utilizing an LLM, comprising a system, control module, method, and computer program product according to this application.

[0016] Figure 2 This is a flowchart illustrating an exemplary implementation of the operation of a robot system utilizing LLM, comprising a system, control module, method, and computer program product according to this application.

[0017] Figure 3 This is a flowchart illustrating an exemplary operation method of a robot system performing a work objective according to the system, control module, method, and computer program product of this application.

[0018] Figure 4 This is a flowchart illustrating another exemplary method of operation for a robot system to perform a work objective according to the system, control module, method, and computer program product of this application.

[0019] Figure 5 This is an illustrative diagram illustrating an exemplary implementation of a robot with access to an LLM, based on the system, control module, method, and computer program product of this application, wherein the LLM is stored and executed outside the robot (e.g., in the cloud) and invoked or accessed by the robot control system.

[0020] Figure 6 This is an illustrative diagram illustrating an exemplary embodiment of a robot with access to an LLM, based on the system, control module, method, and computer program product of this application, wherein the LLM is stored and executed locally on the robot as part of the robot's control system.

[0021] Figure 7 This is an illustrative diagram of an exemplary robot system that includes various features and components described throughout the system, control module, method, and computer program product description of this application.

[0022] Detailed description The following description sets forth specific details to illustrate and provide an understanding of various implementations and embodiments of the systems, methods, control modules, and computer program products of this application. Those skilled in the art will understand that some of the specific details described herein may be omitted or modified in alternative implementations and embodiments, and that the various implementations and embodiments described herein may be combined with each other and / or with other methods, components, materials, etc., to produce further implementations and embodiments.

[0023] In some cases, well-known structures and / or processes associated with computer systems and data processing are not shown or provided in detail in order to avoid unnecessarily complicating or obscuring the description of implementation methods and embodiments.

[0024] Unless the specific context requires otherwise, throughout this specification and the appended claims, the term “comprise” and its variations (e.g., “comprises” and “comprising”) are used in an open, inclusive sense to mean “including, but not limited to”. Unless the specific context requires otherwise, throughout this specification and the appended claims, the singular forms “a,” “an,” and “the” include the plural referents. For example, references to “an embodiment” and “the embodiment” respectively include “embodiments” and “the embodiments,” and references to “an implementation” and “the implementation” respectively include “implementations” and “the implementations.” Similarly, unless the specific context expressly specifies otherwise, the term “or” is generally used in its broadest sense to mean “and / or.”

[0025] The titles and abstracts of this disclosure are provided for convenience only and are not intended to, and should not be construed as, defining the scope or meaning of the systems, methods, control modules, and computer program products of this application.

[0026] The various implementations described herein provide systems, methods, control modules, and computer program products for enhancing, facilitating, strengthening, or implementing control over one or more robotic systems using one or more LLMs. Exemplary robotic systems that can adopt the teachings of the systems, methods, control modules, and computer program products of this application include, but are not limited to, the general-purpose humanoid robot developed by Sanctuary Cognitive Systems, Inc., various aspects of which are described in the following documents: U.S. Patent Serial No. 18 / 375,943, U.S. Patent Application Serial No. 18 / 513,440, U.S. Patent Application Serial No. 18 / 417,081, U.S. Patent Application Serial No. 16 / 940,566 (Publication No. US 2021-0031383 A1), U.S. Patent Application Serial No. 17 / 023,929 (Publication No. US 2021-0090201 A1), U.S. Patent Application Serial No. 17 / 061,187 (Publication No. US 2021-0122035 A1), U.S. Patent Application Serial No. 17 / 098,716 (Publication No. US 2021-0146553A1), and U.S. Patent Application Serial No. 17 / 111,789 (Publication No. US 2021-0146553A1). US Patent Application Serial No. 17 / 158,244 (Publication No. US 2021-0234997 A1), US Provisional Patent Application Serial No. 63 / 001,755 (Publication No. US 2021-0307170) A1) and / or U.S. Provisional Patent Application Serial No. 63 / 057,461 and U.S. Provisional Patent Application Serial Nos. 63 / 151,044, 63 / 173,670, 63 / 184,268, 63 / 213,385, 63 / 232,694, 63 / 316,693, 63 / 253,591, 63 / 293,968, 63 / 293,973 and / or 63 / 278,817, each of which is incorporated herein by reference in its entirety.

[0027] In some implementations, the robot system or control module may employ a finite instruction set comprising general, reusable work primitives that can be combined (in various combinations and / or permutations) to perform a task. For example, the robot control system may store a library of reusable work primitives, each corresponding to a specific basic subtask or subaction (hereinafter referred to as the instruction set) that the robot can operate to perform autonomously. The work objective can be analyzed to determine a sequence (i.e., combination and / or permutation) of reusable work primitives that, when executed by the robot, will accomplish the work objective. The robot can execute the sequence of reusable work primitives to accomplish the work objective. In this way, the finite instruction set can be used across a wide range of industries to perform a broad variety of tasks and work objectives. This method is described in U.S. Patent Publication No. 2022-0258340, based on U.S. Patent Application Serial No. 17 / 566,589, which is incorporated herein by reference in its entirety.

[0028] To expand upon the foregoing, general-purpose robots are capable of accomplishing a variety of different work objectives. As used throughout this specification and the appended claims, the term "work objective" refers to a specific task, job, assignment, or application with a specified goal and a determinable outcome, typically (though not necessarily) intended to facilitate some economically valuable work. Work objectives exist in many aspects of business, research and development, commercial activities, and personal activities. Exemplary work objectives include, but are not limited to: cleaning a location (e.g., a bathroom) or object (e.g., a bathroom mirror), preparing meals, loading / unloading storage containers (e.g., a truck), taking inventory, collecting one or more samples, performing one or more measurements, constructing or assembling an object, destroying or dismantling an object, delivering items, receiving objects and / or data, and so on. The various embodiments described herein provide robots, systems, control modules, computer program products, and methods for operating robotic systems to perform tasks or work objectives at least semi-autonomously.

[0029] According to the robot, system, control module, computer program product and method of this application, a work objective can be deconstructed or broken down into a "workflow" comprising a set or more "work primitives", wherein successful completion of the work objective involves executing each work primitive in the workflow.

[0030] Depending on the specific implementation, the completion of the work objective can be achieved by: i) executing a corresponding set of work primitives sequentially or continuously; ii) executing a corresponding set of work primitives in parallel; or iii) executing a corresponding set of work primitives in any combination of continuous and parallel execution (e.g., sequential overlap) suitable for the work objective and / or the robot executing the work objective. Therefore, in some implementations, work primitives can be interpreted as lower-level activities, steps, or subtasks that are implemented or executed as a workflow to accomplish higher-level work objectives.

[0031] Advantageously, according to the robots, systems, control modules, computer program products, and methods of this application, a catalog of "reusable" work bases can be defined. A work base is reusable if it can be commonly invoked, executed, adopted, or applied in accomplishing multiple different work objectives. For example, a reusable work base is a work base common to corresponding workflows of multiple different work objectives. In some embodiments, a reusable work base may include at least one variable defined when or before the work base is invoked. For example, "pick up". object "It can be a reusable work primitive, where the 'picking' process can generally be performed at least semi-autonomously to facilitate multiple different work objectives, and the picking can be defined based on the specific work objective being pursued." object .

[0032] As previously stated, the various embodiments described herein provide robots, systems, control modules, computer program products, and methods in which the robot is capable of performing tasks or achieving work objectives at least semi-autonomously. Unless the specific context requires otherwise, the term "autonomous" is used throughout this specification and the appended claims to mean "not under the control of another party," while the term "semi-autonomous" is used to mean "at least partially autonomous." In other words, throughout this specification and the appended claims, the term "semi-autonomous" means "under limited control of another party," unless the specific context requires otherwise. An example of a semi-autonomous robot is one that can independently and / or automatically perform and control some of its own low-level functions (e.g., its mobility and grasping capabilities), but relies on some external control over high-level instructions (e.g., what to do and / or how to do it).

[0033] According to the robots, systems, control modules, computer program products, and methods of this application, a catalog of reusable work primitives can be defined, identified, developed, or constructed such that any given work objective across multiple different work objectives can be accomplished by executing a corresponding workflow, which includes a specific combination and / or arrangement of reusable work primitives selected from the catalog. Once such a catalog of reusable work primitives has been established, one or more robots can be trained to autonomously or automatically execute each individual reusable work primitive in the catalog, without necessarily including the context that: i) the specific reusable work primitive being trained is part of a specific workflow, and / or ii) any other reusable work primitive that may precede or follow the specific reusable work primitive being trained in the specific workflow. In this way, a semi-autonomous robot can operate autonomously or automatically to execute each individual reusable work primitive in a catalog of reusable work primitives, requiring only instructions, directives, or guidance from another party (e.g., an operator, user, or driver) when it comes to deciding which reusable work primitive(s) to execute and / or in what order. In other words, the operator, user, driver, or LLM module can provide the semi-autonomous robot system with a workflow consisting of reusable work primitives, and the semi-autonomous robot system can autonomously or automatically execute the reusable work primitives to accomplish the work objective based on the workflow. For example, a semi-autonomous humanoid robot can operate autonomously to look left when instructed to look left, autonomously open its right-hand actuator when instructed to open it, and so on, without relying on detailed low-level control of such functions by a third party. Once instructions regarding the workflow are given, such a semi-autonomous humanoid robot can autonomously complete the work objective, which details which reusable work primitives it must execute and in what order to accomplish the work objective. Furthermore, according to the robot, system, method, control module, and computer program product of this application, if the robot system is trained or otherwise configured (e.g., via consultation with an LLM module, which can be included in the robot system) to analyze work objectives and independently define the corresponding workflow itself by decomposing the work objectives into a set of reusable work primitives from a library of reusable work primitives operable by the robot system to perform autonomously, then the robot system can operate completely autonomously.

[0034] In the context of a robotic system, a reusable working primitive can correspond to a basic, low-level function that the robotic system can operate to (e.g., autonomously or automatically) perform and that the robotic system can invoke or execute to achieve something. Examples of reusable working primitives for humanoid robots include, but are not limited to: looking up, looking down, looking left, looking right, moving the right arm, moving the left arm, closing the right actuator, opening the right actuator, closing the left actuator, opening the left actuator, moving forward, turning left, turning right, moving backward, etc., as well as cognitive functions such as analysis, calculation, planning, determination, reasoning, etc.; however, those skilled in the art will recognize that: i) the foregoing list of exemplary reusable working primitives for humanoid robots is by no means exhaustive; ii) in the robots, systems, control modules, computer program products, and methods of this application, the high-level functions that the robot can operate to perform are deconstructed or broken down into a set of basic components or constituent parts, referred to throughout this specification and the appended claims as "working primitives". Unless the specific context requires otherwise, a working primitive can be interpreted as a building block for constructing higher-level robotic functions.

[0035] In some implementations, training a robotic system to autonomously execute reusable work primitives can be done in a real-world or simulated environment. Once the robot has been trained to autonomously execute a catalog of reusable work primitives, the robot's operations can be abstracted down to the level of the reusable work primitives; for example, an LLM module that prepares a task plan for the robot can do so by determining which reusable work primitives to execute and, in some implementations, in what order they are executed, and the robot can have sufficient autonomy or automation to perform a complete work objective based on such limited control instructions.

[0036] As mentioned earlier, "cleaning a bathroom mirror" is an illustrative example of a work objective that can be deconstructed into a set of work primitives to achieve the goal, and whose result is deterministic. In this case, the objective is a clean bathroom mirror, and an exemplary set of work primitives (or workflows) for completing the work objective is as follows: Those skilled in the art will understand that the exemplary workflow described above, comprising nine work units, is used as an illustrative example of a workflow that can be deployed to accomplish the task of cleaning a bathroom mirror; however, the precise definition and composition of each work unit, as well as the specific combination and / or arrangement of work units selected / executed to accomplish the task objective (i.e., the specific construction of the workflow), can vary in different implementations according to the robot, system, control module, computer program product, and method. For example, in some implementations, the aforementioned work units 3, 4, and 5 (i.e., positioning the mirror, aiming the cleaning solution at the mirror, and dispensing the cleaning solution onto the mirror) can all be combined into a single higher-level work unit, such as "spraying the cleaning solution onto the mirror," while in other implementations, these same work units can be broken down into additional lower-level work units, such as: Positioning mirror Mark the edges of the mirror Aim the cleaning solution at the first point within the boundary of the mirror and squeeze out the solution. Aim the cleaning solution at the second position within the boundary of the mirror and squeeze out the cleaning solution. etc.

[0037] Based on the above examples and descriptions, those skilled in the art will recognize that the granularity of the work primitives can vary in different embodiments of the robots, systems, control modules, computer program products, and methods of this application. Furthermore, according to the robots, systems, control modules, computer program products, and methods of this application, work primitives are advantageously "reusable" in the sense that each work primitive can be adopted, invoked, applied, or "reused" in the execution of more than one overall work objective. For example, while cleaning a bathroom mirror may involve the work primitive "grabbing cleaning solution," other work objectives may also use the "grabbing cleaning solution" work primitive, such as, for example, "cleaning a toilet," "cleaning a window," and / or "cleaning a floor." In some embodiments, work primitives can be abstracted to become more general. For example, "grabbing cleaning solution" can be abstracted as "grabbing a spray bottle" or "grabbing..." Object 1 ",in Object 1 The variable is defined as " Object 1 = spray bottle", while "positioning mirror" can be abstracted as "positioning the object to be sprayed" or simply "positioning". Object 2 ",in" Object 2 =Mirror". In such cases, the "grab spray bottle" work primitive can be used for tasks that do not involve cleaning, such as "painting walls" (where spray bottle = spray paint), "designing hairstyles" (where spray bottle = hairspray), or "preparing stir-fry meal" (where spray bottle = cooking oil spray).

[0038] Unless the specific context requires otherwise, throughout this specification and the appended claims, references to “LLM” or “LLM module” shall be construed as including one or more LLMs or one or more LLM modules, and / or one or more applications or programs that operate, access, use, or otherwise utilize at least one LLM. For example, references to interaction with an LLM or LLM module (e.g., providing input to an LLM, receiving output from an LLM, querying an LLM, soliciting an LLM, etc.) may be made through applications or interfaces that use an LLM module (e.g., chat applications that access an LLM to interpret input and formulate output, such as OpenAI™’s ChatGPT™ built on the LLM’s GPT-3™ series).

[0039] In some embodiments of the systems, methods, control modules, and computer program products of this application, LLM is used to assist in determining a sequence of reusable working primitives (hereinafter referred to as "instructions") selected from a finite library of reusable working primitives (hereinafter referred to as the "instruction set") that, when executed by the robot, will enable the robot to complete a task or enable the robot to complete a task. In some embodiments, LLM is used to help determine a "workflow." For example, a robot control system may take natural language (NL) commands as input and return a task plan formed by a sequence of permissible instructions extracted from the instruction set, the completion of which fulfills the intent of the NL input. Throughout this specification and the appended claims, unless the specific context requires otherwise, a task plan may include or consist of a workflow depending on the particular implementation. An exemplary application is "assembling" a set of chess pieces comprising sixteen white chess pieces and sixteen black chess pieces. Humans can speak or type commands to the robot, such as "Put all white pieces into the right box and all black pieces into the left box," and LLMs can support fully autonomous systems that translate this input into a sequence of permissible instructions for successful task execution. In this case, an LLM can help allow the robot to perform general tasks specified in a NL (Neutral Level). General tasks include, but are not limited to, all jobs in the current economy.

[0040] Throughout the systems, methods, control modules, and computer program products of this application, the term "natural language" refers to any language that has evolved naturally among humans, and includes, but is not limited to: English, French, Spanish, Chinese (Mandarin, Cantonese, Wu dialect, etc.), Portuguese, Japanese, Russian, Korean, Arabic, Hebrew, German, Polish, Hindi, Bengali, Italian, Punjabi, Vietnamese, Hausa, Swedish, Finnish, etc.

[0041] Figure 1 This is a flowchart illustrating an exemplary implementation of an automated task planner as method 100. LLM 101 (such as, but not limited to, OpenAI™’s text-davinci-03) is provided or otherwise combined with: i) a description of the current scene 102 generated using the robot system’s perception system, ii) a description of an instruction set 103 formed by parameterized reusable working primitives or instructions that can be executed by the robot system, and iii) a wrapper that incorporates NL phrases into prompts 104 to automatically generate a task plan 105, which can then be executed by the robot system to realize the intent of the NL input.

[0042] Although Figure 1 The scene description 102 and instruction set 103 are shown as being contained "inside" the robot (i.e., within the dashed box representing the robot), and the NL cue 104, LLM 101, and task plan 105 (and the conversion to robot control signals) are shown as being "outside" the robot (i.e., outside the dashed box representing the robot). However, this is not intended to be limiting and is done for illustrative purposes only. While it is true in some embodiments that any or all of the NL cue 104, LLM 101, and / or task plan 105 (and the conversion to robot control signals) may be stored, generated, accessed, or otherwise present outside the robot (or outside or away from the robot), in other embodiments, any or all of the NL cue 104, LLM 101, and / or task plan 105 (and the conversion to robot control signals) may be stored, generated, accessed, or otherwise present inside the robot (or on the robot).

[0043] Here is a concrete example of a prompt and response pair, obtained by running a Python script: hint: On the table are White Pawn 1, White Pawn 2, Black Pawn 1, Black Pawn 2, White Rook, Black Bishop, and two blue boxes. I am facing a person. My goal is to do as this person says. This person says, "Put all the White pieces in the box on the right and all the Black pieces in the box on the left." I can only choose strictly from the following options: 1.) Fetching objects 2.) Place the object on the object 3.) Look at the object 4.) Slice objects 5.) Greet that person. 6.) Put the object into the object The object should be replaced with the appropriate object on the table. Here is a list of the commands I should execute above to achieve my goal: response: 1. Capture White's pawn number 1 2. Place White's Pawn 1 into the box on the right. 3. Capture White's pawn number 2. 4. Place White's Pawn #2 into the box on the right. 5. Capture Black's pawn number 1. 6. Place Black's Pawn 1 into the box on the left. 7. Capture Black's pawn number 2. 8. Place Black's Pawn #2 into the box on the left. 9. Capture the white chess rook 10. Place the white chess rook into the box on the right. 11. Capture the black chess bishop. 12. Place the black bishop into the box on the left. Finish.

[0044] In the example above, the response provided by the LLM corresponds to the task plan. If the robot system executes the sequence of instructions specified in the task plan, the task specified in the NL via prompts will be successfully completed by the robot system. Throughout this disclosure, the term "motion plan" may be used instead of "task plan." In this respect, the sequence of instructions specified in the task plan (motion plan) may include instructions that cause the robot to undergo a series of movements or motions.

[0045] Figure 2 This is a flowchart illustrating an exemplary method 200 for operating a robot system. Figure 2 Method 200 is similar in at least some respects to Figure 1Method 100. Generally speaking, Figure 2 Method 200 in the document describes how it can be implemented. Figure 1 Detailed implementation of method 100. Method 200 is a robotic system (such as referenced in the image). Figure 5The discussion concerns the operation methods of the robot system 500. Generally, throughout this specification and the appended claims, the operation methods of the robot system are methods in which at least some (if not all) of various actions are performed by the robot system. For example, certain actions of the operation methods of the robot system may be performed by at least one processor or processing unit (hereinafter referred to as "processor") of the robot system on a non-transitory processor-readable storage medium communicatively coupled to the robot system (collectively referred to as the robot controller of the robot system), and in some embodiments, certain actions of the operation methods of the robot system may be performed by peripheral components of the robot system communicatively coupled to at least one processor, such as one or more physically actuated parts (e.g., arms, legs, end effectors, grippers, hands), one or more sensors (e.g., optical sensors, audio sensors, tactile sensors, haptic sensors), mobility systems (e.g., wheels, legs), communication and networking hardware (e.g., receivers, transmitters, transceivers), and so on. The non-transitory processor-readable storage medium of the robot system may store data (including, for example, at least one reusable library of working primitives and at least one associated library of senses) and / or processor-executable instructions, which, when executed by at least one processor, cause the robot system to perform a method and / or cause the at least one processor to perform those actions of the method executed by the at least one processor. The robot system may communicate with remote systems and / or remote non-transitory processor-readable storage media via communication and networking hardware communicatively coupled to at least one processor of the robot system. Therefore, unless the specific context requires otherwise, references to the non-transitory processor-readable storage medium of the robot system and the data and / or processor-executable instructions stored in the non-transitory processor-readable storage medium are not intended to limit the physical location of the non-transitory processor-readable storage medium relative to at least one processor of the robot system and the remainder of the robot hardware. In other words, unless the specific context requires otherwise, the non-transitory processor-readable storage medium of the robot system may include non-transitory processor-readable storage media located on the robot body of the robot system and / or non-transitory processor-readable storage media located remotely from the robot body. Furthermore, the operation method of a robot system, such as method 200 (or any other method discussed herein), can be implemented as a robot control module or a computer program product.Such a control module or computer program product includes processor-executable instructions or data, which, when stored on a non-transitory processor-readable storage medium of the robot system and executed by at least one processor of the robot system, causes the robot system to perform actions of the method.

[0046] return Figure 2 Method 200 includes three actions 204, 208, and 210, although those skilled in the art will understand that in alternative embodiments, certain actions may be omitted and / or additional actions may be added. Those skilled in the art will also understand that the illustrated order of actions is shown for illustrative purposes only and may be changed in alternative embodiments.

[0047] At point 202, sensor data representing information about the environment of the robot body of the robot system is captured. For this purpose, the robot body may carry at least one exemplary sensor for capturing sensor data, as referred to later. Figure 5 The discussion focuses on the following: In some implementations, the captured sensor data can be comprehensive, providing a detailed and relatively complete representation of the robot's environment (e.g., the entire field of view around the robot body at a distance, with a detailed representation of objects or features in the environment surrounding the robot body). However, such detailed sensor data is not necessary. In other implementations, the sensor data may represent only a portion of the environment surrounding the robot body (e.g., a limited field of view visible to the robot body's image sensors). In still other implementations, the sensor data may be even more limited, such as representing only a single object or feature of the environment. For a given application, the level of detail in the sensor data representing information about the environment can be precisely determined.

[0048] At point 204, at least one processor of the robot system generates a natural language (NL) description of at least one aspect of the environment based on sensor data. This NL description is... Figure 1 The NL description of at least one aspect of the environment does not need to describe every feature or object present in the sensor data, but can focus on one or more particularly relevant features or objects.

[0049] In an exemplary implementation, at least one processor executes an object or feature detection model (e.g., a classification module, such as the YOLO model or any other suitable model) that identifies objects or features in the environment (as represented in sensor data) and assigns text labels to such features or objects. Such text labels may be “robot language.” Throughout this disclosure, the term “robot language” or the like refers to a language used as a result of or intended for use within a robot or programming context, as opposed to natural human language used by humans to communicate with each other. Referring to the preceding chess suite example, a particular chess pawn could be identified in robot language as “chess_pawn_54677”. This is an example of robot language because underscores are used instead of spaces, and the numerical identifier for a pawn is significantly higher than the identifiers used by humans in normal contexts.

[0050] Regardless, there are useful commonalities (especially in common vocabulary) between robot language and human language. In the example of “chess_pawn_54677”, the terms “chess” and “pawn” are also used in human natural language. To generate an NL description of at least one aspect of the environment, at least one processor can execute a text string matching module that matches text in the robot language text label with NL vocabulary. For example, the NL description of “chess_pawn_54677” can be generated as “chess pawn 1”. Furthermore, objects or features identified in the environment can also be associated with metadata, which can be used to generate the NL description of the environment. For example, the label “chess_pawn_54677” can be associated with metadata indicating the color of a chess pawn (typically “white” or “black”). For example, at least one processor can use this metadata to generate “chess_pawn_54677” as the NL description of “white chess pawn 1”. However, including metadata is not mandatory. For example, a label can also indicate information such as "white_chess_pawn_54677".

[0051] Additional NL descriptions for other aspects of the environment can also be generated. Referring to the exemplary tips discussed above, NL descriptions can be generated for several different chess pieces, boxes, players, and the table. Such NL descriptions can be generated in a similar manner to those discussed above.

[0052] Furthermore, the NL description of the generated environment is not necessarily limited to the NL description of objects or features within the generated environment. In some implementations, the location or placement of such objects or features can also be described. Referring to the exemplary prompt discussed above, the sentence “There are white pawn 1, white pawn 2, black pawn 1, black pawn 2, white rook, black bishop, and two blue boxes on the table, and a person is standing opposite me” describes several objects in the environment and their positions. Such NL descriptions can be generated by at least one processor by, for example, populating a template description with a list of objects based on the positions of the objects that fit within the template.

[0053] At position 206, the NL query is provided to the Large Language Model (LLM) module. In some embodiments, the LLM module is a software or data module stored on at least one non-transitory processor-readable storage medium of the system (at the robot body or at a robot controller remotely to the robot body). In such embodiments, the NL query may be prepared by at least one processor of the robot system and provided as input to the LLM module. In other embodiments, the LLM module may be a software or data module stored on a non-transitory processor-readable storage medium of a device separate from the robot system. In still other embodiments, the LLM module may refer to a hardware module that receives input prompts and performs LLM module operations on the input. In such other embodiments, the NL query may be prepared by at least one processor of the robot system and provided to the device where the LLM module resides via the robot system's communication interface. As a specific example, the LLM module may be stored on one or more servers remotely to the robot body and may receive prompts as input via a website, form, or appropriate API. At least one processor of the robot system may prepare the NL query in an appropriate format, and the robot system may send the NL query via the robot system's communication interface.

[0054] The NL query provided to the LLM module includes an NL description of at least one aspect of the environment, as generated at 204. Additionally, the NL query includes an NL description of the work objective, an NL description of the set of instructions executable by the robot system, and an NL request for the task plan. Each of these NL descriptions is described in detail below.

[0055] As mentioned earlier, a work objective typically refers to a specific task, job, assignment, or application with a designated purpose and a determinable outcome. The NL description of such a work objective is an expression of it in a natural human form. Referring to the example of assembling a chess set discussed earlier, the prompt includes the phrase "My objective is to do as this person says." This can be considered an NL description of the work objective on its own, but further input provides more detail about the specific purpose the robotic system should accomplish. In this example, the prompt also includes the phrase "This person said, 'Put all the white pieces in the right-hand box and all the black pieces in the left-hand box.'" This can also be considered an NL description of the work objective and provides specific instructions about what the robotic system is expected to do. In some implementations, the NL description of the work objective includes the full content of both phrases: "My objective is to do as this person says. This person said, 'Put all the white pieces in the right-hand box and all the black pieces in the left-hand box.'"

[0056] The NL description of a work objective can be based on various information or data. In some implementations, the indication of the work objective can be explicitly available to the robot system (e.g., sent to the robot system from a management device or server, or stored in at least one non-transitory processor-readable storage medium of the robot system). The indication of the work objective can be available to the robot system in NL format, such that at least one processor only needs to access the indication of the work objective and provide it to the LLM module. In this sense, at least one processor of the robot system does not need to generate the NL description of the work objective, but can provide an existing NL description of the work objective to the LLM module. Alternatively, the indication of the work objective may not be in NL format (e.g., it can be robot language), and at least one processor can generate the NL description of the work objective based on the indication of the work objective (e.g., by performing a robot language conversion module, such as a text string matching module similar to those previously discussed). In other implementations, at least one processor of the robot system can generate the NL description of the work objective based on other information (such as the role in which the robot is deployed). In this context, "role" generally refers to the category of purpose in which the robot can serve within the relevant environment. For example, a cleaning robot can act as a cleaner of a specific area or facility. In this case, referring to the previous example of assembling a chess set, at least one processor can generate an NL description of the working objective as "cleaning up the scattered chess pieces and putting them into the appropriate boxes".

[0057] The work objective's NL description can be generated based on any appropriate additional information. Alternatively, when generating the work objective's NL description, the robot system's capabilities can be considered. For example, a robot lacking motion elements may only be able to successfully complete the work objective within the area near the robot body.

[0058] As previously mentioned, the set of instructions executable by the robot system can be a reusable library of working primitives, such as “grab an object,” “place an object on an object,” or any other suitable action. These examples are presented here in natural language, but can be stored and accessed in robot language, such as “grasp(object)” or “place(object1,object2)” (as non-limiting examples). In the example of assembling a chess set discussed earlier, “options” 1, 2, 3, 4, 5, and 6 represent the NL description of the set of instructions executable by the robot system. Exemplary hints also include the qualifying statements “I can only strictly choose from the following options:” and “the position where the object should be replaced by the appropriate object on the table,” which provide the LLM module with additional information about how the NL description of the instruction set should be interpreted, used, or applied. Such qualifying statements can be added, removed, or modified appropriately for a given application and a given instruction set. Such qualifying statements can also be included in NL queries, for example, by being included in the template on which the NL query is based, as discussed in more detail later.

[0059] In some implementations, the NL description of the instruction set can be pre-generated and loaded into the robot system, allowing the robot system to provide the pre-generated NL description of the instruction set to the LLM module at point 206. For example, a management device, server, or configuration device can generate the NL description of the instruction set, which can be stored in a non-transitory processor-readable storage medium of the robot system for subsequent access (e.g., during the configuration or deployment of the robot system). As a specific example, the reusable work primitive "place(object1, object2)" can be stored along with metadata of the NL description of the reusable work primitive as "place object on object". This NL description of the instruction can be provided manually by a human or can be generated by at least one processor (and may be reviewed and / or modified by a human to ensure accuracy).

[0060] In some implementations, the NL description of the instruction set can be generated by the robot system. Regardless of where the generation of the NL description of the instruction set is performed (in the example where the NL description is generated by at least one processor), the at least one processor performing this generation can execute a robot language conversion module that generates an NL description for each instruction in the instruction set based on the corresponding instructions expressed in robot language. Similar to what has been discussed previously, such a robot language conversion module can include a text string matching module operable to compare robot language instructions in the instruction set with natural language vocabulary representing actions that can be performed by the robot system. Matching text strings can be identified for inclusion in the NL description of the instruction set. As an example, for the instruction “grasp(object)”, the text string matching module can identify “grasp” and “object” as corresponding NL words. Furthermore, at least one processor can infer, based on the general structure of the programming function, that the intent of the instruction is to cause the robot to “grasp” the input “object”. For this purpose, an NL description “grasp object” can be generated for the instruction. As another example, for the instruction "place(object1,object2)", the text string matching module can recognize "place" and "object" as corresponding NL words. Furthermore, at least one processor can infer from the general structure of the programming function that the instruction's intent is for the robot to "place" the input "object1" on top of the input "object2". To this end, an NL description of "place object on object" can be generated for this instruction.

[0061] A task planning NL request typically refers to a statement or phrase designed to tell the LLM module how to process additional information in an NL query. Referring to the example of assembling a chess set discussed above, the phrase "Here is a list of the above commands that I should execute to accomplish my purpose:" is intended to inform the LLM module that it will generate a list of commands selected from the instruction set to accomplish the stated purpose (as described earlier in the work objective). For example, a task planning NL request can be generated by at least one processor of the robot system based on a request template. As another example, an NL request can be included in an NL query template used to construct or format an NL query, as described below.

[0062] In some implementations, at least one non-transitory processor-readable storage medium of the robot system stores at least one pre-generated NL template. Such an NL template may include any or all of the following: an NL description of at least one aspect of the environment, an NL description of the work objective, an NL description of the instruction set, and / or a corresponding template aspect of an NL request for the task plan. Exemplary templates are discussed below with reference to the previously discussed example hint generation for assembling chess sets. However, other exemplary templates may be used to generate different NL queries in different scenarios. The non-limiting exemplary NL templates discussed may be: "There exists [object_array_1] at [position_1], [object_array_2] at [position_2], ..., [object_array_j] at [position_j]. My goal is [work_objective], and I can only strictly choose from the following options:" 1.) [reusable_work_primitive_1(variable)] ... k.) [reusable_work primitive_k(variable)] Where [variable] should be replaced with the appropriate object at [position_1]...[position_j]. Here is a list of the above actions that I should perform to achieve my purpose:

[0063] In the template above, the elements within the square brackets can be populated by at least one processor inserting appropriate NL descriptions. Specifically, object_array_1 represents at least one array of objects at position_1 (e.g., an array of chess pieces on a table). In this example, at least one processor can replace the text [object_array_1] with the NL description of the chess pieces and replace the text [position_1] with the NL description of the table. Furthermore, in this example, object_array_2 represents a person standing at position_2 opposite the robot body. In this example, at least one processor can replace the text [object_array_2] with the NL description of the person and replace the text [position_2] with the NL description of the person's position. Additionally, from the text [(,)(and)], at least one processor can choose either "," or "and" to naturally connect the text about object_array_1 and object_array_2 (based on the presence or absence of object_array_3). In this example, there is no additional object array (no object_array_3 or object_array_j), so at least one processor chooses to connect the text "and". Furthermore, since there is no additional object array, at least one processor removes or ignores (e.g., replaces it with no text) the text "[object_array_j]" at [position_j].

[0064] As a result of the steps described above, the first statement of the template-generated NL query could be "There are white pawn 1, white pawn 2, black pawn 1, black pawn 2, white rook, black bishop, and two blue boxes at the table, and a person is standing opposite me." This is similar to the exemplary prompt described earlier, but the chess pieces are described as "at the table" instead of "on the table." To improve the generation of NL queries, the template can include options for transition words or location words (such as "on" or "at"), allowing at least one processor to select the most natural word for a given scenario.

[0065] Returning to the template above, at least one processor can replace the text [work_objective] with an NL description of the robot system's work objective. In this example, at least one processor can replace the text [work_objective] so that the second statement of the NL query is "My objective is to do as this person says. This person says 'Put all the white pieces in the right-hand box and all the black pieces in the left-hand box'" similar to the hint in the example discussed earlier.

[0066] Furthermore, at least one processor can replace the text of the available instruction “1.)[reusable_work_primitive_1(variable)]…k.)[reusable_work_primitive_k(variable)]” with the NL description of each available reusable work primitive. In the example scenario, the text for the available instructions could be replaced with “1.) grab object”, “2.) place object on object”, “3.) look at object”, “4.) slice object”, “5.) greet that person”, and “6.) put object in object”, as shown in the exemplary hints discussed earlier.

[0067] In the example, the “variable” for each reusable working primitive is replaced with the appropriate text for “object” or “person”, depending on what the given reusable working primitive applies to. Furthermore, the text [variable] and [position_1]...[position_j] in the penultimate statement of the template are also replaced with the appropriate text for “object”, along with the object’s associated position. In this respect, the penultimate statement of the generated NL query reads as “the position where the object should be replaced by the appropriate object on the table,” as in the exemplary hint presented earlier.

[0068] Given the above, by replacing or inputting the selection elements in the pre-generated template, an NL query suitable for provision to the LLM module is generated.

[0069] While the NL template has been described above as text in which some elements are "replaced" or "inputted," this is not strictly necessary in implementation. For example, instead of literally "replacing" text, an NL template can also be implemented as a set of instructions or functions (e.g., a program or script) that concatenate basic statements with related elements in a segmented manner. In such an example, the NL query is "assembled" into fragments, rather than literally "replaced" elements. In this sense, the presented NL template is intended as a logical representation of how elements can be concatenated, rather than a strict process in which text generation actually occurs.

[0070] Return to Figure 2In method 200, at point 208, the robot system receives the task plan from the LLM module. That is, after providing the NL query to the LLM module at point 206, the LLM module generates the task plan (expressed as NL) and provides the generated task plan to the robot system.

[0071] At 210, the robot system executes the task plan. For example, at least one processor of the robot controller can cause at least one element (e.g., an actuable element) to perform any action specified in the task plan.

[0072] As described above, the task plan provided by the LLM module is expressed in NL. For example, the task plan may indicate at least one action, expressed in NL, that the robot system can perform. In order for the robot system to execute the action plan, at least one processor may first generate a robot language task plan based on the task plan expressed in NL. The robot language task plan may include a set of robot control instructions that, when executed by at least one processor, cause the robot system to perform at least one action indicated in the task plan. For example, the set of robot control instructions may include a library or set of at least one reusable working primitive that is executable by the robot system. Furthermore, the at least one action indicated in the task plan expressed in NL may include an NL description of a specific reusable working primitive (e.g., grabbing chess pawn 1), while the robot control instructions in the robot language task plan may include actions from the NL task plan, but specified in a language format available to the robot system (e.g., grasp(chess_pawn_54677)).

[0073] Similar to the preceding description, generating a robot language task plan may include executing a robot language conversion module that converts at least one action, expressed in NL and executable by the robot system, into at least one reusable working primitive from a set of instructions executable by the robot system. Referring to an example where the NL task plan includes the action "grab chess pawn 1" expressed in NL, the robot language conversion module may match the text string in the action expressed in NL with a text string available in a reusable working primitive that the robot system can use (e.g., `grasp(object)`) or an object in the environment with which the robot system can interact (e.g., `chess_pawn_54677`). Thus, at least one processor may generate a robot language action, such as `grasp(chess_pawn_54677)`.

[0074] In some implementations, the LLM module can be used to autonomously troubleshoot task plans. For example, if a given task plan fails to execute (i.e., fails to be validated, fails to proceed to completion, and / or fails to complete the intended task) or encounters an error, an NL prompt can be sent (returned) to the LLM module. This NL prompt includes all successful portions of the executed or validated task plan, describes the failed portions with additional wording, and asks the LLM module what to do next. Additionally, an external checker can review or validate a proposed plan and reject it for some reason. As a non-limiting example, the external checker can be a logic-based system or inference engine, such as the CYC® Machine Inference AI platform from Cycorp. An inference engine (sometimes called an inference engine) can utilize logical rules, statements, terms, knowledge fragments, or similar libraries and can derive logical conclusions based on these. In this way, the task plan mentioned in method 200 can be validated by the inference engine by comparing the task plan against a set of rules (or similar rules) at least partially specified by the inference engine. In other words, at least a portion of the inference engine's logic can be applied to the task plan to verify its logical validity and / or identify any logical inconsistencies or impossibilities. For example, a rejection might be due to a safety violation related to robot safety or the safety of any human or other living being. In the event of a rejection, an NL hint can be sent back to the LLM module and modified to prevent the plan from failing external checks.

[0075] In some implementations, an LLM can help autonomously assign parameters or definitions to generalized and / or parameterized objects in a robot control system. For example, parameterized working primitives or "instructions" can be assigned by the LLM, as in the chess suite example above. As another example, if a task plan executes successfully, the successful task plan can be stored and then re-parameterized to become generalized. When the robot encounters a future instance of a similar task, it can invoke the stored successful task plan and request the LLM module (e.g., via a simple NL hint) to replace the parameterized object of the previously successful instance of the task plan with a new object specific to the current instance of the task plan. For example, if a plan is generated to successfully sort two types of specific objects, the robot can reuse the plan by requesting the LLM to replace those objects with different objects.

[0076] Various embodiments of the systems, methods, control modules, and computer program products of this application relate to controlling the functions and operations of a robot using NL expressions (descriptions) (e.g., via NL prompts, which can be directly entered by the user in text form, or can be spoken by the user and converted to text by an intervening speech-to-text system), wherein an LLM module can provide an interface between the NL expression and the robot control system. This framework can be particularly advantageous when certain elements of the robot control architecture employ programming and / or instructions that can be expressed using NL. A suitable, but not limiting, example is the instruction set described above. For example, as previously stated, the task planning output of the LLM module can be parsed by finding words that match the commands of the instruction set (e.g., autonomously parsed by the robot control system), and the arguments of the instruction set can be found by string matching within the input NL prompts (e.g., via a text string matching module as previously described). In some embodiments, a 1-1 mapping can be generated between the arguments used in the robot control system and the NL variants to increase the chances of the LLM module correctly processing the text. For example, even if an object is represented as `chess_pawn_54677` in the robot control system (e.g., in the world model environment section of the robot control system), it can also be referred to as "chess pawn 1" in the NL prompt. In this case, if the returned task plan contains the phrase "grasp chess pawn 1", this can match the instruction set "grasp" and the object "chess pawn 1", thus mapping the phrase to `grasp(chess_pawn_54677)`. This parsing and / or word matching (e.g., text string matching modules) can be used in any case where the robot language discussed herein is converted to natural language or natural language is converted to robot language.

[0077] In some implementations, the robot control system can generate and / or employ a scene graph describing the robot's environment, and can apply functions to act on the scene graph and create NL cues or descriptions that describe the scene from the robot's perspective (e.g., in the context of action 204 of method 200). This automatically generated NL cues or descriptions can then be used as input to an LLM module to facilitate various operations such as reasoning, fact checking, and task planning.

[0078] In some implementations, the quality of the task plan may depend at least in part on the robot's understanding of its environment; therefore, the robot control system can periodically check and compare its scene map and internal world model in the background. According to the systems, methods, control modules, and computer program products of this application, this checking and comparison of the scene map (e.g., actual data from the robot's external environment) and the internal world model (e.g., a simulation of the robot's external environment) can be accomplished by automatically generating NL cues or descriptions for each and feeding these NL cues or descriptions through an LLM module.

[0079] In some implementations, the LLM module, acting as a task planner, can be frequently activated by the robot control system to answer the question, “What can I (the robot) do here / now?”. For example, the robot control can automatically generate an NL description of at least one aspect of its environment (scene graph) and capabilities (instruction set), and feed these NL descriptions into the LLM module along with queries such as: What can I do? : or “What should I do? :” (or similar variations, such as “What is the most useful thing for me?”, “What are the productive things I can do?”, etc.). Then, a set of answers to this question or similar questions can be generated individually by generating a task plan (e.g., as referenced above). Figure 2 The robot system performs each of these tasks (as described in Method 200) to achieve these objectives. Each of these task plans can be checked against some constraints or principles (e.g., verified by an inference engine), and the verified tasks are now a set of things that the robot system can do spontaneously without being asked. This type of behavior is a form of agency because the robot system utilizes embodied artificial general intelligence (AGI) to create its own grounded task plans based on real-world contexts and then executes them. In some implementations, the robot system presents an output describing what its plan does in an NL presentation to request permission from a human (or driver / supervisor) to execute the plan. A separate inference system, such as Cyc®, can be used to interpret each step in the plan. The plan can be provided at any desired level of abstraction, where the highest level may be the most useful descriptor for a human supervisor to determine whether the plan should be executed. The LLM module can also be used to check whether the plan passes a certain criterion, for example, by issuing an NL prompt to ask whether the plan at the highest level is compatible with a set of desired constraints.

[0080] Figure 3This is a flowchart illustrating an exemplary method of operation 300 of a robot system performing a work objective according to the system, control module, method, and computer program product of this application. Generally, throughout this specification and the appended claims, the method of operation of a robot system is a method in which at least some (if not all) of various actions are performed by the robot system. For example, certain actions of the method of operation of a robot system may be performed by at least one processor or processing unit (hereinafter referred to as a "processor") of the robot system communicatively coupled to a non-transitory processor-readable storage medium of the robot system, and in some embodiments, certain actions of the method of operation of a robot system may be performed by peripheral components of the robot system communicatively coupled to at least one processor, such as one or more physically actuated parts of the robot body (e.g., arms, legs, end effectors, grippers, hands), one or more sensors (e.g., optical sensors, audio sensors, tactile sensors, motion sensors), mobility systems (e.g., wheels, legs), communication and networking hardware (e.g., receivers, transmitters, transceivers), etc. The non-transitory processor-readable storage medium of the robot system may store data (including, for example, a reusable library of working primitives) and / or processor-executable instructions, which, when executed by at least one processor, cause the robot system to perform a method and / or cause the at least one processor to perform those actions of the method executed by the at least one processor. The robot system may communicate with or include remote systems and / or remote non-transitory processor-readable storage media via communication and network hardware communicatively coupled to at least one processor of the robot system. Therefore, unless the specific context requires otherwise, references to the non-transitory processor-readable storage medium of the robot system and the data and / or processor-executable instructions stored in the non-transitory processor-readable storage medium are not intended to limit the physical location of the non-transitory processor-readable storage medium relative to at least one processor of the robot system and the robot body. In other words, the non-transitory processor-readable storage medium of the robot system may include non-transitory processor-readable storage media located on the robot body and / or non-transitory processor-readable storage media located remotely from the robot body, unless the specific context requires otherwise.

[0081] return Figure 3 Method 300 includes four actions 301, 302, 303, and 304, and two optional actions 305a and 305b. However, those skilled in the art will understand that in alternative embodiments, certain actions and / or conditions may be omitted and / or additional actions and / or conditions may be added. Those skilled in the art will also understand that the illustrated order of actions is shown for illustrative purposes only and may be changed in alternative embodiments.

[0082] At 301, at least one sensor of the robot system captures sensor data representing information about the environment of the robot body. In some embodiments, action 301 of method 300 may be substantially similar to action 202 of method 200.

[0083] At 302, at least one processor of the robot system generates a natural language (NL) description of at least one aspect of the environment based on sensor data. In some embodiments, action 302 of method 300 may be substantially similar to action 204 of method 200.

[0084] At 303, a first NL query is provided to the Large Language Model (LLM) module. The first NL query may include an NL description of at least one aspect of the robot's environment generated at 302, an NL description of a set of instructions executable by the robot system (e.g., action 206 similar to method 200 in some embodiments), and an NL request for at least one work objective. In some embodiments, the NL request for at least one work objective may take the form of a simple query, such as “What can I do here?” or “What can I do now?” or variations thereof, as described above. In some embodiments, the NL request for at least one work objective may include adjectives or qualifiers, such as “What is the most useful thing I can do in this situation?” or “Give me a work objective to accomplish productive work in this scenario,” and so on. In some embodiments, the NL request for at least one work objective may include NL requests for more than one work objective, such as “List five things I can do given the environment description and set of instructions,” regardless of whether there is a request to prioritize or rank the work objectives (e.g., “Rank the top five things I can do now”).

[0085] At 304, at least a first work objective (expressed as NL) is received from the LLM module. The first work objective may include, for example, a concise description of a high-level task, activity, action, job, or function that the robot system can operate to perform within the context of an environmental description in order to achieve a meaningful work or interaction. The result of actions 301, 302, 303, and 304 is that the robot system, through interaction with the LLM module, identifies activities it can perform without instructions, guidance, or intervention from an operator, which, depending on the implementation, may or may not be a component of the robot system. In other words, the robot system not only autonomously performs the work objective but also identifies one or more work objectives to be performed.

[0086] In various embodiments of method 300, method 300 may proceed from action 304 to either optional action 305a or 305b (or neither, or one or more other actions). At 305a, the robot system completes the first work objective, for example, without any instructions, guidance, or confirmation from the robot system's operator. In the case of 305a, the robot system may identify and perform the first work objective by interacting with an LLM module (i.e., autonomously), which may or may not be a component of the robot system depending on the implementation. At 305b, the robot system sends a request to the robot system's operator to confirm that the robot system should complete the first work objective. Based on the operator's response to the request, the robot system may or may not proceed to complete the first work objective. Thus, in the case of 305b, the robot system may autonomously identify the first work objective by interacting with an LLM module (which may or may not be a component of the robot system depending on the implementation), but waits for verification or confirmation from the operator before proceeding to complete the first work objective.

[0087] Throughout this specification and the appended claims, unless the specific context requires otherwise, the term "operator" (e.g., as in "operator of a robot system") means any individual or system that influences the operation of a robot system but resides outside the robot system's own control system. Examples of operators of robot systems include human drivers of robot systems and robot queuing management systems, the latter of which may include any number of human users and / or software-implemented control algorithms.

[0088] According to the robot system, method, control module, and computer program product of this application, the autonomous identification of work objectives achieved by method 300 can be extended to autonomously generating task plans for accomplishing such work objectives.

[0089] Figure 4 This is a flowchart illustrating an exemplary operational method 400 of a robotic system performing a work objective according to the system, control module, method, and computer program product of this application. Method 400 represents an extension of method 300 and therefore includes each of actions 301, 302, 303, and 304 of method 300. Starting with action 304 of method 300, method 400 continues to include two actions 401 and 402 and two optional actions 403a and 403b; however, those skilled in the art will understand that in alternative embodiments, certain actions and / or conditions may be omitted and / or additional actions and / or conditions may be added. Those skilled in the art will also understand that the illustrated order of actions is shown for illustrative purposes only and may be changed in alternative embodiments.

[0090] At 401, a second NL query is provided to the LLM module. In some embodiments, the second NL query provided at 401 of method 400 may be substantially the same as the NL query provided at action 206 of method 200, using the first work objective received at 304 of method 300 as the work objective. For example, the second NL query of action 401 may include an NL description of at least one aspect of the environment generated at 302, the first work objective received at 304, an NL description of a set of instructions executable by the robot system (e.g., the same NL description of the set of instructions used in the first query provided at 303), and an NL request for a first task plan to complete the first work objective.

[0091] At 402, the first task plan (expressed as NL) is received from the LLM module. In some implementations, action 402 of method 400 may be substantially similar to action 208 of method 200.

[0092] In various embodiments of method 400, method 400 may proceed from action 402 to either optional action 403a or 403b (or neither, or one or more other actions). At 403a, the robot system completes the first task plan, for example, without any instructions, guidance, or confirmation from the operator of the robot system. In the case of 403a, the robot system may identify the first work objective and design and execute the first task plan by interacting with an LLM module (i.e., autonomously) without any interaction with the operator of the robot system, which may or may not be a component of the robot system, depending on the implementation. At 403b, the robot system sends a request to the operator of the robot system to confirm that the robot system should complete the first task plan. Based on the response from the operator to the request, the robot system may or may not proceed to complete the first task plan. Thus, in the case of 403b, the robot system may autonomously identify the first task plan by interacting with an LLM module, but waits for verification or confirmation from the operator before continuing to execute the first task plan, which may or may not be a component of the robot system, depending on the implementation.

[0093] In some implementations, the robot system may validate a first task plan to determine whether the first task plan violates any constraints in a set of constraints. This set of constraints may be stored in a task plan validation module (including processor-executable instructions and / or data stored in a non-transitory processor-readable memory of the robot system), which may or may not include an inference engine. For example, the set of constraints may be pre-specified by the operator and include quantifiable requirements such as "the task plan must be able to be executed using less power than the current amount of power available to the robot system"; etc. In this case, the validation module may not require an inference engine. However, in some implementations, the set of constraints may include higher-level conceptual constraints such as "the task plan must not cause harm or damage to persons or property"; "the task plan must not contain or lead to failure scenarios"; and "the task plan must comply with all physical laws," in which case an inference engine may be included to assist in validating such constraints.

[0094] As an example, a failure scenario could include one where the execution of a step in the task plan produces a result inconsistent with the task objective. In the exemplary use case where the task objective is for the robot system to win at chess, the execution of the step where the robot system moves the king to a checkmate position is a failure scenario that produces a result inconsistent with the task objective. This is because moving the king to a checkmate position would result in an immediate defeat of the game.

[0095] As another example, a failure scenario can include a scenario in which the execution of a step in the task plan prevents the execution of at least one other step in the task plan. In an exemplary use case, a failure scenario is one in which the execution of a step that disrupts the execution of an object required for at least one other step in the task plan (e.g., a part of a robot or an object that the robot needs to interact with) is prevented.

[0096] As another example, a failure scenario can include a scenario in which the execution of a step in the task plan has at least one unacceptable impact on the environment. In an exemplary use case, the execution of a step that results in environmental damage (even if the damage is not consistent with the work objectives and does not inhibit other steps in the task plan) is a failure scenario that has at least one unacceptable impact on the environment. For example, the robot body may collide with a wall, leaving a hole, dent, or other unacceptable damage, even if the integrity of the wall is irrelevant to achieving the robot system's work objectives or performing other steps in the task plan.

[0097] In some implementations, verifying the task plan requires simulating the execution of at least one step of the task plan. In one example, an environment model representing the environment of the robot body is accessed. This environment model may be stored on at least one non-transitory processor-readable medium of the robot body and / or generated or updated based on sensor data captured by at least one sensor of the robot body (e.g., data captured at 202). For example, the environment model can be populated by collecting data on individual objects (e.g., tactile or visual data). Based on the collected data, individual objects can be identified, and corresponding profiles of the identified objects can be accessed in a database. The environment model is then populated with visual and / or tactile representations of each individual object based on the corresponding profiles accessed for that object. Detailed embodiments for generating and / or populating environment models, as discussed in U.S. Patent No. 11,717,974, are incorporated herein by reference in their entirety.

[0098] Then, at least one processor simulates at least one step of the task plan in an environment model. For each step of the task plan to be simulated, when simulating the execution of the corresponding step, at least one processor identifies whether each step of the task plan generates a failure scenario.

[0099] Furthermore, if no corresponding failure scenario is identified in the simulation of each step of at least one step of the task plan, at least one processor can simulate at least one additional step of the task plan in an environment model. For each step in the at least one additional step of the task plan, when simulating the execution of the corresponding step, at least one processor can identify whether each step in the at least one step of the task plan generates a corresponding failure scenario. In this way, the task plan can be evaluated in stages, where the selected steps of the task plan are simulated individually.

[0100] As an example, a failure scenario could include a scenario where the simulated execution of a step produces a simulation result inconsistent with the work objective. In an exemplary use case where the work objective is for the robot system to win at chess, the simulated execution of the step where the robot system moves the king to a checkmate state is a failure scenario that produces a simulation result inconsistent with the work objective. This is because moving the king to a checkmate state would result in an immediate loss of the game.

[0101] As another example, a failure scenario can include a scenario in which the simulated execution of a step prohibits the execution of at least one other step in the task plan. In an exemplary use case, a failure scenario is one in which the simulated execution of a step requiring an object (e.g., a part of a robot or an object that the robot needs to interact with) to disrupt at least one other step in the task plan is prohibited from execution.

[0102] As another example, a failure scenario can include a scenario where the simulated execution of a step produces at least one unacceptable simulated effect on the environment. In an exemplary use case, a failure scenario where the simulated execution of a step causes damage to the simulated environment (even if the damage is not consistent with the work objectives and does not inhibit other steps in the task plan) is a failure scenario that produces at least one unacceptable effect on the environment. For example, the robot body could be simulated as colliding with a wall, leaving a hole, dent, or other unacceptable damage, even if the integrity of the wall is irrelevant to achieving the work objectives of the robot system or performing other steps in the task plan.

[0103] By simulating the execution of at least one step of a task plan, efficiency can be improved by avoiding wasting effort or time actually executing an incorrect task plan. Furthermore, in cases where the task plan would lead to such damage, actual damage to the object can be avoided.

[0104] In some implementations, verifying the task plan may include determining whether the task plan violates at least one rule or constraint from a set of rules or constraints specified in at least a portion of the inference engine, as previously described. In some examples, at least one processor of the robot system utilizes the inference engine to verify the task plan based on inference engine data stored on at least one non-transitory processor-readable storage medium of the robot system. In some implementations, the inference engine may be external, and the robot system may send the task plan to a device that stores and executes the inference engine (e.g., a server or peripheral device) for verification.

[0105] Task plans can be reviewed or validated using an inference engine or other logic-based system, and approved or rejected for one or more reasons. As a non-limiting example, a logic-based system or inference engine could be the CYC® Machine Inference AI platform from Cycorp. An inference engine (sometimes called an deduction engine) can utilize logical rules, statements, terms, knowledge fragments, or similar libraries, and can draw logical conclusions based on these. In this way, a task plan can be validated by the inference engine by comparing the task plan against a set of rules (or similar rules) at least partially specified by the inference engine. That is, at least a portion of the logic of the inference engine can be applied to the task plan to verify whether the task plan has logical meaning, and / or to identify any logical inconsistencies or impossibilities in the task plan. For example, a reason for rejection could be a safety violation related to robot safety or the safety of any human or other living being. In the case of rejection, additional NL queries can be sent back to the LLM module to prevent the task plan from failing external checks.

[0106] In some implementations, receiving at least a first work objective from the LLM module at 304 may include receiving multiple work objectives from the LLM module, wherein each corresponding work objective among the multiple work objectives is represented by NL. In such implementations, the robot system may autonomously complete all or a subset of the multiple work objectives, or the robot system may rank the multiple work objectives based on a certain criterion and autonomously complete only a subset of the work objectives that achieve a certain ranking, or the robot system may (similar to action 305b) send a request to the operator of the robot system to confirm which of the multiple work objectives the robot system should complete.

[0107] In some embodiments where multiple work objectives are received from the LLM module at point 304, method 400 can be extended to include determining a corresponding task plan for each (or at least two) of the multiple work objectives received from the LLM module. This may involve performing corresponding iterations of actions 401 and 402 for each (or for at least two) of the multiple work objectives. For example, for each of the at least multiple work objectives, action 401 can be extended to include providing a corresponding NL query to the LLM module, the corresponding NL query including an NL description of at least one aspect of the environment, the work objective expressed in NL, an NL description of a set of instructions executable by the robot system, and a corresponding NL request for a corresponding task plan for completing the work objective; and action 402 can be extended to include receiving a corresponding task plan from the LLM module for each work objective, the corresponding task plan for each work objective expressed in NL.

[0108] In an implementation of method 400, where multiple task plans are received from the LLM module at 402 (i.e., the multiple task plans correspond to respective work objectives among the multiple work objectives received from the LLM module at 304), the robot system can proceed to execute one or more of the task plans. In some implementations, the robot system can select (e.g., autonomously) which task plans among the task plans to execute and in what order. For example, for each of at least two work objectives among the multiple work objectives, the robot system can receive a corresponding task plan from the LLM module, and the robot system can select which corresponding task plan to execute first. Alternatively, the robot system can send a request to the operator of the robot system to determine which corresponding task plan the robot system should execute first, and receive from the operator the selection of the corresponding task plan to execute first. In either case, the robot system can proceed to execute the selected / chosen task plan. In this context, determining which task plan to execute "first" allows, but does not necessarily require, the execution of any number of additional task plans, such as "second," "third," "fourth," etc.

[0109] Some task plans may contain steps that cannot be resolved into instruction set elements and are computationally oriented in nature. For example, a task plan might require calculating integrals or some other computational process, which might be impossible given a particular instruction set. In these cases, the robotic system can send these task plan steps to an LLM-based system or LLM module, which requests the generation of a piece of code, such as a Python script, that produces functions to perform the task. In some implementations, this script can then reside in a "codebase" where a human engineer can review all automatically generated scripts produced by a background "What can I do here?" process and check if they have done what was intended. Such scripts generated by LLM-based devices or modules can provide new instruction set elements that can be invoked to "unlock" task plans blocked by the inability to access the appropriate instructions, or can otherwise be accessed by the robotic system to be incorporated into the task plan.

[0110] In some implementations, the LLM module can be stored and executed outside the robot (e.g., in the cloud) and invoked or accessed by the robot system (e.g., ...). Figure 5 (As shown in the example). Specifically, Figure 5 This is a schematic diagram of the robot body 500, which accesses the LLM module 520 via the cloud 510 (decoupling the LLM module from the robot system including the robot body 500). In other embodiments, the LLM module can be stored and executed locally on the robot system as part of the robot's control system (e.g., Figure 6(As shown in the example). Specifically, Figure 6 This is a schematic diagram of a robot body 600, which has an LLM module 620 locally located at a non-transitory processor-readable storage medium. Both implementations are referred to herein as "robots with accessible LLMs".

[0111] The various implementations described herein include systems, methods, control modules, and computer program products for utilizing one or more LLMs in a robot control system, including, for example, establishing an NL interface between the LLM and the robot control system and invoking the LLM to help autonomously instruct the robot to do what. Example applications of this approach include task planning, motion planning, reasoning about the robot's environment (e.g., "What can I do now?"), and so on. Such implementations are particularly suitable for robot control systems for which at least some control parameters and / or instructions (e.g., the instruction set previously described) are suitable for being specified in the NL. Therefore, some implementations may include converting or translating robot control instructions and / or parameters into the NL for such communication with the LLM via the NL interface.

[0112] Figure 7This is an illustrative diagram of an exemplary robot system 700, including various features and components described throughout the system, robot, method, control module, and computer program product description of this application. For example, robot system 701 can perform method 200, method 300, and / or method 400, as well as associated actions as previously described (and not repeated for brevity). Robot system 700 includes a robot body 701 having a first physically actuated component 702a and a second physically actuated component 702b mechanically coupled to the body 701. In the illustrated embodiment, the first physically actuated component 702a and the second physically actuated component 702b each correspond to a respective robot hand, although those skilled in the art will understand that in alternative embodiments, the physically actuated components may take other forms (e.g., arms or legs, non-hand-like end effectors (e.g., cutters or suction tubes), or any other form useful for the specific application the robot intends to perform). Robotic hand 702a mimics a human hand and includes multiple fingers 721a, 722a, 723a, and 724a, as well as a relative thumb 725a. Robotic hand 702b is similar to a mirror image of robotic hand 702a, while no corresponding details are labeled for robotic hand 702b to reduce clutter. Robotic hands 702a and 702b can be physically actuated in various ways (including electromechanical actuation, cable-driven actuation, magnetorheological fluid-based actuation, and / or hydraulic actuation). Some exemplary details of actuation techniques that can be used for physically actuated robotic hands 702a and 702b are described in U.S. Patent Application Serial No. 17 / 491,577 and U.S. Patent Application Serial No. 17 / 749,536, which are incorporated herein by reference in their entirety.

[0113] The robot body 701 also includes at least one sensor 703, which detects and / or collects data about the environment and / or objects in the environment of the robot system 700 (e.g., including people, such as customers). In the illustrated embodiment, sensor 703 corresponds to a sensor system including a camera, a microphone, and an initial measurement unit, the initial measurement unit itself including three orthogonal accelerometers, a magnetometer, and a compass. However, any suitable sensor may be included in at least one sensor 703 or excluded, which is appropriate for a given application. Sensor data, such as that captured in action 202 of method 200 and / or action 301 of method 300, may be captured by sensor 703, for example.

[0114] For illustrative purposes, Figure 7This includes details of certain exemplary components carried or located within the robot body 701, which are part of the system, robot, method, control module, and computer program product according to this application. Such components include at least one processor 730 and at least one non-transitory processor-readable storage medium or "memory" 740 communicatively coupled to the processor 730. The memory 740 stores data 741 and processor-executable instructions 742 (e.g., together as a robot control module or computer program product), which, when executed by the processor 730, cause the robot body 701 (including any one or both of applicable actuated components, such as robot hands 702a and / or 702b) to perform actions and / or functions associated with the system, method, control module, and computer program product of this application. The at least one processor 730 and the at least one non-transitory processor-readable storage medium 740 together can be considered a robot controller.

[0115] In some implementations, actions or processes can be performed entirely locally at the robot body 701. For example, in some implementations, the entire method 200, method 300, and / or method 400 can be performed locally at the robot body 701. In such implementations, at least one sensor 703 captures sensor data in action 202 (and / or 301), and at least one processor 730 generates an NL description of at least one aspect of the environment in action 204 (and / or 302). At least one processor 730 may further generate any other NL descriptions included in the NL query, as previously discussed. Furthermore, in such implementations, memory 740 also stores an LLM module to which an NL query is provided in action 206 (and / or 303). Providing an NL query in this context may refer to at least one processor 730 executing an LLM module with the NL query as input. Additionally, receiving a task plan from the NL in action 208 of method 200 (and / or 304 of method 400) may include at least one processor 730 receiving a task plan output by an LLM module. Executing a task plan, such as in action 210 (and / or 403a), includes at least one processor 730 executing instructions to cause the robot body 701 to perform actions specified in the task plan.

[0116] In some implementations, the action or process can be performed locally at the robot body 701 or by a separate device detached from the robot body 701. In this regard, at least one processor 730 is also communicatively coupled to a wireless transceiver 750, via which the robot body 701 transmits and receives wireless communication signals 770.

[0117] Figure 7The system also includes a standalone device 780, which is part of the robot system 700 but physically separate from the robot body 701. As a non-limiting example, the standalone device 780 may be a processing unit located adjacent to the robot body 701 (e.g., in the same room), or it may be a remote server located away from the robot body 701. The standalone device 780 includes a wireless transceiver 781 through which it transmits and receives wireless communication signals 770. The wireless transceivers 750 and 781 may be referred to as the communication interface (together or separately) through which the robot body 701 and the standalone device 780 communicate. Furthermore, transceivers 750 and 781 do not necessarily communicate directly (although they can); for example, they may communicate with each other via a network or the Internet. Additionally, transceivers 750 and 781 do not necessarily have to be wireless; in some embodiments, either transceiver may be replaced with a wired communication interface. In some implementations where the LLM module is not stored in memory 740, the robot body 701 can access the LLM module via transceivers 750 and 781 through wireless signal 770.

[0118] Specifically, the standalone device 780 is also shown as including at least one processor 782 communicatively coupled to a wireless transceiver 781, and at least one non-transitory processor-readable storage medium 790 (or "memory" 790) communicatively coupled to the at least one processor 782. The memory 790 stores data 791 and processor-executable instructions 792 (e.g., together as a robot control module or computer program product), which, when executed by the processor 782, cause the standalone device 780 (or components thereof) to perform actions and / or functions associated with the system, robot, method, robot control module, and computer program product of this application. The memory 790 may also store an LLM module. Alternatively, the standalone device 780 may access an LLM module stored at another device (e.g., a cloud-based or internet-based LLM module).

[0119] The methods or processes discussed in this article (e.g., Figure 2 Method 200 Figure 3 Method 300 and / or Figure 4Method 400 can be performed by a combination of robot body 701 and stand-alone device 780. In an exemplary embodiment, at least one sensor 703 captures sensor data in action 202 (and / or 301), and at least one processor 730 generates an NL description of at least one aspect of the environment in action 204 (and / or 302). At least one processor 730 can further generate any other NL descriptions included in the NL query, as previously discussed. In this exemplary embodiment, the memory 790 of stand-alone device 780 stores the LLM module, to which the NL query is provided in action 206 (and / or 303). In this example, providing the NL query means that robot body 701 transmits the NL query to stand-alone device via transceivers 750 and 781 (communication interface). At least one processor 782 then executes the LLM module with the NL query as input. Furthermore, receiving a task plan from the NL, as in action 208 of method 200 (and / or 304 of method 400), includes the robot body 701 receiving a task plan output by the LLM module, which is transmitted from a separate device 780 by transceivers 750 and 781 for reception by the robot controller (or at least one processor 730). Executing the task plan, as in action 210 (and / or 403a), includes at least one processor 730 executing instructions to cause the robot body 701 to perform actions specified in the task plan.

[0120] Throughout this specification and the appended claims, the term "communicative" (as in "communicative coupling" and in variations such as "communicatively coupled") is generally used to refer to any engineered arrangement for transmitting and / or exchanging information. For example, communication coupling can be implemented in the form of various different media and / or communication paths, including but not limited to: conductive paths (e.g., conductive wires, conductive traces), magnetic paths (e.g., magnetic media), wireless signal transmission (e.g., radio frequency antennas), and / or optical paths (e.g., optical fibers). Exemplary communication couplings include, but are not limited to: electrical coupling, magnetic coupling, radio frequency coupling, and / or optical coupling.

[0121] Throughout this specification and the appended claims, the infinitive verb form is frequently used. Examples include, but are not limited to, “encode,” “provide,” “store,” etc. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, i.e., as “to at least encode,” “to at least provide,” “to at least store,” etc.

[0122] This specification, including the accompanying drawings and abstract, is not intended to be an exhaustive or limiting description of all implementations and embodiments of the systems, methods, control modules, and computer program products of this application. Those skilled in the art will understand that the various descriptions and drawings provided can be modified without departing from the spirit and scope of this disclosure. In particular, the teachings herein are not intended to be limited to or restricted to the illustrative examples of computer systems and computing environments provided.

[0123] This specification provides various implementations and embodiments in the form of block diagrams, schematic diagrams, flowcharts, and examples. Those skilled in the art will understand that any function and / or operation in such block diagrams, schematic diagrams, flowcharts, or examples can be implemented individually and / or collectively by a wide range of hardware, software, and / or firmware. For example, the various embodiments disclosed herein, in whole or in part, can be equivalently implemented in one or more of the following: application-specific integrated circuits (i.e., ASICs); standard integrated circuits; computer programs executed by any number of computers (e.g., programs running on any number of computer systems); programs executed by any number of controllers (e.g., microcontrollers); and / or programs executed by any number of processors (e.g., microprocessors, central processing units, graphics processing units); and firmware and any combination of the foregoing.

[0124] Throughout this specification and the appended claims, "memory" or "storage medium" refers to a processor-readable medium, which is an electronic, magnetic, optical, electromagnetic, infrared, semiconductor, or other physical device or apparatus that contains or stores processor data, data objects, logic, instructions, and / or programs. When data, data objects, logic, instructions, and / or programs are implemented as software and stored in memory or storage medium, they can be stored in any suitable processor-readable medium for use by any suitable processor-related instruction execution system, apparatus, or device, such as a computer-based system, a processor-integrated system, or other system capable of retrieving data, data objects, logic, instructions, and / or programs from memory or storage medium and performing various actions or operations (i.e., processing steps) thereon and / or in response to them. Therefore, a "non-transitory processor-readable storage medium" can be any element that stores data, data objects, logic, instructions, and / or programs for use by or in conjunction with an instruction execution system, apparatus, and / or device. As a specific, non-limiting example, a processor-readable medium may be: a portable computer floppy disk (magnetic, compact flash memory card, secure digital device, etc.), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, EEPROM, or flash memory), portable optical disc read-only memory (CDROM), digital magnetic tape, and / or any other non-transitory medium.

[0125] The claims of this disclosure are appended. This disclosure is intended to support, implement, and describe the claims, but is not intended to limit the scope of the claims to any particular implementation or embodiment. In general, the claims should be interpreted to include all possible implementations and embodiments together with the full scope of equivalents to which these claims are entitled.

Claims

1. A method for operating a robot system including a robot body, the method comprising: Sensor data representing information about the environment of the robot body is captured by at least one sensor of the robot system; The robot system generates a natural language (NL) description of at least one aspect of the environment based on the sensor data through at least one processor of the robot system; Provide a first NL query to the large language model LLM module, the first NL query including an NL description of at least one aspect of the environment, an NL description of a set of instructions that the robot system can execute, and an NL request for at least one work objective; Receive at least a first working objective from the LLM module, the first working objective being expressed in NL; A second NL query is provided to the LLM module, the second NL query including an NL description of at least one aspect of the environment, the first work objective expressed in NL, an NL description of the instruction set that can be executed by the robot system, and an NL request for a first task plan for completing the first work objective; The first task plan is received from the LLM module, and the first task plan is expressed in NL. as well as The first work objective is achieved through the robot system.

2. The method according to claim 1, further comprising: A request is sent to the operator of the robot system to confirm that the robot system should complete the first work objective.

3. The method according to claim 1, further comprising: The first task plan is executed by the robot system.

4. The method according to claim 1, further comprising: A request is sent to the operator of the robot system to confirm that the robot system should execute the first task plan.

5. The method according to claim 1, further comprising: Verify the first task plan to determine whether the first task plan violates any constraints in a set of constraints.

6. The method of claim 1, wherein, Receiving at least the first working target from the LLM module includes receiving multiple working targets from the LLM module, and the method further includes: A request is sent to the operator of the robot system to confirm which of the plurality of work objectives the robot system should complete.

7. The method of claim 1, wherein, Receiving at least the first working target from the LLM module includes receiving a plurality of working targets from the LLM module, each of the plurality of working targets being represented by an NL, the method further comprising: For each of at least two of the plurality of work objectives: Provide the LLM module with a corresponding NL query, the corresponding NL query including an NL description of at least one aspect of the environment, the work objective expressed in NL, an NL description of the instruction set executable by the robot system, and a corresponding NL request for a corresponding task plan for completing the work objective; and The corresponding task plan is received from the LLM module, and the corresponding task plan is expressed in NL.

8. The method according to claim 7, further comprising: Choose which corresponding task plan to execute first; and Execute the selected task plan.

9. The method according to claim 7, further comprising: Send a request to the operator of the robot system to determine which corresponding task plan the robot system should execute first; Receive the selection of the corresponding task plan to be executed first by the robot system; as well as Execute the selected task plan.

10. A robot control module, the robot control module comprising at least one non-transitory processor-readable storage medium storing processor-executable instructions or data, the processor-executable instructions or data causing the robot system to perform the following when executed by at least one processor of the robot system: Sensor data representing information about the environment of the robot body is captured by at least one sensor carried by the robot body of the robot system; The at least one processor generates a natural language (NL) description of at least one aspect of the environment based on the sensor data; Provide a first NL query to the large language model LLM module, the first NL query including an NL description of at least one aspect of the environment, an NL description of a set of instructions that the robot system can execute, and an NL request for at least one work objective; Receive at least a first working objective from the LLM module, the first working objective being expressed in NL; A second NL query is provided to the LLM module, the second NL query including an NL description of at least one aspect of the environment, the first work objective expressed in NL, an NL description of the instruction set that can be executed by the robot system, and an NL request for a first task plan for completing the first work objective; The first task plan is received from the LLM module, and the first task plan is expressed in NL. as well as Complete the first work objective.

11. The robot control module of claim 10, wherein, When the processor executes instructions or data, it also causes the robot system to send a request to the operator of the robot system to confirm that the robot system should complete the first work objective.

12. The robot control module of claim 10, wherein, The processor can execute instructions or data, which, when executed by at least one processor of the robot system, also cause the robot system to execute the first task plan.

13. The robot control module of claim 10, wherein, When the processor can execute instructions or data, it also causes the robot system to send a request to the operator of the robot system to confirm that the robot system should execute the first task plan.

14. The robot control module of claim 13, the processor-executable instructions or data, when executed by at least one processor of the robot system, cause the robot system to receive at least the first work goal from the LLM module, further cause the robot system to receive a plurality of work goals from the LLM module, each respective work goal of the plurality of work goals expressed in NL, and wherein, The processor can execute instructions or data, which, when executed by at least one processor of the robot system, further enable the robot system to: For each of at least two of the plurality of work objectives: The LLM module is provided with a corresponding NL query, which includes an NL description of at least one aspect of the environment, the work objective expressed in NL, an NL description of the instruction set that can be executed by the robot system, and a corresponding NL request for a corresponding task plan for completing the work objective. and The corresponding task plan is received from the LLM module, and the corresponding task plan is expressed in NL.

15. The robot control module of claim 14, wherein, The processor can execute instructions or data, which, when executed by at least one processor of the robot system, also cause the robot system to: Choose which corresponding task schedule to execute first; and Execute the selected task plan.

16. The robot control module of claim 14, wherein, The processor can execute instructions or data, which, when executed by at least one processor of the robot system, also cause the robot system to: Send a request to the operator of the robot system to determine which corresponding task plan the robot system should execute first; Receive the selection of the corresponding task plan to be executed first by the robot system; as well as Execute the selected task plan.

Images