TASK AND MOVEMENT PLANNING: LIMITATIONS DERIVED FROM LANGUAGE MODEL

Abstract

Classical task and motion planning (TAMP) systems are capable of solving complex and wide-ranging tasks by using models of a robot and its environment to explicitly justify both discrete and continuous values ​​in the robotics problem. While such systems are powerful on the set of problems for which they were designed, they are not transferable to novel problems for which their models are not specified. The present disclosure integrates a language model together with a TAMP system for solving novel robotics problems, including the use of the language model to derive constraints on a specified robot goal, which can then be used by the TAMP system to generate a motion plan to achieve the robot goal.

Description

TECHNICAL AREA

[0001] The present disclosure relates to task and motion planning for robotics. BACKGROUND

[0002] The advent of Foundation models trained on internet-scale data has led to unprecedented advances in traditionally difficult tasks in image processing and natural language. Current large-language models (LLMs) and vision-language models (VLMs) are capable of completing text from partial specifications, answering questions about images, and even solving challenging word problems that require reasoning and common sense. This impressive performance has inspired several systems attempting to leverage existing pre-trained models in robotics. Such systems exhibit remarkable flexibility: unlike classical robotics approaches, they are capable of achieving novel goals specified by natural language or images. However, no publicly available Foundation models currently exist that can handle continuous values ​​(e.g.,can directly output joint angles, handles, placements) that are sufficient for the complete control of a robot to interact with the physical world.

[0003] In contrast, classical task and motion planning (TAMP) systems are capable of solving complex and wide-ranging tasks, from setting a dining table to 3D printing complex structures. These systems use planning models of the robot and its environment to explicitly reason for both discrete and continuous values ​​in robotics problems. While such systems are powerful for the set of problems for which they were designed, they are not transferable to novel problems for which their models are not specified. Enabling a TAMP system to solve novel problems often requires manually extending the underlying model, which is cumbersome and not scalable when operating in unstructured human environments.

[0004] There is a need to address these problems and / or other problems associated with the state of the art. For example, there is a need to use language models to derive constraints for a specified robot goal, which can then be used by a TAMP system to generate a motion plan to achieve the robot goal. SUMMARY

[0005] A method, a computer-readable medium, and a system for generating a robot motion plan are disclosed. A natural language prompt describing a robot goal is processed using a language model to generate constraints for a task and motion planning (TAMP) system. The TAMP system then generates a motion plan that respects the constraints and achieves the robot goal. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a method for generating a robot motion plan according to one embodiment. Fig. Figure 2 illustrates a system for generating a robot motion plan according to one embodiment. Fig. Figure 3 illustrates an exemplary implementation of the system. Fig. 2 according to one embodiment. Fig. Figure 4 illustrates a hybrid constraint network according to one embodiment. Fig. Figure 5 illustrates a system that includes a subsystem, as in Fig. 2 shown, according to one embodiment. Fig. Figure 6 illustrates a method of a robot system using a robot motion plan according to one embodiment. Fig. 7A illustrates inference and / or training logic according to at least one embodiment. Fig. 7B illustrates inference and / or training logic according to at least one embodiment. Fig. Figure 8 illustrates the training and use of a neural network according to at least one embodiment. Fig. Figure 9 illustrates an exemplary data center system according to at least one embodiment. DETAILED DESCRIPTION

[0006] Fig. Figure 1 illustrates a method 100 for generating a robot motion plan according to one embodiment. In one embodiment, the method 100 can be performed by a device that may consist of a processing unit, a program, a custom circuit, or a combination thereof. In another embodiment, a system consisting of non-volatile memory containing instructions and one or more processors communicating with the memory can execute the instructions to perform the method 100. In yet another embodiment, a non-volatile, computer-readable medium can store computer instructions which, when executed by one or more processors of a device, cause the device to perform the method 100.

[0007] In process 102, a natural language prompt describing a robot goal is processed using a language model to generate constraints for a task and motion planning (TAMP) system. The natural language prompt refers to text describing the robot goal and is entered in natural language. In one embodiment, the natural language prompt can be entered by a user. In another embodiment, the natural language prompt can refer to a specific (e.g., type of) robot system that is to achieve the robot goal. In various embodiments, the robot system, also referred to herein as a "robot," can be a real-world robot system, such as a robotic arm or an autonomous vehicle, or it can be a virtual robot system, such as a character or other moving object in a video game.

[0008] The robot goal refers to a task that a robot system is to perform. The robot goal can define "what" the robot system should do, such as an end result that the robot system should achieve or a final state for the robot system. The robot goal can, at least partially, exclude details about "how" the robot system can or should achieve the robot goal, such as the steps the robot system must take to achieve the robot goal.

[0009] As mentioned, the natural language prompt describing the robot's goal is processed using a language model to generate constraints for a TAMP system. The language model refers to a machine learning model trained to predict constraints for a TAMP system based on a robot's goal. In one embodiment, the language model can be a large-language model (LLM). In another embodiment, the language model can be a vision-language model (VLM).

[0010] The constraints refer to criteria used to anchor the TAMP system in the creation of a motion plan for achieving the robot's goal. Thus, in one embodiment, the constraints can be generated (i.e., supported by) a vocabulary of the TAMP system. The motion plan, which can also be considered a manipulation plan for a robot interacting with a world, is described in more detail below.

[0011] In one embodiment, the constraints can be on the motion plan. In another embodiment, the constraints can be requirements for the motion plan. In another embodiment, the constraints can define target conditions. For example, the constraints can include at least one continuous constraint on a decision variable to define the target conditions. In another embodiment, the constraints can define a sub-plan (e.g., a sub-motion plan). For example, the constraints can include at least one discrete constraint on an action sequence to specify the sub-plan.

[0012] In one embodiment, the language model can be anchored to a set of attainable actions and a set of attainable literals representing attainable states. In another embodiment, both the set of attainable actions and the set of attainable literals can be anchored to a given initial state. The given initial state can refer to a physical state of the robot system. For example, as a first constraint, the language model can specify a subset of the attainable literals that must hold contiguously to achieve the robot goal described by the natural language prompt, and as a second constraint, it can specify a subplan consisting of a subset of the attainable actions that achieve the robot goal described by the natural language prompt.

[0013] In process 104, a motion plan that takes the constraints into account and achieves the robot's goal is generated by the TAMP system. The TAMP system refers to a system (e.g., software and / or hardware) that is preconfigured to provide task and motion planning for at least one type of robot system. The motion plan refers to a definition of one or more steps (e.g., actions, movements, etc.) that the robot system must take to achieve the robot's goal. In one embodiment, the motion plan can define a sequence for the one or more steps. In another embodiment, the motion plan can be generated using a vocabulary of the robot system.

[0014] In one embodiment, the search space of the TAMP system can be restricted (i.e., according to the restrictions) when generating the motion plan to reach the robot's goal. For example, if the restrictions include a subplan, the TAMP system can be restricted to generating a motion plan that includes the subplan as a subsequence. As another example, if the language model predicts a subplan and connecting reachable goal literals, these predictions can be used to "restrict" the TAMP search space, ensuring that it generates a motion plan that reaches the robot's goal.

[0015] In addition to the method 100 described herein, the motion plan can be output to the robot system. In one embodiment, outputting the motion plan to the robot system can cause the robot system to move according to the motion plan in order to achieve the robot goal. For example, the robot system can perform the one or more steps defined by the motion plan to achieve the robot goal.

[0016] In addition to the method described herein, the language model can be iteratively re-queried to refine the constraints. In this embodiment, the refined constraints can be used to anchor (e.g., restrict, etc.) the generation of the motion plan by the TAMP system. In one embodiment, the language model can be re-queried based on a function that tests a set of sampled continuous variables of a domain of constraint parameters. In one embodiment, the function can be generated by the language model or by another language model (e.g., another LLM or VLM). By way of example only, the actions and reachable literals can optionally have undefined continuous constraints that the language model may be required to implement in the form of a test by writing code.Additional embodiments regarding the restriction refinement are described in detail below.

[0017] Further embodiments are now provided in the description of the following figures. It should be noted that the embodiments described herein refer to Method 100 of Fig. 1 disclosed embodiments apply to any of the embodiments of the remaining figures below and / or can be used in combination with them.

[0018] Fig. Figure 2 illustrates a System 200 for generating a robot motion plan according to one embodiment. System 200 can be implemented to perform Method 100 of Fig. 1 in one embodiment. The definitions and embodiments provided above may equally apply to the present description.

[0019] As shown, System 200 includes a Language Model 202. The Language Model 202 can be an LLM or VLM, which may be implemented in software and / or hardware. System 200 also includes a TAMP System 204. The TAMP System 204 may be implemented in software and / or hardware. In one embodiment, the Language Model 202 and the TAMP System 204 may reside on the same computing system. In another embodiment, the Language Model 202 and the TAMP System 204 may reside on different computing systems and may communicate over a network (i.e., exchange input and / or output data), as described herein.

[0020] The language model 202 is configured to process a natural language prompt describing a robot goal in order to generate constraints for the TAMP system 204. The TAMP system 204 is configured to generate a motion plan that takes the constraints into account and achieves the robot goal.

[0021] The present System 200 combines the complementary advantages of a language model (e.g., Foundation model) 202 and a TAMP system 200 to handle wide-ranging manipulation tasks that are open-world, i.e., where the vocabulary of targets is unlimited. In particular, the robot target can be specified in natural language, which may include concepts that the underlying TAMP system 204 does not have built in, but which can be achieved by chaining together robot movement primitives that the TAMP system 204 possesses.

[0022] As an example, a TAMP system 204, capable of performing pick-up and place tasks, expects goals in the form of logical expressions containing predicates such as On(apple, plate). However, a natural language request stating "Place the orange on the table where the apple is initially located" cannot be expressed in terms of On, and thus there would be no way for the TAMP system 204 to solve it, although it could be accomplished through a sequence of pick-up and place primitives. A pure language model 202-based system would also struggle with this task, as it would not only have to predict that the apple needs to be moved out of the way before the orange can be placed, but also predict the continuous robot movements required to accomplish this.

[0023] However, in the present system 200, the discrete continuous planning of the TAMP system 204 and the common-sense reasoning of the language model 202 are integrated through the contract of constraints. In particular, the language model 202 is capable of mapping a very wide range of open-world expressions into discrete action sequences (e.g., that a potato must be cooked before it can be served) and code that represents continuous constraints on key decision variables (e.g., valid poses of the egg so that it is in an oven). These constraints can be easily integrated into existing constraints (e.g., avoiding collisions, considering kinematics) within the (e.g., commercially available) TAMP system 204.Thus, the overall system 200 is able to generate solutions that not only take into account limitations derived from the robot's open-world goal, but are also physically feasible on the robot system.

[0024] The present System 200 can be described as OWL-TAMP (open-world language-based TAMP), which integrates open-world concepts into the TAMP System 204 with traditional robot operations and constraints via constraint generation. Implementations of this framework, as described in more detail below, may include: (1) a method for generating constraints on action sequences to specify subplans with language descriptions; (2) a method for generating constraints on continuous variables affected within the subplan of (1); and (3) combining both (1) and (2) within the TAMP System 204. The System 200 enables a robot system, whether a real robot or a virtual robot, to perform complex, wide-ranging manipulation tasks specified by language directly from sensor input. Problem structure

[0025] A model-based mixed discrete continuous planning approach is applied to control a robot to solve open-world tasks. A planning model of the TAMP system 204 is used, which contains common manipulation primitives applicable across a very wide range of tasks, and a language model 202 (which may also be referred to herein as VLM) is used to extend the planning model to consider novel, task-specific dynamics and constraints.

[0026] The underlying planning model is configured to capture generic dynamics and constraints (e.g., kinematic constraints and reachability, collision constraints) that apply to any task a robot might face, while the language model 202 is configured to provide additional task-specific constraints (e.g., that an object must be placed in a pan in order for it to be "cooked", that serving coffee in a cup requires the cup to be upright) that serve to specialize the planning model for the given situation.

[0027] In one embodiment, System 200 can be modeled using a Planning Domain Definition Language (PDDL) in the style of a factorized action language, representing states and actions in the form of predicates. However, System 200 is not limited to this representative choice and can also be implemented using any of several different planning frameworks, such as PDDLStream and SeSaME. In PDDL, state variables are represented as literals, true or false evaluations of predicates for specific values ​​of their parameters.

[0028] In the following description, a single robot operating in a simplified manipulation domain is used as an educational walking example. Since robotics inherently involves continuous values, both discrete and continuous parameter types are considered, namely: obj - a discrete, manipulable object o, conf - a continuous robot configuration q ∈ R d , traj - a continuous robot trajectory consisting of a sequence n of configurations τ ∈ R nd consists, grasp - a continuous object-grasping pose g ∈ SE(3), and pose - a continuous object placement pose p ∈ SE(3). The fluid predicates, i.e., predicates with truth values ​​that can change over time, are: AtConf(q: conf) - the robot is currently in configuration q, HandEmpty() - the robot's hand is currently empty, AtPose(o: obj, p: pose) - Object o is currently in placement pose p, and AtGrasp(o: obj, g: grasp) - Object o is currently being captured with the capture pose g.

[0029] These predicates can be used to describe states that are represented by proper literals. For example, the initial state in a domain with a single object apple could be: s0 = [AtConf(q0), HandEmpty(), AtPose(apple, p0), ...].

[0030] Parameterized actions that the robot can apply to influence a change in a state are defined by a name, a list of typed parameters, a list of static literal constraints (con) that the parameters must satisfy, a list of flowing literal preconditions (pre) that must be true before applying the action, and a list of flowing literal effects (eff) that apply to the state after applying the action. The actions "move" (Example 1) and "attach" (Example 2) model the robot moving between two configurations and attaching an object to itself, for example, by grasping it. Example 1 Example 2

[0031] attach(o : obj, p : pose, g : grasp, q : conf) con: [Kin(q, o, g, p)] pre: [AtPose(o, p), HandEmpty(), AtConf(q)] eff: [AtGrasp(o, g), ¬AtPose(o, p), ¬HandEmpty()]

[0032] Anchoring action instances of these parameterized actions must satisfy the following static predicates: Motion(q1: conf, τ : traj, q2: conf) - τ is a valid trajectory connecting configurations q1 and q2, and Kin(q: conf, o: obj, g: grasp, p: placement) - configuration q satisfies a kinematic constraint with placement pose p when object o is grasped with grasp pose g. Predicates and actions of the open world

[0033] A small and finite set of traditional TAMP predicates and actions was described above. These correspond to generic dynamics and constraints that a robot encounters in the physical world due to its design. However, as also described above, System 200 is configured to model and plan using open-world concepts that are environment- or task-specific. To support open-world concepts, selected predicates and actions are parameterized with an additional type, a description d. Descriptions modify the semantics of predicates and actions to accommodate an open-world natural language instruction. Descriptions help to specialize the overly general robot interactions (e.g., moving without collision, stable grasping) within the traditional planning model of TAMP System 204 to achieve novel outcomes.Overall, this strategy can be viewed as bootstrapping an unlimited set of predicates and actions from a finite set by using the language itself as a parameter.

[0034] Consider the VLMPose(d: description, o: obj, p: pose) constraint, which is true if the object o, when placed p, satisfies the description d. Some example descriptions d are: "Orange in the middle of the table," "Orange at the starting location of the apple," and "Orange as far away from the robot as possible." Using this constraint, a release action is formulated (Example 3) that involves the robot releasing the object o according to the description d. This could be placing the object on a surface, stacking the object on top of another object, dropping the object into a container, inserting the object into an outlet, etc. Example 3

[0035] detach(d : description, o : obj, g : grasp, p : pose, q : conf) con: [Kin(q, o, g, p), VLMPose(d, o, p)] pre: [AtPose(o, p), HandEmpty(), AtConf(q), ¬ o', p'. AtPose(o', p') Λ Collision(o, p, o', p')] eff: [AtGrasp(o, g), ¬AtPose(o, p), ¬HandEmpty()]

[0036] Additional parameterized actions that model different types of interaction can also be defined, such as an action that moves a cup through waypoints to fill it or pour it out.

[0037] System 200 enables planning with both traditional robot constraints and task-specific open-world limitations. The problem is considered in Fig. 3, where the goal is to “place the orange on the table where the apple is initially located”. Fig. Figure 4 (left) shows the simplified constraint network, a two-part graph of free action parameters (bold) to the action constraints in which they are involved (conf), induced by a plan that directly picks up and places the apple: π=[…,attach(apple,p0A,g,q1),…,unlock("where the green block is",apple,p*A,g,q2)].

[0038] This constraint network is unsatisfactory because the VLMPose constraint limits the set of placements that will fulfill the task, and the collision constraint prevents unsafe placements. However, by using the TAMP System 200, this approach can work backward through candidate plans that first move the apple to eventually find a satisfactory constraint network and ultimately a solution. TAMP with open world concepts

[0039] System 200 addresses TAMP problems 〈s0,A,g〉, which are described by an initial state s0, a set of parameterized actions A, and a goal g. Unlike traditional TAMP problems, the goal g is not a logical formula over literals, but rather a goal description provided in natural language text (e.g., English). Therefore, solving such problems requires translating g into a form that can be used within TAMP System 204.

[0040] One approach to this translation would be to directly request a VLM to output a logical formula about literals (which we will call G) from the goal description g. Given this, one could simply call an off-the-shelf TAMP system to achieve G. While this approach is simple and powerful, it is limited in the types of tasks it is able to express in at least two ways: (1) it can only define a goal state to be achieved and cannot specify intermediate behaviors or states that must occur prior to the goal, and (2) it can only express goals in terms of predicates that are already built into the TAMP system 204.

[0041] Consider a TAMP system capable of solving generalized reordering problems that include the predicate: Support(o1, o2), where Support corresponds to o1 being either on or inside o2. Suppose the goal description is provided: "Cook the strawberry by placing it in the pan, then finally serve it in the bowl." The correct goal translation would be Support(strawberry, bowl), but this does not capture the fact that the strawberry must first be placed in the pan. Now, suppose the goal description: "Can you put the mug on the table so I can pour coffee into it properly?" is provided separately. The TAMP system 204 has no predicate corresponding to Upright(o1): the closest possible translation would be Support(mug, table), which does not fully capture the intent of the goal description (and is also true in the initial state).

[0042] System 200 addresses these limitations in the expressiveness of direct translation by instead translating g into more flexible discrete and continuous constraints (as in Fig. (3 shown). In particular, the language model 202 is first queried to provide a set of discrete constraints on open-world action orders, and then to induce continuous constraints in the form of code for certain predicates (such as VLMPose) that appear in the effects or constraints of action definitions used as part of our first stage. These constraints are then integrated into the TAMP system 204 so that it only produces plans that satisfy these constraints. Intuitively, these constraints will be task-specific, enabling the system 200 to accomplish tasks it otherwise could not. Conversely, by using a TAMP system 204, OWL-TAMP inherits theoretical guarantees regarding the non-language model constraints, such as plan reliability, which is critical for safety, and probabilistic completeness.In the cooking task mentioned above, creating a discrete constraint that every valid plan should perform a detach(strawberry, pan) action before a detach(strawberry, bowl) action would suffice to allow the TAMP system 204 to solve the task. Similarly, in the fruit sorting task, only a continuous constraint on the outcome of each detach(fruit) is required for a TAMP system 204 to achieve the underlying goal.

[0043] The following describes the procedure for generating discrete constraints as well as the procedure for generating continuous constraints given initial discrete constraints. Generating discrete planning constraints using a language model 202

[0044] Given a goal description g, the language model is instructed to generate a subplan that serves as a discrete constraint on the space of solutions of the TAMP system 204. To enable this, a natural language description is assigned to each available action for that specific action. Although the language model 202 could be directly instructed for relevant actions and goals without a list of candidates, the language model 202 is likely to be syntactically and semantically inaccurate. Instead, the set of attainable actions A and literals L available to the TAMP system 204 is anchored before the language model 202 is instructed to return values ​​in these sets. Relaxed planning from the initial state s0 can be used to simultaneously anchor and explore the sets of attainable actions A and literals L. Similarly, placeholders for descriptive parameters can be used. Algorithm 1 represents the subplan generation pseudocode of language model 202.

[0045] It takes a TAMP problem 〈s0, A, g〉, where g is a textual goal description. It first anchors the set of actions A reachable by s0 using GROUND-ACTIONS. Then it accumulates the set of reachable literals L by taking the effects of all actions A. These sets can be filtered by action or predicate type if desired to focus the support of Language Model 202 on specific aspects of the planning problem. It then requests QUERY-VLM for a subplan [a1, ..., a n , l1, ..., l k ] using actions a i ∈ A and target literals l j ∈ L mon, which achieve the target description g. Importantly, the language model 202 populates the description parameter d for each of these actions. The original TAMP problem is then transformed to force solutions that allow the subplan as a subsequence. In particular, a predicate EXECUTED is created that models whether the i-th action in the plan has been executed, and EXECUTED is added to the effects of action a. i and the preconditions of the action a i+1 added. Finally, the planning goal is defined as G = {l i , ..., l m} ⊆ L and EXECUTED(n) is defined, which indicates that all actions have been performed and the transformed TAMP problem is solved using a generic TAMP algorithm of the TAMP system 204.

[0046] Consider the cooking problem mentioned above, where g = "Cook the strawberry by placing it in the pan, then finally serve it in the bowl." Assume that language model 202 does not return a target literal, but only the subplan: π→ π→=[del¨sen(make sure the apple is firmly in the frying pan, apple,…),del¨sen("place on the plate", apple,…),…].

[0047] Although the VLM plan π→ If the task intent is captured (i.e., to place the apple in the pan before serving), the plan is not valid because the object must be picked up with the detach action before it can be detached. Fortunately, the underlying TAMP system 204 models this, so providing this subplan, along with the generated Executed predicates, to the TAMP system 204 results in the TAMP system 204 generating valid plans that are at least 8 actions long. Anchoring continuous constraints with a language model

[0048] The embodiments described above generate actions with fully specified language parameters. However, to apply these actions correctly, the way in which the language parameter should influence the legal action parameter values ​​must be interpreted. In particular, an implementation can be provided for all constraint streams (such as the flowing VLMPose(d, o, p) introduced above) that use the language description d.

[0049] Consider, for example, the coffee task (i.e., if g = "Can you place the cup stably on the table so I can pour coffee into it correctly?"), and assume that the discrete generation process has generated a plan containing the following action: detach ("Place the cup stably on the table, making sure it is upright and positioned to receive the coffee," cup, ...). To implement this action correctly, it must be ensured that the placement pose p of the detach action obeys the description d of being "stably on the table and upright." For this purpose, language model 202, or another language model 206, is queried to generate code to directly implement a test on the pose p that outputs a Boolean value (and can thus be used as part of the VLMPose), as shown in Example 4. Example 4

[0050] def test_poses(p) -> bool: ontop_table_bounds = modify_pose_bounds_to_be_ontop _of_object('Becher', 'Tisch') mug_on_table = position_within_bounds(mug.pose, ontop_table_bounds) upright_orientation = abs(mug.pose.roll) < 0,1 und abs(mug.pose.pitch) < 0,1 return mug_on_table und upright_orientation

[0051] Given such a function, the VLMPose(d, o, p) predicate can be implemented by simply calling this function and passing the pose p, where the cup object is placed, as a parameter. The description d is passed to language model 202 or 206 to generate this function. Given this implementation on VLMPose, the TAMP system 204 is restricted to solutions that accommodate this continuous constraint according to the intent of the task. Although the description here focuses on Boolean functions as action constraints, this approach can also be applied to non-negative functions as action costs, for example, to minimize the distance from a placement to a table edge.

[0052] In one embodiment, the language model 202 can also output continuous constraints corresponding to the target description g itself, and these can then be used to output constraints for each of the discrete actions. Its output is then input by this step as part of the prompts for it, in order to output constraints for each other action with description d and a constraint stream that requires an implementation of the language model 202.

[0053] Fig. Figure 5 illustrates a System 500 that includes a subsystem, as in Fig. Figure 2 shows one embodiment. System 200 can be implemented to carry out Method 100 of Fig. 1 in one embodiment. The definitions and embodiments provided above may equally apply to the present description.

[0054] As above in relation to Fig. As described in section 2, the present system 500 includes, as a subsystem, both a language model 202 and a TAMP system 204. The language model 202 is configured to process a natural language prompt describing a robot goal in order to generate constraints for the TAMP system 204. The TAMP system 204 is configured to generate a motion plan that takes the constraints into account and achieves the robot goal.

[0055] The present system 500 also includes a robot system 502. The robot system 502 can be implemented in software and / or hardware. The language model 202 and / or the TAMP system 204 can be implemented as components of the robot system 502. For example, the language model 202 and / or the TAMP system 204 can be implemented within a computer system of the robot system 502. In another embodiment, the language model 202 and / or the TAMP system 204 can reside on a different computing system than the robot system 502, in which case such a computing system and the robot system 502 communicate via a network (i.e., exchange input and / or output data), as described herein.

[0056] The robot system 502 is configured to move according to the motion plan to achieve the robot goal. In one embodiment, the robot system 502 can be a real robot system moving in the real world according to the motion plan to achieve the robot goal. For example, the real robot system 502 can be a robot arm (e.g., an articulated arm) that moves, grasps real objects, transports real objects, repositions real objects, etc., according to the motion plan. As another example, the real robot system 502 can be an autonomously driving vehicle moving in the real world according to the motion plan (e.g., accelerating, decelerating, stopping, turning, changing lanes, etc.).

[0057] In another embodiment, the robot system 502 can be a virtual robot system that moves in a virtual world according to the motion plan to achieve the robot's goal. For example, the virtual robot system 502 can be a character or other movable object in an application that moves according to the motion plan (e.g., as depicted in a user interface). The application can be a video game, a virtual reality application, an augmented reality application, a simulation application, etc.

[0058] Fig. Figure 6 illustrates a method 600 of a robot system using a robot motion plan according to one embodiment. The method 600 can be performed by the robot system 502 of Fig. 5. Again, the definitions and embodiments provided above can apply equally to the present description.

[0059] In process 602, a movement plan is received. The movement plan can be processed according to procedure 100 of Fig. 1 and / or through the system 200 of Fig. 2. In the present embodiment, the motion plan consists of a sequence of steps to be executed (e.g., performed, etc.) by the robot system. In operation 604, a first step in the motion plan is executed (e.g., performed, etc.). In decision 606, it is determined whether the motion plan includes a next step. If it is determined that the motion plan includes a next step, then the next step is executed in operation 608, and procedure 600 then returns to decision 606. As soon as it is determined that the motion plan does not include a next step, then procedure 600 ends. Machine learning

[0060] Deep neural networks (DNNs), including deep learning models developed on processors, have been used for a variety of applications, from self-driving cars to faster drug development, from automatic image labeling in online image databases to intelligent real-time speech translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, continuously learning, becoming increasingly intelligent, and delivering faster and more accurate results over time. A child is initially taught by an adult to correctly identify and classify different shapes, eventually becoming able to identify shapes without coaching.Similarly, a deep learning or neural learning system must be trained in object recognition and classification to become more intelligent and efficient at identifying basic objects, hidden objects, etc., while also assigning context to objects.

[0061] At the simplest level, neurons in the human brain consider various inputs they receive, assign importance levels to each input, and pass the output to other neurons to respond to. An artificial neuron, or perceptron, is the most basic model of a neural network. For example, a perceptron might receive one or more inputs representing different features of an object that the perceptron is trained to recognize and classify. Each of these features is assigned a specific weight based on its importance in defining the object's shape.

[0062] A deep neural network (DNN) model comprises multiple layers of many interconnected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained on enormous amounts of input data to solve complex problems quickly and with high accuracy. In one example, a first layer of the DNN model decomposes an input image of a car into different sections and looks for basic patterns such as lines and angles. The second layer assembles the lines to search for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final few layers generate a label for the input image that identifies the model of a specific car brand.

[0063] Once the DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited in ATMs, identifying pictures of friends in photos, providing movie recommendations to over fifty million users, identifying and classifying different types of automobiles, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0064] During training, data flows through the DNN in a forward propagation phase until a prediction is generated that displays a label matching the input. If the neural network does not correctly label the input, errors between the correct label and the predicted label are analyzed, and the weights for each feature are adjusted during a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset. Training complex neural networks requires massive amounts of parallel computing power, including floating-point multiplication and addition.Inference is less computationally intensive than training because it is a latency-sensitive process in which a trained neural network is applied to new inputs it has not seen before to classify images, translate language, and generally derive new information. INFERENCE AND TRAINING LOGIC

[0065] As noted above, a deep learning or neural learning system must be trained to generate inferences from input data. Details regarding inference and / or training logic 715 for a deep learning or neural learning system are given below in conjunction with Fig. 7A and / or Fig. 7B provided.

[0066] In at least one embodiment, the inference and / or training logic 715 can, without limitation, include a data store 701 to store forward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the data store 701 stores weight parameters and / or input / output data of each layer of a neural network that are trained or used in conjunction with one or more embodiments during the forward propagation of input / output data, and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, any part of the data memory 701 can be contained in another on-chip or off-chip data memory, including the L1, L2 or L3 cache or system memory of a processor.

[0067] In at least one embodiment, any part of the data memory 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the data memory 701 can be a cache memory, a dynamic directly addressable memory (“DRAM”), a static directly addressable memory (“SRAM”), a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the choice of whether the data memory 701 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, flash, or another type of memory, can depend on available on-chip memory versus off-chip memory, latency requirements of training and / or inference functions being performed, batch size of data used in the inference and / or training of a neural network, or a combination of these factors.

[0068] In at least one embodiment, the inference and / or training logic 715 can, without limitation, include a data store 705 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the data store 705 stores weight parameters and / or input / output data of each layer of a neural network that are trained or used in conjunction with one or more embodiments during the backward propagation of input / output data, and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, any part of the data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor. In at least one embodiment, any part of the data memory 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the data memory 705 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory.In at least one embodiment, the choice of whether the data storage 705 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, Flash or another type of memory, may depend on available on-chip memory versus off-chip memory, latency requirements of training and / or inference functions being performed, batch size of data being used in inference and / or training of a neural network, or a combination of these factors.

[0069] In at least one embodiment, the data memory 701 and the data memory 705 can be separate memory structures. In at least one embodiment, the data memory 701 and the data memory 705 can be the same memory structure. In at least one embodiment, the data memory 701 and the data memory 705 can be partly the same memory structure and partly separate memory structures. In at least one embodiment, any part of the data memory 701 and the data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.

[0070] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include one or more arithmetic logic unit(s) (“ALU(s)”) 710 to perform logical and / or mathematical operations that are at least partially based on or specified by training and / or inference code, the result of which may lead to activations (e.g., output values ​​of layers or neurons within a neural network) stored in an activation memory 720, which are functions of input / output and / or weighting parameter data stored in the data memory 701 and / or the data memory 705.In at least one embodiment, activations stored in the activation memory 720 are generated according to linear algebraic and / or matrix-based mathematics, performed by the ALU(s) 710 in response to the execution of instructions or other code, wherein weight values ​​stored in the data memory 705 and / or the data 701 are used as operands together with other values, such as deviation values, gradient information, pulse values, or other parameters or hyperparameters, any or all of which may be stored in the data memory 705 or the data memory 701 or in another memory on or off the chip.In at least one embodiment, the ALU(s) 710 is / are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, the ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses it (e.g., a coprocessor). In at least one embodiment, the ALU(s) 710 may be contained within the execution units of a processor or otherwise in a bank of ALUs that can be accessed by the execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed-function units, etc.).In at least one embodiment, the data memory 701, the data memory 705, and the activation memory 720 can be located on the same processor or other hardware logic device or circuit, whereas in another embodiment, they can be located on different processors or other hardware logic devices or circuits, or a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any part of the activation memory 720 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, inference and / or training code can be stored with other code that a processor or other hardware logic or circuitry can access and that is retrieved and / or processed using the fetch, decode, schedule, execute, retract and / or other logical circuitry of a processor.

[0071] In at least one embodiment, the activation memory 720 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 720 can be located wholly or partially inside or external to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 720 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, flash, or another type of memory, can depend on available on-chip memory versus off-chip memory, the latency requirements of training and / or inference functions being performed, the batch size of data used in the inference and / or training of a neural network, or a combination of these factors.In at least one embodiment, the inference and / or training logic 715, which is described in . Fig. Figure 7A illustrates the use of the inference and / or training logic 715 in conjunction with an application-specific integrated circuit (“ASIC”), such as Google’s Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, the inference and / or training logic 715, which is illustrated in Figure 7A, can be used ... Fig. 7A illustrates how they can be used in conjunction with hardware of a central processing unit (“CPU”), hardware of a graphics processing unit (“GPU”) or other hardware, such as field-programmable gate arrays (“FPGAs”).

[0072] Fig. Figure 7B illustrates inference and / or training logic 715 according to at least one embodiment. In at least one embodiment, the inference and / or training logic 715 can, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the inference and / or training logic 715, which is described in Fig. Figure 7B illustrates this, in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the inference and / or training logic 715, which is illustrated in Figure 7B, can be used ... Fig. Figure 7B illustrates the use of the inference and / or training logic 715 in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, the data memory 701 and the data memory 705, which can be used to store weight values ​​and / or other information, including deviation values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment, which is illustrated in Fig. As illustrated in Figure 7B, each of the data storage 701 and the data storage 705 is assigned to a dedicated computing resource, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, each of the computing hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the data storage 701 and the data storage 705, respectively, the result of which is stored in the activation memory 720.

[0073] In at least one embodiment, each of the data storage devices 701 and 705 and the corresponding computing hardware 702 and 706, respectively, corresponds to different layers of a neural network, such that the resulting activation from one "memory / computing pair 701 / 702" of the data storage device 701 and the computing hardware 702 is provided as an input for the next "memory / computing pair 705 / 706" of the data storage device 705 and the computing hardware 706, in order to mirror the conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 701 / 702 and 705 / 706 can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) may be included after or in parallel to the memory / computing pairs 701 / 702 and 705 / 706 in the inference and / or training logic 715. TRAINING AND USE OF A NEURAL NETWORK

[0074] Fig. Figure 8 illustrates another embodiment for training and deploying a deep neural network. In at least one embodiment, the untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is a PyTorch framework, whereas in other embodiments, the training framework 804 is a TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 804 trains an untrained neural network 806 and enables it to be trained using processing resources described herein to generate a trained neural network 808. In at least one embodiment, weights can be selected randomly or by pretraining using a deep belief network.In at least one embodiment, the training can be carried out in a supervised, partially supervised or unsupervised manner.

[0075] In at least one embodiment, the untrained neural network 806 is trained using supervised learning, wherein the training dataset 802 contains an input paired with a desired output for an input, or wherein the training dataset 802 contains an input with a known output and the output of the neural network is manually graded. In at least one embodiment, the untrained neural network 806 is trained in a supervised manner, processes inputs from the training dataset 802, and compares resulting outputs with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back by the untrained neural network 806. In at least one embodiment, the training framework 804 adjusts weights that control the untrained neural network 806.In at least one embodiment, the training framework 804 includes tools to monitor how well the untrained neural network 806 converges to a model, such as the trained neural network 808, that is suitable for generating correct answers, as in result 814, based on known input data, such as new data 812. In at least one embodiment, the training framework 804 repeatedly trains the untrained neural network 806 while adjusting weights to refine an output of the untrained neural network 806 using a loss function and an adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 achieves a desired accuracy.In at least one embodiment, the trained neural network 808 can then be used to implement any number of machine learning operations.

[0076] In at least one embodiment, the untrained neural network 806 is trained using unsupervised learning, wherein the untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, the training dataset 802 for unsupervised learning contains input data without associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 806 can learn groupings within the training dataset 802 and can determine how individual inputs relate to the untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing map, which is a type of trained neural network 808 capable of performing operations useful in reducing the dimensionality of new data 812.In at least one embodiment, unsupervised training can also be used to perform anomaly detection, enabling the identification of data points in a new data set 812 that deviate from normal patterns of the new data set 812.

[0077] In at least one embodiment, semi-supervised learning can be used, which is a technique in which the training dataset 802 contains a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 804 can be used to perform incremental learning, such as through transferred learning techniques. In at least one embodiment, incremental learning enables the trained neural network 808 to adapt to new data 812 without forgetting knowledge that was embedded in the network during the initial training. DATA CENTER

[0078] Fig. Figure 9 illustrates an exemplary data center 900 in which at least one embodiment can be used. In at least one embodiment, the data center 900 includes a data center infrastructure layer 910, a framework layer 920, a software layer 930, and an application layer 940.

[0079] In at least one embodiment, as in Fig. As shown in Figure 9, the data center infrastructure layer 910 can include a resource orchestrator 912, clustered compute resources 914, and node compute resources (“node CRs”) 916(1)-916(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 916(1)-916(N) can include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units, etc.), storage devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output devices (“NW I / O” devices), network switches, virtual machines (“VMs”), power modules, and cooling modules. In at least one embodiment, one or more node CRs can be composed of the node CRss 916(1)-916(N) be a server that has one or more of the above-mentioned computing resources.

[0080] In at least one embodiment, the grouped compute resources 914 can include separate groupings of node CRs located in one or more racks (not shown) or many racks located in data centers at different geographic locations (also not shown). Separate groupings of node CRs within the grouped compute resources 914 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs containing CPUs or processors can be grouped in one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks can also contain any number of power modules, cooling modules and network switches in any combination.

[0081] In at least one embodiment, the resource orchestrator 922 can configure or otherwise control one or more node CRs 916(1)-916(N) and / or grouped computing resources 914. In at least one embodiment, the resource orchestrator 922 can include a software design infrastructure (“SDI”) management entity for the data center 900. In at least one embodiment, the resource orchestrator can include hardware, software, or a combination thereof.

[0082] In at least one embodiment, as in Fig. As shown in Figure 9, the framework layer 920 includes a job scheduler 932, a configuration manager 934, a resource manager 936, and a distributed file system 938. In at least one embodiment, the framework layer 920 can include a framework to support software 932 of software layer 930 and / or one or more application(s) 942 of application layer 940. In at least one embodiment, software 932 or application(s) 942 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 920 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which can, but is not limited to, using the distributed file system 938 for large-scale data processing (e.g., "Big Data").In at least one embodiment, the job scheduler 932 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 900. In at least one embodiment, the configuration manager 934 can be able to configure different layers, such as the software layer 930 and the framework layer 920, which include Spark and the distributed file system 938, to support data processing at scale. In at least one embodiment, the resource manager 936 can be able to manage clustered or grouped compute resources that are mapped to or allocated to support the distributed file system 938 and the job scheduler 932. In at least one embodiment, clustered or grouped compute resources can include the grouped compute resource 914 on the data center infrastructure layer 910.In at least one embodiment, the resource manager 936 can coordinate with the resource orchestrator 912 to manage these mapped or allocated computing resources.

[0083] In at least one embodiment, the software 932 contained in software layer 930 may include software used by at least parts of the node CRs 916(1)-916(N), the grouped computing resources 914, and / or the distributed file system 938 of framework layer 920. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0084] In at least one embodiment, the application(s) 942 contained in the application layer 940 may include one or more types of applications used by at least parts of the node CRs 916(1)-916(N), the grouped compute resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include any number of a genomic application, a cognitive compute application, and a machine learning application, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0085] In at least one embodiment, any configuration manager 934, resource manager 936, and resource orchestrator 912 can implement any number and any type of self-modifying actions based on any set and any type of data acquired in any technically feasible way. In at least one embodiment, self-modifying actions can relieve a data center operator of data center 900 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly performing parts of a data center.

[0086] In at least one embodiment, the Data Center 900 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or deriving information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weighting parameters according to a neural network architecture using software and computing resources described above in relation to the Data Center 900.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to derive or predict information using resources described above in relation to the Computing Center 900, by using weighting parameters calculated by one or more training techniques described herein.

[0087] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0088] The inference and / or training logic 615 is used to perform inference and / or training operations that are associated with one or more embodiments. In at least one embodiment, the inference and / or training logic 615 can be implemented in system Fig. 9. can be used to infer or predict processes that are based at least partially on weighting parameters calculated using neural network training processes, neural network functions and / or architectures, or neural network use cases described herein.

[0089] As described herein, a method, a computer-readable medium, and a system for generating a robot motion plan are disclosed. According to the Fig. Embodiments 1-6 can provide machine learning models that can be used to perform inference operations and provide inferred data. The machine learning models can be stored (partially or completely) in one or both of the data stores 701 and 705 in the inference and / or training logic 715, as shown in the Fig. 7A and Fig. 7B is shown. The training and deployment of the machine learning models can be done as in Fig. The distribution of the machine learning models can be carried out using one or more servers in a 900 data center, as shown in section 8 and described herein. Fig. 9 shown and described herein.

Claims

[1] Procedure, encompassing: in a device: Processing a natural request describing a robot goal, using a language model to generate constraints for a task and motion planning system, TAMP system; and Generating a movement plan using the TAMP system that takes the constraints into account and achieves the robot's goal. [2] Method according to claim 1, wherein the language model is a Large Language Model, LLM. [3] Method according to claim 1 or 2, wherein the language model is a Vision-Language Model, VLM. [4] Method according to any of the preceding claims, wherein the restrictions are generated in a vocabulary of the TAMP system. [5] Method according to any of the preceding claims, wherein the restrictions define target conditions. [6] Method according to claim 5, wherein the constraints include at least one continuous constraint over a decision variable to define the target conditions. [7] Method according to any of the preceding claims, wherein the restrictions define a sub-plan. [8] Method according to claim 7, wherein the restrictions include at least one discrete restriction over an action sequence to specify the subplan. [9] Method according to any of the preceding claims, wherein the language model is anchored with a set of attainable actions and a set of attainable literals representing attainable states. [10] Method according to claim 9, wherein the set of attainable actions and the set of attainable literals are both anchored to a given initial state. [11] Method according to claim 10, wherein the language model: As a first of the restrictions, a subset of the attainable literals is determined that must hold together in order to fulfill the robot goal described by the natural language prompt, and As a second of the constraints, a subplan is determined, consisting of a subset of the achievable actions that achieve the robot goal described by the natural language prompt. [12] Method according to claim 11, wherein the TAMP system is restricted to generating a motion plan which includes the subplan as a subsequence. [13] Method according to any one of the preceding claims, further comprising, in the apparatus: Outputting the movement plan to a robot system. [14] Method according to claim 13, wherein the output of the motion plan to the robot system causes the robot system to move according to the motion plan in order to reach the robot goal. [15] Method according to claim 13 or 14, wherein the robot system is a real robot system. [16] Method according to claim 15, wherein the real robot system is a robot arm. [17] Method according to claim 15, wherein the real robot system is an autonomously driving vehicle. [18] Method according to any one of claims 13 to 16, wherein the robot system is a virtual robot system. [19] Method according to claim 18, wherein the robot system is a character or other movable object in a video game. [20] Method according to any one of the preceding claims, further comprising, in the apparatus: Iterative re-inquiry of the language model to refine the constraints. [21] Method according to claim 20, wherein the language model is re-queried based on a function that tests a set of sampled continuous variables of a domain of constraint parameters. [22] Method according to claim 21, wherein the function is generated by the language model or by another language model. [23] System, encompassing: a non-volatile memory containing instructions; and one or more processors in communication with memory, wherein the one or more processors execute the instructions to: Processing a natural language request describing a robot goal, using a language model to generate constraints for a task and motion planning system, TAMP system; and Generating a movement plan using the TAMP system that takes the constraints into account and achieves the robot's goal. [24] System according to claim 23, wherein the language model is one of a Large Language Model, LLM, or a Vision Language Model, VLM, and wherein the constraints are generated in a vocabulary of the TAMP system. [25] System according to claim 23 or 24, wherein the constraints define target conditions and a sub-plan. [26] System according to any one of claims 23 to 25, wherein the one or more processors further execute the instructions to: Outputting the motion plan to a robot system to cause the robot system to move according to the motion plan in order to achieve the robot's goal. [27] Non-volatile, computer-readable medium that stores computer instructions which, when executed by one or more processors of a device, cause the device to: a natural language request describing a robot goal, processed using a language model to generate constraints for a task and motion planning system, TAMP system; and The TAMP system generates a movement plan that takes the constraints into account and achieves the robot's goal. [28] Non-volatile computer-readable medium according to claim 27, wherein the language model is one of a Large Language Model (LLM) or a Vision Language Model (VLM), and wherein the constraints are generated in a vocabulary of the TAMP system. [29] Non-volatile computer-readable medium according to claim 27 or 28, wherein the constraints define target conditions and a sub-plan. [30] Non-volatile computer-readable medium according to any one of claims 27 to 29, wherein the device is further caused to: outputs the movement plan to a robot system to cause the robot system to move according to the movement plan in order to achieve the robot's goal.

Images