Application programming interface for heterogeneous robots through robotic foundation model prompting

Abstract

A method includes receiving, by a server hosting a foundation model and via an application programming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot. The method also includes receiving, by the server and via the API, a query from the robot for instructions to complete a task. The method also includes generating, by the server and using the foundation model, at least one command in response to the query. The at least one command is executable by the robot to complete the task, and the at least one command is generated based on the current set of affordances associated with the robot. The method also includes sending, by the server and via the API, the at least one command to the robot.

Description

APPLICATION PROGRAMMING INTERFACE FOR HETEROGENEOUS ROBOTS THROUGH ROBOTIC FOUNDATION MODEL PROMPTINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of U.S. Provisional Patent Application No. 63 / 579,367, filed August 29, 2023. The contents of which are hereby incorporated by reference in their entirety.BACKGROUND

[0002] Robots can have different form factors and capabilities. Additionally, robots can operate using different sets of commands. As a non-limiting example, a first robot having a first configuration can operate according to a first set of commands, and a second robot having a second configuration can operate according to a second set of commands. In some scenarios, the first robot may not be responsive to the second set of commands, and the second robot may not be responsive to the first set of commands. Thus, challenges can exist for generating commands for different types of robots.SUMMARY

[0003] A cloud-based robotic foundation model can return robot-specific commands for heterogeneous robots based on API prompts from the robots. For example, a particular robot can send a prompt to an API associated with the cloud-based robotic foundation model. The prompt can indicate a current set of affordances associated with the particular robot. Based on the current set of affordances, the cloud-based robotic foundation model can generate robotspecific commands for the particular robot when queried by the particular robot. For example, after sending the current set of affordances to the cloud-based robotic foundation model via the API, the particular robot can send a query to the API associated with the cloud-based robotic foundation model. In response to the query, the cloud-based robotic foundation model can generate one or more robot-specific commands for the particular robot based on the affordances provided by the particular robot. Because the cloud-based robotic foundation model uses the current set of affordances to generate the one or more robot-specific commands for the particular robot, the one or more robot-specific commands can have a command style (e.g., a command format) that is recognizable by the particular robot. As a result, the one or more robot-specific commands are returned to the particular robot as viable actions.

[0004] To illustrate, a first robot may have a movable base and a short arm, and a second robot may have a stationary base and a long arm. Each robot can send a prompt to the API to indicate a current set of affordances. As a non-limiting example, the first robot can send code snippets for moving the base and code snippets for manipulating the short arm, and the second robot can send code snippets for manipulating the long arm. The cloud-based foundation model can generate robot-specific commands for each robot based on the corresponding code snippets. To illustrate, each robot can send a query to the API for instructions to pick up an object. In response to the query from the first robot, the cloud-based foundation model can generate a first set of robot-specific commands for the first robot based on the current set of affordances the first robot sent to the API. For example, the first set of robot-specific commands can include at least one command for moving the base of the first robot close to the object and at least one command for manipulating the short arm to pick up the object. In response to the query from the second robot, the cloud-based foundation model can generate a second set of robot-specific commands for the second robot based on the current set of affordances the second robot sent to the API. For example, the second set of robotspecific commands can include at least one command for manipulating the long arm to pick up the object. However, the second set of robot-specific commands may not include a command to move a base of the second robot because the current set of affordances did not indicate that the second robot can move its base. Because the first and second set of robot-specific commands are tailored to the first and second robots, respectively, based on affordances sent to the API, the syntax of the first set of commands may not be recognizable (e.g., executable) by the second robot, and the syntax of the second set of commands may not be recognizable by the first robot.

[0005] In a first example embodiment, a method includes receiving, by a server hosting a foundation model and via an application programming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot. The method also includes receiving, by the server and via the API, a query from the robot for instructions to complete a task. The method also includes generating, by the server and using the foundation model, at least one command in response to the query. The at least one command is executable by the robot to complete the task, and the at least one command is generated based on the current set of affordances associated with the robot. The method also includes sending, by the server and via the API, the at least one command to the robot.

[0006] In a second example embodiment, a server includes a memory and processor coupled to the memory. The processor is configured to receive, via an applicationprogramming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot. The processor is also configured to receive, via the API, a query from the robot for instructions to complete a task. The processor is also configured to generate, using a foundation model, at least one command in response to the query. The at least one command is executable by the robot to complete the task, and the at least one command is generated based on the current set of affordances associated with the robot. The processor is also configured to send, via the API, the at least one command to the robot.

[0007] In a third example embodiment, a non-transitory computer-readable medium includes instructions that, when executed by a processor within a server, cause the processor to perform operations. The operations include receiving, via an application programming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot. The operations also include receiving, via the API, a query from the robot for instructions to complete a task. The operations also include generation, using a foundation model, at least one command in response to the query. The at least one command is executable by the robot to complete the task, and the at least one command is generated based on the current set of affordances associated with the robot. The operations also include sending, via the API, the at least one command to the robot.

[0008] In a fourth example embodiment, a system may include various means for carrying out each of the operations of the first example embodiment.

[0009] In a fifth example embodiment, a method includes sending, by a robot, a prompt to a server via an application programming interface (API). The prompt indicates a current set of affordances associated with the robot. The method also includes sending, by the robot, a query to the server via the API. The query corresponds to a request for instructions to complete a task. The method also includes receiving, by the robot, at least one command from the server via the API in response to sending the query. The at least one command is generated by the server using a foundation model based on the current set of affordances associated with the robot. The method also includes executing, by the robot, the at least one command to complete the task.

[0010] In a sixth example embodiment, a robot includes a memory and a processor coupled to the memory. The processor is configured to send a prompt to a server via an application programming interface (API). The prompt indicates a current set of affordances associated with the robot. The processor is also configured to send a query to the server via the API. The query corresponds to a request for instructions to complete a task. The processor is also configured to receive at least one command from the server via the API in response tosending the query. The at least one command is generated by the server using a foundation model based on the current set of affordances associated with the robot. The processor is also configured to execute the at least one command to complete the task.

[0011] In a seventh example embodiment, a non-transitory computer-readable medium includes instructions that, when executed by a processor within a robot, cause the processor to perform operations. The operations include sending a prompt to a server via an application programming interface (API). The prompt indicates a current set of affordances associated with the robot. The operations also includes sending a query to the server via the API. The query corresponds to a request for instructions to complete a task. The operations also include receiving at least command from the server via the API in response to sending the query. The at least one command is generated by the server using a foundation model based on the current set of affordances associated with the robot. The operations also include executing the at least one command to complete the task.

[0012] In an eighth example embodiment, a system may include various means for carrying out each of the operations of the fifth example embodiment.

[0013] These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 illustrates an example of a system that enables a cloud-based robot foundation model to return robot-specific commands to a robot based on API prompts from the robot, in accordance with examples described herein.

[0015] Figure 2 illustrates another example of a system that enables a cloud-based robot foundation model to return robot-specific commands to a robot based on API prompts from the robot, in accordance with examples described herein.

[0016] Figure 3 illustrates an example of a system that enables a cloud-based robot foundation model to return robot-specific commands to heterogeneous robots based on API prompts from the robots, in accordance with examples described herein.

[0017] Figure 4 illustrates an example of a system that enables a cloud-based robot foundation model to return robot-specific commands to a robot based on modified API prompts from the robot, in accordance with examples described herein.

[0018] Figure 5 is a diagram illustrating training and inference phases of a machinelearning model, in accordance with examples described herein.

[0019] Figure 6 illustrates a flow chart, in accordance with examples described herein.

[0020] Figure 7 illustrates a flow chart, in accordance with examples described herein.DETAILED DESCRIPTION

[0021] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example,” “exemplary,” and / or “illustrative” is not necessarily to be construed as preferred or advantageous over other embodiments or features unless stated as such. Thus, other embodiments can be utilized and other changes can be made without departing from the scope of the subject matter presented herein.

[0022] Accordingly, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0023] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.

[0024] Particular embodiments are described herein with reference to the drawings. In the description, common features are designated by common reference numbers throughout the drawings. In some figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically distinct, the same reference number is used for each, and the different instances are distinguished by addition of a letter to the reference number. When the features as a group or a type are referred to herein (e.g., when no particular one of the features is being referenced), the reference number is used without a distinguishing letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the distinguishing letter. For example, referring to Figure 1, multiple affordances are illustrated and associated with reference numbers 142 A and 142B. When referring to a particular one of these affordances, such as the affordance142A, the distinguishing letter “A” is used. However, when referring to any arbitrary one of these affordances or to these affordances as a group, the reference number 142 is used without a distinguishing letter.

[0025] Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order. Unless otherwise noted, figures are not drawn to scale.I. Overview

[0026] Robots can utilize a cloud-based application programming interface (API) to receive commands to perform a variety of different functions. However, providing a unified cloud-based API for robotic applications can be relatively difficult because robots come in different shapes and can perform different functions.

[0027] For example, each robot attempting to use the cloud-based API can have different sets of affordances. According to one scenario, a first robot may have an arm with a gripper for manipulation, a second robot may have two arms with grippers for manipulation, and a third robot may not have any arms. According to another scenario, a first robot may be a mobile wheeled robot, a second robot may be a mobile legged robot, and a third robot may be a fixed robot. According to yet another scenario, a first robot may utilize a first type of perception system, a second robot can utilize a second type of perception system, and a third robot may use a third type of perception system.

[0028] In the above scenarios, it may be relatively difficult to provide a detailed or specific cloud-based API that can serve each robot because of the different sets of affordances associated with the robots. As a result, generic APIs that are not particularly expressive are typically provided. As a non-limiting example, the cloud-based API can be used for generic functions, such as object detection, and more expressive functions are controlled using clientside intelligence.

[0029] The techniques described herein enable a cloud-based robotic foundation model to return client-specific commands (e.g., robot-specific commands) to heterogeneous robots based on API prompts from the robots. As used herein, a “foundation model” corresponds to a large machine-learning (ML) model trained on a vast quantity of data at scale such that it can be adapted to a wide range of downstream tasks. In some implementations, a “foundation model” as used herein can correspond to a large language model (LLM). In other implementations, a “foundation model” can correspond to a sequence model that inputs oroutputs sequences of data, such as text streams, audio clips, images, video clips, time-series data, etc. The cloud-based robotic foundation model can be a large, pre-trained robotic foundation model hosted as a cloud service for robots with arbitrary capabilities and affordances. As non-limiting examples of robots with arbitrary capabilities, the cloud-based robotic foundation model can be trained to return client-specific commands for robots having a single arm with a gripper for manipulation, robots having two arms with grippers for manipulation, robots having no arms, mobile robots, wheeled robots, legged robots, fixed robots, etc. As non-limiting examples of robots with arbitrary affordances, the cloud-based robotic foundation model can be trained to return client-specific commands for robots that move according to position-based commands (e.g., based on the Cartesian coordinate system), robots that move according to velocity-based commands, etc. In some implementations, the affordances of a particular robot can be selected from a predefined set of affordances.

[0030] To determine how to communicate viable commands to a particular robot, the cloud-based robotic foundation model can use an API to receive a current set of affordances associated with the particular robot. For example, the particular robot can send a prompt to the API that indicates the current set of affordances associated with the particular robot. In some embodiments, the current set of affordances can include viable code snippets that are actionable by the particular robot and corresponding natural language associated with the code snippets. As a non-limiting example, if the particular robot is a mobile robot, the particular robot can prompt the API by stating “robot.move(l)”. In this example, the viable code snippet (e.g., “robot.move(l)”) can be used by the cloud-based robotic foundation model to determine a viable command style or format for instructing the particular robot to move one meter ahead. Optionally, the prompt can also include a natural language representation of the code snippet. For example, the current set of affordances sent to the API as a prompt can also include the text “#move 1 meter ahead”. Based on the above affordances, the cloud-based robotic foundation model can generate other viable commands for moving the particular robot other distances. For example, the cloud-based robotic foundation model can understand that it can return “robot.move()” as a viable action for the particular robot if the particular robot sends a query to the API to move a particular distance. Additionally, if the robot sends a query for instructions to perform another task that inherently requires the robot to move, the cloud-based robotic foundation model can return “robot.move()” as one of a plurality of commands to perform the other task.

[0031] As another non-limiting example, if the particular robot is a robot having a fixed manipulator arm, the particular robot can prompt the API by stating “#detect apple”, “x, y, z =robot.detect(‘apple’)”, “#pick the detected apple”, and “robot.pick_object(x, y, z)”. In this example, the viable code snippet (e.g., “robot.detect(‘apple’)”) can be used by the cloud-based robotic foundation model to determine a viable command style for instructing the particular robot to detect an apple, as described in the natural language portion (e.g., “#detect apple”) in the current set of affordances. Additionally, in this example, the viable code snippet (e.g., “robot.pick_object(x, y, z)”) can be used by the cloud-based robotic foundation model to determine a viable command style for instructing the fixed manipulator arm of the particular robot to pick up the detected apple, as described in the natural language portion (e.g., “#pick the detected apple”) in the current set of affordances. Thus, based on the above affordances, the cloud-based robotic foundation model can use the viable command styles to generate other viable commands for detecting objects and picking up the detected objects. For example, the cloud-based robotic foundation model can understand that it can return “detect” and “pick object” as viable actions for the particular robot if the particular robot sends queries to the API to detect an object or pick up a detected object, respectively.

[0032] Thus, the techniques described herein provide the particular robot (e.g., a client robot) with the ability to prompt the cloud-based robotic foundation model, via the API, to declare its affordances in its current context. In return, in response to a query, the cloud-based robotic foundation model can generate and return, via the API, viable robot-specific commands to the particular robot based on the declared affordances. Because the commands are generated by the cloud-based robotic foundation model, the API does not require a deep level of integration with the clients (e.g., the robots) to match what it provides to client requirements.

[0033] One benefit to the above-described techniques is the ability for the particular robot to modify its affordances at any time, even dynamically, in a manner that is easy and flexible. For example, the particular robot can send a prompt to the API that indicates a modified current set of affordances associated with the particular robot. The modified current set of affordances can include updated code snippets that are actionable by the particular robot and corresponding natural language associated with the updated code snippets. In some implementations, the particular robot may modify its affordances in response to a system update. For example, if new code is installed on the particular robot that modifies or adds to the affordances of the particular robot, the particular robot may modify its affordances to reflect the new code. In other implementations, the particular robot may modify its affordances in response to operating conditions. As a non-limiting example, if the particular robot can no longer perform a task because of insufficient battery power, the particular robot may modify its affordances to reflect as much. The examples described above are for illustrative purposesand should not be construed as limiting. In other implementations, the particular robot can modify its affordances for other reasons. Based on the modified current set of affordances, the cloud-based robotic foundation model can generate updated or modified robot-specific commands that can be returned as viable actions.

[0034] Additionally, because client-specific (e.g., robot-specific) commands are generated by the cloud-based robotic foundation model, the API can serve heterogeneous robots having different sets of affordances. For example, the API can be used for detailed or specific functions, reducing the need for more expressive functions to be controlled using client-side intelligence. As a result, memory requirements and processing complexity associated with client-side intelligence can be relaxed.II. Example Systems

[0035] Figure 1 illustrates an example of a system 100 that enables a cloud-based robot foundation model to return robot-specific commands to a robot based on API prompts from the robot.

[0036] The system 100 includes a server 110 and a robot 130. The server 110 includes a memory 112, a processor 114 coupled to the memory 112, and an API 116 communicatively coupled to the processor 114. The memory 112 can correspond to a non-transitory computer- readable medium that includes instructions 113 executable by the processor 114 to perform the operations described herein.

[0037] The server 110 can be configured to host a foundation model 120. In the example of Figure 1, the foundation model 120 is hosted by the processor 114. The foundation model 120 can be a large, pre-trained cloud-based robotic foundation model hosted as a cloud service for robots with arbitrary capabilities and affordances. As described below, the foundation model 120 can be used to generate client-specific (e.g., robot-specific) commands that are returned to heterogeneous robots via the API 116.

[0038] The robot 130 includes a memory 132 and a processor 134 coupled to the memory 132. The memory 132 can correspond to a non-transitory computer-readable medium that includes instructions 133 executable by the processor 134 to perform the operations described herein.

[0039] The robot 130 can have unique capabilities that require specific commands to execute functions associated with the capabilities. As a non-limiting example, in one embodiment, the robot 130 can have a single arm with a gripper for manipulation. In this embodiment, to manipulate the gripper of the single arm to perform a specific task (e.g., pick up a particular object), specific commands are required to be provided to the robot 130, asopposed to generic commands. As another non-limiting example, in one embodiment, the robot 130 can be a legged robot. In this embodiment, to perform a specific task (e.g., kick a right leg), specific commands are required to be provided to the robot 130, as opposed to generic commands. Although two possible capabilities of the robot 130 are highlighted above, it should be understood that the above-described capabilities are merely for illustrative purposes and are not intended to be limiting. In other embodiments, the robot 130 can have other unique capabilities and features that require specific commands to viably perform tasks associated with the capabilities and features. As non-limiting examples, the robot 130 may have the capability to bend a particular way in response to receiving a specific command, the robot 130 may have the capability to move its head a particular way in response to receiving a specific command, the robot 130 may have the capability to blink in response to receiving a specific command, etc.

[0040] As described below, the robot 130 can prompt the API 116 of the server 110 to declare affordances 142 associated with the robot 130. Based on the affordances 142 and in response to a query 150, the foundation model 120 can return, via the API 116, one or more specific viable commands 170 for the robot to perform a task associated with the query 150.

[0041] To illustrate, the processor 134 of the robot 130 can be configured to declare a current set of affordances 140 associated with the robot 130. As illustrated in Figure 1, the current set of affordances 140 includes an affordance 142 A and an affordance 142B. Although two affordances 142 are depicted in Figure 1, in other embodiments, the current set of affordances 140 can include additional (or fewer) affordances 142. As a non-limiting example, in one embodiment, the current set of affordances 140 can include thirty (30) affordances 142. As another non-limiting example, in one embodiment, the current set of affordances 140 can include a single affordance 142. In some implementations, the affordances 140 of the robot 130 can be selected from a predefined set of affordances. For example, to declare its affordances 140, the robot 130 can send data (e.g., bits) that indicate which affordances from a predefined set of affordances are applicable to the robot 130.

[0042] The affordance 142 A includes an example command 144 A and can optionally include a natural language description 146 A associated with the example command 144 A. As a non-limiting example, if the robot 130 has a fixed manipulator arm, the example command 144 A can correspond to a command for the fixed manipulator arm to perform a specific task. To illustrate, the example command 144A can state “robot.detect(‘apple’)”. Optionally, the natural language description 146 A associated with the example command 144 A can be included in the affordance 142A and can state “#detect apple”. In this example, the examplecommand 144A is a viable command for the robot 130 to detect an apple. The affordance 142B includes an example command 144B and can optionally include a natural language description 146B associated with the example command 144B. Using the same non-limiting example whereby the robot 130 has a fixed manipulator arm, the example command 144B can state “robot.pick_object(x, y, z)”. Optionally, the natural language description 146B associated with the example command 144B can be included in the affordance 142B and can state “#pick the detected apple”. In this example, the example command 144B is a viable command for the robot 130 to pick up an object.

[0043] It should be understood that the above-described example commands 144 and the accompanying natural language descriptions 146 are for illustrative purposes and should not be construed as limiting. In other embodiments, the example commands 144 and the accompanying natural language descriptions 146 can differ. The example commands 144 can correspond to viable code snippets that can be executed by the processor 134 to initiate performance of the associated tasks indicated in the natural language descriptions 146. Thus, by way of the example commands 144, the affordances 140 can indicate viable command styles (or formats) for commands to perform corresponding tasks.

[0044] The processor 134 of the robot 130 can be configured to generate a prompt 160 that indicates the current set of affordances 140. The robot 130 can be configured to send the prompt 160 to the server 110 via the API 116. As described below, the foundation model 120 can use the current set of affordances 140 indicated in the prompt 160 to generate viable commands 170 for the robot 130 in response to robot queries 150 or queries from a separate user device.

[0045] For example, after the robot 130 sends the prompt 160 to the server 110 via the API 116, the processor 134 of the robot 130 can generate a query 150 associated with one or more tasks 152. The query 150 can include a natural language representation 154 of the one or more tasks 152 to be performed by the robot 130. As a non-limiting example, the query 150 can include the phrase “#pick up a detected chair”. In this example, the phrase “pick up a detected chair” is the natural language representation 154 of two tasks 152 (e.g., (1) detect a chair and (2) pick up the chair). The robot 130 can be configured to send the query 150 to the server 110 via the API 116.

[0046] In response to receiving the query 150, the processor 114 can generate at least one command 170 using the foundation model 120. In particular, the query 150 can be provided to the foundation model 120 to identify the one or more tasks 152 based on the natural language representation 154 of the one or more tasks 152. After identifying the tasks 152associated with the query 150, the foundation model 120 can determine how to present the commands 170 associated with the tasks 152 in a viable format that can be understood by the robot 130. For example, based on the current set of affordances 140 associated with the robot 130, the foundation model 120 can be used to determine a command style (or format) for the at least one command 170. To illustrate, based on the current set of affordances 140 associated with the robot 130, the foundation model 120 can understand that it can return “robot.detect()” and “robot.pick_object()” as viable actions for the robot 130 if the query 150 indicates that an object should be picked up. As a result, the foundation model 120 can generate commands 170 according to the above-mentioned command style to enable the robot 130 to perform the tasks. For example, the commands 170 generated by the foundation model 120 can include “robot.detect(‘ chair’)” and “robot.pick_object(x, y, z)”.

[0047] The server 110 can be configured to send (e.g., return) the commands 170 to the robot 130 via the API 116. In response to receiving the commands 170, the processor 134 can execute the commands 170 to initiate performance of the tasks 152 by the robot 130.

[0048] Thus, the techniques described with respect to Figure 1 enables the server 110 to provide robot-specific commands 170 to the robot 130 based on the current set of affordances 140 associated with the robot 130. For example, the foundation model 120 can be trained to generate robot-specific commands 170 for the robot 130 based on the current set of affordances 140. Because the robot-specific commands 170 are generated by the foundation model 120, the API 116 can serve heterogeneous robots having different sets of affordances. For example, the API 116 can be used for detailed or specific functions, reducing the need for more expressive functions to be controlled using client-side intelligence. As a result, memory requirements and processing complexity associated with client-side intelligence at the robot 130 can be relaxed.

[0049] Figure 2 illustrates another example of a system 200 that enables a cloud-based robot foundation model to return robot-specific commands to a robot based on API prompts from the robot.

[0050] The system 200 includes the foundation model 120, the API 116, and a robot 130A. According to one implementation, the foundation model 120 and the API 116 can be integrated into the server 110, as depicted in Figure 1. The robot 130A in Figure 2 corresponds to a mobile robot that includes a camera. Thus, the robot 130A can have unique capabilities, such as the capability to capture images and the capability to move. According to one implementation, the robot 130A can correspond to the robot 130 of Figure 1.

[0051] In Figure 2, the robot 130A can be configured to prompt 160 A the API 116 to declare affordances 142 associated with the robot 130A. For example, as illustrated in Figure 2, the prompt 160A includes a natural language description 146 that states “#Go To Table” and three example commands 144 (e.g., code snippets) to perform the task indicated in the natural language description 146. In particular, the prompt 160A includes commands 144 that state “IMG = Get_Image()”, “Table = Find(“Table”, IMG)”, and “Robot.Nav(Table)”. It should be understood that the prompt 160A is merely for illustrative purposes and should not be construed as limiting. In other embodiments, the prompt 160 A can include different natural language descriptions 146, different example commands 144, more complex natural language descriptions, more complex commands, etc. As a non-limiting example, in other embodiments, the prompt 160 A can be more generic and can describe an API of the robot 130A. For example, the prompt 160A can state “#Get camera image: image = robot.get_image()” and “Navigate to a location: robot.navigate(location)”. Based on the generic prompt, the foundation model 120 can generate relevant commands.

[0052] The robot 130A can be configured to send the prompt 160 A to the API 116, and the foundation model 120 can access the prompt 160A via the API 116. As described below, the foundation model 120 can use the affordances 140 (e.g., the code snippets and the corresponding natural language) indicated in the prompt 160 A to generate viable robot-specific commands 170A for the robot 130A in response to a query 150A.

[0053] For example, after the robot 130A sends the prompt 160 A to the foundation model 120 via the API 116, the robot 130A can generate a query 150A associated with one or more tasks 152 for the robot 130A to perform. The query 150A can include a natural language representation 154 of the one or more tasks 152 to be performed by the robot 130A. In Figure 2, the query 150A includes the phrase “go to chair”.

[0054] In response to receiving the query 150A via the API 116, the foundation model 120 can generate robot-specific commands 170A for the robot 130A. In particular, the foundation model 120 can determine how to present the commands 170A in a viable format that can be understood by the robot 130A. For example, based on the current set of affordances 140 in the prompt 160 A, the foundation model 120 can determine a command style (or format) for the commands 170A. To illustrate, based on the current set of affordances 140 associated with the robot 130A, the foundation model 120 can understand that it can return “IMG = Get_Image()”, “Object = Find(“Object”, IMG)” and “Robot.Nav()” as viable actions for the robot 130A if the query 150A indicates that the robot 130A is to go to a particular object. As a result, the foundation model 120 can generate commands 170A according to the above-mentioned command style to enable the robot 130A to perform the tasks. For example, the commands 170A generated by the foundation model 120 can include “IMG = Get_Image()”, “Chair = Find(“Chair”, IMG)” and “Robot.Nav(Chair)”.

[0055] The API 116 can be configured to send (e.g., return) the commands 170A to the robot 130A. In response to receiving the commands 170A, the robot 130A can execute the commands 170 A to go to the chair.

[0056] Thus, the techniques described with respect to Figure 2 enables the foundation model 120 and the API 116 to provide robot-specific commands 170A to the robot 130A based on the current set of affordances 140 in the prompt 160 A. For example, the foundation model 120 can be trained to generate robot-specific commands 170A for the robot 130A based on the current set of affordances 140. Because the robot-specific commands 170A are generated by the foundation model 120, the API 116 can serve heterogeneous robots having different sets of affordances. For example, the API 116 can be used for detailed or specific functions, reducing the need for more expressive functions to be controlled using client-side intelligence. As a result, memory requirements and processing complexity associated with client-side intelligence at the robot 130A can be relaxed.

[0057] In some embodiments, the prompt 160 A may corresponds to information from a robot manual. For example, the prompt 160A may include the “GET IMAGE” to return a current image. The prompt 160A may also include “FIND(text_description, image)”, where “text description” is the object to find (e.g., the “table”). As a result, the coordinates of the objects may be returned. The prompt 160A may also include “ROBOT.NAV(coord)” to navigate the robot 130A to the specified coordinates.

[0058] Figure 3 illustrates an example of a system 300 that enables a cloud-based robot foundation model to return robot-specific commands to heterogeneous robots based on API prompts from the robots.

[0059] The system 300 includes the foundation model 120, the API 116, a robot 130B, and a robot 130C. According to one implementation, the foundation model 120 and the API 116 can be integrated into the server 110, as depicted in Figure 1. The robot 130B in Figure 2 corresponds to a mobile robot. Thus, the robot 130B can have unique capabilities, such as the capability to move. According to one implementation, the robot 130B can correspond to the robot 130 of Figure 1. The robot 130C in Figure 2 corresponds to a single arm with a gripper for manipulation. Thus, the robot 130C can have unique capabilities, such as the capability to lift an arm and grip different objects. According to one implementation, the robot 130C can correspond to the robot 130 of Figure 1.

[0060] In Figure 3, the robot 130B can be configured to prompt 160B the API 116 to declare affordances 142 associated with the robot 13 OB. For example, as illustrated in Figure 3, the prompt 160B includes a natural language descriptions 146 that state “#Move 5 Meters” and “#Turn Right”. The prompt 160B also includes two example commands 144 (e.g., code snippets) to perform the tasks indicated in the natural language description 146. In particular, the prompt 160B includes commands 144 that state “Robot.Move(5)” and “Robot.Right(90)”. It should be understood that the prompt 160B is merely for illustrative purposes and should not be construed as limiting. In other embodiments, the prompt 160B can include different natural language descriptions 146, different example commands 144, more complex natural language descriptions, more complex commands, etc.

[0061] The robot 130B can be configured to send the prompt 160B to the API 116, and the foundation model 120 can access the prompt 160B via the API 116. As described below, the foundation model 120 can use the affordances 140 (e.g., the code snippets and the corresponding natural language) indicated in the prompt 160B to generate viable robot-specific commands 170B for the robot 130B in response to a query 150B.

[0062] For example, after the robot 130B sends the prompt 160B to the foundation model 120 via the API 116, the robot 130B can generate a query 150B associated with one or more tasks 152 for the robot 130B to perform. The query 150B can include a natural language representation 154 of the one or more tasks 152 to be performed by the robot 130B. In Figure 3, the query 150B includes the phrase “Turn Left and Advance 10 Meters”.

[0063] In response to receiving the query 150B via the API 116, the foundation model 120 can generate robot-specific commands 170B for the robot 130B. In particular, the foundation model 120 can determine how to present the commands 170B in a viable format that can be understood by the robot 130B. For example, based on the current set of affordances 140 in the prompt 160B, the foundation model 120 can determine a command style (or format) for the commands 170B. To illustrate, based on the current set of affordances 140 associated with the robot 130B, the foundation model 120 can understand that it can return “Robot.Direction()” and “Robot.Move()” as viable actions for the robot 130B if the query 150B indicates that the robot I30B is to move for a particular distance in a particular direction. As a result, the foundation model 120 can generate commands 170B according to the above- mentioned command style to enable the robot 130B to perform the tasks. For example, the commands 170B generated by the foundation model 120 can include “Robot.Left(90)” and “Robot.Move(lO)”.

[0064] The API 116 can be configured to send (e.g., return) the commands 170B to the robot 130B. In response to receiving the commands 170B, the robot 130B can execute the commands 170B to turn left and advance ten (10) meters.

[0065] The foundation model 120 can also generate robot-specific commands 170C for the robot 130C using a similar process. To illustrate, in Figure 3, the robot 130C can be configured to prompt 160C the API 116 to declare affordances 142 associated with the robot 130C. For example, as illustrated in Figure 3, the prompt 160C includes a natural language descriptions 146 that state “#Move Down 1 cm” and “#Close Gripper”. The prompt 160C also includes two example commands 144 (e.g., code snippets) to perform the tasks indicated in the natural language description 146. In particular, the prompt 160C includes commands 144 that state “Arm.Down(l)” and “Arm.Gripper.Close()”. It should be understood that the prompt 160C is merely for illustrative purposes and should not be construed as limiting. In other embodiments, the prompt 160C can include different natural language descriptions 146, different example commands 144, more complex natural language descriptions, more complex commands, etc.

[0066] The robot 130C can be configured to send the prompt 160C to the API 116, and the foundation model 120 can access the prompt 160C via the API 116. As described below, the foundation model 120 can use the affordances 140 (e.g., the code snippets and the corresponding natural language) indicated in the prompt 160C to generate viable robot-specific commands 170C for the robot 130C in response to a query 150C.

[0067] For example, after the robot 130C sends the prompt 160C to the foundation model 120 via the API 116, the robot 130C can generate a query 150C associated with one or more tasks 152 for the robot 130C to perform. The query 150C can include a natural language representation 154 of the one or more tasks 152 to be performed by the robot 130C. In Figure 3, the query 150C includes the phrase “Lift the Arm 10 cm”.

[0068] In response to receiving the query 150C via the API 116, the foundation model 120 can generate at least one robot-specific command 170C for the robot 130C. In particular, the foundation model 120 can determine how to present the at least one command 170C in a viable format that can be understood by the robot 130B. For example, based on the current set of affordances 140 in the prompt 160C, the foundation model 120 can determine a command style (or format) for the at least one command 170C. To illustrate, based on the current set of affordances 140 associated with the robot 130C, the foundation model 120 can understand that it can return “Arm.Direction()”as a viable actions for the robot 130C if the query 150C indicates that the robot 130C is to move the arm. As a result, the foundation model 120 can generate atleast one command 170C according to the above-mentioned command style to enable the robot 130C to perform the tasks. For example, the at least one command 170C generated by the foundation model 120 can include “Arm.Up(lO)”.

[0069] The API 116 can be configured to send (e.g., return) the at least one command 170C to the robot 130C. In response to receiving the at least one command 170C, the robot 130C can execute the at least one command 170C to lift the arm ten (10) centimeters.

[0070] Thus, the techniques described with respect to Figure 3 enables the foundation model 120 and the API 116 to provide robot-specific commands 170AB, 170C to the heterogeneous robots 130B, 130C based on the arbitrary affordances 140 in the prompts 160B, 160C. For example, the foundation model 120 can be trained to generate robot-specific commands 170B, 170C for the heterogeneous robots 130B, 130C based on the different sets of affordances 140. Because the robot-specific commands 170B, 170C are generated by the foundation model 120, the API 116 can serve the heterogeneous robots 130B, 130C having different sets of affordances. For example, the API 116 can be used for detailed or specific functions, reducing the need for more expressive functions to be controlled using client-side intelligence. As a result, memory requirements and processing complexity associated with client-side intelligence at the robots 130B, 130C can be relaxed.

[0071] It should be understood that in some implementations, the same task can be given to different robots with a different set of affordances. As a non-limiting example, a vacuum robot can send a query that states “clean trash from the floor” to the API 116, and a legged robot with an arm can also send a query that states “clean trash from the floor” to the API 116. In response to receiving the queries, the foundation model 120 can generate two appropriate commands 170 (e.g., code snippets) for each robot to perform the task. For example, for the vacuum robot, the foundation model 120 can generate commands that state “robot.navigate()” and “robot.run_vacuum()” to vacuum up the trash. For the legged robot with the arm, the foundation model 120 can generate a command that states “robot. walt_to()” to move towards the trash, a command that states “robot. arm_to()” to move the arm towards the trash, and commands that state “robot.gripper_close()” and “robot.arm.lift()” to pick up the trash.

[0072] Figure 4 illustrates an example of a system 400 that enables a cloud-based robot foundation model to return robot-specific commands to a robot based on modified API prompts from the robot. The system 400 includes the server 110 and the robot 130.

[0073] The system 400 can operate in a substantially similar manner as the system 100 described with respect to Figure 1; however, in Figure 4, the robot 130 can dynamically modifythe current set of the affordances 140 to generate a modified current set of affordances 440. As a result, the foundation model 120 can generate commands 470 based on the modified current set of affordances 440 as opposed to the old set of affordances 140 associated with the robot 130.

[0074] To illustrate, the processor 134 of the robot 130 can be configured to declare a modified current set of affordances 440 associated with the robot 130. As illustrated in Figure 4, the modified current set of affordances 440 includes an affordance 442A and the affordance 142B. In some embodiments, the affordance 442A is an updated / modified version of the affordance 142 A.

[0075] The affordance 442A includes an example command 444A and a natural language description 146 A associated with the example command 444A. As a non-limiting example, if the robot 130 has a fixed manipulator arm, the example command 444 A can correspond to a command for the fixed manipulator arm to perform a specific task. To illustrate, the example command 444 A can state “robot.find(‘ apple’)” (e.g., a modified version of the example command 144A that states “robot.detect(‘apple’)”) and the natural language description 146 A associated with the example command 444 A can state “#detect apple”. In this example, the example command 444A is a viable command for the robot 130 to detect an apple.

[0076] The processor 134 of the robot 130 can be configured to generate a prompt 460 that indicates the modified current set of affordances 440. The robot 130 can be configured to send the prompt 460 to the server 110 via the API 116. As described below, the foundation model 120 can use the modified current set of affordances 440 indicated in the prompt 160 to generate viable commands 470 for the robot 130 in response to robot queries 150.

[0077] For example, after the robot 130 sends the prompt 460 to the server 110 via the API 116, the processor 134 of the robot 130 can generate the query 150 associated with one or more tasks 152. As a non-limiting example, the query 150 can include the phrase “#pick up a detected chair”. In this example, the phrase “pick up a detected chair” is the natural language representation 154 of two tasks 152 (e.g., (1) detect a chair and (2) pick up the chair). The robot 130 can be configured to send the query 150 to the server 110 via the API 116.

[0078] In response to receiving the query 150, the processor 114 can generate at least one command 470 using the foundation model 120. In particular, the query 150 can be provided to the foundation model 120 to identify the one or more tasks 152 based on the natural language representation 154 of the one or more tasks 152. After identifying the tasks 152 associated with the query 150, the foundation model 120 can determine how to present thecommands 470 associated with the tasks 152 in a viable format that can be understood by the robot 130. For example, based on the modified current set of affordances 440 associated with the robot 130, the foundation model 120 can be used to determine a command style (or format) for the at least one command 170. To illustrate, based on the modified current set of affordances 440 associated with the robot 130, the foundation model 120 can understand that it can return “robot.find()” and “robot.pick_object()” as viable actions for the robot 130 if the query 150 indicates that an object should be picked up. As a result, the foundation model 120 can generate commands 470 according to the above-mentioned command style to enable the robot 130 to perform the tasks. For example, the commands 470 generated by the foundation model 120 can include “robot.find(‘chair’)” and “robot.pick_object(x, y, z)”. Thus, because the affordance 142 A was modified / updated to the affordance 442 A, the at least one command 470 (“robot.find(‘ chair’)”) has a command style that is different from the command style of the at least one command 170 (“robot.detect(‘chair’)”).

[0079] The server 110 can be configured to send (e.g., return) the commands 470 to the robot 130 via the API 116. In response to receiving the commands 470, the processor 134 can execute the commands 470 to initiate performance of the tasks 152 by the robot 130.

[0080] The techniques described with respect to Figure 4 enable the robot 130 to modify its affordances at any time, even dynamically, in a manner that is easy and flexible. For example, the robot 130 can send a prompt to the API 116 that indicates a modified current set of affordances 440 associated with the robot 130. The modified current set of affordances 440 can include updated code snippets (e.g., the example command 444A) that are actionable by the robot 130 and corresponding natural language associated with the updated code snippets. Based on the modified current set of affordances 440, the foundation model 120 can generate updated or modified robot-specific commands 470 can be returned as viable actions.III. Example Machine-Learning Process For Large Language Models

[0081] Figure 5 shows a diagram 500 illustrating a training phase 502 and an inference phase 504 of trained machine-learning model(s) 532, in accordance with example embodiments. According to some examples, the trained machine-learning model(s) 532 can correspond to the large language model 162 and / or the large language model 182. Some machine-learning techniques involve training one or more machine-learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine-learning algorithm can be termed as a trained machine-learning model. For example, Figure 5 shows the training phase 502 where machine-learning algorithm(s) 520 are being trained on trainingdata 510 to become trained machine-learning model(s) 532. Then, during the inference phase 504, the trained machine-learning model(s) 532 can receive input data 530 and one or more inference / prediction requests 540 (perhaps as part of the input data 530) and responsively provide as an output one or more inferences and / or prediction(s) 550.

[0082] As such, the trained machine-learning model(s) 532 can include one or more models of machine-learning algorithm(s) 520. The machine-learning algorithm(s) 520 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine-learning algorithm, and / or a heuristic machinelearning system). The machine-learning algorithm(s) 520 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning.

[0083] In some examples, the machine-learning algorithm(s) 520 and / or the trained machine-learning model(s) 532 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up the machine-learning algorithm(s) 520 and / or the trained machinelearning model(s) 532. In some examples, the trained machine-learning model(s) 532 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0084] During the training phase 502, the machine-learning algorithm(s) 520 can be trained by providing at least the training data 510 as training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of the training data 510 to the machinelearning algorithm(s) 520 and the machine-learning algorithm(s) 520 determining one or more output inferences based on the provided portion (or all) of the training data 510. Supervised learning involves providing a portion of the training data 510 to the machinelearning algorithm(s) 520, with the machine-learning algorithm(s) 520 determining one or more output inferences based on the provided portion of the training data 510, and the output inference(s) are either accepted or corrected based on correct results associated with the training data 510. In some examples, supervised learning of the machine-learning algorithm(s) 520 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of the machinelearning algorithm(s) 520.

[0085] Semi-supervised learning involves having correct results for part, but not all, of the training data 510. During semi-supervised learning, supervised learning is used for a portion of the training data 510 having correct results, and unsupervised learning is used for a portion of the training data 510 not having correct results. Reinforcement learning involves the machine-learning algorithm(s) 520 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, the machinelearning algorithm(s) 520 can output an inference and receive a reward signal in response, where the machine-learning algorithm(s) 520 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, the machine-learning algorithm(s) 520 and / or the trained machine-learning model(s) 532 can be trained using other machinelearning techniques, including but not limited to, incremental learning and curriculum learning.

[0086] In some examples, the machine-learning algorithm(s) 520 and / or the trained machine-learning model(s) 532 can use transfer learning techniques. For example, transfer learning techniques can involve the trained machine-learning model(s) 532 being pre-trained on one set of data and additionally trained using the training data 510. More particularly, the machine-learning algorithm(s) 520 can be pre-trained on data from one or more computing devices and a resulting trained machine-learning model provided to a particular computing device, where the particular computing device is intended to execute the trained machinelearning model during the inference phase 504. Then, during the training phase 502, the pretrained machine-learning model can be additionally trained using the training data 510, where the training data 510 can be derived from kernel and non-kernel data of the particular computing device. This further training of the machine-learning algorithm(s) 520 and / or the pre-trained machine-learning model using the training data 510 of the particular computing device’s data can be performed using either supervised or unsupervised learning. Once the machine-learning algorithm(s) 520 and / or the pre-trained machine-learning model has been trained on at least the training data 510, the training phase 502 can be completed. The trained resulting machine-learning model can be utilized as at least one of the trained machine-learning model(s) 532.

[0087] In particular, once the training phase 502 has been completed, the trained machine-learning model(s) 532 can be provided to a computing device, if not already on the computing device. The inference phase 504 can begin after training the machine-learning model(s) 532 are provided to the particular computing device.

[0088] During the inference phase 504, the trained machine-learning model(s) 532 can receive the input data 530 and generate and output one or more corresponding inferences and / or prediction(s) 550 about the input data 530. As such, the input data 530 can be used as an input to the trained machine-learning model(s) 532 for providing corresponding inference(s) and / or prediction(s) 550 to kernel components and non-kernel components. For example, the trained machine-learning model(s) 532 can generate inference(s) and / or prediction(s) 550 in response to one or more inference / prediction requests 540. In some examples, the trained machinelearning model(s) 532 can be executed by a portion of other software. For example, the trained machine-learning model(s) 532 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. The input data 530 can include data from the particular computing device executing the trained machine-learning model(s) 532 and / or input data from one or more computing devices other than the particular computing device.

[0089] If the trained machine-learning model 532 corresponds to the foundation model 120, the input data 530 can include data associated with different affordances 142. Other types of input data are possible as well. Inference(s) and / or prediction(s) 650 can include other output data produced by the trained machine-learning model(s) 532 operating on the input data 530 (and the training data 510). In some examples, the trained machine-learning model(s) 532 can use output inference(s) and / or prediction(s) 550 as input feedback 560. The trained machinelearning model(s) 532 can also rely on past inferences as inputs for generating new inferences.

[0090] Convolutional neural networks and / or deep neural networks used herein can be an example of the machine-learning algorithm(s) 520. After training, the trained version of a convolutional neural network can be an example of the trained machine-learning model(s) 532. In this approach, an example of the one or more inference / prediction requests 540 can be a request to predict robot-specific commands 170 for heterogeneous robots 130.IV. Additional Example Operations

[0091] Figure 6 illustrates a flow chart of a method 600 related to a new technology. The method 600 may be carried out by the server 110 among other possibilities. The embodiments of Figure 6 may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures or otherwise described herein.

[0092] The method 600 includes receiving, by a server hosting a foundation model and via an API, a prompt from a robot that indicates a current set of affordances associated with the robot, at block 602. For example, referring to Figure 1, the server 110 receives the prompt160 from the robot 130 that indicates the current set of affordances 140 associated with the robot 130. The server 110 hosts the foundation model 120, and the prompt 160 is received via the API 116. The foundation model 120 can be trained for heterogeneous robots having arbitrary capabilities and arbitrary affordances. According to one implementation of the method 600, the API 116 is usable to define arbitrary affordances. In some scenarios, the affordances are selected from a predefined list.

[0093] The method 600 also includes receiving, by the server and via the API, a query from the robot for instructions to complete a task, at block 604. For example, referring to Figure 1, the server 110 receives the query 150 from the robot 130 via the API 116 for instructions to complete the task 152.

[0094] The method 600 also includes generating, by the server and using the foundation model, at least one command in response to the query, at block 606. The at least one command is executable by the robot to complete the task, and the at least one command is generated based on the current set of affordances associated with the robot. For example, referring to Figure 1, the server 110 uses the foundation model 120 to generate the command(s) 170 in response to the query 150. The at least one command 170 is generated based on the current set of affordances 140 associated with the robot 130.

[0095] The method 600 also includes sending, by the server and via the API, the at least one command to the robot, at block 608. For example, referring to Figure 1, the server 110 sends, via the API 116, the command(s) 170 to the robot 130.

[0096] According to one implementation of the method 600, generating the at least one command 170 includes determining a command style for the at least one command 170 based on the current set of affordances 140 associated with the robot 130. The command style can be recognizable by the robot 130. Generating the at least one command 170 can also include generating the at least one command 170 according to the command style. According to one implementation, the at least one command 170 is executable by the robot 130 when the at least one command 170 is generated according to the command style.

[0097] According to one implementation of the method 600, to indicate the current set of affordances 140 associated with the robot 130, the prompt 160 includes at least one example command 144 that is recognizable by the robot 130. According to one implementation of the method 600, the query 150 corresponds to a natural language version of the task 152.

[0098] According to one implementation, the method 600 includes receiving, by the server 110 via the API 116, a second prompt from the robot that indicates a modified current set of affordances associated with the robot. The second prompt is received after the prompt,and the modified current set of affordances corresponds to an updated version of the current set of affordances. The method 600 also includes receiving, by the server and via the API, a second query from the robot for instructions to complete a second task after receiving the second prompt from the robot. The method 600 also includes generating, by the server and using the foundation model, at least one second command in response to the second query. The at least one second command is executable by the robot to complete the second task, and the second command is generated based on the modified current set of affordances associated with the robot. The method 600 also includes sending, by the server and via the API, the at least one second command to the robot. According to one implementation, the at least one second command has a second command style that is different from a command style of the at least one command.

[0099] According to one implementation, the method 600 includes receiving, by the server 110 and via the API 116, a second prompt 160C from a second robot 130C that indicates a current set of affordances associated with the second robot 130C. The method 600 also includes receiving, by the server 110 and via the API 116, a second query 150C from the second robot 130C for instructions to complete a second task. The method 600 also includes generating, by the server 110 and using the foundation model 120, at least one second command 170C in response to the second query 150C. The at least one second command 170C is executable by the second robot 130C to complete the second task, and the at least one second command 170C is generated based on the current set of affordances associated with the second robot 130C. The method 600 also includes sending, by the server 110 and via the API 116, the at least one second command 170C to the second robot 130C. According to one implementation, the current set of affordances associated with the robot 130B is different from the current set of affordances associated with the second robot 130C. According to one implementation, the at least one command 170B corresponds to at least one first robot-specific command for the robot 130B that is not viable for the second robot 130C, and the at least one second command 170C corresponds to at least one second robot-specific command for the second robot 130C that is not viable for the robot 130B.

[0100] The method 600 of Figure 6 enables the server 110 to provide robot-specific commands 170 to the robot 130 based on the current set of affordances 140 associated with the robot 130. For example, the foundation model 120 can be trained to generate robot-specific commands 170 for the robot 130 to complete the task 152 based on the current set of affordances 140. Because the robot-specific commands 170 are generated by the foundation model 120, the API 116 can serve heterogeneous robots having different sets of affordances.For example, the API 116 can be used for detailed or specific functions, reducing the need for more expressive functions to be controlled using client-side intelligence. As a result, memory requirements and processing complexity associated with client-side intelligence at the robot 130 can be relaxed.

[0101] Figure 7 illustrates a flow chart of a method 700 related to a new technology. The method 700 may be carried out by the robot 130 among other possibilities. The embodiments of Figure 7 may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures or otherwise described herein.

[0102] The method 700 includes sending, by a robot, a prompt to a server via an API, at block 702. The prompt indicates a current set of affordances associated with the robot. For example, referring to Figure 1, the robot 130 sends the prompt 160 to the server 110 via the API 116. The prompt 160 indicates the current set of affordances 140 associated with the robot 130. According to one implementation, to indicate the current set of affordances associated with the robot, the prompt includes at least one example command that is recognizable by the robot.

[0103] The method 700 also includes sending, by the robot, a query to the server via the API, at block 704. The query corresponds to a request for instructions to complete a task. For example, referring to Figure 1, the robot 130 sends the query 150 to the server 110 via the API 116. The query 150 corresponds to a request for instructions to complete the task 152.

[0104] The method 700 also includes receiving, by the robot, at least one command from the server via the API in response to sending the query. The at least one command is generated by the server using a foundation model based on the current set of affordances associated with the robot. For example, referring to Figure 1, the robot 130 receives the at least one command 170 from the server 110 via the API 116 in response to sending the query 150. The at least one command 170 is generated by the server 110 using the foundation model 120 based on the current set of affordances 140 associated with the robot 130.

[0105] The method 700 also includes executing, by the robot, the at least one command to complete the task, at block 708. For example, referring to Figure 1, the robot 130 can execute the at least one command 170 to complete the task 152. According to one implementation, the query corresponds to a natural language representation of the task.

[0106] According to one implementation, the method 700 includes dynamically updating, by the robot, the current set of affordances associated with the robot to generate a modified current set of affordances associated with the robot. The method 700 also includessending, by the robot, a second prompt to the server via the API. The second prompt indicates the modified current set of affordances associated with the robot. The method 700 also includes sending, by the robot, a second query to the server via the API. The second query corresponds to a request for instructions to complete a second task. The method 700 also includes receiving, by the robot, at least one second command from the server via the API in response to sending the second query. The at least one second command is generated by the server using the foundation model based on the modified current set of affordances associated with the robot. According to one implementation of the method 700, the at least one second command has a second command style that is different from a command style of the at least one command.

[0107] The method 700 of Figure 7, robot-specific commands 170 to be provided to the robot 130 based on the current set of affordances 140 associated with the robot 130. For example, the foundation model 120 can be trained to generate robot-specific commands 170 for the robot 130 based on the current set of affordances 140. Because the robot-specific commands 170 are generated by the foundation model 120, the API 116 can serve heterogeneous robots having different sets of affordances. For example, the API 116 can be used for detailed or specific functions, reducing the need for more expressive functions to be controlled using client-side intelligence. As a result, memory requirements and processing complexity associated with client-side intelligence at the robot 130 can be relaxed.V. Conclusion

[0108] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0109] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures,can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0110] With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and / or communication can represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages can be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or operations can be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.[OHl] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

[0112] The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non-transitory computer readable media that store program code and / or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compactdisc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0113] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules inthe same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0114] The particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Further, some of the illustrated elements can be combined or omitted. Yet further, an example embodiment can include elements that are not illustrated in the figures.

[0115] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for the purpose of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: receiving, by a server hosting a foundation model and via an application programming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot; receiving, by the server and via the API, a query from the robot for instructions to complete a task; generating, by the server and using the foundation model, at least one command in response to the query, wherein the at least one command is executable by the robot to complete the task, and wherein the at least one command is generated based on the current set of affordances associated with the robot; and sending, by the server and via the API, the at least one command to the robot.

2. The method of claim 1, wherein generating the at least one command comprises: determining a command style for the at least one command based on current set of affordances associated with the robot, wherein the command style is recognizable by the robot; and generating the at least one command according to the command style.

3. The method of claim 1, wherein, to indicate the current set of affordances associated with the robot, the prompt includes at least one example command that is recognizable by the robot.

4. The method of claim 1, wherein the query corresponds to a natural language representation of the task.

5. The method of claim 1, further comprising: receiving, by the server via the API, a second prompt from the robot that indicates a modified current set of affordances associated with the robot, wherein the second prompt is received after the prompt, and wherein the modified current set of affordances corresponds to an updated version of the current set of affordances;receiving, by the server and via the API, a second query from the robot for instructions to complete a second task after receiving the second prompt from the robot; generating, by the server and using the foundation model, at least one second command in response to the second query, wherein the at least one second command is executable by the robot to complete the second task, and wherein the at least one second command is generated based on the modified current set of affordances associated with the robot; and sending, by the server and via the API, the at least one second command to the robot.

6. The method of claim 5, wherein the at least one second command has a second command style that is different from a command style of the at least one command.

7. The method of claim 5, wherein the task is similar to the second task.

8. The method claim 1, further comprising: receiving, by the server and via the API, a second prompt from a second robot that indicates a current set of affordances associated with the second robot; receiving, by the server and via the API, a second query from the second robot for instructions to complete a second task; generating, by the server and using the foundation model, at least one second command in response to the second query, wherein the at least one second command is executable by the second robot to complete the second task, and wherein the at least one second command is generated based on the current set of affordances associated with the second robot; and sending, by the server and via the API, the at least one second command to the second robot.

9. The method of claim 8, wherein the current set of affordances associated with the robot is different from the current set of affordances associated with the second robot.

10. The method of claim 8, wherein the at least one command corresponds to at least one first robot-specific command for the robot that is not viable for the second robot, andwherein the at least one second command corresponds to at least one second robot-specific command for the second robot that is not viable for the robot.

11. The method of claim 1, wherein the API is usable to define arbitrary affordances.

12. A server comprising: a memory; and a processor coupled to the memory, the processor configured to: receive, via an application programming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot; receive, via the API, a query from the robot for instructions to complete a task; generate, using a foundation model, at least one command in response to the query, wherein the at least one command is executable by the robot to complete the task, and wherein the at least one command is generated based on the current set of affordances associated with the robot; and send, via the API, the at least one command to the robot.

13. The server of claim 12, wherein, to indicate the current set of affordances associated with the robot, the prompt includes at least one example command that is recognizable by the robot.

14. The server of claim 12, wherein the query corresponds to a natural language representation of the task.

15. A method compri sing : sending, by a robot, a prompt to a server via an application programming interface (API), wherein the prompt indicates a current set of affordances associated with the robot; sending, by the robot, a query to the server via the API, wherein the query corresponds to a request for instructions to complete a task; and receiving, by the robot, at least one command from the server via the API in response to sending the query, wherein the at least one command is generated by theserver using a foundation model based on the current set of affordances associated with the robot; and executing, by the robot, the at least one command to complete the task.

16. The method of claim 15, wherein, to indicate the current set of affordances associated with the robot, the prompt includes at least one example command that is recognizable by the robot.

17. The method of claim 15, further comprising: dynamically updating, by the robot, the current set of affordances associated with the robot to generate a modified current set of affordances associated with the robot; sending, by the robot, a second prompt to the server via the API, wherein the second prompt indicates the modified current set of affordances associated with the robot; sending, by the robot, a second query to the server via the API, wherein the second query corresponds to a request for instructions to complete a second task; receiving, by the robot, at least one second command from the server via the API in response to sending the second query, wherein the at least one second command is generated by the server using the foundation model based on the modified current set of affordances associated with the robot; and executing, by the robot, the at least one second command to complete the second task.

18. The method of claim 17, wherein the at least one second command has a second command style that is different from a command style of the at least one command.

19. The method of claim 15, wherein the query corresponds to a natural language representation of the task.

20. A non-transitory computer-readable medium comprising instructions that, when executed by a processor within a server, cause the processor to perform operations comprising: receiving, via an application programming interface (API), a prompt from a robot that indicates a current set of affordances associated with the robot;receiving, via the API, a query from the robot for instructions to complete a task; generating, using a foundation model, at least one command in response to the query, wherein the at least one command is executable by the robot to complete the task, and wherein the at least one command is generated based on the current set of affordances associated with the robot; and sending, via the API, the at least one command to the robot.

21. A robot comprising: a memory; and a processor coupled to the memory, the processor configured to: send a prompt to a server via an application programming interface (API), wherein the prompt indicates a current set of affordances associated with the robot; send a query to the server via the API, wherein the query corresponds to a request for instructions to complete a task; receive at least one command from the server via the API in response to sending the query, wherein the at least one command is generated by the server using a foundation model based on the current set of affordances associated with the robot; and execute the at least one command to complete the task.

22. A non-transitory computer-readable medium comprising instructions that, when executed by a processor within a robot, cause the processor to perform operations comprising: sending a prompt to a server via an application programming interface (API), wherein the prompt indicates a current set of affordances associated with the robot; sending a query to the server via the API, wherein the query corresponds to a request for instructions to complete a task; receiving at least one command from the server via the API in response to sending the query, wherein the at least one command is generated by the server using a foundation model based on the current set of affordances associated with the robot; and execute the at least one command to complete the task.

Images