Techniques for automating programming of robotics tasks using language models

Abstract

The present invention relates to techniques for automating programming of robot tasks using language models. One embodiment of a computer-implemented method for generating program code to control a robot includes: receiving user input specifying a task to be performed by the robot; processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 587,887, filed on October 4, 2023, entitled “AUTOMATED PROGRAMMING FOR ROBOTIC ASSEMBLY USING CHATGPT” The subject matter of this related application is hereby incorporated herein by reference. Background Art Technical Field

[0003] Embodiments of the present invention generally relate to robotic control, artificial intelligence (AI), and machine learning, and more specifically to techniques for automating programming of robotic tasks using language models.

[0005] Robots are increasingly being used to automatically perform tasks in a variety of environments. One method for controlling a robot to perform a given task is to program the robot to repeatedly perform the same action. For example, in electronics assembly, a robot can be programmed to perform tasks such as soldering circuit boards or placing microchips with high precision.

[0006] A conventional method for programming a robot to perform a given task uses a behavior script that includes a specific sequence of commands. The specific command sequence directs the robot to perform the movements and actions required for the given task. Generally speaking, the behavior script is developed by a programmer who has a detailed understanding of the mechanisms and capabilities of each robot, and the precision and specificity of the command sequence in the behavior script is critical to correctly control the robot to perform the given task.

[0007] One drawback of conventional robotic systems is that such systems cannot easily adapt to modifications to the tasks that the robot is controlled to perform. For example, when an assembly process is modified to assemble different parts or to assemble parts in a different order, the conventional robot must be reprogrammed with a new behavior script to control the robot to perform the modified assembly process. Typically, reprogramming is very time-consuming and requires a lot of engineering expertise, including understanding the robot kinematics, the requirements of the robot to perform the task, and the control programming language.

[0008] As the foregoing indicates, there is a need in the art for more efficient techniques for controlling robots to perform tasks. Summary of the invention

[0009] One embodiment of the present disclosure sets forth a computer-implemented method for generating program code to control a robot. The method includes receiving a user input specifying a task to be performed by the robot. The method further includes processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task. Additionally, the method further includes, for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.

[0010] Other embodiments of the present disclosure include, but are not limited to, one or more computer-readable media including instructions for performing one or more aspects of the disclosed technology and one or more computing systems for performing one or more aspects of the disclosed technology.

[0011] At least one technical advantage of the disclosed technology over the prior art is that, unlike conventional robotic systems that cannot perform new tasks without extensive manual reprogramming, the disclosed technology automatically generates executable scripts for controlling a robot to perform new tasks. Using the disclosed technology, a robot can be adapted to various scenarios requiring the robot to perform different tasks without requiring extensive manual reprogramming and expertise. These technical advantages represent one or more technical advances over prior art approaches. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to be able to understand in detail the manner of the above features of various embodiments, the inventive concept briefly summarized above can be described in more detail by reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings only show typical embodiments of the inventive concept and therefore should not be considered to limit the scope in any way, and there are other equivalent embodiments.

[0013] Figure 1 is a block diagram illustrating a robotic control system configured to implement one or more aspects of an embodiment of the present invention;

[0014] Figure 2 According to various implementation schemes Figure 1 A more detailed illustration of a computing device;

[0015] Figure 3 According to various implementation schemes Figure 1 A more detailed illustration of the task decomposition modules in the robot control application of

[0016] Figure 4 According to various implementation schemes Figure 1 A more detailed illustration of the script generation module in the robot control application of

[0017] Figure 5 is a flow chart of method steps for controlling a robot according to various embodiments; and

[0018] Figure 6 is a flow chart of method steps for generating a subtask script according to various embodiments. DETAILED DESCRIPTION

[0019] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0020] System Overview

[0021] Figure 1 A block diagram of a computer-based system 100 configured to implement one or more aspects of at least one embodiment is shown. As shown, the system 100 includes a computing device 110, a robot 150, and a sensor 140.

[0022] The robot control application 126 is executed on one or more processors 112 of the computing device 110 and is stored in the system memory 114 of the computing device 110. The processor 112 receives user input from an input device such as a keyboard or mouse. The processor 112 may include one or more main processors of the computing device 110 to control and coordinate the operation of other system components. In operation, the processor 112 may issue commands to control the operation of one or more graphics processing units (GPUs) (not shown) and / or other parallel processing circuit systems (e.g., parallel processing units, deep learning accelerators, etc.), which incorporate circuit systems optimized for graphics and video processing, such as video output circuit systems. The GPU may deliver pixels to a display device, which may be any conventional cathode ray tube, liquid crystal display, light emitting diode display, etc.

[0023] The system memory 114 of the computing device 110 stores content such as software applications and data for use by the processor 112 and the GPU and / or other processing units. The system memory 114 can be any type of memory capable of storing data and software applications, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash ROM), or any suitable combination of the foregoing. In some embodiments, a storage device (not shown) can supplement or replace the system memory 114. The storage device may include any number and type of external memory accessible to the processor 112 and / or GPU. For example, but not limited to, the storage device may include a secure digital card, an external flash memory, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, and / or any suitable combination of the foregoing.

[0024] The computing device 110 shown herein is for illustration purposes only and may be varied and modified without departing from the scope of the present disclosure. For example, the number of processors 112, the number of GPUs and / or other processing unit types, the number of system memories 114, and / or the number of applications included in the system memories 114 may be modified as desired. In addition, the computing device 110 may be modified as desired. Figure 1 In some embodiments, any combination of processor 112, system memory 114, and / or GPU may be included in and / or replaced by any type of virtual computing system, distributed computing system, or cloud computing environment such as a public cloud system, a private cloud system, or a hybrid cloud system.

[0025] In some embodiments, the robot control application 126 includes a language model 131, a task decomposition module 132, and a script generation module 133. In operation, the robot control application can control the robot 150 to perform one or more tasks based on sensor data captured by one or more sensors 140 (e.g., force sensors, cameras, etc.). Figures 3 to 6 The techniques that the robot control application 126 can use to control the robot 150 are discussed in more detail. Although shown as being included in the robot control application 126, in some embodiments, the language model 131 can be located outside the robot control application 126 ( For example , the language model 131 can be executed in a cloud computing environment) and can be accessed in any technically feasible manner, such as via an application programming interface (API).

[0026] As shown, the robot 150 includes a plurality of links 151, 153, and 155 as rigid members, and joints 152, 154, and 156 as movable components that can be actuated to cause relative movement between adjacent links. In addition, the robot 150 includes a plurality of fingers 158. i (collectively referred to herein as fingers 168 and individually as fingers 168), the plurality of fingers can be controlled to grasp an object. For example, in some embodiments, the robot 150 may include a locked wrist and a plurality of fingers. Although an exemplary robot 150 is shown for illustrative purposes, in some embodiments, the techniques disclosed herein can be applied to control any technically feasible robot ( For example , a robot with multiple arms, a robot including multiple robots that collaborate to perform tasks, etc.).

[0027] Figure 2 According to various implementation schemes Figure 1 1 is a more detailed illustration of a computing device 110 of FIG. Computing device 110 may include any type of computing system, including but not limited to a server machine, a server platform, a desktop machine, a laptop machine, a handheld / mobile device, a digital kiosk, an in-vehicle infotainment system, and / or a wearable device. In some embodiments, computing device 110 is a server machine operating in a data center or cloud computing environment that provides scalable computing resources as a service over a network.

[0028] In some embodiments, computing device 110 includes, but is not limited to, processor 112 and memory 144 coupled to parallel processing subsystem 212 via memory bridge 205 and communication path 213. Memory bridge 205 is further coupled to I / O (input / output) bridge 207 via communication path 206, and I / O bridge 207 is in turn coupled to switch 216.

[0029] In some embodiments, I / O bridge 207 is configured to receive user input information, sensor data analysis (e.g., evaluating gestures, voice, or other information about one or more uses in the field of view or sensory field of one or more sensors), etc. from an optional input device 208 such as a keyboard, mouse, touch screen, etc., and forward the input information to processor 112 for processing. In some embodiments, computing device 110 can be a server machine in a cloud computing environment. In such embodiments, computing device 110 may not include input device 208, but can receive equivalent input information by receiving commands in the form of messages transmitted over a network and received via network adapter 218 (e.g., in response to one or more inputs from a remote computing device). In some embodiments, switch 216 is configured to provide connections between I / O bridge 207 and other components of machine learning server 110 (such as network adapter 218 and various expansion cards 220 and 221).

[0030] In some embodiments, I / O bridge 207 is coupled to system disk 214, which may be configured to store content as well as applications and data for use by processor 112 and parallel processing subsystem 212. In one embodiment, system disk 214 provides non-volatile storage for applications and data, and may include a fixed or removable hard drive, flash memory devices, and CD-ROM (disc read only memory), DVD-ROM (digital versatile disc-ROM), Blu-ray disc, HD-DVD (high definition DVD), or other magnetic, optical, or solid state storage devices. In some embodiments, other components (such as a universal serial bus or other port connection, optical disk drive, digital versatile disc drive, film recording device, etc.) may also be connected to I / O bridge 207.

[0031] In some embodiments, memory bridge 205 may be a northbridge chip and I / O bridge 207 may be a southbridge chip. Additionally, communication paths 206 and 213, as well as other communication paths within computing device 110, may be implemented using any technically suitable protocol, including but not limited to AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0032] In some embodiments, parallel processing subsystem 212 includes a graphics subsystem that delivers pixels to an optional display device 210, which can be any conventional cathode ray tube, liquid crystal display, light emitting diode display, etc. In such embodiments, parallel processing subsystem 212 can incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry. Such circuitry can be incorporated across one or more parallel processing units (PPUs) (also referred to herein as parallel processors) included within parallel processing subsystem 212.

[0033] In some embodiments, the parallel processing subsystem 212 incorporates circuitry that is optimized for general-purpose processing and / or computational processing (e.g., optimized). Likewise, such circuitry may be incorporated across one or more PPUs included within the parallel processing subsystem 212, which are configured to perform such general-purpose operations and / or computational operations. In yet other embodiments, one or more PPUs included within the parallel processing subsystem 212 may be configured to perform graphics processing, general-purpose processing, and / or computational processing operations. The system memory 114 includes at least one device driver that is configured to manage processing operations of one or more PPUs within the parallel processing subsystem 212. Additionally, the system memory 114 stores the robot control application 126 and the operating system 204 on which the robot control application 126 runs. The operating system 204 may be For example Microsoft or Although primarily described herein with respect to computing device 110 , the techniques disclosed herein may also be implemented in whole or in part in other software and / or hardware, such as in parallel processing subsystem 212 .

[0034] In some embodiments, the parallel processing subsystem 212 can be used with Figure 2 For example, parallel processing subsystem 212 may be integrated with processor 112 and other connected circuitry on a single chip to form a system on a chip (SoC).

[0035] In some embodiments, processor 112 comprises the main processor of computing device 110 to control and coordinate the operation of other system components. In some embodiments, processor 112 issues commands to control the operation of the PPUs. In some embodiments, communication path 213 is a PCI Express link in which a dedicated lane is allocated to each PPU. Other communication paths may also be used. The PPUs advantageously implement a highly parallel processing architecture, and the PPUs may be provided with any amount of local parallel processing memory (PP memory).

[0036] It should be understood that the system shown herein is illustrative, and variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of CPUs 202, and the number of parallel processing subsystems 212, may be modified as desired. For example, in some embodiments, system memory 114 may be connected to processor 112 directly rather than through memory bridge 205, and other devices may communicate with system memory 114 via memory bridge 205 and processor 112. In other embodiments, parallel processing subsystem 212 may be connected to I / O bridge 207 or directly to processor 142 rather than being connected to memory bridge 205. In still other embodiments, I / O bridge 207 and memory bridge 205 may be integrated into a single chip rather than existing as one or more discrete devices. In some embodiments, there may be no Figure 2 For example, the switch 216 may be omitted, and the network adapter 218 and the expansion cards 220, 221 may be directly connected to the I / O bridge 207. Finally, in some embodiments, Figure 2 One or more of the components shown may be implemented as virtualized resources in a virtual computing environment, such as a cloud computing environment. Specifically, in some embodiments, parallel processing subsystem 212 may be implemented as a virtualized parallel processing subsystem. For example, parallel processing subsystem 212 may be implemented as a virtual graphics processing unit (vGPU) that renders graphics on a virtual machine (VM) executing on a server machine whose GPU and other physical resources are shared across one or more VMs.

[0037] Using language models to automate programming of robotics tasks

[0038] Figure 3 According to various implementation schemes Figure 1 FIG. 1 is a more detailed diagram of the task decomposition module 132 in the robot control application 126 of FIG. As shown in the figure, the task decomposition module 132 uses the language model 131 to process the robot task information 301, one or more task decomposition examples 302 and one or more formatting rules 303, and the task decomposition module 132 generates robot subtask information 304.

[0039] Illustratively, the robot control application 126 receives as input robot task information 301. In some embodiments, the robot task information 301 includes: (1) the overall goal of the robot task to be performed, e.g., a natural language description of the task, such as "assemble axes" or the like; (2) the environment in which the robot operates; and (3) any constraints or parameters specific to the task, such as time limits, accuracy requirements, safety considerations, etc. In some embodiments, spatial relationships of common objects are extracted from computer-aided design (CAD) software and stored in a file ( For example , JavaScript Object Notation (JSON) file), which can also be included in the robot task information 301.

[0040] The language model 131 is a trained machine learning model that interprets the robot task information 301 in operation and translates the robot task information 301 into robot subtask information 304, which can be used by the script generation module 133 to generate a script for performing a subtask. In some embodiments, any technically feasible machine learning model can be used. In some embodiments, the language model 131 can be a large language model (LLM). In some embodiments, the script can include program code, such as a call to a specific function provided by an API for controlling the robot, which can be translated into a low-level command that prompts the robot to make a physical action. In some other embodiments, the script can include a low-level command that prompts the robot to make a physical action. The language model 131 is able to generate a script for performing a subtask because the language model 131 understands possible robot behaviors consistent with the goals of the robot task information 301, which can be collected based on extensive training on various data sets. In some embodiments, the task decomposition module 132 uses a small number of shot prompts with one or more task decomposition examples 302 to constrain the output of the language model 131.

[0041] The task decomposition examples 302 include templates, previous examples of decomposing similar tasks, and the like. In some embodiments, the task decomposition examples 302 include examples of how similar tasks were decomposed into subtasks in the past, which provide references for the task decomposition module 132. The task decomposition module 132 uses the task decomposition examples 302 to inform the language model 131 how to decompose the task into subtasks. Each subtask is a task in itself, and each subtask can be assigned a behavior label, which is a primitive corresponding to a common robot behavior, such as "move", "pick up", "place", "insert", etc. In some embodiments, one or more of the task decomposition examples 302 can be included in a few shot prompts that require the language model 131 to decompose the task into subtasks.

[0042] One or more formatting rules 303 define how the decomposed subtasks should be constructed. The task decomposition module 132 uses the formatting rules 303 to inform the language model 131 of the correct format of the language model 131 output. In some embodiments, if the robot operates based on a certain behavior scripting language (such as python, C++, etc.), the formatting rules 303 ensure that the decomposed tasks included in the robot subtask information 304 are compatible with the syntax and semantics of the behavior scripting language. The output of the task decomposition module 132 is the robot subtask information 304 including a series of smaller, discrete robot actions, so that when executed in sequence, the robot task included in the robot task information 301 is completed. For example, the robot task "assemble axis" can be decomposed into the following subtasks: "detect axis", "pick up axis", "move axis" and "insert axis". In some embodiments, the task decomposition module 132 continuously checks whether the robot subtask information 304 of all robot subtasks has been generated.

[0043] Figure 4 According to various implementation schemes Figure 1 1 is a more detailed illustration of the script generation module 133 in the robot control application 126 of FIG. As shown, the script generation module 133 includes a text replacer module 401, a subtask context module 402, a script verification module 403, and a simulation module 404. Although a single script generation module 133 is shown for illustrative purposes, in some embodiments, the script generation module 133 can be implemented using multiple agents that execute in parallel to generate scripts for different subtasks specified in the robot subtask information 304, and then simulate and debug the scripts in sequence to ensure that the tasks can be successfully executed.

[0044] The script generation module 133 uses the language model 131 to process the robot subtask information 304, one or more scripted rules 408, and one or more scripted examples 409 to generate a subtask script 420, which is a script including program code executable to perform the subtask specified in the robot subtask information 304. Although one language model 131 is shown for illustrative purposes, in some embodiments, the language model used by the script generation module 133 may be different from the large language model used by the task decomposition module 131. For example, in some embodiments, the task decomposition module 131 and the script generation module 133 may use different language models that are fine-tuned for decomposing the robot task into subtasks using appropriate training data and writing program code for the robot, respectively.

[0045] The text replacer module 401 is configured to prevent sensitive information from being inadvertently disclosed when the subtask context module 402 interacts with the language model 131. In operation, the text replacer module 401 replaces certain terms that may contain proprietary or confidential data with predetermined public or private aliases. In some embodiments, the aliases are designed to be unique and irreversible, which means that once the terms are replaced, the original information cannot be inferred from the aliases. In some embodiments, the text replacer module 401 operates by referencing a dictionary of terms that require aliases. When processing a script or data packet containing any term, the text replacer module 401 scans the content and automatically performs the replacement.

[0046] The subtask context module 402 includes subtask information 405, script history 406, and simulation history 407. Illustratively, the subtask context module 402 receives the robot subtask information 304, one or more scripting rules 408, and one or more scripting examples 409 generated by the task composition module 132 and input into the script generation module 133. The scripting rules 408 provide a structural, grammatical, and semantic framework for the subtask script 410 generated by the script generation module 133 using the language model 131, and the subtask script can be in various programming languages, such as Python, C++, etc. For example, the scripting rules 408 of the Python-based script may include "reply only with Python code", "remember to import '***'", "put all code in the main function", etc. The scripting examples 409 include, but are not limited to, previously tested and correct robot subtask scripts. In some embodiments, the scripting examples 409 include scripting examples with different levels of complexity to increase the diversity of the scripting examples 409. In some embodiments, the scripting examples 409 can be obtained in any technically feasible manner. For example, in some embodiments, the behavior label assigned to the subtask ( For example, "pick") can be used to find scripted examples associated with the same behavior label in a set of files or databases.

[0047] The subtask context module 402 stores the robot subtask information 304, the scripted rules 408, and the scripted examples 409 as subtask information 405. In some embodiments, the subtask information 405 indicates (1) one or more subtasks into which the user-specified task is decomposed, and (2) the behavior label assigned to each subtask. In some embodiments, the subtask information 405 may also include reference documents for the robot subtask, such as technical documents, operation guides, specific programming references that further assist in the script development of the robot subtask, and the like.

[0048] The subtask context module 402 sends the subtask context 410 including the content of the subtask information 405, the script history 406 and the simulation history 407 to the text replacer module 401. Given such input, the text replacer module 401 replaces the proprietary confidential terms (if any) therein with public aliases. The subtask context with the replaced text is then sent to the language model 131, which generates a subtask script that may include errors. The text replacer module 401 can also replace the text in the subtask script generated by the language model 131 to generate a temporary script 413, which the text replacer module 401 sends to the subtext context module 402 to be stored in the script history 406. The temporary script 413 is a script generated by the language model 131, which has been verified by the script verification module 403 for syntax, formatting errors, runtime exceptions, etc. The verification results are also stored in the script history 406. Once the temporary script 413 is verified, the simulation module 404 simulates the temporary script 413 in a virtual environment to check whether the robot subtask can be successfully implemented using the script. For example, the virtual environment may include simulated work units that execute the script, and control errors such as collisions or failure to reach the desired final state may cause the simulation result to fail. "Failed" or "successful" simulation results are also stored in the simulation history 407. If the simulation result fails, the subtask context module 402 appends the script history 406 and the simulation history 407 to the subtask context 410 and resends the subtask context 410 to the language model 131 to generate the script. The script generation module 133 continuously updates the temporary script ( For example , temporary script 413), until a temporary script with no verification and simulation errors is generated during runtime or a subtask script 410 of all subtasks is generated.

[0049] For example, in the case of a robot assembly task, the initial chat context input into the language model 131 can be expressed as (R, L, S, P, B, E), where R is a role, such as "You are a chat agent that divides the robot task into robot subtasks", L is a formatting rule, S is an assembly sequence as a dictionary, P is a list of part names, B is a list of available behavior primitives, and E is a set of high-quality examples for a small number of shot prompts. The user can then provide a language description of the task T, such as "Assemble a toy car". The script generation module 133 uses the language model 131 to generate and debug scripts for subtasks and associated behavior labels ( For example , "Pick the baseplate" subtask and "Pick" behavior label, etc.). For a given subtask, the initial chat context input into the language model 131 can be expressed as (R, L, A, W, P, E), where R is a role, such as or "You are a chat agent generating code for a bot", L is a scripted rule, W is an assembly context as a dictionary, D is a reference document for the programming language API, and E is a scripted example associated with a behavior label. The generated script S and any syntax errors or runtime exceptions X can be attached to such a context during each iteration to improve the previous version: (R, L, A, W, D, E, S1, X 1 ,...,S n ).

[0050] Figure 5 is a flow chart of method steps for controlling a robot according to various embodiments. Figures 1 to 4 The method steps are described herein as a system, but one skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the embodiments.

[0051] As shown, the method 500 begins at step 501, where the robot control application 126 receives a natural language input specifying a robot task. A user may enter a robot task using one or more input devices 208, and the robot task is communicated to the robot control application 126. In some embodiments, the input may include a text description of the task, such as "assemble axis".

[0052] At step 502, the task decomposition module 132 of the robot control application 126 processes the robot task specified in the natural language input via the language model 131 to decompose the robot task into robot subtasks. The task decomposition module 132 uses the language model 131 to interpret the natural language description of the robot task and transforms the natural language description into a set of structured robot subtasks, such as the robot subtasks in the robot subtask information 304. The task decomposition module 132 uses the task decomposition examples 302 (which may include templates or historical data on how to decompose similar robot tasks into subtasks) to guide the language model 131 on how to decompose the robot task into subtasks. In addition, the task decomposition module 132 uses the formatting rules 303 to ensure that the output of the language model 131 has a format suitable for processing by the script generation module 133. The output of the task decomposition module 132 is a series of robot subtask information, which includes but is not limited to discrete, executable robot actions "detect axis", "pick up axis", "move axis" and "insert axis", which are consistent with the primary goals outlined in the robot task specified in the natural language input. In some embodiments, the output of the task decomposition module 132 may also include behavior labels, such as "detect," "pick up," "move," and "insert," assigned to each subtask and corresponding to common robot behaviors.

[0053] At step 503, the script generation module 133 of the robot control application processes the robot subtask information via the language model 131 to generate a subtask script ( For example , subtask script 401). Combined with Figure 6 Step 503 is described in more detail.

[0054] At step 504, the robot control application 126 executes each subtask script in sequence to control the robot ( For example , robot 150). In some embodiments, the robot control application 126 can translate the subtask script into specific commands that prompt the robot to perform physical actions. In some embodiments, the robot can be controlled to perform any suitable action. For example, the robot can be controlled to grasp and manipulate objects, such as picking up an object, placing an object, and / or inserting an object into another object. For another example, the robot can be controlled to assemble objects together. In various embodiments, the robot control application 126 uses one or more sensors ( For example , sensors 140) to monitor the robot's movements to obtain feedback to confirm accuracy and / or otherwise correct each movement.

[0055] Figure 6 is a flow chart of method steps for generating a subtask script according to various embodiments. Figures 1 to 5The method steps are described herein as a system, but one skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the embodiments.

[0056] As shown in the figure, at step 601, the subtask context module 402 receives robot subtask information ( For example , robot subtask information 304). The subtask context module 402 also receives one or more scripted rules ( For example , scripted rule 408) and one or more scripted examples ( For example , scripted example 409). The scripting rules include grammatical and semantic guidelines to ensure that the script generated by the script generation module 133 is coherent and executable. For example, if the script is to be generated in Python, the scripting rules may include standards such as using Python specific code, including basic imports, or encapsulating logic within a main function. Similarly, for C++ scripts, the scripting rules may include guidelines for including header files, using appropriate namespace declarations, and adhering to object-oriented principles. The scripted example is an example of an executable script for a subtask similar to the subtask specified in the robot subtask information. For example, in some embodiments, the behavior label ( For example , "pick") can be used to search within a set of files or database examples associated with the same behavior tag. In some embodiments, the scripted examples include scripts with different levels of complexity, thereby enriching the diversity and applicability of the scripted examples. The subtask context module 402 combines the robot subtask information with the subtask information ( For example , the scripted rules and scripted examples within the subtask information 405) are stored together.

[0057] At step 602, the script generation module 133 uses the language model 131 based on the subtask information and the script history ( For example , script history 406) and simulation history ( For example , simulation history 407))'s subtask context to generate a temporary script for the robot subtask ( For example , temporary script 413). The subtask context module 402 communicates the subtask context to the text replacer module 401, which systematically replaces any proprietary and confidential terms with public counterparts. The text-replaced subtask context is then input into the language model 131, which generates a script that may or may not include errors. After script generation, the preliminary script is subjected to another round of text replacement by the text replacer module 401 to ensure that all terms are non-proprietary, thereby generating a temporary script, which is then stored in the script history ( For example , script history 406).

[0058] At step 603, the script verification module 403 verifies the temporary script for various parameters such as correct syntax, proper format, and / or absence of runtime exceptions. In some embodiments, the script verification module 403 also checks to ensure that the temporary script is a script that includes program code.

[0059] At step 604, if there are validation errors, the method 500 proceeds to step 605, where the script generation module 133 appends the errors to the subtask context. In some embodiments, validation errors are stored in the script history ( For example , script history 406), and simulation errors are stored in the simulation history ( For example , simulation history 407). Then, the method 500 returns to step 602, where the script generation module 133 generates another temporary script for the robot subtask based on the robot subtask context.

[0060] If there are no verification errors, the method 500 directly proceeds to step 606, where the simulation module 404 simulates the temporary script. The simulation module 404 uses the temporary script to control the robot to perform a virtual test run to evaluate the temporary script in the guiding robot subtask.

[0061] At step 607, if there is a simulation error, the method 500 proceeds to step 605, where the script generation module 133 appends the error to the subtask context. On the other hand, if there is no simulation error, the method proceeds directly to step 608, where the script generation module 133 saves the subtask script.

[0062] At step 609, the script generation module 133 checks whether there are more robot subtasks. If there are no more robot subtasks, the method 500 continues to step 504, where the robot control application 126 executes the subtask scripts in sequence to control the robot, as described above in conjunction with Figure 5 If there are more robot subtasks, the method 500 returns to step 602, where the script generation module 133 generates a temporary script for another robot subtask based on the subtask context using the language model 131. Although shown as an iterative process in which subtasks generate subtask scripts one by one, in some embodiments, the script generation module 133 can be implemented using multiple agents that execute in parallel to generate scripts for different subtasks specified in the robot subtask information received in step 601, and then simulate and debug the scripts in sequence to ensure that the tasks can be successfully executed.

[0063] In summary, a technology for automatically programming a robot to perform a task is disclosed. Given a natural language input specifying a robot task, a language model is used to design and generate an executable script to control the robot to perform the task. In some embodiments, the robot control application prompts the language model to decompose the task specified in the natural language input into a series of simpler subtasks. Then, the robot control application prompts the language model to generate an executable script for each of the subtasks. The robot control application verifies the generated script to identify compile-time errors and runtime errors. Verification can include simulating the execution of the script in sequence to control the robot within a virtual environment. When an error is identified during verification, the robot control application prompts the language model to modify the script to eliminate the error. After the executable script is generated and verified, the script can be executed in sequence to control the physical robot to perform the task.

[0064] At least one technical advantage of the disclosed technology over the prior art is that, unlike conventional robotic systems that cannot perform new tasks without extensive manual reprogramming, the disclosed technology automatically generates executable scripts for controlling a robot to perform new tasks. Using the disclosed technology, a robot can be adapted to various scenarios requiring the robot to perform different tasks without requiring extensive manual reprogramming and expertise. These technical advantages represent one or more technical advances over prior art approaches.

[0065] 1. In some embodiments, a computer-implemented method for generating program code to control a robot includes: receiving user input specifying a task to be performed by the robot; processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.

[0066] 2. A computer-implemented method according to clause 1, further comprising: executing one or more operations based on the program code for controlling the robot to perform the subtasks to simulate the robot performing each subtask included in the multiple subtasks; and in response to one or more errors when simulating the robot performing a first subtask included in the multiple subtasks, processing the first subtask and the one or more errors via the second machine learning model to generate additional program code for controlling the robot to perform the first subtask.

[0067] 3. A computer-implemented method according to clause 1 or 2, wherein the user input is natural language input and each of the first machine learning model and the second machine learning model is a language model.

[0068] 4. A computer-implemented method according to any one of clauses 1 to 3, further comprising: for each subtask included in the multiple subtasks, selecting one or more program code examples associated with the subtask from multiple program code examples, wherein processing the subtask via the second machine learning model includes inputting the subtask and the one or more program code examples associated with the subtask into the second machine learning model.

[0069] 5. A computer-implemented method according to any one of clauses 1 to 4, wherein processing the user input via the first machine learning model includes inputting at least one of the following into the first machine learning model: a role of the first machine learning model for generating the plurality of subtasks, one or more rules for outputting the first machine learning model, one or more parameters associated with the robot, or one or more parameters associated with the task.

[0070] 6. A computer-implemented method according to any one of clauses 1 to 5, wherein processing the subtask via the second machine learning model includes inputting at least one of the following into the second machine learning model: a role of the second machine learning model for generating program code, one or more rules for outputting the second machine learning model, one or more parameters associated with the robot, one or more parameters associated with the task, one or more definitions associated with a programming language associated with the program code, or one or more program code examples.

[0071] 7. A computer-implemented method according to any one of clauses 1 to 6, wherein processing the subtask via the second machine learning model further includes inputting at least one of one or more previous outputs of the second machine learning model or one or more errors from one or more simulations of the robot performing the subtask into the second machine learning model.

[0072] 8. The computer-implemented method of any one of clauses 1 to 7, further comprising performing one or more operations to verify that the program code for controlling the robot to perform the subtask is program code.

[0073] 9. A computer-implemented method according to any one of clauses 1 to 8, wherein the first machine learning model is the second machine learning model.

[0074] 10. The computer-implemented method of any of clauses 1 to 9, further comprising causing the robot to move based on the program code for each subtask.

[0075] 11. In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by at least one processor, cause the at least one processor to perform the following steps: receiving user input specifying a task to be performed by the robot; processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.

[0076] 12. One or more non-transitory computer-readable media according to claim 11, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: performing one or more operations based on the program code for controlling the robot to perform the subtasks to simulate the robot performing each subtask included in the multiple subtasks; and in response to one or more errors when simulating the robot performing the first subtask included in the multiple subtasks, processing the first subtask and the one or more errors via the second machine learning model to generate additional program code for controlling the robot to perform the first subtask.

[0077] 13. One or more non-transitory computer-readable media according to clause 11 or 12, wherein the one or more errors include at least one of a runtime exception, a syntax error, a format error, a conflict, or a failure to reach a final state associated with the task.

[0078] 14. One or more non-transitory computer-readable media according to any one of clauses 11 to 13, wherein each of the first machine learning model and the second machine learning model comprises a large language model (LLM).

[0079] 15. One or more non-transitory computer-readable media according to any one of clauses 11 to 14, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: for each subtask included in the multiple subtasks, select one or more program code examples associated with the subtask from a plurality of program code examples, wherein processing the subtask via the second machine learning model includes inputting the subtask and the one or more program code examples associated with the subtask into the second machine learning model.

[0080] 16. One or more non-transitory computer-readable media according to any one of clauses 11 to 15, wherein processing the user input via the first machine learning model includes inputting at least one of the following into the first machine learning model: a role of the first machine learning model for generating the plurality of subtasks, one or more rules for outputting the first machine learning model, one or more parameters associated with the robot, or one or more parameters associated with the task.

[0081] 17. One or more non-transitory computer-readable media according to any one of clauses 11 to 16, wherein processing the subtask via the second machine learning model includes inputting at least one of the following into the second machine learning model: a role of the second machine learning model for generating program code, one or more rules for outputting the second machine learning model, one or more parameters associated with the robot, one or more parameters associated with the task, one or more definitions associated with a programming language associated with the program code, or one or more program code examples.

[0082] 18. One or more non-transitory computer-readable media according to any one of clauses 11 to 17, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: generating the one or more program code examples based on the behavior tags associated with the subtask.

[0083] 19. One or more non-transitory computer-readable media according to any one of clauses 11 to 18, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: causing the robot to move based on the program code for each subtask.

[0084] 20. In some embodiments, a system includes one or more memories storing instructions and one or more processors, wherein the one or more processors are coupled to the one or more memories and are configured, when executing the instructions, to: receive user input specifying a task to be performed by the robot; process the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and for each subtask included in the plurality of subtasks, process the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.

[0085] Any and all combinations of any of the claim elements in any of the claims and / or any elements described in this application in any manner fall within the contemplated scope of this disclosure and protection.

[0086] Descriptions of various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0087] Aspects of embodiments of the present invention may be embodied as systems, methods, or computer program products. Thus, aspects of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software aspects with hardware aspects, which implementations may all be generally referred to herein as "modules" or "systems." In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0088] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0089] Aspects of the present disclosure are described above with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure.It should be understood that each frame of the flowchart illustration and / or block diagram and the frame combination of the flowchart illustration and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine.When executed via the processor of a computer or other programmable data processing device, the instruction causes the function / action specified in one or more frames of the flowchart and / or block diagram to be realized.Such processors can be but are not limited to general-purpose processors, special-purpose processors, application-specific processors or field programmable gate arrays.

[0090] The flow charts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram may represent a module, segment or part of a code, and the code includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the box may not appear in the order mentioned in the accompanying drawings. For example, the two boxes shown in succession can actually be executed substantially simultaneously, or these boxes can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each box illustrated in the block diagram and / or flow chart and the combination of boxes in the block diagram and / or flow chart can be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or operation.

[0091] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope of the disclosure, and the scope of the disclosure is determined by the claims that follow.

Claims

1. A computer-implemented method for generating program code to control a robot, the method comprising: receiving user input specifying a task to be performed by the robot; processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and For each subtask included in the plurality of subtasks, the subtask is processed via a second machine learning model to generate a program code for controlling the robot to perform the subtask.

2. The computer-implemented method of claim 1 , further comprising: performing one or more operations based on the program code for controlling the robot to perform the subtasks to simulate the robot performing each subtask included in the plurality of subtasks; and In response to simulating one or more errors when the robot performs a first subtask included in the multiple subtasks, the first subtask and the one or more errors are processed via a second machine learning model to generate additional program code for controlling the robot to perform the first subtask.

3. A computer-implemented method according to claim 1, wherein the user input is a natural language input, and each of the first machine learning model and the second machine learning model is a language model.

4. The computer-implemented method of claim 1 , further comprising: For each subtask included in the multiple subtasks, one or more program code examples associated with the subtask are selected from multiple program code examples, wherein processing the subtask via the second machine learning model includes inputting the subtask and the one or more program code examples associated with the subtask into the second machine learning model.

5. A computer-implemented method according to claim 1, wherein processing the user input via the first machine learning model includes inputting at least one of the following into the first machine learning model: a role of the first machine learning model for generating the plurality of subtasks, one or more rules for outputting the first machine learning model, one or more parameters associated with the robot, or one or more parameters associated with the task.

6. A computer-implemented method according to claim 1, wherein processing the subtask via the second machine learning model includes inputting at least one of the following into the second machine learning model: a role of the second machine learning model for generating program code, one or more rules for outputting the second machine learning model, one or more parameters associated with the robot, one or more parameters associated with the task, one or more definitions associated with a programming language associated with the program code, or one or more program code examples.

7. A computer-implemented method according to claim 1, wherein processing the subtask via the second machine learning model further includes inputting at least one of one or more previous outputs of the second machine learning model or one or more errors from one or more simulations of the robot performing the subtask into the second machine learning model.

8. The computer-implemented method of claim 1, further comprising performing one or more operations to verify that program code for controlling the robot to perform the subtask is program code.

9. The computer-implemented method of claim 1 , wherein the first machine learning model is the second machine learning model.

10. The computer-implemented method of claim 1, further comprising causing the robot to move based on the program code for each subtask.

11. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the following steps: receiving user input specifying a task to be performed by the robot; processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and For each subtask included in the plurality of subtasks, the subtask is processed via a second machine learning model to generate a program code for controlling the robot to perform the subtask.

12. The one or more non-transitory computer-readable media of claim 11, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: performing one or more operations based on the program code for controlling the robot to perform the subtasks to simulate the robot performing each subtask included in the plurality of subtasks; and In response to simulating one or more errors when the robot performs a first subtask included in the multiple subtasks, the first subtask and the one or more errors are processed via a second machine learning model to generate additional program code for controlling the robot to perform the first subtask.

13. The one or more non-transitory computer-readable media of claim 12, wherein the one or more errors include at least one of a runtime exception, a syntax error, a format error, a conflict, or a failure to reach a final state associated with the task.

14. The one or more non-transitory computer-readable media of claim 11, wherein each of the first machine learning model and the second machine learning model comprises a large language model (LLM).

15. One or more non-transitory computer-readable media according to claim 11, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: for each subtask included in the multiple subtasks, select one or more program code examples associated with the subtask from a plurality of program code examples, wherein processing the subtask via the second machine learning model includes inputting the subtask and the one or more program code examples associated with the subtask into the second machine learning model.

16. One or more non-transitory computer-readable media according to claim 11, wherein processing the user input via the first machine learning model includes inputting at least one of the following into the first machine learning model: a role of the first machine learning model for generating the plurality of subtasks, one or more rules for outputting the first machine learning model, one or more parameters associated with the robot, or one or more parameters associated with the task.

17. One or more non-transitory computer-readable media according to claim 11, wherein processing the subtask via the second machine learning model includes inputting at least one of the following into the second machine learning model: a role of the second machine learning model for generating program code, one or more rules for outputting the second machine learning model, one or more parameters associated with the robot, one or more parameters associated with the task, one or more definitions associated with a programming language associated with the program code, or one or more program code examples.

18. The one or more non-transitory computer-readable media of claim 11, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the following steps: generating the one or more program code examples based on a behavior tag associated with the subtask.

19. The one or more non-transitory computer-readable media of claim 11, wherein when executed by the at least one processor, the instructions further cause the at least one processor to perform the step of causing the robot to move based on the program code for each subtask.

20. A system comprising: one or more memories storing instructions; and one or more processors coupled to the one or more memories and configured, when executing the instructions, to: receiving user input specifying a task to be performed by the robot, processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task, and For each subtask included in the plurality of subtasks, the subtask is processed via a second machine learning model to generate a program code for controlling the robot to perform the subtask.

Images