Automatic code generation using generative ai and reinforcement learning

Abstract

Methods for programming automation systems and other devices using reinforcement learning from human feedback and corresponding systems and computer-readable mediums. A method includes receiving (210), by a generative artificial intelligence (AI) system (204) implemented by one or more computer systems (100), a user prompt (168) including high-level task description. The method includes receiving (210), by the AI system (204), simulation environment information (158) corresponding to the user prompt (168), wherein the simulation environment information (158) includes configuration information (160). The method includes generating (220) a machine program (162), by the AI system (204), corresponding to the user prompt (168) and in accordance with the configuration information (160). The method includes simulating (222) the machine program (162) in a simulation environment (208) implemented by the one or more computer systems (100).

Description

AUTOMATIC CODE GENERATION USING GENERATIVE Al AND REINFORCEMENT LEARNINGTECHNICAL FIELD

[0001] The present disclosure is directed, in general, to systems and methods for generating executable code, including but not limited to control code for low-level controls in automation systems.BACKGROUND OF THE DISCLOSURE

[0002] One of the key challenges when configuring, assembling, or repurposing automation system equipment is programming the low-level controls, e.g., sensors and actuators, of the machines to accomplish the processing goal. The process of breaking down requirements to setting up a process plan, programming, testing and validation of automation system requires a certain level of expertise and can also take significant amount of time depending on its complexity. Current approaches rely on manual programming of individual devices and systems. Improved systems are desirable.SUMMARY OF THE DISCLOSURE

[0003] Various disclosed embodiments include methods for programming automation systems and other devices using reinforcement learning from human feedback and corresponding systems and computer-readable mediums. A method includes receiving, by a generative artificial intelligence (Al) system implemented by one or more computer systems, a user prompt including high-level task description. The method includes receiving, by the Al system, simulation environment information corresponding to the user prompt, wherein the simulation environment information includes configuration information. The method includes generating a machine program, by the Al system, corresponding to the user prompt and in accordance with the configuration information. The method includes simulating the machine program in a simulation environment implemented by the one or more computer systems.

[0004] Various embodiments also include comprising executing the machine program by at least one physical device. Various embodiments also include generating a low-level action plan, by the Al system, corresponding to the user prompt and in accordance with the configuration information, and when a user approves the low-level action plan, then generating the machine program in accordance with the low-level action plan, and when the user does not approve the low-level action plan, then repeating the receiving the user prompt.

[0005] Various embodiments also include, when a user approves the simulated machine program, then storing the user prompt and the machine program, and when the user does not approve the simulated machine program, then repeating the receiving the user prompt.

[0006] In various embodiments, the configuration information includes information that defines inputs, outputs, settings, and other parameters that are available for a physical machine to be programmed. In various embodiments, the configuration information includes information that defines the operations that can be performed by a physical machine to be programmed. In various embodiments, the Al system also creates a semantic model of a physical machine to be programmed.

[0007] Disclosed embodiments also include a computer system comprising a processor and an accessible memory, particularly configured to perform processes as disclosed herein. Disclosed embodiments also include a non-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more computer systems to perform processes as disclosed herein.

[0008] The foregoing has outlined rather broadly the features and technical advantages of the present disclosure so that those skilled in the art may better understand the detailed description that follows. Additional features and advantages of the disclosure will be described hereinafter that form the subject of the claims. Those skilled in the art will appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the art will also realize that such equivalent constructions do not depart from the spirit and scope of the disclosure in its broadest form.

[0009] Before undertaking the DETAILED DESCRIPTION below, it may be advantageous to set forth definitions of certain words or phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or” is inclusive, meaning and / or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, whether such a device is implemented in hardware, firmware, software or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words and phrases. While some terms may include a wide variety ofembodiments, the appended claims may expressly limit these terms to specific embodiments.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a more complete understanding of the present disclosure, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like numbers designate like objects, and in which:

[0011] FIG. 1 illustrates a block diagram of a computer system in which an embodiment can be implemented;

[0012] FIG. 2 illustrates an example of a process in accordance with disclosed embodiments;

[0013] FIG. 3 illustrates an example of an input / output (I / O) configuration interface in accordance with disclosed embodiments;

[0014] FIG. 4 illustrates an example of a schema interface in accordance with disclosed embodiments; and

[0015] FIG. 5 illustrates an example of a simulation in accordance with disclosed embodiments.DETAILED DESCRIPTION

[0016] FIGS. 1 through 5, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged device. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

[0017] Disclosed embodiments improve on current techniques for configuring and programming automation systems and devices. A process as disclosed herein can combine Reinforcement Learning with Human Feedback (RLFH) techniques along with Large Language Models (LLM) to achieve efficient and accurate automated code generation.

[0018] In various embodiments, the system can receive, from a user, business requirements and machine configurations which are used as a prompt for an LLM. The system can then use the LLM to generate low-level action plans. In some cases, the system may interact with the user to refine the prompts to get a better-suited plan or to validate a low-level action plan.

[0019] The system can then convert the action plan into a machine program, for example but not limited to a structured control language (SCL) code or structured text (ST) code. The system can test and validate such code in a simulation environment using RLFH or other techniques. Similarly, the system can again receive user modifications or validation. The system can then transmit the machine program to actual physical hardware for execution. The high-quality prompts which aided in generating optimized machine programs can be saved as template and can be reconfigured when the machine is repurposed for a different requirement.

[0020] FIG. 1 illustrates a block diagram of a computer system in which an embodiment can be implemented, for example as a computer system particularly configured by software or otherwise to perform the processes as described herein, and in particular aseach one of a plurality of interconnected and communicating systems as described herein. The computer system depicted includes a processor 102 connected to a level two cache / bridge 104, which is connected in turn to a local system bus 106. Local system bus 106 may be, for example, a peripheral component interconnect (PCI) architecture bus. Also connected to local system bus in the depicted example are a main memory 108 and a graphics adapter 110. The graphics adapter 110 may be connected to display 111.

[0021] Other peripherals, such as local area network (LAN) / Wide Area Network / Wireless (e.g. WiFi) adapter 112, may also be connected to local system bus 106. Expansion bus interface 114 connects local system bus 106 to input / output (I / O) bus 116. I / O bus 116 is connected to keyboard / mouse adapter 118, disk controller 120, and I / O adapter 122. Disk controller 120 can be connected to a storage 126, which can be any suitable machine usable or machine readable storage medium, including but not limited to nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), magnetic tape storage, and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs), and other known optical, electrical, or magnetic storage devices.

[0022] Storage 126 can store any data that may be necessary or useful for performing processes as described herein, including (but not limited to) executable code 152, Al system(s) 154, simulation system(s) 156, simulation environment information 158, configuration information 160, machine program(s) 162, action plan(s) 164, semantic model(s) 166, user prompt 168, and other data 170.

[0023] Also connected to I / O bus 116 in the example shown is audio adapter 124, to which speakers (not shown) may be connected for playing sounds. Keyboard / mouse adapter 118 provides a connection for a pointing device (not shown), such as a mouse, trackball, trackpointer, touchscreen, etc.

[0024] Those of ordinary skill in the art will appreciate that the hardware depicted in FIG. 1 may vary for particular implementations. For example, other peripheral devices, such as an optical disk drive and the like, also may be used in addition or in place of thehardware depicted. The depicted example is provided for the purpose of explanation only and is not meant to imply architectural limitations with respect to the present disclosure.

[0025] A computer system in accordance with an embodiment of the present disclosure includes an operating system employing a graphical user interface. The operating system permits multiple display windows to be presented in the graphical user interface simultaneously, with each display window providing an interface to a different application or to a different instance of the same application. A cursor in the graphical user interface may be manipulated by a user through the pointing device. The position of the cursor may be changed and / or an event, such as clicking a mouse button, generated to actuate a desired response.

[0026] One of various commercial operating systems, such as a version of Microsoft Windows™, a product of Microsoft Corporation located in Redmond, Wash, may be employed if suitably modified. The operating system is modified or created in accordance with the present disclosure as described.

[0027] LAN / WAN / Wireless adapter 112 can be connected to a network 130 (not a part of computer system 100), which can be any public or private computer system network or combination of networks, as known to those of skill in the art, including the Internet. Computer system 100 can communicate over network 130 with server system 140, which is also not part of computer system 100, but can be implemented, for example, as a separate computer system 100.

[0028] FIG. 2 illustrates an example of a process 200 for using reinforcement learning with human feedback to improve the quality of user prompts in an automation setting where a generative Al system generate PLC programs after receiving high level actions and simulation environment information as inputs from a user.

[0029] The example of FIG. 2 illustrates various aspects of actions that may be performed, for example, between a human user 202, a generative Al system 204 implemented by one or more computer systems, real machines 206 such as PLCs or devices being controlled by generated PLC code, automation systems, or other devices,and simulation systems 208 that also can be implemented by one or more computer systems. This example will be further illustrated in the context of developing code to deliver a workpiece on a conveyor system, but the disclosed processes can be applied to any automation system or to other devices and systems.

[0030] In the context of this example, the configuration information may include a vendor provided machine I / O configuration such as:• 10 : Magazine empty• Il : Workpiece at beginning of conveyor• 12 : Workpiece at end of conveyor• 13 : Workpiece in the middle of conveyor• 14 : Slide advanced• 15 : Slide retracted• Q0 : Move conveyor forward• QI : Reverse conveyor• Q2 : Advance Slide

[0031] In this example, I / O labels starting with “I” are sensors and starting with “Q” are actuators.

[0032] At 210, a user enters, and the generative Al system 204 implemented by one or more computer systems (the “Al system”) receives, a user prompt including one or more high-level tasks, which can include simulation environment information. “Receiving,” as used herein, can include loading from storage, receiving from another device or process, receiving via an interaction with a user, or otherwise. The simulation environment information can include configuration information for the machines 206, such as information that defines what inputs, outputs, settings, and other parameters are availablefor a given machine 206, which of those can be configured, what operations each machine, component, or device can perform, and other information.

[0033] One example of a high-level task received from a user may be, “Move items in magazine one by one to conveyor for processing.”

[0034] At 212, the Al system 204 generates a low-level action plan according to the user prompt and returns the low-level action plan to the user. As part of 212, if configuration information for the machines 206 were not received as part of the simulation environment information, the generative Al system 204 can receive such configuration information as necessary to develop the low-level action plan or generate the programs as described below.

[0035] In the exemplary context, generative Al system 204 may produce a low-level action plan such as:1. Check if magazine is not empty2. If Step 1 is True, Move conveyor forward. Otherwise do nothing.3. Slide advanced4. Check if workpiece at end of conveyor5. If Step 4 is True, repeat Step 1.

[0036] At 214, the Al system 204 receives an indication as to whether the user agrees with the low-level action plan. If the user does not agree, then at 216, the Al system 204 can receive user edits to the prompt, the configuration information, or other relevant data. At 218, the Al system 204 uses the new inputs to create a new low-level action plan and returns to 214.

[0037] One the user has agreed to the low-level action plan at 214 (if user confirmation is sought), then at 220, the Al system 204 generates at least one machine program for the low-level actions, which can be, for example, PLC code for specific PLCs or devices.The generated programs can include any configuration information necessary or useful for simulating operation of the machines 206 or for operating the machines 206.

[0038] At 222, a simulation system 208 implemented by one or more computer systems (the “simulation system,” which can be implemented using the same computer system(s) as the Al system 204) simulates the machine program in a simulation environment. The simulation includes execution of the generated programs. In various embodiments, this can include simulating the programs in the SIMIT Simulation Platform by Siemens and can include using the Mechatronics Concept Designer (MCD) software by Siemens.

[0039] FIG. 3 illustrates an example of an input / output (I / O) configuration interface 300 that could be used for a simulation, for example, using the SIMIT Simulation Platform and the MCD software. This example illustrates both inputs 302 and outputs 304 of the simulated system (in this example, a conveyor system). The various settings and parameters of the system or devices being simulated (the machines 206) can be manipulated in the simulation system 208 to accomplish the simulation, such as by user 202, the Al system 204, or the generated programs. I / O (and other) configurations as may be presented in a configuration interface also define exactly what inputs, outputs, settings, and other parameters are available for a given machine 206 and which of those can be configured.

[0040] In many cases, the I / O configuration of hardware resources, including sensors and actuators, are defined by the equipment vendor. In such cases, when the user 202 gives a high-level task to the generative Al system 204, including the I / O configuration details for the specific machines 206, the Al system 204 will use the information to create a semantic model of automation system and can then create low-level action plans using available functionalities only. Creation of the semantic model can be performed, for example, as part of step 212.

[0041] FIG. 4 illustrates an example of a schema 400 for creating a SIMIT interface for simulation testing using RLFH. In this example, the schema 400 is implemented using a spreadsheet (where 402a continues to 402b).

[0042] FIG. 5 illustrates an example of a simulation 500 of the operations of machines 206 (in this case, conveyor system components) using the generated programs. This example shows SIMIT interface 502 that can be implemented according to the configuration information, the generated programs, and / or the schema 400. This example also shows an MCD visual simulation 504 corresponding to the SIMIT interface 502, the configuration information, the generated programs, and / or the schema 400.

[0043] Returning to the process of FIG. 2, at 224, the Al system can receive an indication as to whether the user agrees with or approves the results of the simulation. If the user does not, the process returns to 216.

[0044] If the user approves the simulation results at 224 (if approval is required), then at 226, the Al system 204 stores the prompt, the low-level action plan, and the generated programs, and can store any other data that is useful or necessary.

[0045] Thereafter, at 228, the generated machine programs are executed by at least one physical device. Real machines 206 can be used to perform real hardware testing using the generated programs. Once the programs are complete, physical machines, devices, and systems can be operated using the generated programs.

[0046] In this example, the Al system 204 uses Reinforcement Learning with Human Feedback by receiving user feedback at 214 and 224, based on the action plans produced at 212 and the simulations at 222, and by receiving revised prompts from the user at 216.

[0047] As described above, the Al system 204 receives and uses machine configuration information as part of the disclosed process. This provides a significant advantage over other systems by enabling the generated programs to be specifically written to control specific devices or overall automation systems, which in turn controls specialized machine(s) with very specific and limited functionalities.

[0048] These functionalities are described in the configuration information such as vendor provided I / O configuration of hardware resources (sensors and actuators). Therefore, when user gives a high-level task to Al system 204 with configuration information, Al system 204 will use the information to create a semantic model ofautomation system and create plans low-level action plans using available functionalities only.

[0049] In other cases, the Al system 204 can also generate several unique plans using same high level task and machine configuration information as inputs. The multiple generated plans can then be simulated as described above and the Al system 204 can receive a user selection of the most suited plan on the basis of simulation results, skipping the validation of Al system 204 generated low-level action plan sequence.

[0050] Of course, those of skill in the art will recognize that, unless specifically indicated or required by the sequence of operations, certain steps in the processes described above may be omitted, performed concurrently or sequentially, or performed in a different order. In particular, as described herein, the process can include automatically generating and simulating the machine program based on the user prompt, so that the user can review the simulation(s) directly rather than providing human feedback at each step.

[0051] Those skilled in the art will recognize that, for simplicity and clarity, the full structure and operation of all computer systems suitable for use with the present disclosure is not being depicted or described herein. Instead, only so much of a computer system as is unique to the present disclosure or necessary for an understanding of the present disclosure is depicted and described. The remainder of the construction and operation of computer system 100 may conform to any of the various current implementations and practices known in the art.

[0052] It is important to note that while the disclosure includes a description in the context of a fully functional system, those skilled in the art will appreciate that at least portions of the mechanism of the present disclosure are capable of being distributed in the form of instructions contained within a machine-usable, computer-usable, or computer- readable medium in any of a variety of forms, and that the present disclosure applies equally regardless of the particular type of instruction or signal bearing medium or storage medium utilized to actually carry out the distribution. Examples of machine usable / readable or computer usable / readable mediums include: nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electricallyprogrammable read only memories (EEPROMs), and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs).

[0053] Although an exemplary embodiment of the present disclosure has been described in detail, those skilled in the art will understand that various changes, substitutions, variations, and improvements disclosed herein may be made without departing from the spirit and scope of the disclosure in its broadest form.

[0054] None of the description in the present application should be read as implying that any particular element, step, or function is an essential element which must be included in the claim scope: the scope of patented subject matter is defined only by the allowed claims. Moreover, none of these claims are intended to invoke 35 USC §112(f) unless the exact words "means for" are followed by a participle. The use of terms such as (but not limited to) “mechanism,” “module,” “device,” “unit,” “component,” “element,” “member,” “apparatus,” “machine,” “system,” “processor,” or “controller,” within a claim is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or enhanced by the features of the claims themselves, and is not intended to invoke 35 U.S.C. §112(f).

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving (210), by a generative artificial intelligence (Al) system (204) implemented by one or more computer systems (100), a user prompt (168) including high-level task description; receiving (210), by the Al system (204), simulation environment information (158) corresponding to the user prompt (168), wherein the simulation environment information (158) includes configuration information (160); generating (220) a machine program (162), by the Al system (204), corresponding to the user prompt (168) and in accordance with the configuration information (160); and simulating (222) the machine program (162) in a simulation environment (208) implemented by the one or more computer systems (100).

2. The method of claim 1, further comprising executing (228) the machine program (162) by at least one physical device.

3. The method of claim 1, further comprising: generating (212) a low-level action plan (164), by the Al system (204), corresponding to the user prompt (168) and in accordance with the configuration information (160), and when a user (202) approves the low-level action plan (164), then generating (220) the machine program (162), wherein the machine program is also in accordance with the low-level action plan (164); and when the user (202) does not approve the low-level action plan (164), then repeating the receiving (210) the user prompt (168).

4. The method of claim 1 , further comprising when a user (202) approves the simulated machine program (162), then storing (226) the user prompt (168) and the machine program (162); and when the user (202) does not approve the simulated machine program (162), then repeating the receiving (210) the user prompt (168).

5. The method of claim 1, wherein the configuration information (160) includes information that defines inputs, outputs, settings, and other parameters that are available for a physical machine to be programmed.

6. The method of claim 1, wherein the configuration information (160) includes information that defines the operations that can be performed by a physical machine to be programmed.

7. The method of claim 1, wherein the Al system (204) also creates (212) a semantic model (166) of a physical machine to be programmed.

8. A computer system (100) comprising a processor (102) and an accessible memory (108), the computer system (100) particularly configured to perform a process as in any of claims 1 -7.

9. A non-transitory computer-readable medium (126) encoded with executable instructions (152) that, when executed, cause one or more computer systems (100) to perform a process as in any of claims 1 -7.

Images