Automated code generation using generative AI and reinforcement learning

By combining generative AI and reinforcement learning techniques, automated systems can automatically generate and simulate machine programs, solving the problem of complex and inefficient low-level control programming in existing technologies, and achieving efficient and accurate configuration of automated systems.

CN123125573APending Publication Date: 2026-11-17SIEMENS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202480090979.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-04-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies require a great deal of expertise and time to configure and program automated system equipment, especially in low-level control, and rely on manual programming, which is inefficient.

Method used

By combining generative artificial intelligence (AI) systems and reinforcement learning (RLFH) technology, machine programs are generated and simulated by receiving user prompts and simulated environment information, enabling efficient programming of automated systems.

Benefits of technology

It enables efficient and accurate automated system programming, reduces manual intervention, and improves programming efficiency and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125573A_ABST
    Figure CN123125573A_ABST
Patent Text Reader

Abstract

Methods for programming automated systems and other devices using reinforcement learning based on human feedback, as well as corresponding systems and computer-readable media. One method includes: receiving (210) a user prompt (168) including a high-level task description by a generative artificial intelligence (AI) system (204) implemented through one or more computer systems (100). The method includes: receiving (210) simulation environment information (158) corresponding to the user prompt (168) by the AI ​​system (204), wherein the simulation environment information (158) includes configuration information (160). The method includes: generating (220) a machine program (162) corresponding to the user prompt (168) and consistent with the configuration information (160) by the AI ​​system (204). The method includes: simulating (222) the machine program (162) in the simulation environment (208) implemented by the one or more computer systems (100).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for generating executable code, including but not limited to control code for low-level control in automation systems. Background Technology

[0002] One of the key challenges in configuring, assembling, or modifying automated system equipment is programming the low-level control of the machine (such as sensors and actuators) to achieve processing objectives. Breaking down requirements into processes for planning, programming, testing, and validating automated systems requires a degree of expertise and can be time-consuming, depending on their complexity. Current approaches rely on manual programming of individual devices and systems. Improvements are needed. Summary of the Invention

[0003] Various disclosed implementations include methods for programming automated systems and other devices using reinforcement learning from human feedback, as well as corresponding systems and computer-readable media. One method includes: a generative artificial intelligence (AI) system implemented by one or more computer systems receiving user prompts including a high-level task description. The method includes: the AI ​​system receiving simulation environment information corresponding to the user prompts, wherein the simulation environment information includes configuration information. The method includes: the AI ​​system generating a machine program corresponding to the user prompts and consistent with the configuration information. The method includes: simulating the machine program in a simulation environment implemented by one or more computer systems.

[0004] Various implementations also include: executing a machine program by at least one physical device. Various implementations also include: generating a low-level action plan by an AI system that corresponds to user prompts and is consistent with configuration information; generating a machine program based on the low-level action plan when the user approves it; and repeatedly receiving user prompts when the user does not approve the low-level action plan.

[0005] Various implementations also include: storing user prompts and machine programs when the user approves the simulated machine program, and repeatedly receiving user prompts when the user does not approve the simulated machine program.

[0006] In various implementations, the configuration information includes information defining the inputs, outputs, settings, and other parameters available to the physical machine to be programmed. In various implementations, the configuration information includes information defining the operations that the physical machine to be programmed can perform. In various implementations, the AI ​​system also creates a semantic model of the physical machine to be programmed.

[0007] The disclosed embodiments also include a computer system comprising a processor and accessible memory, specifically configured to perform the processes disclosed herein. The disclosed embodiments also include a non-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more computer systems to perform the processes disclosed herein.

[0008] The features and technical advantages of this disclosure have been summarized rather broadly above to enable those skilled in the art to better understand the following detailed descriptions. Additional features and advantages of this disclosure will be described below, and these features and advantages form the subject matter of the claims of this disclosure. Those skilled in the art will understand that they can readily use the disclosed concepts and detailed descriptions as the basis for modifying or designing other structures to achieve the same purpose as this disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the spirit and scope of this disclosure in its broadest form.

[0009] Before proceeding to the detailed embodiments below, it is advantageous to define certain words or phrases used throughout this patent document: the terms “comprising” and “including” and their derivatives mean to include without limitation; the term “or” is inclusive, meaning and / or; the phrases “associated with” and “related to” and their derivatives may mean to include, to be included within, to be interconnected with, to contain, to be contained within, to be connected to or linked to, to be coupled to or coupled to, to be able to communicate with, to cooperate with, to interleave, to be juxtaposed, to be adjacent to, to be bound to or bound to, to have, to have the attributes of, etc.; and the term “controller” means any device, system or part thereof that controls at least one operation, whether the device is implemented in hardware, firmware, software or some combination thereof. It should be noted that the functionality associated with any particular controller can be centralized or distributed, whether local or remote. Definitions of certain words and phrases are provided throughout this patent document, and those skilled in the art will understand that such definitions apply in many (if not most) cases to the prior and future uses of these defined words and phrases. While some terms may encompass a wide variety of implementations, the appended claims may expressly limit these terms to a particular implementation. Attached Figure Description

[0010] To provide a more complete understanding of this disclosure and its advantages, reference is now made to the following description in conjunction with the accompanying drawings, wherein like numerals denote like objects, and wherein: Figure 1 A block diagram of a computer system in which one implementation method can be carried out is shown; Figure 2 An example of a process according to the disclosed implementation method is shown; Figure 3 An example of an input / output (I / O) configuration interface according to the disclosed embodiment is shown; Figure 4 An example of a modal interface according to the disclosed implementation is shown; and Figure 5 An example of a simulation based on the disclosed implementation is shown. Detailed Implementation

[0011] The following discussion Figures 1 to 5 The various embodiments used to describe the principles of this disclosure are for illustrative purposes only and should not be construed as limiting the scope of this disclosure in any way. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged device. Numerous innovative teachings of this application will be described with reference to exemplary, non-limiting embodiments.

[0012] The disclosed implementation improves upon current techniques for configuring and programming automated systems and devices. The processes disclosed herein combine reinforcement learning (RLFH) techniques with human feedback with large language models (LLM) to achieve efficient and accurate automated code generation.

[0013] In various implementations, the system may receive business requirements and machine configurations from the user, which are used as prompts for the LLM (Local Management Model). The system can then use the LLM to generate low-level action plans. In some cases, the system may interact with the user to refine the prompts, thereby obtaining a more suitable plan or validating the low-level action plan.

[0014] The system can then translate the action plan into a machine program, such as, but not limited to, Structured Control Language (SCL) code or Structured Text (ST) code. The system can then test and validate this code in a simulated environment using RLFH or other techniques. Similarly, the system can again receive user modifications or validations. The system can then transmit the machine program to the actual physical hardware for execution. High-quality hints that help generate optimized machine programs can be saved as templates and can be reconfigured as the machine is adapted to meet different needs.

[0015] Figure 1A block diagram of a computer system in which one implementation can be carried out is shown, for example as a computer system specifically configured by software or otherwise to perform the processes described herein, and particularly as each of the plurality of interconnected and communicating systems described herein. The depicted computer system includes a processor 102 connected to a secondary cache / bridge 104, which in turn is connected to a local system bus 106. The local system bus 106 may be, for example, a peripheral component interconnect (PCI) architecture bus. In the depicted example, main memory 108 and a graphics adapter 110 are also connected to the local system bus. The graphics adapter 110 may be connected to a display 111.

[0016] Other peripheral devices, such as LAN / WAN / wireless (e.g., WiFi) adapter 112, can also be connected to the local system bus 106. An expansion bus interface 114 connects the local system bus 106 to the input / output (I / O) bus 116. The I / O bus 116 connects to a keyboard / mouse adapter 118, a disk controller 120, and an I / O adapter 122. The disk controller 120 can be connected to a memory 126, which can be any suitable machine-usable or machine-readable storage medium, including but not limited to non-volatile, hard-coded media such as read-only memory (ROM) or erasable electrically programmable read-only memory (EEPROM), magnetic tape storage, and user-recordable media such as floppy disks, hard disk drives, and compact disc read-only memory (CD-ROM) or digital universal disc (DVD), as well as other known optical, electrical, or magnetic storage devices.

[0017] The memory 126 may store any data that may be necessary or useful for performing the process described herein, including (but not limited to) executable code 152, AI system 154, simulation system 156, simulation environment information 158, configuration information 160, machine program 162, action plan 164, semantic model 166, user prompts 168, and other data 170.

[0018] In the example shown, audio adapter 124 is also connected to I / O bus 116, and a speaker (not shown) can be connected to this adapter to play sound. Keyboard / mouse adapter 118 provides connectivity for clicking devices (not shown), such as a mouse, trackball, trackpoint clicker, touchscreen, etc.

[0019] Those skilled in the art will understand that Figure 1 The hardware depicted may vary depending on the specific implementation. For example, other peripheral devices, such as optical disc drives, may be used to complement or replace the depicted hardware. The examples depicted are provided for illustrative purposes only and are not intended to imply any architectural limitations on the present disclosure.

[0020] A computer system according to embodiments of this disclosure includes an operating system employing a graphical user interface. The operating system allows multiple display windows to be presented simultaneously in the graphical user interface, each providing an interface for a different application or different instances of the same application. A cursor in the graphical user interface can be manipulated by the user by clicking the device. The cursor position can be changed and / or events, such as clicking a mouse button, can be generated to trigger a desired response.

[0021] With appropriate modifications, one of various commercially available operating systems may be used, such as a version of Microsoft Windows™, a product of Microsoft Corporation located in Redmond, Washington. The operating system is modified or created in accordance with the contents of this disclosure.

[0022] LAN / WAN / wireless adapter 112 can connect to network 130 (not part of computer system 100), which can be any public or private computer system network or combination of networks known to those skilled in the art, including the Internet. Computer system 100 can communicate with server system 140 via network 130, which is also not part of computer system 100, but can be implemented as a separate computer system 100, for example.

[0023] Figure 2 The illustration shows an example of a process 200 for improving the quality of user prompts in an automated environment using reinforcement learning with human feedback, in which a generative AI system generates a PLC program after receiving a high-level task description and simulated environment information from the user as input.

[0024] Figure 2 The example illustrates actions that can be performed, for example, between a human user 202, a generative AI system 204 implemented by one or more computer systems, a real machine 206 (such as a PLC or device controlled by generated PLC code), an automation system or other equipment, and a simulation system 208, which may also be implemented by one or more computer systems. This example will be further illustrated in the context of developing code to deliver workpieces on a conveyor system, but the disclosed process can be applied to any automation system or other equipment and system.

[0025] In the context of this example, configuration information may include vendor-provided machine I / O configurations, such as: I0: The hopper is empty. I1: The workpiece is at the starting end of the conveyor belt. I2: The workpiece is at the end of the conveyor belt I3: The workpiece is in the middle of the conveyor belt. I4: Slider moves forward I5: Slider retracts Q0: The conveyor belt moves forward. Q1: Reverse conveyor belt Q2: Propel the slider In this example, I / O labels starting with "I" are sensors, and those starting with "Q" are actuators.

[0026] At 210, a user input occurs, and a generative AI system 204 (“AI system”) implemented by one or more computer systems receives a user prompt comprising one or more high-level tasks, the description of which may include simulation environment information. As used herein, “receive” can include loading from storage, receiving from another device or process, receiving through interaction with a user, or otherwise receiving. The simulation environment information may include configuration information for machine 206, such as information defining what inputs, outputs, settings, and other parameters are available for a given machine 206, which of these can be configured, what operations each machine, component, or device can perform, and other information.

[0027] An example of a high-level task received from a user could be, “Move the items in the silo one by one onto the conveyor belt for processing.”

[0028] At 212, AI system 204 generates a low-level action plan based on user prompts and returns the low-level action plan to the user. As part of 212, if the configuration information of machine 206 is not received as part of the simulation environment information, generative AI system 204 may receive such configuration information when needed to develop low-level action plans or generate programs described below.

[0029] In the exemplary context, generative AI system 204 can generate the following low-level action plan: 1. Check if the hopper is not empty. 2. If step 1 is true, the conveyor belt moves forward. Otherwise, no action is taken.

[0030] 3. The slider moves forward.

[0031] 4. Check if the workpiece is at the end of the conveyor belt.

[0032] 5. If step 4 is true, then repeat step 1.

[0033] At point 214, AI system 204 receives an instruction regarding whether the user agrees to the low-level action plan. If the user disagrees, at point 216, AI system 204 may receive the user's edits to prompts, configuration information, or other relevant data. At point 218, AI system 204 creates a new low-level action plan using the new input and returns to point 214.

[0034] Once the user agrees to the low-level action plan at 214 (if user confirmation is sought), at 220, the AI ​​system 204 generates at least one machine program for these low-level actions. This machine program may, for example, be PLC code specific to a particular PLC or device. The generated program may include any configuration information necessary or useful for simulating the operation of machine 206 or operating machine 206.

[0035] At 222, a simulation system 208 (“simulation system”, which may be implemented using the same computer system as AI system 204) implemented by one or more computer systems simulates the machine program in a simulation environment. This simulation includes the execution of the generated program. In various implementations, this may include simulating these programs in Siemens’ SIMIT simulation platform and may include using Siemens’ Mechatronics Concept Designer (MCD) software.

[0036] Figure 3 An example of an input / output (I / O) configuration interface 300 that can be used for simulation, such as using the SIMIT simulation platform and MCD software, is illustrated. This example shows the inputs 302 and outputs 304 of the system being simulated (in this example, a conveyor belt system). Various settings and parameters of the simulated system or device (machine 206) can be manipulated within the simulation system 208 to complete the simulation, such as by the user 202, the AI ​​system 204, or a generated program. The I / O (and other) configurations that can be presented in the configuration interface also precisely define what inputs, outputs, settings, and other parameters are available for a given machine 206 and which of these can be configured.

[0037] In many cases, the I / O configuration of hardware resources (including sensors and actuators) is defined by the equipment vendor. In such cases, when user 202 provides a high-level task description, including details of the I / O configuration of a specific machine 206, to generative AI system 204, AI system 204 uses this information to create a semantic model of the automated system, and can then create low-level action plans using only the available functionality. The creation of the semantic model can, for example, be performed as part of step 212.

[0038] Figure 4 An example of pattern 400 for creating a SIMIT interface for simulation testing using RLFH is shown. In this example, pattern 400 is implemented using a spreadsheet (where 402a continues into 402b).

[0039] Figure 5An example of a simulation 500 of the operation of machine 206 (in this case, a conveyor system component) using a generated program is shown. This example illustrates a SIMIT interface 502 that can be implemented based on configuration information, the generated program, and / or mode 400. This example also illustrates an MCD visual simulation 504 corresponding to the SIMIT interface 502, configuration information, the generated program, and / or mode 400.

[0040] Back Figure 2 In the process, at point 224, the AI ​​system can receive instructions on whether the user agrees to or approves the simulation results. If the user does not agree, the process returns to point 216.

[0041] If the user approves the simulation results at 224 (if approval is required), then at 226, the AI ​​system 204 stores the prompts, low-level action plans, and the generated program, and may store any other useful or necessary data.

[0042] Subsequently, at point 228, the generated machine program is executed by at least one physical device. Real hardware tests can be performed using the generated program on a real machine 206. Once the program is complete, it can be used to operate physical machines, devices, and systems.

[0043] In this example, AI system 204 uses reinforcement learning with human feedback by receiving user feedback at 214 and 224 (based on the action plan generated at 212 and the simulation at 222) and receiving revision prompts from the user at 216.

[0044] As described above, AI system 204 receives and uses machine configuration information as part of the disclosed process. This offers a significant advantage over other systems because it allows the generated programs to be specifically written to control particular devices or overall automated systems, which in turn control specialized machines with very specific and limited functionality.

[0045] These functionalities are described in the configuration information, such as the I / O configuration provided by the hardware resources (sensors and actuators). Therefore, when a user gives the AI ​​system 204 a high-level task with configuration information, the AI ​​system 204 will use the information to create a semantic model of the automation system and create low-level action plans using only the available functionalities.

[0046] In other cases, AI system 204 can also use the same high-level task and machine configuration information as input to generate multiple unique plans. The generated plans can then be simulated as described above, and AI system 204 can receive the user's selection of the most suitable plan based on the simulation results, thereby skipping the verification of the low-level action plan sequence generated by AI system 204.

[0047] Of course, those skilled in the art will recognize that, unless otherwise specifically instructed or required, certain steps described in the above process may be omitted, performed concurrently or sequentially, or performed in a different order. In particular, as described herein, the process may include automatically generating and simulating machine programs based on user prompts, allowing the user to directly review the simulation without having to provide manual feedback at each step.

[0048] Those skilled in the art will recognize that, for the sake of brevity, this document does not depict or describe the complete structure and operation of all computer systems suitable for use with this disclosure. Rather, only the parts of the computer system relevant to or necessary for understanding this disclosure are depicted and described. The remaining construction and operation of the computer system 100 may take any of the various current implementations and practices known in the art.

[0049] It is important to note that although this disclosure is described in the context of a fully functional system, those skilled in the art will appreciate that at least a portion of the mechanisms of this disclosure can be distributed in any of a variety of forms, in the form of instructions contained in a machine-usable, computer-usable, or computer-readable medium, and that this disclosure applies equally regardless of the particular type of instruction or signal-bearing medium or storage medium actually used to perform the distribution. Examples of machine-usable / readable or computer-usable / readable media include: non-volatile, hard-coded media such as read-only memory (ROM) or erasable, electrically programmable read-only memory (EEPROM), and user-recordable media such as floppy disks, hard disk drives, and optical disc read-only memory (CD-ROM) or digital versatile discs (DVDs).

[0050] Although exemplary embodiments of this disclosure have been described in detail, those skilled in the art will understand that various changes, substitutions, variations, and modifications can be made to the content disclosed herein without departing from the spirit and scope of this disclosure in its broadest form.

[0051] Nothing described in this application should be construed as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims: the scope of the patentable subject matter is limited only by the permissible claims. Furthermore, none of these claims are intended to invoke 35 USC §112(f) unless the exact phrase “means for…” is followed by a participle. Use of terms such as (but not limited to) “mechanism,” “module,” “device,” “unit,” “component,” “element,” “part,” “construct,” “device,” “machine,” “system,” “processor,” or “controller” in the claims is understood to refer to structures known to a person skilled in the art that are further modified or enhanced by the features of the claims themselves, and is not intended to invoke 35 USC §112(f).

Claims

1. A method comprising: A generative artificial intelligence (AI) system (204) implemented by one or more computer systems (100) receives (210) a user prompt (168) including a high-level task description. The AI ​​system (204) receives (210) simulation environment information (158) corresponding to the user prompt (168), wherein the simulation environment information (158) includes configuration information (160). A machine program (162) generated (220) by the AI ​​system (204) that corresponds to the user prompt (168) and is consistent with the configuration information (160); and The machine program (162) is simulated (222) in a simulation environment (208) implemented by the one or more computer systems (100).

2. The method of claim 1, further comprising executing (228) the machine program (162) by at least one physical device.

3. The method according to claim 1, further comprising: The low-level action plan (164) generated (212) by the AI ​​system (204) corresponds to the user prompt (168) and is consistent with the configuration information (160), and When the user (202) approves the low-level action plan (164), then the machine program (162) is generated (220), wherein the machine program is also consistent with the low-level action plan (164); and When the user (202) does not approve the low-level action plan (164), the user prompt (168) is then repeatedly received (210).

4. The method according to claim 1, further comprising: When the user (202) approves the simulated machine program (162), then the user prompt (168) and the machine program (162) are stored (226); and When the user (202) does not approve the simulated machine program (162), the user prompt (168) is then repeatedly received (210).

5. The method of claim 1, wherein, The configuration information (160) includes information that limits the inputs, outputs, settings and other parameters available to the physical machine to be programmed.

6. The method of claim 1, wherein, The configuration information (160) includes information that defines the operations that can be performed by the physical machine to be programmed.

7. The method according to claim 1, wherein, The AI ​​system (204) also creates (212) a semantic model (166) of the physical machine to be programmed.

8. A computer system (100) including a processor (102) and an accessible memory (108), said computer system (100) being particularly configured to perform the process according to any one of claims 1 to 7.

9. A non-transitory computer-readable medium (126) encoded with executable instructions (152) that, when executed, cause one or more computer systems (100) to perform the process according to any one of claims 1 to 7.