Automatic verification of artificial intelligence-generated automation code

Abstract

A method for generating and automatically verifying automation code includes receiving a code request including a natural language specification for a component of an automation system to execute a desired behavior. The code request is distilled using an LLM-based prompt interface to create separate prompts for an LLM-based code generator and an LLM-based code critic, based on which, the code generator generates an initial automation code, and the code critic formulates suggested aspects of the generated automation code to be tested. The method includes, based on the suggested aspects, generating and executing code-level test cases to verify the logic of the automation code and simulation-based test cases to verify a physical behavior of the component. Results of the test cases are processed by the code critic to formulate a code revision request. The automation code is modified by the code generator based on the code revision request and subsequently outputted.

Description

AUTOMATIC VERIFICATION OF ARTIFICIAL INTELLIGENCE-GENERATED AUTOMATION CODETECHNICAL FIELD

[0001] The present disclosure relates generally to the field of industrial automation, and in particular, to automation engineering systems for automatically verifying artificial intelligence (AI)- generated code for automation components.BACKGROUND

[0002] Large Language Models (LLMs) and other Generative Al (GenAI) technologies have recently shown impressive capabilities to automatically generate code, even for specialized languages like programmable logic controller (PLC) automation code. However, due to the known issues with LLMs regarding their tendencies to “hallucinate,” the generated code cannot always be trusted to be functionally accurate or optimal. Nevertheless, many industrial automation companies are developing LLM-based engineering assistant tools (e.g., Industrial Copilot for Engineering®, developed by Siemens AG and Microsoft Corporation) that can automatically generate PLC code based on a natural language prompt from the user. Ensuring accuracy and efficiency in the generated code typically requires additional tests for verification and validation (V&V) to be carried out.

[0003] A known approach to improve the accuracy and efficiency of Al-generated automation code involves a “brute force” approach to simply train a better LLM by collecting more training data (prompt and code), and refine the model through retraining (e.g., fine-tuning). This approach can require a large manual effort to build up a dataset of high-quality prompt & code pairs, which is especially difficult to find for specialized languages, such as programming languages for PLCs.

[0004] An alternative approach is to use a GenAI-based system (e.g., ChatGPT developed by OpenAI Inc.) to generate code-level test cases and use standard V&V test frameworks (e.g. Gunit® developed by Guidewire Software Inc., TestRail® developed by Gurock Software GmbH, TIA Portal® Test Suite developed by Siemens AG, etc.) to evaluate the generated code. However, this approach may not catch all issues in the generated code, such as violation of physics-based constraints, or system integration issues, among others. Additionally, it is often then a manualprocess to leverage the test results to improve the generated code.SUMMARY

[0005] Aspects of this disclosure provide methods, systems, and computer program products that address and overcome one or more of the above-described technical challenges. The disclosed methodology provides a multi-agent architecture including a number of LLM-based agents and leverages virtual commissioning tools, such as digital twins and physics-based simulation tools, to automatically verify and validate Al -generated automation code.

[0006] A first aspect of this disclosure provides a computer-implemented method for generating and automatically verifying automation code. The method comprises receiving a code request comprising a natural language specification for a component of an automation system to execute a desired behavior. The method further comprises processing the code request by a prompt interface comprising a first trained LLM to generate a first prompt for a code generator and a second prompt for a code critic. The method further comprises based on the first prompt, generating an initial automation code by the code generator comprising a second trained LLM. The method comprises based on the second prompt, formulating suggested aspects of the generated automation code to be tested, by the code critic comprising a third trained LLM. The method further comprises, based on the suggested aspects, generating and executing code-level test cases to verify a logic of the automation code and simulation-based test cases to verify a physical behavior of the component. The method further comprises processing results of the executed test cases by the code critic to formulate a code revision request. The method further comprises modifying the automation code by the code generator based on the code revision request. The method further comprises outputting the modified automation code.

[0007] Further aspects of this disclosure provide a computing system and a computer program product embodying the described method.

[0008] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0010] FIG. 1 is a high-level block diagram of an automation engineering system according to one or more embodiments of the disclosure.

[0011] FIG. 2 illustrates a multi-agent architecture for automatic verification of Al-generated automation code according to one or more embodiments of the disclosure.

[0012] FIG. 3 illustrates a block diagram of a computer system in which embodiments of the disclosure may be implemented.DETAILED DESCRIPTION

[0013] FIGS. 1 through 3, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged device. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

[0014] A large language model (LLM) essentially includes a deep learning model that can recognize and generate text, among other tasks. LLMs can include a very large number (often billions) of model parameters are capable of ingesting massive amounts of data, often from the Internet. LLMs typically include a type of neural network architecture called transformers. Transformer LLMs are capable of unsupervised training on huge sets of data through a now well- known concept of self-attention, to detect subtle ways that elements (“tokens”) in a sequence relate to each other. It is through this process that transformers learn to understand basic grammar, natural languages, and knowledge. Thus, on some level, LLMs can "understand" semantics in that they can associate words and concepts by their meaning, having seen them grouped together in that waymillions or billions of times. Any large, complex data set can be used to train LLMs, including programming languages. Some LLMs can help programmers write code. For example, they can write functions upon request — or, given some code as a starting point, they can finish writing a program based on the context of the code being typed.

[0015] Recent developments in LLMs offer an alternative to traditional automation programming. However, given the irregularities in LLM code generation, naive use of LLMs in the automation engineering workflow, signified by inefficient prompting and blind execution of unverified output code, can lead to unsafe operation. Yet, foundational models such as the GPT-4 LLM and LLama 2 LLM are challenging traditional approaches to automation and programming. Especially noteworthy is these models’ instructional (i.e., “chat-instruct”) capabilities, which allow for dynamic prompting based on a conversational input paradigm, which can enable automated feedback mechanisms. Additionally, the application of fine-tuning methods (process of taking pretrained LLMs and further training them on smaller, specific datasets) have made domain-specific training easier and significantly reduced compute and data requirements for these tasks. Recently, the LLM research community has pioneered “prompt-engineering,” - a practice of optimizing prompts resulting in more accurate and relevant LLM responses.

[0016] Verification and validation of LLM-generated automation code poses the challenge of not only verifying the logic of the code itself, but also verifying the physical behavior of one or more components of the automation system, such as controllers (e.g., PLC), robots, conveyors, sensors, among others. For example, if the behavior of an automation component involves violation of a physical constraint (e.g., collision), or a faulty physical interaction with another component, it may indicate that there is a bug in the automation code. However, such bugs cannot be uncovered by code-level testing that include tests such as unit tests, functional tests, integration tests, etc.

[0017] Virtual commissioning is a process for integration and acceptance testing of automation code written by experts. Simulation tools, such as Siemens Process Simulate®, Siemens NX MCD®, among others, provide virtual environments for safely verifying that the automation logic performs correctly under planned and unplanned conditions.

[0018] The present disclosure exploits and improves on these advances, in combination with LLM-based code generation techniques for industrial control systems, to produce an innovative,effective, and efficient automation process. The disclosed methodology provides a multi-agent architecture including a number of LLM-based agents and leverages virtual commissioning tools, such as digital twins and physics-based simulation tools, to automatically verify and validate AI- generated automation code.

[0019] A key difference between the disclosed embodiments and the state-of-the-art is the use of multiple LLM-based agents, including a prompt interface, a code generator and a code critic, among others, which can cooperate to distill a user’s intent specified in a natural language code request, to not just generate automation code, but also identify what the code needs to be appropriately tested for by formulating suggested aspects of the automation code to be tested, and then automatically generate and execute (i) code-level test cases to verify the logic of the automation code and (ii) simulation-based test cases to verify a physical behavior of the component. Certain types of issues with the generated automation code (e.g., system integration issues or violations of physics-based constraints) may only be uncovered by executing the automation code in a simulation environment, for example by investigating all execution paths (both normal and atypical). Furthermore, specialized test cases such as corner cases and fault recovery sequences (e.g., robot ESTOP recovery) can be effectively enumerated in test cases and validated in the simulation environment. The test results from both the code-level and simulation-based test cases are used to inform the code critic to generate a code revision request and generate a modified automation code by the code generator based on the code revision request.

[0020] In embodiments, the automation code may be iteratively verified by executing the codelevel and / or simulation-based test cases and modifying the automation code based on the returned test results until a convergence criterion is met.

[0021] The tested and validated code may be outputted and installed on the intended component of the automation system. The component can then be operated by executing the installed automation code.

[0022] As an added feature, since the generated automation code will be fully tested and verified by the system, the combination of user prompt and resulting code can be taken as additional training data for the code generator.

[0023] According to disclosed embodiments, a multi-agent architecture may be implemented using several intelligent agents that can generate responses in natural language or code and interact with other agents by calling respective APIs and may furthermore have access to necessary runtime environments of other system to execute code. These agents may include, a prompt interface, a code generator, a code critic, a test case generator, a code-level test interface and a simulation interface. The agents may interact in sequential or asynchronous patterns.

[0024] An “agent”, as described herein, is defined by a combination of software and hardware. The software may incorporate models, such as LLMs or may include algorithmic modules. The hardware may include memory storing such software and processor(s) to execute the software. The processing capability may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements.

[0025] According to disclosed embodiments, at least some (or all) of the agents may comprise a respective trained LLM. For example, a “trained” LLM may include a deep neural network such as a transformer, pre-trained by unsupervised learning on a large, diverse dataset, which may be configured to the specific task by fine-tuning model parameters using input-output pairs from a domain-specific dataset. Alternately, a “trained” LLM may include an off-the-shelf, fully trained large language model (e.g., GPT-4) that can be configured to the specific task by appropriate prompt engineering, such as by providing detailed context and / or relevant training examples in the prompt (one shot or few-shot learning). For example, some of the agents may be “pre-prompted” to provide context information when formulating requests or prompts for other agents. The “pre-prompts” include additional prompts that may be implemented, for example, as system-wide configurations or user-specific “profiles” for the system prior to runtime of the code verification workflow.

[0026] In some embodiments, one or more of the agents may comprise algorithmic modules (e.g., static code) configured to programmatically execute the specified task(s).

[0027] Turning now to the drawings, FIG. 1 shows an example of an automation engineering system 100 according to one or more embodiments. The automation engineering system 100 can include one or more processors and memory having stored thereon applications, agents, and computer program modules including, for example, an engineering tool 102, an engineeringassistant tool (“copilot”) 104, a digital twin creator 106, a simulation environment 110 and a virtual controller 112.

[0028] The engineering tool 102 may include an application that provides a framework for programmers to design and test automation code for various components (e.g., PLCs, robots, etc.), design human machine interfaces (HMI), and may additionally provide access to services such as planning, integrated engineering, operations, etc. A non-limiting example of an engineering tool is the Total Integrated Automation or HA portal® developed by Siemens AG.

[0029] The engineering copilot 104 may include a GenAI powered assistant for industrial engineering that can expedite the engineering effort by offloading a sizeable majority of the tasks to a system of dedicated LLM-based agents to generate and automatically verify automation code according to the disclosed embodiments. As mentioned above, many industrial automation companies are developing LLM-based engineering assistant tools (e.g., Industrial Copilot for Engineering®, developed by Siemens AG and Microsoft Corporation) that can automatically generate PLC code based on a natural language prompt from the user. The disclosed embodiments may be implemented as improvements to such tools, among other implementations.

[0030] In some embodiments, the engineering tool 102 may include a code verification and validation framework, that can run a suite of tests on a piece of automation code to verify the automation logic. To verify the physical behavior of the automation component, the disclosed embodiments leverage virtual commissioning tools, such as the digital twin creator 106, the simulation environment 110 and the virtual controller 112.

[0031] The digital twin creator 106 may be used to build a digital twin 108 of the automation system, for example, by a system integration engineer. To illustrate, in the case of a robotic cell for assembling parts, the digital twin 108 may comprise digitized 3D representations of one or more robots, fixtures, tools, conveyors, parts to be assembled, etc. The digital twin creator 106 may also be used to create a digital twin of a data acquisition system based on information such as number, placement and orientation of cameras, design of fixture to mount cameras, and camera settings such as exposure, color mode, resolution, analog gain, field of view, depth of focus, etc.

[0032] The simulation environment 110 may include a physics-based simulation engine to runsimulations of the automation system utilizing the digital twin 108. The initial conditions for the physics-based simulation environment 110 to run the simulations may be set based on the generated test case scenarios. In example embodiments, the simulation environment may comprise commercially available simulation software, such as Siemens Process Simulate®, Siemens NX MCD®, among others.

[0033] To run simulations to verify a piece of automation code, such as PLC code, the code may be installed and executed on a virtual controller 112 interacting with the physics-based simulation environment 110. The virtual controller 112 may include a virtualization of a physical controller, such as a PLC. In some embodiments, the virtual controller 112 may be configured to execute automation code running within a virtual machine that virtualizes the hardware of the physical controller / PLC. The virtual machine may be managed, for example, by a real-time hypervisor in a server that can support the execution of deterministic real-time control loops. In a non-limiting example, the virtual automation controller 126 may include a SIMATIC S7-1500® virtual PLC (vPLC) developed by Siemens AG.

[0034] Referring now to FIG. 2, an example of a multi-agent architecture 200 is illustrated. The shown architecture 200 depicts a number of agents 204, 210, 214, 218, 224 and 230, which may be included as components of an automation engineering system for generating and automatically verifying automation code. For example, in some embodiments, the agents 204, 210, 214, 218, 224 and 230, or at least a subset thereof, may be integrated into an engineering assistant tool, such as the engineering copilot 104 shown in FIG. 1.

[0035] The workflow begins by receiving a code request 202 including a natural language specification for a component of an automation system to execute a desired behavior. “Receiving,” as used herein, can include loading from storage, receiving from another device or process, receiving via an interaction with a user, or otherwise. In preferred embodiments, this includes receiving instructions as natural language prompt from a user. This can also include receiving text from documents such as a plant specification, that outline the behavior (including conditions, constraints, etc.) of the industrial process that the automation code is intended to control.

[0036] To illustrate with an example use-case, the code request 202 may include a specification for a PLC for executing an industrial assembly operation, which may include a sequence of stepsunder specified conditions and subject to specified constraints. The code request 202 may, in this case, include a prompt by a user to create an automation state machine for a PLC in Structured Text (ST). The code request 202 may further include other specifications, such as interface to other automation components (e.g. robots, sensors and other controllers and / or actuators), error cases and handling (e.g., robot ESTOP recovery, mechanical failures, sensor failures, etc.), HMI screen definition, among others.

[0037] The code request 202 may be received and processed by a prompt interface 204 including a first trained LLM that is configured to distill the natural language specification and spawn two sub-requests, namely: (i) a sub-request 206 to generate automation code based on specified behavior of the component and (ii) a sub-request 208 to suggest aspects of the generated automation code to critique or test. The prompt interface 204 may create the sub-request 206 as a first natural language prompt comprising explicit instructions for a code generator 210 to generate an initial automation code based on the specification in the code request 202. The prompt interface 204 may create the sub-request 208 as a second natural language prompt comprising explicit instructions for a code critic 214 to suggest aspects of the automation code to be tested based on the interpreted user’s objective in the code request 202.

[0038] In some embodiments, the prompt interface 204 may be configured (e.g., by preprompting or fine-tuning) to provide additional context information in the first prompt 206 and the second prompt 208, enabling the code generator 210 and the code critic 214 to formulate their responses and execute their tasks by themselves. For example, the prompt interface 204 may be preprompted or fine-tuned to formulate the first prompt 206 to specify information such as the language of the requested automation code (e.g., Structured Text in this use case), the specific project in the engineering tool (e.g. TIA Portal®), internal (or company-specific) standard conventions for PLC program organization, data structures, tag names, among others. Additionally, or alternately, the prompt interface 204 may be pre-prompted or fine-tuned to formulate the second prompt 208 to instruct the code critic 214 to create suggested aspects of the automation code to be tested based on the specified sequence of steps that the automation code is meant to perform, and address it as a natural language prompt to a test case generator 218 (e.g., at a specified IP address). In some embodiments, the prompt interface 204 may also provide additional context information to the code critic 214, such as which system qualities to prioritize, e.g., error handling, safety or runtime performance, among others.

[0039] The code generator 210 may comprise a second trained LLM that can generate an initial automation code 212 in the specified language based on the first prompt 206. In examples, the second trained LLM may comprise a model such as GPT-4, LLama 2, among others, that can generate different kinds of automation code, e.g., Structured Text, Structured Control Language, Ladder Logic, etc., for PLCs, Python or C++ for ROS-based systems, or even vendor-specific programming languages (e.g., Yaskawa® Inform, Kuka® Robot Language, URScript, etc.), whereby it can be used to generate automation code for different components of an automation system.

[0040] The code critic 214 may evaluate the generated automation code 212 utilizing the second prompt 208 to formulate a response 216 including suggested aspects of the generated automation code to be tested. The code critic 214 may comprise a third trained LLM, which may be configured for this task by fine-tuning model parameters using code- suggestions pairs (e.g., from a domain specific dataset) or based on one-shot or few-shot learning examples provided via appropriate prompting (e.g., provided as pre-prompts). To illustrate, continuing with the example use-case, the output of the code critic 214 in this example can include an enumeration of all the ways in which the automation state machine can transition from one state to another to make sure that a condition has not been missed, i.e., to validate that a transition between a pair of states occurs under all required conditions and no anomalous condition. The response 216 may be provided as a request to a test case generator 218 to generate test cases based on the enumerated aspects.

[0041] The test case generator 218 may process the request 216 to design code-level test cases and simulation-based test cases. Code-level test cases include those tests that are designed to verify the logic of the generated automated code. Code-level test cases can include, for example, unit tests (testing logically isolated units of a piece of code), functional tests (testing the code against functional requirements), integration tests (testing interaction between two or more unit-tested components of a piece of code), among others. Simulation-based tests include those tests that are designed to verify a physical behavior of the component of the automation system on which the generated automation code is to be executed. Simulation-based tests can be used, for example, to verify that physical constraints are not violated, test for system integration issues or faulty interactions between physical components of the automation system, test for equipment failure cases, test interactions between simulated sensor systems and the automation code, fault recovery, among others.

[0042] In some embodiments, the test case generator 218 may comprise a fourth trained LLM that can generatively design the code-level test cases and the simulation-based test cases as natural language response to the request 216 from the code critic 214. For example, the test case generator 218 may include a trained model such as GPT-4, which may be pe-prompted to split the requested test cases into code-level and simulation-based test cases. In some embodiments, the test case generator 218 may further be provided with access to a database with a large number of automation projects with test cases. The test case generator 218 may also be prompted with context information (e.g., test cases for Siemens PLC code) and furthermore, may be prompted by one-shot or few-shot learning examples to generate accurate and relevant test cases.

[0043] In other embodiments, the test case generator 218 may be implemented as an algorithmic module that can generate the code-level test cases and the simulation-based test cases using a heuristic or rule-based method. For example, in this case, the test case generator 218 may look for keywords to identify physics-related and logic-related connotations in the request 216. To generate the test cases, the test case generator 218 may have access to a database containing an ontology including descriptions of different items that can be tested (e.g., an enumeration of possible system behaviors, situations, physical parameters, etc.). This implementation may be suitable especially for relatively less complex automation systems (including smaller number of components and interactions).

[0044] The generated simulation-based test cases 220 may be dispatched as a natural language prompt to a simulation interface 224. The simulation-based test cases may be executed via the simulation interface 224 interacting with the physics-based simulation environment 110 representing the automation system. To execute the test cases, the simulation interface 224 may formulate API calls to the simulation environment 110 to configure initial conditions of the simulation environment 110 for running simulations based on the dispatched simulation-based test cases 220. Continuing with the example use-case, based on the dispatched simulation-based test cases 220, the simulation interface 224 may force the variables defining the initial conditions of the simulation environment 110 by formulating appropriate API calls, to exhaustively test the paths in which the automation state machine can transition between states, and verify that the component behaves as expected in the simulation environment 110. Running the simulations may include executing the generated automation code on a virtual controller (such as a vPLC in this example) interacting with the physics-based simulation environment 110 (see FIG. 1). The digital twin of the data acquisitionsystem may be utilized to generate simulated sensor signals indicative of the outcome of the simulations, for verifying functionality of the generated automation code.

[0045] In some embodiments, the simulation interface 224 may include a fifth trained LLM. Given an API specification of the simulation environment 110, the LLM-based simulation interface 224 may be capable of formulating API calls to the simulation environment 110 based on the prompt 220. Based on the outcome of the executed simulations, the simulation interface 224 may generate a natural language response 228 returning the simulation-based test results to the code critic 214, requesting it to suggest revisions to the automation code based on those results. In other embodiments, depending on the complexity of the automation system, the simulation interface 224 may be implemented as an algorithmic module configured to programmatically make API calls to interact with the simulation environment 110 to execute the simulation-based test cases and return the simulation-based test results 228 to the code critic 214.

[0046] The generated code-level test cases 222 may be dispatched as a natural language prompt to a code-level test interface 230. In some embodiments, the code-level test interface 230 may be implemented as a sixth trained LLM , which may be configured to process the prompt 222 to formulate API calls to interact with a code test framework 232, such as Gunit®, TestRail®, HA Portal® Test Suite , among others, for executing a batch of code-level test cases in the appropriate runtime for the automation code under test. Based on the outcome of the executed code-level tests, the code-level test interface 230 may generate a natural language response 234 returning the codelevel test results to the code critic 214, requesting it to suggest revisions to the automation code based on those results. The code-level test interface 230 may be further configured to formulate a request 236 for the code generator 210 to append certain unit tests to the automation code. In other embodiments, the code-level test interface 230 may be implemented as an algorithmic module fully scripted to make API calls to interact with the test framework 232 to execute the code-level test cases, return the code-level test results 234 to the code critic 214 and to formulate requests for the code generator 210 to append certain unit tests to the automation code.

[0047] The code critic 214 may evaluate the simulation-based test results and code-level test results by processing the requests 228 and 234 and generate a code revision request 238 for the code generator 210. In some embodiments, the code-critic 214 may formulate the code revision request 238 based on unsuccessful results (“failure”) from the executed code-level and / or simulation-basedtest cases, i.e., test cases in which the logic of the automation code or the simulated behavior of the component did not produce a desired result. Based on the failing test cases, the code critic may identify specific aspects of the automation code that need to be revised or debugged and formulate a code revision request 238 as a natural language prompt requesting the code generator 210 to revise the automation code based on the identified aspects.

[0048] In some embodiments, the code critic 214 may be pre-prompted to look for issues around code complexity and / or memory efficiency in formulating the code revision request. For example, the code critic 214 may be prompted (regardless of the user input) to always suggest refactoring steps to the code generator to simplify code complexity. A typical objective can be to minimize duplicated code by instead generating reusable functions (function blocks) and applying object- oriented programming principles. Additionally, or alternatively, the code critic 214 may be preprompted to look for issues around style or conventions for the generated automation code, to ensure that it matches internal (or company-specific) or industry standards.

[0049] The code generator 210 may process the code revision request 238 to modify the automation code and generate a revised version of the automation code 212. The revised automation code 212 may be verified again by executing the code-level and / or simulation-based test cases and further revised based on the returned test results, as described above. The code verification process may be carried out automatically for a number of iterations until convergence is reached. A convergence criterion may be defined, for example, by a maximum number of iterations, or having no failed test result, or any other heuristic.

[0050] In some embodiments, the interactions between the various agents may be synchronized via an orchestrator 240. The orchestrator 240 may include a software service communicating with the agents that can programmatically sequence the order of tasks carried by the agents in each iteration (i.e., code generationsuggesting aspects of the code to testtest case generation test cases executioncode revision). For example, the orchestrator 240 may ensure that in each iteration, the code critic 214 waits for the test results from the simulation interface 224 as well as the code-level test interface 230 before generating a code revision request 238. In other embodiments, the agents may interact between themselves in asynchronous patterns. For example, as soon as the code-level test interface 230 returns results to the code critic 214, the code critic 214 may request revisions to the automation code, even if simulation-based tests are still ongoing.

[0051] In a typical scenario, an automation system may include multiple components including one or more controllers (PLCs), robots, conveyors, tooling, sensors, and other devices, each of which is operable by executing respective pieces of automation code. The code generator 210 can be prompted, in each case to generate specific pieces of automation code for different components of the automation system, each of which may then be evaluated and refined by the code critic 214. Each generated program can then be tested using the test case generator 218 and simulation interface 224 against a simulation model of the automation system. For example, the generated pieces of automation code may be verified via execution of simulation-based test cases by running simulations of individual components (component testing), and / or pairs of components (pair- wise integration testing), and / or the automation system as a whole (full system integration testing).

[0052] Each tested and validated piece of automation code may be outputted and installed on the respective component of the automation system. The respective component can then be operated by executing the installed piece of automation code.

[0053] FIG. 3 illustrates an example of a computing environment within which embodiments of the present disclosure may be implemented. A computing environment 300 includes a computer system 310 that may include a communication mechanism such as a system bus 321 or other communication mechanism for communicating information within the computer system 310. The computer system 310 further includes one or more processors 320 coupled with the system bus 321 for processing the information.

[0054] The processors 320 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as described herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and / or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may include any type of suitable processing unit including, but not limitedto, a central processing unit, a microprocessor, a Reduced Instruction Set Computer (RISC) microprocessor, a Complex Instruction Set Computer (CISC) microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), a digital signal processor (DSP), and so forth. Further, the processor(s) 320 may have any suitable microarchitecture design that includes any number of constituent components such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read / write operations to cache memory, branch predictors, or the like. The microarchitecture design of the processor may be capable of supporting any of a variety of instruction sets. A processor may be coupled (electrically and / or as comprising executable components) with any other processor enabling interaction and / or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0055] The system bus 321 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may permit exchange of information (e.g., data (including computerexecutable code), signaling, etc.) between various components of the computer system 310. The system bus 321 may include, without limitation, a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, and so forth. The system bus 321 may be associated with any suitable bus architecture including, without limitation, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA (EISA), a Video Electronics Standards Association (VESA) architecture, an Accelerated Graphics Port (AGP) architecture, a Peripheral Component Interconnects (PCI) architecture, a PCI-Express architecture, a Personal Computer Memory Card International Association (PCMCIA) architecture, a Universal Serial Bus (USB) architecture, and so forth.

[0056] Continuing with reference to FIG. 3, the computer system 310 may also include a system memory 330 coupled to the system bus 321 for storing information and instructions to be executed by processors 320. The system memory 330 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 331 and / or random access memory (RAM) 332. The RAM 332 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 331 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). Inaddition, the system memory 330 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 320. A basic input / output system 333 (BIOS) containing the basic routines that help to transfer information between elements within computer system 310, such as during start-up, may be stored in the ROM 331. RAM 332 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 320. System memory 330 may additionally include, for example, operating system 334, application modules 335, and other program modules 336. Application modules 335 may include aforementioned modules described for FIG. 1 and may also include a user portal for development of the application program, allowing input parameters to be entered and modified as necessary.

[0057] The operating system 334 may be loaded into the memory 330 and may provide an interface between other application software executing on the computer system 310 and hardware resources of the computer system 310. More specifically, the operating system 334 may include a set of computer-executable instructions for managing hardware resources of the computer system 310 and for providing common services to other application programs (e.g., managing memory allocation among various application programs). In certain example embodiments, the operating system 334 may control execution of one or more of the program modules depicted as being stored in the data storage 340. The operating system 334 may include any operating system now known or which may be developed in the future including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

[0058] The computer system 310 may also include a disk / media controller 343 coupled to the system bus 321 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 341 and / or a removable media drive 342 (e.g., floppy disk drive, compact disc drive, tape drive, flash drive, and / or solid state drive). Storage devices 340 may be added to the computer system 310 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire). Storage devices 341, 342 may be external to the computer system 310.

[0059] The computer system 310 may include a user input interface or graphical user interface (GUI) 361, which may comprise one or more input devices, such as a keyboard, touchscreen, tablet and / or a pointing device, for interacting with a computer user and providing information to theprocessors 320.

[0060] The computer system 310 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 320 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 330. Such instructions may be read into the system memory 330 from another computer readable medium of storage 340, such as the magnetic hard disk 341 or the removable media drive 342. The magnetic hard disk 341 and / or removable media drive 342 may contain one or more data stores and data files used by embodiments of the present disclosure. The data store 340 may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores in which data is stored on more than one node of a computer network, peer-to-peer network data stores, or the like. Data store contents and data files may be encrypted to improve security. The processors 320 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 330. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0061] As stated above, the computer system 310 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 320 for execution. A computer readable medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 341 or removable media drive 342. Non-limiting examples of volatile media include dynamic memory, such as system memory 330. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 321. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0062] Computer readable medium instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machineinstructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0063] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable medium instructions.

[0064] The computing environment 300 may further include the computer system 310 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 380. The network interface 370 may enable communication, for example, with other remote devices 380 or systems and / or the storage devices 341, 342 via the network 371. Remote computing device 380 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 310. When used in a networking environment, computer system 310 may include modem 372 for establishing communications over a network 371, such as the Internet. Modem 372 may be connected to system bus 321 via user network interface 370, or via another appropriate mechanism.

[0065] Network 371 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 310 and other computers (e.g., remote computing device 380). The network 371 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 371.

[0066] It should be appreciated that the program modules, applications, computer-executable instructions, code, or the like depicted in FIG. 3 as being stored in the system memory 330 are merely illustrative and not exhaustive and that processing described as being supported by any particular module may alternatively be distributed across multiple modules or performed by a different module. In addition, various program module(s), script(s), plug-in(s), Application Programming Interface(s) (API(s)), or any other suitable computer-executable code hosted locally on the computer system 310, the remote device 380, and / or hosted on other computing device(s) accessible via one or more of the network(s) 371, may be provided to support functionality provided by the program modules, applications, or computer-executable code depicted in FIG. 3 and / or additional or alternate functionality. Further, functionality may be modularized differently such that processing described as being supported collectively by the collection of program modules depicted in FIG. 3 may be performed by a fewer or greater number of modules, or functionality described as being supported by any particular module may be supported, at least in part, by another module. In addition, program modules that support the functionality described herein may form part of one or more applications executable across any number of systems or devices in accordance with any suitable computing model such as, for example, a client-server model, a peer-to-peer model, and so forth. In addition, any of the functionality described as being supported by any of the program modules depicted in FIG. 3 may be implemented, at least partially, in hardware and / or firmware across any number of devices.

[0067] It should further be appreciated that the computer system 310 may include alternateand / or additional hardware, software, or firmware components beyond those described or depicted without departing from the scope of the disclosure. More particularly, it should be appreciated that software, firmware, or hardware components depicted as forming part of the computer system 310 are merely illustrative and that some components may not be present or additional components may be provided in various embodiments. While various illustrative program modules have been depicted and described as software modules stored in system memory 330, it should be appreciated that functionality described as being supported by the program modules may be enabled by any combination of hardware, software, and / or firmware. It should further be appreciated that each of the above-mentioned modules may, in various embodiments, represent a logical partitioning of supported functionality. This logical partitioning is depicted for ease of explanation of the functionality and may not be representative of the structure of software, hardware, and / or firmware for implementing the functionality. Accordingly, it should be appreciated that functionality described as being provided by a particular module may, in various embodiments, be provided at least in part by one or more other modules. Further, one or more depicted modules may not be present in certain embodiments, while in other embodiments, additional modules not depicted may be present and may support at least a portion of the described functionality and / or additional functionality. Moreover, while certain modules may be depicted and described as sub-modules of another module, in certain embodiments, such modules may be provided as independent modules or as sub-modules of other modules.

[0068] Although specific embodiments of the disclosure have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the disclosure. For example, any of the functionality and / or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the disclosure, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this disclosure. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like can be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for generating and automatically verifying automation code, comprising: receiving a code request comprising a natural language specification for a component of an automation system to execute a desired behavior, processing the code request by a prompt interface comprising a first trained LLM to generate a first prompt for a code generator and a second prompt for a code critic, based on the first prompt, generating an initial automation code by the code generator comprising a second trained LLM, and based on the second prompt, formulating suggested aspects of the generated automation code to be tested, by the code critic comprising a third trained LLM, based on the suggested aspects, generating and executing code-level test cases to verify a logic of the automation code and simulation-based test cases to verify a physical behavior of the component, processing results of the executed test cases by the code critic to formulate a code revision request, modifying the automation code by the code generator based on the code revision request, and outputting the modified automation code.

2. The method according to claim 1, wherein the suggested aspects of the automation code to be tested are processed by a test case generator to design the code-level and simulation-based test cases.

3. The method according to claim 2, comprising executing the simulation-based test cases via a simulation interface interacting with a physics-based simulation environment representing the automation system and returning simulation-based test results by the simulation interface to the code critic.

4. The method according to claim 3, wherein the simulation interface is configured to formulate API calls to the physics-based simulation environment to configure initialconditions of the physics-based simulation environment for running simulations based on the simulation-based test cases provided by the test case generator.

5. The method according to claim 4, wherein running the simulations includes executing the generated automation code on a virtual controller interacting with the physicsbased simulation environment.

6. The method according to any of claims 2 to 5, comprising executing the codelevel test cases via a code-level test interface interacting with a code test framework and returning code-level test results by the code-level test interface to the code critic,7. The method according to claim 6, wherein the code-level test interface is configured to formulate API calls to the code test framework for executing a batch of codelevel test cases in the appropriate runtime for the automation code under test, and formulate a request for the code generator to append some of the code-level test cases to the automation code.

8. The method according to any of claims 1 to 7, wherein the code revision request is formulated by the code critic based on unsuccessful results from the code-level and / or simulation-based test cases.

9. The method according to claim 8, wherein the code critic is pre-prompted to look for issues around code complexity and / or memory efficiency in formulating the code revision request.

10. The method according to any of claims 6 to 9, wherein each of the prompt interface, the code generator, the code critic, the test case generator, the simulation interface and the code-level test interface is configured as an intelligent agent capable of generating responses in natural language or code and interaction with other agents by calling respective APIs.

11. The method according to claim 10, wherein interactions between the agents are synchronized via an orchestrator.

12. The method according to any of claims 1 to 11, comprising iteratively verifying the automation code by executing the code-level and / or simulation-based test cases and modifying the automation code based on the returned test results until a convergence criterion is met.

13. The method according to any of claims 1 to 12, comprising operating the component by installing the outputted automation code on the component and executing the installed automation code.

14. The method according to any of claims 1 to 13, wherein the automation system comprises multiple different components, wherein the code generator is configured to generate specific pieces of automation code for the different components, and wherein the generated pieces of automation code are verified via execution of simulation-based test cases based on a simulation of individual components, and / or pairs of components, and / or the automation system as a whole.

15. A non-transitory computer-readable storage medium including instructions that, when processed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 14.

16. A system for generating and automatically verifying automation code, comprising: one or more processors, and memory storing instructions executable by the one or more processors to perform a method according to any of claims 1 to 14.

Images