Generative ai industrial automation visual runtime

Abstract

The present disclosure relates to generative AI industrial automation visualizations runtime. A human-machine interface (HMI) development system utilizes a generative AI model to assist in the development of HMI projects according to specified functional requirements, which can be provided to the development system as intuitive natural language speech or written text. The system can formulate and implement HMI project edits during design time based on analysis of the natural language design input. After the HMI project is deployed as a runtime application, the system can also receive and process natural language requests to modify the runtime HMI according to described modification criteria.

Description

Technical Field

[0001] The topics disclosed in this article generally relate to industrial automation systems, and for example, to the development and deployment of industrial human-machine interfaces (HMIs). Background Technology

[0002] Industrial human-machine interfaces (HMIs) include computer terminals with the display capabilities to execute HMI runtime applications. These HMI runtime applications define the display screens presented to the operator of the industrial automation system, navigation structures for navigating between display screens, and data links or bindings between graphical elements and corresponding data labels in the controller's data tables. HMI developers typically use HMI development platforms to design these aspects of the HMI. These HMI development platforms support graphical and menu-driven development workflows, where developers select graphical display and control elements from a library to include on each display interface, and manipulate these selected elements, for example, via drag-and-drop interaction on a model of the display interface to produce the desired layout. Summary of the Invention

[0003] The following is a simplified overview to provide a basic understanding of some of the aspects described in this paper. This overview is not an extensive review, nor is it intended to identify key / important elements or to depict the scope of the various aspects described herein. Its sole purpose is to present some concepts in a simplified form as an introduction to the more detailed descriptions that follow.

[0004] In one or more embodiments, a system is provided, comprising: a human-machine interface (HMI) deployment component configured to deploy an HMI application to an HMI terminal for execution, wherein the HMI application presents on the HMI terminal a display screen including graphical objects displaying data generated by an industrial automation system; a user interface component configured to receive natural language input describing a new display screen to be added to the HMI application via interaction with the HMI application; a generative artificial intelligence (AI) component configured to formulate a prompt for a generative AI model in response to receiving the natural language input, the prompt being designed to obtain a response from the generative AI model including information used by the generative AI component to generate the new display screen, wherein the prompt is generated based on analysis of one or more custom models trained with training data and the natural language input; and an HMI generation component configured to add the new display screen to the HMI application and present the new display screen on the HMI terminal.

[0005] Furthermore, one or more embodiments provide a method comprising: deploying a human-machine interface (HMI) application to an HMI terminal for execution by a system including a processor, wherein the HMI application presents a display screen on the HMI terminal including graphical objects displaying data generated by an industrial automation system; receiving natural language input describing a new display screen to be added to the HMI application by the system via interaction with the HMI application; in response to receiving the natural language input, formulating a prompt for a generative artificial intelligence (AI) model by the system, the prompt being designed to obtain a response from the generative AI model including information used by the system to generate the new display screen, wherein the prompt is generated based on analysis of one or more custom models trained with training data and the natural language input; and modifying the HMI application by the system to add the new display screen to the HMI application.

[0006] Furthermore, according to one or more embodiments, a non-transitory computer-readable medium having instructions stored thereon is provided, the instructions causing a human-machine interface (HMI) development system to perform operations in response to execution, the operations including: deploying an HMI application to an HMI terminal for execution, wherein the HMI application presents a display screen on the HMI terminal including graphical objects displaying data generated by an industrial automation system; receiving natural language input describing a new display screen to be added to the HMI application via interaction with the HMI application; formulating a prompt for a generative artificial intelligence (AI) model in response to receiving the natural language input, the prompt being designed to obtain a response from the generative AI model including information used by the system to generate the new display screen, wherein the prompt is generated based on analysis of one or more custom models trained with training data and the natural language input; and modifying the HMI application to add the new display screen to the HMI application.

[0007] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and figures. These aspects indicate various modes that can be practiced, all of which are intended to be covered herein. Other advantages and novel features will become apparent when considered in conjunction with the figures, based on the following detailed description. Attached Figure Description

[0008] Figure 1 This is a block diagram of an example industrial control environment.

[0009] Figure 2 It is a diagram of a general architecture that includes industrial controllers and human-machine interfaces (HMIs).

[0010] Figure 3 This is a block diagram of an example HMI development system.

[0011] Figure 4This is a diagram illustrating a sample data flow associated with creating an HMI project using an HMI development system.

[0012] Figure 5 This is a graph showing the training of a custom model.

[0013] Figure 6 This diagram illustrates the debugging of an HMI application onto an HMI terminal.

[0014] Figure 7 This diagram illustrates how a HMI application can be dynamically edited using natural language prompts during runtime when developing an HMI system.

[0015] Figure 8a This is a flowchart of the first part of an example method for modifying an HMI runtime application using natural language design input.

[0016] Figure 8b This is a flowchart of the second part of an example method for modifying an HMI runtime application using natural language design input.

[0017] Figure 9a This is a flowchart of the first part of an example method for dynamically creating or updating standard operating procedures for resolving alarm conditions in industrial automation systems based on monitored operator behavior.

[0018] Figure 9b This is a flowchart of the second part of an example method for dynamically creating or updating standard operating procedures for resolving alarm conditions in industrial automation systems based on monitored operator behavior.

[0019] Figure 10 This is a flowchart of an example method for presenting data log information as a natural language summary.

[0020] Figure 11 This is an example computing environment.

[0021] Figure 12 This is an example of a networked environment. Detailed Implementation

[0022] This disclosure will now be described with reference to the accompanying drawings, in which similar reference numerals are consistently used to refer to similar elements. In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding of this disclosure. However, it will be apparent that this disclosure can be practiced without these specific details. In other instances, well-known structures and apparatuses are shown in block diagram form to facilitate description.

[0023] As used herein, the terms “component,” “system,” “platform,” “layer,” “controller,” “terminal,” “station,” “node,” and “interface” are intended to refer to a computer-related entity or an entity related to or part of an operating device having one or more specific functions, wherein such an entity may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to: a process running on a processor; a processor; a hard disk drive; multiple storage drives (optical or magnetic storage media) including fixed (e.g., secured with screws or bolts) or removable fixed solid-state drives; an object; an executable file; an executing thread; a computer-executable program, and / or a computer. For illustration, a server and an application running on a server can both be components. One or more components may reside within an executing process and / or thread, and components may reside on one computer and / or be distributed among two or more computers. Furthermore, the components described herein may be executed from various computer-readable storage media on which various data structures are stored. These components may communicate via local and / or remote processes, for example, based on signals having one or more data packets (e.g., data from a component interacting with another component in a local system, a distributed system, and / or interacting with other systems via signals through a network such as the Internet). As another example, a component may be a device having specific functions provided by mechanical parts operated by an electrical or electronic circuitry system, which is operated by software or firmware applications executed by a processor, wherein the processor may be internal or external to the device and execute at least a portion of the software or firmware application. As another example, a component may be a device providing specific functions through electronic components rather than mechanical parts, the electronic components including a processor to execute software or firmware that at least partially provides the functions of the electronic components. As yet another example, an interface may include input / output (I / O) components and associated processors, applications, or application programming interface (API) components. While the foregoing examples pertain to aspects of components, the illustrative aspects or features also apply to systems, platforms, interfaces, layers, controllers, terminals, etc.

[0024] As used herein, the terms “infer” and “inference” generally refer to the process of reasoning or inferring the state of a system, environment, and / or user based on a set of observations captured via events and / or data. For example, inference can be used to identify specific situations or actions, or it can generate probability distributions of states. Inference can be probabilistic, i.e., calculating the probability distribution of states of interest based on considerations of data and events. Inference can also refer to techniques used to compose higher-level events from a set of events and / or data. Such inference enables the construction of new events or actions from a set of observed events and / or stored event data, regardless of whether the events are closely related in time or whether the events and data come from one or more event and data sources.

[0025] Furthermore, the term "or" is intended to mean inclusive "or" rather than exclusive "or". That is, unless otherwise stated or clearly understood from the context, the phrase "X uses A or B" is intended to mean any natural inclusive arrangement. That is, the phrase "X uses A or B" is satisfied in any of the following cases: X uses A; X uses B; or X uses both A and B. Additionally, unless otherwise stated or clearly understood from the context that the article refers to the singular form, the articles "a" and "an" as used in this application and the appended claims should generally be interpreted as meaning "one or more".

[0026] Furthermore, as used herein, the term "set" excludes an empty set, such as a set containing no elements. Therefore, "set" as used in this disclosure includes one or more elements or entities. For illustration, a set of controllers includes one or more controllers; a set of data resources includes one or more data resources; and so on. Similarly, as used herein, the term "group" refers to a collection of one or more entities; for example, a group of nodes refers to one or more nodes.

[0027] Various aspects or features will be presented according to the system, which may include multiple devices, components, modules, etc. It should be understood and appreciated that various systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Combinations of these methods may also be used.

[0028] Figure 1This is a block diagram of an example industrial control environment 100. In this example, multiple industrial controllers 118 are deployed throughout the industrial plant environment to monitor and control relevant industrial systems or processes related to product manufacturing, processing, motion control, batch processing, material handling, or other such industrial functions. The industrial controllers 118 typically execute corresponding control programs to facilitate the monitoring and control of industrial units 120 that constitute controlled industrial assets or systems (e.g., industrial machines). One or more industrial controllers 118 may also include software controllers executing on a personal computer or other hardware platform or on a cloud platform. Some hybrid devices may also combine controller functionality with other functions (e.g., visualization). The control programs executed by the industrial controllers 118 may include any conceivable type of code for processing input signals read from the industrial unit 120 and controlling output signals generated by the industrial controllers, including but not limited to ladder logic, sequential function charts, function block diagrams, or structured text.

[0029] Industrial device 120 may include: input devices that provide data relating to the controlled industrial system to industrial controller 118; and output devices that respond to control signals generated by industrial controller 118 for controlling aspects of the industrial system. Example input devices may include telemetry devices (e.g., temperature sensors, flow meters, level sensors, pressure sensors, etc.), manual operator control devices (e.g., buttons, selector switches, etc.), safety monitoring devices (e.g., safety mats, safety cords, light curtains, etc.), and other such devices. Output devices may include motor drivers, pneumatic actuators, signaling devices, robot control inputs, valves, etc. Industrial device 120 is an example of such a device. M Some industrial units can operate autonomously on the factory network 116 without being controlled by the industrial controller 118.

[0030] Industrial controller 118 can communicatively interface with industrial device 120 via hardwired or network connection. For example, industrial controller 118 may be equipped with native hardwired inputs and outputs for communicating with industrial device 120 to control these devices. Local controller I / O may include: digital I / O that sends discrete voltage signals to and receives discrete voltage signals from field devices; or analog I / O that sends analog voltage or current signals to and receives analog voltage or current signals from devices. Controller I / O may communicate with the controller's processor via a backplane, allowing digital and analog signals to be read into and controlled by the control program. Industrial controller 118 may also communicate with industrial device 120 via factory network 116 using, for example, a communication module or integrated networking port. Exemplary networks may include the Internet, intranet, Ethernet, DeviceNet, ControlNet, Data Highway and Data Highway Plus (DH / DH+), remote I / O, fieldbus, Modbus, Profibus, wireless networks, serial protocols, etc. The industrial controller 118 may also store persistent data values ​​that can be referenced by the control program and used for control decisions. These persistent data values ​​include, but are not limited to, measured or calculated values ​​representing the operating status of the controlled machine or process (e.g., tank level, position, alarms, etc.) or captured time-series data collected during the operation of the automated system (e.g., status information at multiple time points, diagnostic occurrences, etc.). Similarly, some intelligent devices—including, but not limited to, motor drives, instruments, or condition monitoring modules—may store data values ​​used for control and / or visualization of operating status. Such devices may also capture time-series data or events in a log for later retrieval and viewing.

[0031] Industrial automation systems typically include one or more personal machine interface (HMI) terminals 114 that enable factory personnel to view telemetry and status data associated with the automation system and control aspects of system operation. The HMI terminals 114 can communicate with one or more industrial controllers 118 via a factory network 116 and exchange data with the industrial controllers to visualize information related to the controlled industrial process on one or more pre-developed operator interface screens. The HMI terminals 114 can also be configured to allow operators to submit data to designated data tags or memory addresses of the industrial controllers 118, providing a means for operators to issue commands to the controlled system (e.g., cycle start commands, device actuation commands, etc.), modify setpoint values, etc. The HMI terminals 114 execute HMI runtime applications that generate one or more display screens through which operators interact with the industrial controllers 118, thereby interacting with the controlled process and system. Example display screens can use graphical representations of processes displaying measured or calculated values ​​to visualize the current status of an industrial system or its associated devices, employing color or position animations based on the status, presenting alarm notifications, or other such techniques to present relevant data to the operator. Data presented in this manner is read from the industrial controller 118 by the HMI terminal 114 and displayed on one or more display screens according to the display format selected by the HMI developer. The HMI terminal 114 may include a fixed-position or mobile device with a user-installed or pre-installed operating system and user-installed or pre-installed graphical application software.

[0032] Some industrial environments may also include other systems or devices related to specific aspects of the controlled industrial system. These systems or devices may include, for example, one or more data historians 110 that aggregate and store production information collected from the industrial controller 118 and other industrial devices.

[0033] Industrial units 120, industrial controllers 118, HMI terminals 114, associated controlled industrial assets, and other factory floor systems such as data history devices 110, vision systems, and other such systems operate at the operational technology (OT) level in the industrial environment. More advanced analytics and reporting systems can operate at a higher enterprise level within the information technology (IT) domain of the industrial environment; for example, on an office network 108 or a cloud platform 122. These more advanced systems may include, for example, an enterprise resource planning (ERP) system 104 that integrates and centrally manages advanced business operations such as finance, sales, order management, marketing, human resources, or other such business functions. A manufacturing execution system (MES) 102 can monitor and manage control operations at the control level based on higher-level business considerations, thereby driving those control-level operations toward results that meet defined business objectives (e.g., order fulfillment, resource tracking and management, asset utilization tracking, etc.). A reporting system 106 can collect operational data from industrial units on the factory floor and generate daily or shift reports summarizing operational statistics of controlled industrial assets.

[0034] Figure 2 This is a diagram of a general architecture including industrial controller 118 and HMI 114. Industrial facilities may include one or more controlled processes 2101 to 210 associated with product manufacturing, processing, motion control, batch processing, material handling, or other such industrial functions. N As described above, execution processes 2101 to 210 are performed. N Devices and machines (e.g., Figure 1 The device 120 and its associated machinery can be monitored and controlled by an industrial controller 118, which executes a control program 204 to facilitate the control of processes 2101 to 210. N The monitoring and control of the controller 118. The control program 204 can be virtually any type of code used to process input signals read into the controller 118 and control output signals from the controller 118, including but not limited to ladder logic programming, sequential function charts, function block diagrams, or structured text. Data read into or generated by the controller 118 can be stored in data table 206 within the controller's memory.

[0035] Controller 118 can be connected to the controlled processes 2101 to 210 via factory network 116 or via another hardwired or network connection. NThe controller exchanges data between its input and output devices. For example, the controller 118 may be equipped with local hardwired input and output points that exchange digital and analog signals with field devices to control the devices. Local controller I / O may include: digital I / O that sends discrete voltage signals to and receives discrete voltage signals from field devices; or analog I / O that sends analog voltage or current signals to and receives analog voltage or current signals from devices. The controller 118 converts the input signals from the controlled process 210 into digital and analog data values ​​stored in the controller's data table 206. The control program 204 processes these input data values ​​according to a user-defined control algorithm and sets the values ​​of the controller's digital and analog output signals based on this processing. The values ​​of the output signals, as well as any other values ​​calculated by the control program 204, are stored in the data table 206.

[0036] HMI 114 uses data stored in data table 206 of the controller to connect with controlled processes 2101 to 210 N The relevant information is visualized as graphics and alphanumeric data. For this purpose, HMI 114 communicates with controller 118 via factory network 116 or via a direct connection, and reads and writes data to data table 206 through this connection. HMI 114 presents controlled processes 2101 to 210. N A navigable interface display screen showing the current operation or status information. In some implementations, the display screen can present the execution of controlled processes 2101 to 210. N The machine provides a graphical representation, and these graphical representations can be animated based on the current state of the corresponding machine, as determined by data values ​​contained in data table 206 of the controller. These animations may include, for example, setting the color of a graphical element based on the state of the corresponding machine component, changing the height of a fill graphic based on the corresponding fill level of a tank, setting the position or orientation of a graphical element based on the corresponding position or orientation of a machine component, displaying alphanumeric text conveying measurement values ​​(e.g., temperature, pressure, flow rate, etc.), or other such animations.

[0037] The operator can also interact with the HMI display screen to send changes to the controlled processes 2101 to 210 to the controller 118. NThe commands issued by the HMI 114 may include, for example, changing control setpoints, initiating start or stop commands, changing the operating mode of the machine or process, clearing alarm messages presented by the HMI 114, or other such commands. To provide a means of issuing these commands, the display screen may include interactive graphical controls, such as graphical buttons, data input fields, or other such controls, linked to corresponding data tags defined in datasheet 206. Through interaction with these controls, the operator can write digital or analog values ​​to these data tags, and these values ​​are controlled by the control program 204 in conjunction with the industrial processes 2101 to 2102. N To handle it.

[0038] Typically, HMI 114 includes a computer terminal with the display capability to execute HMI runtime application 202. HMI runtime application 202 defines the display screens presented to the operator (including the definitions of graphical elements and controls included on each display screen and the arrangement of these elements), the navigation structure for navigating between display screens, and the data links or bindings between graphical elements and corresponding data labels in the controller's data table 206. HMI developers typically use an HMI development platform to design these aspects of the HMI, which compiles the design into HMI runtime application 202, which can be downloaded to and executed on the HMI terminal. These HMI development platforms typically support graphical and menu-driven development workflows, where developers select graphical display and control elements from a library to include on each display interface, and manipulate these selected elements on a model of the display interface, for example, via drag-and-drop interaction, to produce the desired layout. For elements whose appearance or behavior is a function of the values ​​of data labels defined in the controller's data table 206, developers typically define bindings to appropriate data labels by invoking the element's properties window and specifying the data label in the appropriate property field of the window. Similarly, for elements designed to write data to controller 118—such as graphical buttons and data input fields—users typically configure the data labels to which those elements write their data by interacting with the element's properties window. This graphical development approach can also be cumbersome and time-consuming.

[0039] As an alternative to this graphical HMI development method, one or more embodiments described herein provide an HMI development system that supports the use of generative artificial intelligence (AI) to assist in the development and use of industrial HMI applications. In one or more embodiments, the HMI development system may support industry-specific cue engineering services that use natural language cueing describing the functional and visual requirements of the HMI to assist users in developing industrial visualization projects. To this end, the HMI development system may utilize a generative AI model and associated neural networks to generate portions of the HMI project based on functional requirements provided to the HMI development system as intuitive natural language input (e.g., spoken or written natural language text), including display screen content and layout, screen navigation structure, links between animated graphics and data sources (e.g., controller data labels), alarm definitions, color settings, and other such aspects. The HMI development system may include a dedicated cue engineering layer and associated custom models—trained using training data such as knowledge of various types of industrial control applications, knowledge of specific types of industrial assets, vertical-domain-specific industry standards and best practices, sample HMI layouts, and others—that generate cueing or meta-cueing based on the user's natural language input to be submitted to a generative AI model such as a large language model (LLM).

[0040] Figure 3 This is a block diagram of an example HMI development system 302 according to one or more embodiments of this disclosure. Aspects of the systems, apparatus, or processes described in this disclosure can constitute machine-executable components contained within a machine, such as machine-executable components contained in one or more computer-readable media (or media) associated with one or more machines. Such components, when executed by one or more machines such as computers, computing devices, automation devices, virtual machines, etc., can cause the machines to perform the described operations.

[0041] HMI development system 302 may include a user interface component 304, an HMI generation component 306, an HMI deployment component 308, a generative AI component 310, a training component 312, one or more processors 318, and a memory 320. In various embodiments, one or more of the user interface component 304, HMI generation component 306, HMI deployment component 308, generative AI component 310, training component 312, one or more processors 318, and memory 320 may be electrically and / or communicatively coupled to each other to perform one or more functions of HMI development system 302. In some embodiments, components 304, 306, 308, 310, and 312 may include software instructions stored on memory 320 and executed by processor 318. HMI development system 302 may also be coupled with… Figure 3It may interact with other hardware and / or software components not described herein. For example, processor 318 may interact with one or more external user interface devices such as a keyboard, mouse, display monitor, touchscreen, or other such interface devices.

[0042] User interface component 304 can be configured to receive user input and present output to the user in any suitable format (e.g., visual, auditory, tactile, etc.). In some embodiments, user interface component 304 can be configured to generate interface displays and provide them to client devices (e.g., laptops, tablets, smartphones, etc.) and exchange data via these interface displays. Input data that can be received via various embodiments of user interface component 304 may include, but is not limited to: natural language chat input or prompts, interactions with the HMI development interface (e.g., selection input, drag-and-drop input, alphanumeric input, etc.), or other such input. Output data presented by various embodiments of user interface component 304 may include natural language responses to chat input or prompts, HMI displays in both design mode and runtime mode, answers to user-submitted questions or requests, HMI development suggestions, or other such output.

[0043] HMI generation component 306 can be configured to create HMI projects based on design input (including natural language requests or functional descriptions) received via user interface component 304. HMI deployment component 308 can be configured to debug the HMI application created by HMI generation component 306 onto an appropriate execution platform, such as an HMI terminal.

[0044] Generative AI component 310 can be configured to use generative AI-assisted HMI generation component 306 to generate or analyze portions of an HMI project, including generating, formatting, and configuring HMI display screens. To this end, generative AI component 310 can use an associated custom model 322 trained with domain-specific industrial training data to implement prompting engineering functions. Generative AI component 310 can generate prompts or meta-prompts and submit them to one or more generative AI models and associated neural networks, where these prompts are generated based on natural language requests or queries submitted by the designer and domain-specific information contained in the custom model 322. Depending on the nature of the designer's request or query, the response returned by the generative AI model in response to the prompt can be used by HMI generation component 306 or user interface component 304 to generate portions of the HMI project, presenting answers to the designer's questions about a portion of the HMI project or about the design platform itself. Training component 312 can be configured to train one or more custom models 322 using various types of relevant training data. The system 302 uses these custom models 322 by performing the following operations: generating and customizing parts of the HMI application, and generating appropriate hints for the generative AI model as needed to assist in the development of the HMI.

[0045] One or more processors 318 may perform one or more of the functions described in the systems and / or methods disclosed herein. Memory 320 may be a computer-readable storage medium storing computer-executable instructions and / or information for performing the functions described in the systems and / or methods disclosed herein.

[0046] Figure 4 This is a diagram illustrating example data flows associated with creating an HMI project 402 using the HMI development system 302, according to one or more implementations. Some implementations of the HMI development system 302 can be implemented on a cloud platform, making it accessible to multiple industrial customers with authorized access to services using the HMI development system. Cloud-based implementations can also facilitate collaborative project development, whereby multiple developers contribute design and programming input to a public HMI project 402. Alternatively, some implementations of the HMI development system 302 can be executed at least partially on local client devices, while accessing remote services and repositories as needed.

[0047] A client device 404 (e.g., a laptop, tablet, desktop, mobile device, wearable AR / VR device, etc.) owned by a user with appropriate authentication credentials can access the project development tools of the HMI development system and utilize these tools to create HMI projects 402—including display screen definitions, layouts of graphical elements on the display screen, screen navigation structures, animation definitions of graphical elements, links to data labels for their value-driven animations, or other such aspects of HMI projects 402—for visualizing the status and operational information of the industrial automation system. Through interaction with a development interface generated by the system's user interface component 304 and delivered to the client device 404, developers can submit design inputs 412 to the HMI development system 302 in various supported formats. Design inputs 412 can include explicit HMI design inputs, such as menu-driven screen creation, selection and placement of graphical elements (e.g., via drag-and-drop interaction with the development interface), manual definition of animation attributes and data links, and other such inputs 412.

[0048] Additionally, the HMI system's development services may include a chat-based interface and associated natural language processing services that utilize generative AI to assist users in creating, editing, or customizing HMI projects 402 for industrial applications, as well as searching for answers to specific questions related to HMI project 402 or the HMI development platform itself. To this end, the HMI development system 302 may include a generative AI component 310 that responds to natural language prompts submitted by the user as part of design input 412. These prompts may include, for example, natural language descriptions of desired visual layouts, screen navigation definitions, graphic content to be included on the display screen and the screen location where the content should reside, alarm requests, data sources or labels for controlling the animation properties of selected graphic objects, questions about development tools supported by the HMI development system 302, requests for HMI design recommendations, or other such prompts. Depending on the nature of the prompts, the generative AI component 310 can create or edit portions of the HMI project 402, generate development recommendations for users to consider, provide answers to questions about the HMI project 402 or about development tools available as part of the development platform of the HMI development system, or other such content designed to assist users in developing the HMI project 402.

[0049] Generative AI component 310 can use an associated custom model 322 trained with domain-specific industrial training data to implement prompting engineering functions, and can interface with generative AI model 406 (e.g., LLM or another type of model) and associated neural network. Figure 5This is a diagram illustrating the training of a custom model 322 used by the generative AI component 310. In some embodiments, the generative AI model 406 may reside and execute outside the HMI development system 302, and the generative AI component 310 may include suitable connectivity tools and protocols, application programming interfaces (APIs), or other services that allow the generative AI component 310 to exchange prompts and responses with the generative AI model 406. The custom model 322 may be trained using a training dataset 502 representing a set of domain-specific industry knowledge, which may assist the generative AI component 310 in generating or modifying portions of the HMI item 402 in a manner with a high probability of satisfying the user's natural language design description and meeting any application-specific or vertical domain-specific requirements. Example training data 502 that can be used to train the custom model 322 includes, but is not limited to: information defining industry standards (e.g., global or vertical-specific safety standards, food and drug standards, design standards such as ISA-88 standards, etc.), technical specifications or design standards for various types of industrial control applications (e.g., batch control processes, die casting, valve control, agitator control, etc.), knowledge of specific industrial verticals (e.g., automotive, food and beverage, pharmaceutical, oil and gas, textile, mining, etc.), knowledge of industrial best practices, technical specifications for various types of industrial equipment or assets (e.g., industrial controllers, motor drives such as variable frequency drives, sensors, etc.), control design rules, sample HMI display layouts for various types of control applications or use cases, customer-specific training data describing internal HMI design preferences or standards (e.g., preferred screen layout format, preferred graphics type, preferred text font, etc.), or other such training data.

[0050] During project development, generative AI component 310 can generate prompts 504 as needed and submit them to generative AI model 406, which is designed to obtain responses 506 that assist in HMI project development tasks. These prompts 504 are generated based on the user's natural language input and industry knowledge and reference data encoded in a trained custom model 322. Generative AI component 310 can refer to the custom model 322 as needed, combined with processing the user's natural language queries or requests (which can be submitted as design input 412), and prompt the generative AI model 406 to obtain responses 506 from the auxiliary user interface component 304 and HMI generation component 306 in handling these requests and queries.

[0051] Return to Figure 4Based on user design input 412, user interface component 304 can present design feedback 418 designed to assist developers in developing HMI project 402. At least some of this design feedback 418 may include natural language chat prompts generated by generative AI component 310 requesting specific information items from the user (e.g., as part of an iterative dialogue with the user aimed at determining the user's design requirements), HMI design recommendations (e.g., recommendations for HMI screen layouts or navigation controls), design views of the HMI project 402 under development, responses to natural language queries submitted by the user regarding HMI project 402 or the system's HMI development tools, or other such feedback.

[0052] Implementations of the HMI development system 302 may use prompting engineering services to process natural language design input 412 submitted by the user interface component 304 (e.g., via a spoken interface or a text-based chatbot). These prompting engineering services may leverage industry knowledge encoded in the customization module 322 (such as that learned from training data 502) and responses 506 prompted by the generative AI model 406 to accurately determine the designer's design requirements and generate portions of the HMI project 402 to address those requirements, or to provide refined answers to design queries.

[0053] When a user submits natural language design input 412 describing an HMI design request to HMI development system 302, generative AI component 306 analyzes the input 412 based on domain-specific industry knowledge and design rules encoded in custom model 322 (i.e., knowledge and rules represented by training data 502). Based on this analysis and depending on the nature of the design input 412, generative AI component 306 creates or modifies a portion of HMI project 402 in a manner that meets the design requirements described by the design input 412. In cases where the design input 412 includes a natural language query about the HMI project 402 being edited or about system 302 itself, generative AI component 310 generates and returns a response to that query as design feedback 418. For example, this response could be an answer to a question about the HMI project 402 being viewed, guidance on appropriate development tools supported by system 302 that can be used to solve the HMI design problem specified by the query, or other such responses.

[0054] In addition to referencing the information contained in the custom model 322, the generative AI component 310 can also prompt the generative AI model 406 as needed to obtain a response 506 (see [link]). Figure 5The response 506 helps to formulate an appropriate HMI design (or natural language response) in response to the user's natural language design input 412. For example, in response to receiving a natural language design request, the generative AI component 310 may determine whether sufficiently accurate HMI design actions can be performed (e.g., creating HMI project 402, creating or modifying a display screen defined by project 402, placing appropriately configured graphical objects on the display screen, removing graphical objects from the display screen, configuring animation data links for animate graphical objects, etc.) solely based on relevant information contained in the custom model 322, or alternatively, whether supplementary information from the generative AI model 406 is needed to determine an appropriate HMI design action with a sufficiently high probability to satisfy the design request described by the natural language design input 412. If the supplementary information from the generative AI model 406 is deemed necessary, the generative AI component 310 may formulate a prompt 504 based on an analysis of the design input 412 and a relevant subset (or a corresponding subset of the training data 502) of industry knowledge encoded in the custom model 322. These prompts 504 are designed to obtain responses 506 from the generative AI model 406, which can be used to generate or modify portions of the HMI project 402 in a manner that satisfies the user's natural language design requests.

[0055] In developing a response to a user’s question about HMI project 402 or about design tools supported by HMI development system 302, generative AI component 310 may aggregate information from a custom model 322 identified as relevant to the query (e.g., knowledge of common design standards for the type of industrial control application of interest, technical or specification data of industrial assets or devices, industrial design standard information, vertical industry-specific industrial standards, knowledge of the development platform of the IDE system, etc.) with language-specific combination or grammatical information obtained from generative AI model 406 as a response 506 to develop a natural language answer to the user’s query.

[0056] If needed, system 302 can also guide iterative natural language chat exchanges with the user to determine the user's HMI design request or increase the probability that the generative AI component 310 will formulate an HMI design action that meets the user's design request. For example, the generative AI component 310 can parse the initial natural language design input 412 to determine the type of design requirement described by the input 412 and refine and contextualize the initial input 412 in a way that is expected to help the custom model 322 or the generative AI model 406 quickly and accurately achieve the desired HMI design solution. If the generative AI component 310 determines that additional information from the user will produce a design solution with a higher probability of meeting the user's initial request (i.e., a probability exceeding a defined threshold), the generative AI component 310 can formulate and present one or more query responses as design feedback 418, which prompt the user with more refined information that will allow the generative AI component 310 to provide a more complete or more accurate HMI design solution for the user's request (i.e., an estimate of a solution with accuracy exceeding a defined threshold). Through iterative chat exchanges, the generative AI component 310 can collaborate with the user to explore potential HMI content and configuration changes that may meet the user's needs. The generative AI component 310 can guide these natural language conversations with the user, in part based on learned knowledge of the types of questions requiring answers, to generate an HMI item 402 or a portion thereof that aligns with the user's needs.

[0057] Users can interact with system 302 using such iterative natural language dialogues to facilitate incremental design and refinement of HMI project 402. For example, using a natural language design request submitted as design input 412, users can initiate the creation of a new HMI project 402 and continue to refine project 402 by adding new display screens (e.g., “create a panel with two instant buttons, gauges, drop-down lists, and a rotating box”), adjusting color schemes (e.g., “change the background color of the station #6 screen to gray”), adding or removing processing stations or other graphic elements (e.g., “add a redundant standby pump in parallel with pump #6”), repositioning or resizing graphic objects (e.g., “move the leak test station to the top of the screen”, “make the tank smaller”, etc.), or performing other such refinements. As needed, the generative AI component 310 can guide the refinement process by presenting natural language prompts as design feedback 418, which requests additional information from the user that may help converge on a suitable HMI design for the user's control system; or by providing suggestions for improving or optimizing HMI project 402 based on the current design status of project 402 (e.g., recommendations for rearranging or resizing graphical objects to make more efficient use of space, recommendations for modifying the display screen navigation structure of project 402, etc.).

[0058] In addition to allowing users to submit free-form natural language design input 412, some implementations of the user interface component 304 can present users with pre-written or pre-loaded prompts for selection and submission to the generative AI component 310. These pre-written prompts can be stored in a prompt library 408 of the HMI development system 302 and represent questions or development tasks typically submitted by users of the HMI development system 302. In an example scenario, the user interface component 304 can present 10 of the most common questions or requests submitted by users of the system 302 as a list of selectable natural language prompts, where selecting a prompt from the list causes it to be submitted to the generative AI component 310 for processing. Where appropriate, the user interface component 304 can allow users to customize one or more parameters of the selected pre-loaded prompt to meet specific needs (e.g., by indicating the specific display screen, graphical object, data label, controller definition, program instruction, or industrial asset that the selected pre-written prompt will point to).

[0059] In another example, the generative AI component 310 can determine the general design theme pointed to by the user's initial design input 412 (e.g., creating a suitable layout for graphical objects on a display screen, designing a suitable screen navigation structure for HMI project 402, etc.) and select a subset of pre-written tips from the tip library 408 that are determined to be helpful to the design theme. If needed, the user interface component 304 can present these tips on the system's development interface for the user to choose from. The selection of a pre-written tip causes the selected tip to be submitted to the generative AI component 310 for processing as design input 412.

[0060] Generative AI component 310 can use a range of methods to process natural language design input 412 submitted by the user and formulate prompts 504 for generative AI model 406, which is designed to produce responses 406 that assist the user's design requests. According to the example method, generative AI component 310 can access archives of chat exchanges between generative AI component 310 and other users of system 302 and identify chat sessions initiated by user queries that are similar to the initial design input 412 submitted by the current user. After identifying these archived chat sessions, generative AI component 310 can analyze these past chat sessions to determine the type of design action ultimately performed by system 302 on HMI project 402 as a result of these sessions (e.g., creating or configuring a display screen with content based on specific keywords from the user query, adding a certain type of graphical object to the display screen, configuring data links controlling the animation properties of the graphical object, adding controller definitions to HMI project 402, defining controllers with which project 402 exchanges data, etc.), and update HMI project 402 based on the results of these past chat sessions, adapting to the user's initial request (or, if necessary).

[0061] Analysis of these archived chat sessions, along with any other relevant industry knowledge or expertise encoded in the custom model 322, can also assist the generative AI component 310 in inferring user needs from the initially ambiguous natural language design input 412 and modifying the design of the HMI project 402 in a way that addresses those needs. If the generative AI component 310 determines that supplementary information from the generative AI model 406 is needed to determine a design modification to the HMI project 402 that has a sufficiently high probability of satisfying the user's request, the generative AI component 310 can also formulate a prompt 504, which is designed to prompt at least a portion of the information that the generative AI model 406 infers to be of interest to the user. For example, this could include formulating a prompt 504 to request a specific type of information from the generative AI model 406 that may not have been specified in the user's design input 412, but which the generative AI component 310 determines will address the user's needs. In this way, the generative AI component 310 and its associated custom model 322 can proactively construct the user's natural language design input 412 in a way that quickly and accurately guides the generative AI model 406 toward the user's desired HMI design solution (e.g., generating an HMI project 402 that meets the design requirements implied by the natural language design input 412).

[0062] In another example approach, the generative AI component 310 may augment the user's natural language design input 412 with additional information from a custom model 322 that contextualizes the user's request, and integrate this additional information with the user's request to generate a prompt 504 submitted to the generative AI model 406. The type of additional contextual information added to the design input 412 may depend on the nature of the design request and may include, but is not limited to, information obtained from a supplier knowledge base or device documents of an industrial device known to be relevant to the user's design request.

[0063] Various example HMI design actions, such as HMI project creation and editing, can now be performed by system 302 based on the processing of a user's natural language design input 412. Typically, system 302 can perform virtually any type of HMI project creation, development, and editing action based on and according to a natural language request or design description submitted as design input 412. For example, in response to receiving a natural language request to create a new HMI project 402 (e.g., "Create new project"), HMI generation component 306 (with the assistance of generative AI component 310 as needed) can initiate a new HMI project 402 within the system's development environment. As described above, the natural language design input 412 can be submitted to system 302 as typed or spoken natural language text, which can be input via a chatbot interface or another type of natural language conversational interface presented by user interface component 304.

[0064] As part of this initial request to create HMI project 402, or via a subsequent separate natural language design request, the user can submit natural language design requirements or configuration requests for the newly created HMI project 402. Example HMI design actions that can be performed by system 302 based on such natural language design input 412 may include, but are not limited to: creating HMI project 402, defining the display screens constituting HMI project 402, defining navigation modes or controls for navigating between display screens (including adding graphical navigation controls to selected display screens and defining which other display screens these navigation controls will invoke), selecting both static and animated graphical objects and adding them to the display screens, defining animation controls for animated graphical elements (including specifying data tags for industrial controllers or another data source controlling the animation state of graphical objects), and specifying the graphical objects on the display screen. Define the layout or position on the display screen, define the color scheme of the display screen, define the conditions that will trigger alarms (e.g., the upper or lower limit of a specified data label to trigger a high-level or low-level alarm), define the text of the corresponding alarm, define multiple versions of a given HMI item 402 to be used in different environmental conditions (e.g., strong light conditions, outdoor use, high particle environment, etc.), define scripts that are triggered in response to specified conditions detected by the HMI during runtime, name the display screen or graphical object, add controller definitions to the HMI item 402 to define the industrial controllers with which the HMI will exchange data values, or define other such attributes of the HMI item 402.

[0065] For example, as sample natural language design input 412, a user could submit "Create a new project where navigation is on the left, help and login are in the upper right, alert summary is centered at the top, and the company logo is on the left. Create three sub-areas across all cells: alert history and top ten alerts." The generative AI component 310 can analyze this input 412 using industry or application knowledge encoded in the custom model 322 and responses 506 prompted by the generative AI model 406 as needed to determine or infer the user's HMI design requirements and instruct the HMI generation component 306 to create or edit the HMI project 402 in a way that meets those requirements. In this example, the HMI generation component 306 would define the HMI project 402, define the display screen within it, add navigation button controls to the left side of the screen, add help and login button controls to the upper right corner of the screen, center the alert summary near the top of the screen, and add the company logo to the left side of the screen.

[0066] The user's natural language design input 412 may include explicit references to portions or elements of HMI item 402 as needed to describe the HMI configuration actions to be performed. These references may include display screen names (e.g., "Add a button for navigating to the overview screen to the lower left of the row 1 screen"), names or identifiers of graphical elements or their attributes (e.g., "Set the open state color of the #1 valve graphic to green and link it to the valve open label of controller 1"), or other such explicit references.

[0067] In some scenarios, users can submit natural language design input 412 describing the visualization requirements of the industrial automation system that needs HMI project 402 in more general terms (i.e., without references to specific elements of HMI project 402), rather than submitting natural language design input 412 describing HMI configuration actions with explicit references to elements or configuration functions of HMI project 402 itself. According to this method, users can describe, via natural language design input 412, the industrial assets, devices, or machines (e.g., tanks, valves, presses, conveyors, processing stations, etc.) that are part of the automation system and want to be visualized on the HMI, the functional relationships between assets (e.g., "I need a tank called tank #1 that feeds material to another tank called tank #2 via an inlet valve called valve #1"), and information about the assets the user wishes to visualize ("Show me the valve status and tank fill level"). Users can also provide other details about the assets or applications, such as tank capacity, maximum flow rates between tanks, or other relevant information about the assets. With the assistance of generative AI component 310, HMI generation component 306 can process these functional descriptions to determine the user's visualization requirements and develop HMI project 402 to meet those requirements. Development actions performed by system 302 in response to the natural language design input 412 may include adding appropriate graphical objects to the HMI display screen, arranging these graphical objects in a manner that reflects the functional relationships between assets represented by the graphical objects, determining and configuring data links between the animation attributes of these objects and their corresponding controller data labels, or other such HMI development functions.

[0068] In other example scenarios, the user's natural language design input 412 can describe the type of industrial automation system or control application to be visualized, as well as application-specific details of the user system that can be used by system 302 to create and customize suitable HMI projects 402 for the automation system. For example, a user can specify via natural language design input 412 an HMI required for batch processing of a specified type of material (e.g., plastics, chemicals, pharmaceuticals, etc.). This design input 412 can describe any functional details that can be used by generative AI component 310 and HMI generation component 306 to determine suitable graphical content, layout, and animations for visualizing the control application.

[0069] The generative AI component 310 can, in part, formulate appropriate HMI configuration actions in response to any of the various types of natural language design inputs 412 described above, based on industry-specific knowledge encoded in the custom model 322 (or the underlying training data 502). The industry knowledge encoded in the custom model 322 can assist the generative AI component 310 and the HMI generation component 306 in determining the preferred or compliant configuration of the HMI project 402, which also meets the user design requirements specified in the natural language design inputs 412. This encoded knowledge alleviates the burden on the user to provide highly granular details of the control application or automation system for which they are designing the HMI, because the system 302 can leverage the domain-specific knowledge recorded in the custom model 322 to infer the user's needs and formulate an appropriate HMI design that meets those inferred needs.

[0070] For example, some of the training data in training data 502 used to train the custom model 322 may include knowledge of standard or general system design for corresponding different types of industrial control applications (e.g., batch processing applications, die casting applications, material handling applications, machining applications, etc.). This knowledge may include: the type, quantity, and configuration of the corresponding types of industrial assets (e.g., machines, monitoring and control devices, conveyors, processing stations, barrels or tanks, pipes, ovens, presses, etc.) typically used in such applications; the functional relationships between these assets (e.g., the arrangement of assets, the direction and sequence of parts or materials flowing through assets, etc.); or other such domain-specific knowledge. In response to receiving a user's natural language design input 412, which describes the functional requirements for visualizing an automation system designed to implement a specific type of industrial control application, the generative AI component 310 can access relevant domain-specific information about the specified type of control application from the custom model 322 and use this information in conjunction with aspects of developing an HMI design with a high probability of meeting the requirements described by the user's natural language design input 412. This can include, for example, inferring a set of industrial assets that are highly likely to be part of an automation system to be visualized (e.g., based on the type of control application or industrial vertical described by the user's design input 412), and generating an HMI project 402 with a display screen having a preliminary layout of graphical objects representing these assets. In such a scenario, even if the user's natural language design input 412 does not specify the identity or type of the assets that will constitute the automation system, the generative AI component 310 and the HMI generation component 306 can infer a suitable set of graphical asset representations and the arrangement of these graphical representations.

[0071] In some implementations, the custom model 322 can be trained using information about a specific custom machine (e.g., a machine or production line built by an OEM for a customer), or the custom model 322 can include a pre-trained model provided by the OEM. The machine-specific training data 502 can include, for example, the names and functions of workstations constituting the machine or production line (e.g., tool stations, quality inspection stations, ovens, etc.), the identification and arrangement of components constituting the machine (e.g., valves, barrels, pumps, industrial robots, actuators, stoppers, etc.), machine specification data, or other such information. The generative AI component 310 can utilize the machine-specific information used to train these models 322 to appropriately configure the HMI project 402 to visualize the custom machine based on user design input 412. The custom model 322, trained using machine-specific training data 502 for machines known to be deployed at a customer facility, can assist the HMI development system 302 in generating an HMI project 402 that accurately reflects the customer's equipment with relatively little design input 412 from the user.

[0072] By supporting HMI development through the use of natural language design inputs 412 that describe the functional requirements of HMI project 402 at varying degrees of granularity or specificity, HMI development system 302 enables users in various roles within an industrial enterprise to contribute to the design of HMI project 402. These roles include plant engineers or control engineers designing the corresponding automation system, maintenance personnel who will maintain and troubleshoot the automation system, and machine operators who will operate the automation system. These diverse user roles may require different information presentations from HMI project 402, and members of these roles can ensure that HMI project 402 meets their specific needs by providing natural language design inputs 412 that describe their role-specific requirements at various levels of specificity. For example, a control engineer can submit natural language design input 412 that describes visual requirements based on specific equipment or device names, clearly identified controller data labels for animated attributes driving graphical objects, or other technical languages ​​and references that machine operators may not be familiar with. Additionally, machine operators can be invited to contribute natural language design input 412 describing their HMI requirements based on the information they wish to see and how they want that information to be organized (e.g., "I want to see the tank fill level and outlet flow rate on the same screen"). In all these scenarios, system 302 can process these natural language inputs, leveraging domain-specific knowledge encoded in a custom model 322 and responses from a generative AI model 406 as needed, to implement development actions on HMI project 402 that are intended to meet the requirements of these diverse users.

[0073] System 302 can support multi-user collaborative development of HMI project 402, receiving natural language design input 412 from users with different roles. In some such collaborative development scenarios where users with different roles contribute to the development of HMI project 402, some implementations of system 302 can process the design input 412 for HMI project 402 according to the role of the user from whom the design input 412 is received. For example, some implementations of HMI development system 302 can create multiple versions of HMI project 402 for the development of a given automation system, wherein each version of project 402 is created using design input 412 received from a user with a single common role (e.g., machine or production line operator, plant engineer, maintenance personnel, etc.). This results in multiple versions of HMI project 402 that are specific to the corresponding different user roles and can be invoked by users with those roles during runtime. In this way, system 302 enables users with different roles to create versions of HMI project 402 that meet the specific requirements of their roles.

[0074] Alternatively, the HMI generation component 306 can integrate design inputs 412 from multiple users with different roles into a single HMI project 402 that satisfies all the requirements specified by the multi-user design inputs 412. In some such implementations, if requests from these roles conflict with each other, the system 302 can prioritize the design request from one role over the design request from another. For example, if the system 302 determines that design inputs 412 received from the plant engineer conflict with design inputs 412 submitted by the machine operator (e.g., in terms of the placement or orientation of graphic objects, color schemes, content to be included in or omitted from the HMI), the system 302 can choose to implement the plant engineer's design inputs 412 in the HMI project 402 while overriding the design inputs 412 received from the machine operator.

[0075] In this regard, the generative AI component 310 can infer the user's level of expertise based on the wording of the natural language design input 412, and formulate design feedback 418 based on this inferred level of expertise (e.g., answers to follow-up questions about the user's design request, questions about HMI design, or questions about development tools provided by system 302). The user's level of expertise can be inferred, for example, based on determining whether the words, phrases, or terms used in the design input 412 are likely to be used by a relatively advanced professional who can be expected to understand the technically more sophisticated design feedback 418, or alternatively, whether more basic information might need to be included in the feedback 418.

[0076] For users at these different skill levels, the generative AI component 310 can express any design feedback 418 or response to design input 412 at a level of assumed understanding that is appropriate for the user's inference, including prompting the user to provide additional information to assist the system 302 in determining appropriate HMI development actions that have the potential to meet the user's request. This can influence the choice of words used in the feedback 418 and the granularity of the response content.

[0077] Some implementations of the HMI development system 302 can also be configured to process design input 412, which includes various types of design documents generated as part of the design process for the automation system to be visualized. For example, a user can submit digital design drawings (e.g., computer-aided design (CAD) drawings) of the automation system or machine requiring the HMI project 402 as design input 412. Example CAD drawings that can be submitted as design input 412 may include electrical drawings, I / O drawings, mechanical drawings, panel layout drawings, or other such documents. With the assistance of the custom model 322 or responses prompted by the generative AI model 406, the generative AI component 310 can determine the appropriate content and format for the corresponding HMI project 402 based on the analysis of these drawings, and generate the HMI project 402 with that content and format.

[0078] In the example scenario, generative AI component 310 can identify the types of machines or devices included in the automation system, and the communication or functional relationships between these machines and devices, based on the analysis of I / O drawings, mechanical drawings, or electrical drawings of the automation system. Generative AI component 310 can also infer the type of industrial processing performed by the automation system based on the analysis of these drawings. Furthermore, based on knowledge of the types of information that operators are typically interested in regarding the inferred type of industrial application (such as that encoded in custom model 322, or based on the analysis of other archived HMI applications of similar types of industrial applications), generative AI component 310 can instruct HMI generation component 306 to generate HMI project 402, which contains graphical objects representing a subset of known machines or devices of general interest (e.g., tanks, valves, motors, conveyors, etc.) and organized to represent the functional relationships between the machines or devices. HMI generation component 306 can also define the animation attributes of these graphical objects and the data links (e.g., as determined from the I / O drawings) between them and their corresponding data tags or addresses in the industrial controller that will monitor and control the automation system.

[0079] Some implementations of the HMI development system also allow users to submit or import previously developed code snippets or portions of previously developed HMI visualizations (e.g., pre-developed HMI displays or completed HMI applications or projects 402), and submit requests for natural language summaries of these pre-developed components. For example, a developer might have an HMI project 402 developed by another developer or another type of industrial visualization application (or a portion of such an application), and might have specific questions about the functionality of an unfamiliar HMI. The developer can submit a pre-developed project 402 along with requests for summaries of HMI functionality or specific questions about the HMI design to system 302. Example questions that a user may submit regarding the submitted HMI project 402 may include, but are not limited to: questions about the type of industrial application targeted by the submitted project 402; questions about the animation or data link attributes of graphical objects contained in the HMI project 402 (e.g., "What is the state of the control bucket #3 graphic?", "What data label does the start button link to?", etc.); questions about the computing resources required to execute the HMI project 402 (e.g., "How much memory is required to run this HMI project?"); questions about the type of information a given display screen is designed to convey; questions about the functionality of control graphics included on the display screen (e.g., "What does sliding the control graphics do?"), or other such questions. In response to these submissions, the generative AI component 310 may generate a natural language answer to the user's question based on an analysis of both the user's question and the submitted HMI project 402, and present the answer as design feedback 418. As in the previous example, the generative AI component 310 may, as needed, utilize industry knowledge contained in the custom model 322 and responses 506 prompted by the generative AI model 406 to determine and formulate answers to the user's questions about the functionality of the HMI project.

[0080] In some scenarios, when a project is submitted to system 302 for analysis, system 302 can provide a general natural language summary of the purpose and function of the HMI project, rather than submitting questions specific to an unfamiliar HMI project 402. This general summary can specify, for example, the overall purpose of the HMI (e.g., the type of machine or industrial process the HMI is designed to visualize), the type of information provided by each of the HMI's display screens, or other such information.

[0081] Some implementations of system 302 may also support similar reverse hinting engineering methods for analyzing and summarizing unfamiliar control code submitted to system 302. Similar to the reverse hinting engineering analysis of unfamiliar HMI items 402 or visualization components, generative AI component 310 may generate a general natural language functional summary of control code segments submitted to system 302 by the user, or it may generate natural language answers to specific questions about the submitted code. Example summaries or answers may include a description of the control function or machine the code is designed to perform (e.g., batch control process, sheet metal stamping machine control, sheet metal tension control, control loops to keep process variables within target ranges, etc.), a description of the language in which the code is written, an identifier of the hardware platform capable of executing the code, I / O requirements for executing the code (e.g., the number of analog inputs, digital inputs, analog outputs, and digital outputs required to interface the code with an automation system to be monitored and controlled by the code), or other such descriptions. Generative AI component 310 can also, based on analysis of unfamiliar code, determine edits that can be made to improve code efficiency or readability, or reduce the total number of lines of code without changing the intended functionality of the code, and generate natural language recommendations describing these edits. Example edits that can be recommended in this way may include, but are not limited to: removing redundant code, merging duplicate code sections into a single routine referenced in the code as needed, modifying variable naming conventions, or other such edits. In any of these reverse-hint engineering scenarios, generative AI component 310 can, as needed, utilize relevant content from custom model 322 and responses 506 from hints from generative AI model 406, combined with analysis of the submitted HMI project or code, and generate a natural language summary or answer.

[0082] Once development on HMI project 402 is complete and HMI generation component 306 is completed, HMI project 402 can be deployed to HMI terminal 114 or other computing platforms with display capabilities for execution. Figure 6This diagram illustrates the debugging of HMI project 402 onto HMI terminal 114. The system's HMI deployment component 308 can communicatively interface with HMI terminal 114 via a direct connection between HMI development system 302 and terminal 114, or via a remote connection to terminal 114 in the case of an implementation where system 302 is executed on a cloud platform or other remote platform. When executed on HMI terminal 114, the compiled HMI application 602 presents an HMI with a display screen and associated graphical elements and behaviors defined by design input 412 and generated and configured at least in part based on the processing of the developer's natural language prompts. Where HMI project 402 also defines communication parameters for HMI terminal 114 itself, the execution of HMI application 602 will configure the terminal's networking or communication settings according to the communication settings defined in HMI project 402. This may include, for example, setting the terminal's networking parameters to operate on the factory network 116, and setting the HMI's communication settings so that the terminal 114 will communicate with industrial devices (e.g., controller 118) with which the HMI terminal 114 will exchange data.

[0083] In some implementations, when a new HMI project 402 has been created, the user can also request a test script via natural language input 412 to test the interaction between the HMI application 602 and the industrial controller 118 with which the application 602 will communicate. Based on this request, the generative AI component 306 can generate such a test script based on any available information about the HMI terminal 114, the industrial controller 118, and the network 116, as well as the communication requirements defined by the HMI project 402 itself, and bundle the test script with the HMI application 602. After the HMI application 602 is deployed, the test script can be executed on the HMI terminal 114 to test the communication link between animated graphical objects on the HMI application's display screen and the corresponding data tags in the controller 118. If errors are found based on the execution of these test scripts—such as incorrect data tag links, incorrectly configured or unconfigured graphical objects, or other such errors—information about these errors can be returned to the system 302, which can use the generative AI component 310 to determine debugging actions to correct these errors. Then, system 302 can implement these corrections in HMI project 402, redeploy application 602 to HMI terminal 114, and re-execute the test script to ensure that all errors are corrected.

[0084] After deploying HMI project 402 as runtime application 602, some implementations of HMI development system 302 can also support dynamic runtime modifications to the resulting HMI application 602 using natural language prompts. Figure 7This diagram illustrates the use of natural language prompts 706 during runtime of the HMI development system 302 to implement dynamic editing 704 of the HMI application 602. At any time during the runtime operation of the HMI application 602, an operator or other user can submit natural language prompts 706 to the application 602 requesting modifications to the visual aspects of the HMI application 602. These natural language prompts 706 can be submitted via a text-based or verbal chat interface integrated into the HMI application 602 and are then routed to the HMI development system 302 for processing. The generative AI component 310 can process these natural language prompts 706 to determine the nature of the user-requested modifications and return HMI edits 704 that implement the requested modifications on the HMI application 602.

[0085] In an example scenario, a user can indicate via natural language prompt 706 that the display screen presented by HMI application 602 is difficult to see due to environmental conditions in the area where the HMI is being used (e.g., bright ambient light or relative darkness). Prompt 706 can specify the nature of the visual difficulty, such as, "I can't see the screen in this bright light," or "This control room is dark; could you brighten the display?" Based on an assessment of prompt 706, generative AI component 310 can adjust application 602 or HMI terminal 114 itself to correct the problem (e.g., by adjusting brightness or contrast levels to counteract the bright ambient conditions). Another example prompt 706 can request countermeasures for visual impairments (e.g., "I can't see blue in this environment; can we correct that?" or "I can't see yellow very clearly; could you choose an alternative color?"), and in response to such prompt 706, generative AI component 310 can determine and implement appropriate countermeasures to compensate for the indicated impairment (e.g., adjusting the brightness, contrast, or color selection on the HMI display).

[0086] Natural language prompt 706 can also be used to change the content of a display rendered by an HMI application. For example, a user can submit natural language prompt 706 requesting the addition or removal of graphical elements, such as animated graphical objects or numbers, from the currently rendered display. In response to receiving and processing the prompt, generative AI component 310 will return an HMI edit 704 that implements the indicated modifications on HMI application 602.

[0087] Users can also dynamically create new custom display screens using natural language prompts 706 as needed. For example, a given machine operator might want to aggregate graphical objects currently distributed across multiple different screens onto a single new display screen. To achieve this, the operator can submit a natural language prompt 706 describing what the operator wants to see (e.g., "Get me the valve from display 2, get me the pump from display 3, and build a new display for me."), and based on the translation of that prompt 706, the generative AI component will return an HMI editor 704 that creates a screen with the described content and adds it to the HMI application 602.

[0088] System 302 can also process natural language prompts 706, which describe the functionality the operator wants to see from a more abstract perspective, rather than explicitly identifying what the operator wants to see with specific graphical objects. For example, natural language prompt 706 might indicate that the operator wants to see a display of the open and closed status of an auxiliary calibration valve (e.g., “Give me the screen for calibrating valve #2”). Based on the analysis of this prompt 706, industry knowledge encoded in the custom model 322 (which may include knowledge of the types of information known to be helpful in performing such calibrations), and analysis of the HMI application 602 itself, the generative AI component 310 can determine the appropriate content and associated data links expected to be useful for performing valve calibration (i.e., links to appropriate controller data labels or other sources of the required data), and return to an editor 704 to create a new screen that includes that content. In some cases, the content of the new screen can be extracted from other screens already defined in the HMI application 602.

[0089] In another scenario, an operator might want to view the identity and status of interlocks or permissions for a specific machine state or control action of interest. Typically, an interlock or permission for a given machine state or control action is a set of conditions that must be true before the control system allows the machine to be placed in a machine state or issue a control action. In the example scenario, an operator might want to view the conditions that prevent the machine from being placed in automatic mode. Even if a dedicated display screen for displaying these interlock states is not currently defined as part of the HMI runtime application 602, the operator can submit a natural language prompt 706 requesting to view the interlocks for the machine state of interest (e.g., “Show me the interlocks for automatic mode of this machine”). In response to the request, the generative AI component 310 may identify the interlocks of interest based on: analysis of the industrial control program executed on the controller 118 that monitors and controls the machine (e.g., by identifying the program conditions that allow the machine to be placed in automatic mode, and descriptions of these conditions inferred from their corresponding comments or I / O addresses), any available design documents accessible to the system 302 (from which the generative AI component 330 may determine any safety devices or other control devices that act as permissions or interlocks for machine states or control actions), and, if necessary, knowledge encoded in the custom model 322 (e.g., proprietary information on machine control design previously submitted by plant engineers, interlocking industry knowledge required for the type of control process performed by the machine, etc.), and a response 506 prompted by the generative AI model 406. Based on the results of this analysis, the HMI generation component 306 may return to an editor 704, which creates a new interlock screen that presents the interlocks along with the current status of each interlock as a graphical or alphanumeric indicator. An example interlocking screen created in this way can display each interlock as a color-coded indicator containing an alphanumeric description of the interlock (e.g., “Mode Switch Auto”, “Safety Door Closed”, “Clamp in Place”, “Pusher Retracted”, “Light Curtain Cleared”, etc.), where the color of the indicator conveys the current status of the interlock (e.g., green for satisfaction and red for dissatisfaction).

[0090] In addition to creating the content and layout of the interlocking screen, editor 704 can also configure the animation properties of the interlocking for each graphical interlocking indicator, linking the color state of the interlocking indicator to the appropriate data source (e.g., controller data labels) representing the current state of the interlocking. To this end, generative AI component 310 can determine, based on analysis of the control program or other information sources, which data labels of the control program executed on controller 118 represent the corresponding current state of the interlocking, and link the animation properties of the graphical interlocking indicator to these data labels.

[0091] Operators can use a similar approach to create dynamic alarm screens that meet the content and formatting preferences specified by the operator's natural language prompt 706. For example, an operator might submit natural language prompt 706 requesting a view of alarms associated with a specified machine station and also requesting a preferred display format for the alarms (e.g., "Show me the alarm grid for the pick-up and place station"). Based on the analysis of this prompt 706, generative AI component 310 can determine the identity and status of alarm conditions defined for the specified station and return to editor 704, which creates a new alarm screen that presents the identified alarms and their respective active or inactive states in a grid (e.g., as color-coded animation). As in the previous example, generative AI component 310 can utilize any necessary information related to identifying the relevant alarms and their current status, including but not limited to control procedures for monitoring and controlling the station of interest, proprietary design information or industry knowledge stored in custom model 322, responses 506 from prompts in generative AI model 406, or other such information. Operators can also use natural language prompt 706 to instruct system 302 to filter the view of alarms as needed. For example, a natural language prompt 706 stating "Show me all priority 1 alarms for the shampoo production line" can enable the generative AI component 310 to identify a subset of active machine alarm conditions associated with the specified production line and having a priority 1 importance level, and return to an editor 704, which creates a new display screen to present the identifiers of these alarms.

[0092] Some implementations of system 302 can also dynamically construct display screens that access and display information from new data sources not currently accessed by the HMI runtime application 602 based on the user's natural language prompt 706. These data sources may include, for example, a work order management system that stores maintenance work orders for opening and closing of a plant, a cloud-based industrial analytics system that collects operational data from automation systems on the factory floor and generates statistical operational statistics based on the analysis of this data, or other such systems. In an example scenario, the user may submit a natural language prompt 706 requesting a link to the work order management system and the presentation of a new display screen containing specified information from that system; for example, “Build me a screen to display the unfinished work orders for the press.” Based on the analysis of this prompt 706, the generative AI component 310 can identify the work order management system containing the required information, retrieve a subset of work order data from that system that satisfies the user's request (e.g., a subset of unfinished work orders for maintenance operations to be performed on the press), and modify the runtime application 602 to create and display a new display screen that presents information on unfinished work orders (including descriptions and statuses of unfinished maintenance activities). The work order management system can be a cloud-based system accessible by the HMI development system 302 or a local system residing on the factory floor. The HMI development system 302 can retrieve relevant work order information from the local system to display on the runtime application 602.

[0093] System 302 can also dynamically link the HMI runtime application 602 to other products or services based on the user's natural language prompts 706. For example, a user can submit natural language prompts 706 requesting information about products or services the factory might need to purchase (e.g., replacement parts or equipment, engineering services, etc.). Based on the analysis of the prompts 706, the generative AI component 310 can identify suitable product or service providers and link the HMI runtime application 602 to the customer service or technical support portal associated with that provider. Depending on the nature of the user's request, system 302 can present information retrieved from the provider's website about the requested product or service (e.g., pricing and availability) on a new display screen, establish real-time communication links with customer support experts associated with the product or service provider, or otherwise dynamically attach the HMI runtime application 602 to the desired product or service platform as needed by the user.

[0094] In a similar manner, system 302 may respond to natural language prompt 706 to create a new display screen and link it to a remote technical support expert or knowledge base article, which requests assistance with operational problems observed by the machine or system monitored by HMI runtime application 602, or requests assistance with the functionality of the HMI system itself.

[0095] Similar to the design-time scenario, system 302 can also utilize pre-loaded or pre-written prompts during the runtime of HMI application 602. For example, generative AI component 310 can retrieve selected pre-written prompts from prompt library 408 when appropriate and present these pre-written prompts to the user via HMI application 602 for selection and processing by generative AI component 310. Generative AI component 310 can select an appropriate subset of pre-written prompts to present to the user based on the current state of HMI application 602 or its associated automation system, or based on the user's interaction pattern with HMI application 602. For example, generative AI component 310 can determine that the user is attempting to perform a specific operational task based on the user's interaction with HMI application 602. Based on this determination, generative AI component 310 can select and present a subset of pre-written prompts determined to be relevant to the operational task from prompt library 408. The user's selection of one of these pre-written prompts allows the selected prompt to be processed by generative AI component 310.

[0096] In some implementations, after system 302 has implemented dynamic runtime modifications to HMI application 602 based on user natural language prompts 706, the user can submit a request to save the modified version of HMI application 602 as a template 702 to be stored together with the HMI project 402 that compiles and runs HMI application 602. Each template 702 represents a version of the underlying HMI project 402 that has been modified according to a given set of natural language prompts 706 as described above, and can be given a name or other type of identifier to distinguish template 702 from other templates 702 associated with project 402. The user can invoke the selected template via interaction with HMI application 602 (e.g., by requesting a selected template via natural language prompts 706). When the selected template 702 is invoked, user interface component 304 can modify the HMI application 602 currently running on HMI terminal 114 according to the selected template 702. In another example scenario, template 702 can record dynamic modifications made to HMI application 602 to make HMI more suitable for a given environment, context, or user. These may include modifications to adapt the HMI to bright light environments, dark environments, visually impaired users, or other such contextual considerations. System 302 may invoke one of these templates 702 in response to an explicit request to adjust the HMI application 602 according to the selected template 702, or automatically in response to system 302 determining that the contextual conditions for creating the template 702 are currently active.

[0097] As another runtime feature, the HMI development system 302 can also handle natural language prompts 706, which include questions about the current operation of the HMI application 602 itself or about the automation system visualized by the application 602. For example, a user can submit a natural language prompt 706 requesting assistance in diagnosing performance problems using the visualized automation system or activity alerts already generated by the HMI application 602. For instance, such a prompt 706 could request suggested countermeasures to resolve a current alert status reported by the HMI or a performance problem observed by the operator and described by the prompt 706. In response to such a prompt 706, the generative AI component 310 can formulate a natural language response 708 to the prompt 706 based on the nature of the problem described by the prompt 706 and knowledge of the specific automation system and its components encoded in the custom model 322 (e.g., if one of the custom models 322 is a machine-specific model, such as a model provided by an OEM), more general knowledge about the type of industrial application performed by the automation system encoded in the custom model 322, or one or more responses 506 from the prompt 406 of the generative AI model 406. The system 302 can present the resulting natural language responses 708 on the HMI via a chat interface for submitting the prompt 706. Depending on the nature of the user query, these responses 708 may describe a recommended workflow for resolving an alarm condition or performance problem, answer questions about the nature of the alarm or problem, or other such information. The ability to process chat-based queries about alarm conditions or performance problems can streamline diagnostic processes and assist machine operators or other plant personnel in quickly resolving performance issues.

[0098] In some implementations, system 302 can monitor workflow or HMI interaction patterns of machine operators or engineers in response to specific alarm conditions or performance issues, and further refine the diagnostic process by training one or more custom models 322 based on these observed interaction patterns and the learned workflow knowledge known to have successfully resolved these performance issues. Generative AI component 310 can then use these trained models 322 to generate recommendations as a natural language response 708 for resolving similar types of alarms or performance issues. For example, generative AI component 310 can determine that when a given alarm condition is triggered on the HMI, the corresponding alarm is resolved most quickly or efficiently when the operator performs a specific set of interactions with the HMI and automation system (e.g., navigating to a specific display screen, changing the status or setpoint value of a binary tag via HMI interaction, performing a series of control panel or HMI interactions, removing parts from the automation system's workstation, etc.). Therefore, training component 312 can utilize this learned optimal interaction pattern to train one or more custom models 322. If an alarm is triggered in a subsequent operational plan, the generative AI component 310 can utilize these updated trained models 322 to generate and present a natural language response 708, which describes the action to be taken by the operator to resolve the alarm situation. This guidance can be presented in response to the detection of an alarm situation or in response to a request from the operator for assistance in resolving the alarm (submitted as a natural language prompt 706). In this way, the system 302 can dynamically generate and maintain up-to-date standard operating procedures (SOPs) to resolve alarm situations or operational anomalies.

[0099] In some implementations, in addition to creating SOPs for alarm resolution workflows based on observations of interaction patterns changing over time, as discussed above, or as an alternative, system 302 may update the SOP more immediately after an operator resolves a given alarm condition. For example, after an alarm condition or anomaly occurs on an automated system and is resolved by a machine or production line operator, system 302 may present a natural language prompt on the HMI asking the operator if they wish to update the automated system's SOP to record the operator's sequence of actions as the preferred response to the alarm condition. If the operator responds by requesting an update (or creation) of the SOP to record the workflow, the generative AI component 310 may create a record of the sequence of actions the operator took to resolve the alarm condition and associate that record with the alarm condition resolved using that operator sequence. As discussed above, this SOP information may be recorded as part of a customized model 322 so that when the alarm condition recurs at a later time, system 302 can present a description of these actions via the HMI runtime application 302 to provide guidance to other operators.

[0100] Because system 302 can track and record interaction patterns of different operators during runtime, it can present these past user interaction patterns in response to the user's natural language prompt 706. For example, if the automated system experiences an alarm condition or performance problem unfamiliar to the current machine operator, the operator can submit a natural language prompt 706 requesting assistance in resolving the problem by explicitly requesting a previous interaction pattern that was successfully resolved by another operator, or by more generally requesting assistance. In response to such a prompt 706, generative AI component 310 can retrieve information about one or more past interaction sequences that are known to have been successfully mitigated, or record relevant portions of the SOP for a previous action sequence used to mitigate the alarm condition, and present this information on the HMI. The interaction sequence may include an ordered sequence of actions performed by a previous operator to resolve the problem. These actions may include interactions with the HMI (e.g., navigating to a specific display screen, interacting with specific HMI controls such as graphical buttons or data input fields), interactions with the control panel of the automation system (e.g., pressing a button, setting the position of a selector switch, placing the automation system in a specific operating mode, manually moving machine components of the automation system to their original positions), or other interactions with the automation system that can be detected by system 302 (e.g., removing a part or debris from a location on a conveyor of the automation system monitored by presence sensors). In response to requests for these past interactions, the system's user interface component 304 may present information about these past interactions on the HMI in any suitable format, including but not limited to natural language descriptions of these interactions or graphical demonstrations that visualize the interactions.

[0101] In another example, a user can submit a natural language prompt 706 that requests information about all interactions performed on the HMI and its associated automation systems within a given timeframe (e.g., interactions for previous shifts, interactions for the past six hours, etc.), rather than requesting a recommended set of interactions to resolve a specific alarm or performance issue, and system 302 can formulate and present descriptions of these interactions via the HMI.

[0102] In some scenarios, HMI runtime application 602 can be configured to automatically generate data logs that record details of operational or technical events related to HMI application 602, HMI terminal 114, or the HMI runtime application 602 itself. Such data logs may also record user interactions with the HMI, diagnostic or health status data of HMI terminal 114 or HMI runtime application 602, or other such information. A given data log may include one or more files (e.g., text files or other formats) containing a list of timestamps recording detected events or states, where each item in the list includes a description of the detected activity and the date and time the activity was detected. HMI runtime application 602 may generate and store these data logs locally on HMI terminal 114, or send these data logs to system 302 for storage.

[0103] If needed, a user can submit a request for a natural language summary of the data logs or portions thereof via interaction with the HMI runtime application 602. For example, a user can submit a natural language prompt 706 requesting a summary of activities within a specified time frame, and / or a subset of activities falling within a specified activity category or related to a specific aspect of the automation system or machine (e.g., “What does the data log indicate regarding the feeder system outage that occurred this morning?”). Based on the analysis and interpretation of this request, the generative AI component 310 can identify a relevant subset of information recorded in the data logs (e.g., a subset of information falling within a specified time frame and corresponding to the category or type of the desired information), generate a natural language summary of the selected data log information, and present the summary as a natural language response 708. As in the previous example, the generative AI component 310 can formulate these data log summaries as needed, utilizing information contained in the custom model 322 and responses 504 from prompts from the generative AI model 406. An example summary developed in this way can describe a series of events preceding an anomaly that occurred on a machine on the production line, the subsequent actions taken by the operator in response to the anomaly, the total amount of time the machine was offline due to the anomaly, the inferred root cause of the anomaly, or other such summaries.

[0104] Figures 8a to 10Various methods according to one or more embodiments of this application are illustrated. Although for the purpose of simplicity, the one or more methods shown herein are shown and described as a series of actions, it should be understood and recognized that the invention is not limited to the order of actions, as some actions may occur in a different order than those shown and described herein and / or simultaneously with other actions according to the invention. For example, those skilled in the art will understand and recognize that the method may alternatively be represented as a series of interrelated states or events, such as in a state diagram. Furthermore, not all actions shown are necessary for implementing the method according to the invention. Additionally, interaction diagrams may represent methodologies or approaches according to this disclosure when different entities formulate different parts of the method. Furthermore, two or more of the disclosed example methods may be implemented in combination with each other to achieve one or more features or advantages described herein.

[0105] Figure 8a The first part of an example method 800a for modifying an HMI runtime application using natural language design input is shown, wherein the HMI runtime application executes on an HMI terminal or another type of client device and is designed to visualize operational and status information of an industrial automation system monitored by the HMI application. Initially, at 802, a natural language request describing desired modifications to the HMI application is received via a chat interface or another type of natural language interface supported by the HMI application. The natural language request can describe virtually any type of modification or functional requirement that the user wishes to invoke on the HMI application. For example, the user can submit a written or verbal natural language description of a new display screen created for the HMI application, along with the type of content to be included on the new screen. The natural language description can appropriately reference specific aspects of the automation system to convey the type of content the user wishes to view. For example, even if the HMI runtime application does not currently present a display screen with that information, the user may want to view the identity and status of interlocks or permissions associated with determining whether a specified control command (e.g., putting a machine in automatic mode, issuing a home command to an actuator in the automation system, etc.) is permitted. Therefore, the user can submit a natural language request to display the interlock status of the control command of interest. Other examples of natural language requests can describe the desired visualization layout, new graphical content to be added to an existing display screen and the screen location where the content should reside, a description of a new alarm definition to be added to the HMI application's alarm database, or other such natural language input. In some scenarios, natural language requests can reference existing elements or attributes of the HMI runtime application, such as graphical objects or data link definitions that have already been added to the project (e.g., "Link the fill animation of tank 2 to the level label of tank 1"; "Move valve 1 to the right side of the screen", etc.).

[0106] At step 804, the cloud-based HMI development system uses a trained custom model or generative AI model to analyze the request received in step 802 to determine whether sufficient information can be inferred from the request to determine modifications to the HMI application that will satisfy the user's request. The custom model can be trained using a training dataset representing a range of domain-specific industry knowledge. Example training data that can be used to train custom models includes, but is not limited to, information defining industry standards (e.g., global or vertical-specific safety standards, food and pharmaceutical standards, design standards such as ISA-88 standards, etc.), technical details or design standards for various types of industrial control applications (e.g., batch control processing, die casting, valve control, agitator control, etc.), knowledge of specific industrial verticals (e.g., automotive, food and beverage, pharmaceutical, oil and gas, textile, mining, etc.), knowledge of industrial best practices, technical specifications for various types of industrial equipment or assets (e.g., industrial controllers, motor drives such as variable frequency drives, sensors, etc.), control design rules, sample HMI display layouts for various types of control applications or use cases, customer-specific training data describing internal HMI design preferences or standards (e.g., preferred screen layout formats, preferred graphics types, preferred text fonts, etc.), customer-specific design documents of automation systems monitored by HMI applications, or other such training data. As part of the analysis, the system can also generate prompts and submit them to the generative AI model, and, when necessary, combine analysis of the user's request with the generation of a natural language response tailored to the user to utilize the content of the generative AI model's response.

[0107] At step 806, it is determined whether more information from the user is needed to determine the user's desired modifications and to apply appropriate modifications or edits to the HMI runtime application determined to meet these requirements. If additional information is needed ("Yes" at step 806), the method proceeds to step 808, where the HMI development system determines the required additional information and presents a natural language prompt designed to guide the user to provide the additional information. In determining the nature of the necessary additional information, the system may refer to industry knowledge encoded in a trained model and responses from prompts from a generative AI model. At step 810, a response to the prompt generated at step 808 is received via a chat interface.

[0108] Steps 806 through 810 are repeated as a natural language dialogue with the user until sufficient information is obtained that can be translated into a set of functional requirements for the requested HMI modification. If no further information from the user is needed ("No" at step 806), the method continues until... Figure 8bThe second part, 800b, is shown. At 812, the HMI development system formulates modifications to the HMI application based on at least one of the analysis of user natural language requests and responses obtained in steps 802 and 810, the content of a trained custom model, or responses prompted by a generative AI model. Modifications may include, for example, creating a new display screen containing graphical objects conveying information the user wishes to see (and having animated links to appropriate controller data labels), adding graphical elements representing industrial assets or devices to an existing display screen, removing graphical elements from a screen, creating or modifying data links between the animation attributes of graphical objects and the controller data labels controlling those animation attributes, deleting a display screen, defining a screen navigation mode or architecture, adding and configuring control objects (e.g., graphical buttons or data input fields), or other such edits. At 814, the modifications determined in step 812 are implemented by the HMI development system on the HMI runtime application.

[0109] Figure 9a The first part of an example method 900a for dynamically creating or updating standard operating procedures for resolving alarm conditions in industrial automation systems based on monitored operator behavior is shown. Initially, at 902, an HMI runtime application that visualizes the status and operational information of the automation system is deployed and executed on an HMI terminal or another type of client device. At 904, it is determined whether the HMI application has detected an alarm condition occurring on the automation system. Examples of such alarm conditions include, but are not limited to, machine downtime events, critical telemetry values ​​of the automation system drifting outside normal tolerance ranges (e.g., high temperature alarms, high pressure alarms, etc.), blockage of parts at the inlet or outlet of the automation system, or other such alarm conditions. If an alarm condition is detected ("Yes" at step 904), the method proceeds to step 906, where the sequence of operator interactions with the HMI application and the automation system is monitored and recorded to resolve the alarm condition. These interactions may include, for example, navigating to a specific display screen of an HMI application, interacting with graphical elements on these screens (e.g., entering settings via data fields on the screen, submitting commands via interaction with graphical control objects such as buttons), placing the machines of the automation system in a specific mode (e.g., semi-automatic mode, automatic mode, etc.), placing movable actuators of the automation system (e.g., pushers, robotic arms, stoppers, grippers, stepper motors, etc.) in a specific position via manual control of actuators, removing parts being processed by the automation system, interacting with control devices on the automation system control panel, or other such interactions.

[0110] At 908, in response to the detection that the alarm condition detected in step 904 has been cleared, the system presents a query on the HMI application regarding whether the interaction sequence monitored and recorded in step 906 should be added to the standard operating procedure (SOP) of the automation system. At 910, it is determined whether the sequence should be added to the SOP. If so ("Yes" at step 910), the method proceeds to step 910, where the record of the interaction sequence is stored in association with the alarm condition as part of the automation system's SOP. This record will be used to generate natural language instructions to address subsequent occurrences of the alarm condition.

[0111] Then, the method proceeds to... Figure 9b The second part, 900b, is shown. At 914, it is determined whether a request for assistance with another instance of the alarm condition has been received. If such a request is received ("Yes" at step 914), the method proceeds to step 916, where a description of the interaction sequence is presented via the HMI application. In this respect, the system can convert the interaction sequence for a specific alarm condition that has occurred, recorded in the SOP, into a natural language description of the steps to be performed in the sequence, and present this natural language description on the HMI application. In an example scenario, the system can present the description step by step, showing a description of each step to be performed, and waiting for the operator to perform the step before presenting a description of the next step in the sequence. If appropriate, the system can also present a graphical representation on the HMI, which is designed to provide guidance for combining the execution of one or more interaction steps recorded in the sequence (e.g., a graphical representation of the machine with a graphical indicator showing where the next step in the sequence should be applied).

[0112] Figure 10 An example method 1000 for presenting data log information as a natural language summary is illustrated. Initially, at 1002, an HMI runtime application for visualizing the status and operational information of the automation system is deployed and executed on the HMI terminal. At 1004, during the operation of the HMI application and its associated automation system, a data log recording events detected by the HMI application is generated and maintained. The data log may record a list of timestamps of detected activities or statuses, where each item in the list includes a description of the detected activity and the date and time the activity was detected.

[0113] At step 1006, it is determined whether a request for a summary of a portion of the data log has been received (e.g., by submitting such a request via an HMI application). This request may specify a time range of interest and a specific type or subset of information to be summarized, such as data related to a specific machine specified in the request, data related to past anomalies occurring on the automated system, or other such information. The request may be submitted as a typed or spoken natural language request via the chat interface of the HMI application. If such a request is received ("Yes" at step 1006), the method proceeds to step 1008, where, based on analysis of at least one of the data log itself, the content of the request, and the content of a custom model trained with domain-specific industrial training data or a response from a generative AI model prompt, a natural language summary of the portion of the data log to be summarized, as determined based on the analysis of the request received at step 1006, is generated.

[0114] In some scenarios, users can also request root cause analysis of a selected subset of data log activities related to past events (e.g., anomalies experienced by an automated system, key performance indicators of an automated system reaching specified values, etc.). In response to this request, the system can infer the root cause of the event of interest based on analysis of the sequence of events or activities recorded in the data log, relevant domain-specific knowledge recorded in a custom model, and responses from generative AI model prompts as needed. A natural language summary can include a description of the results of this root cause analysis, indicating possible root causes of the past event of interest. At 1010, the natural language summary generated in step 1008 is presented on the HMI application.

[0115] The embodiments, systems, and components described herein, as well as the control systems and automation environments that can implement the various aspects set forth in this specification, may include computer or network components, such as servers, clients, programmable logic controllers (PLCs), automation controllers, communication modules, mobile computers, onboard computers for mobile vehicles, wireless components, control components, etc. Computers and servers include one or more processors (electronic integrated circuits that perform logic operations using electrical signals), which are configured to execute instructions stored in media such as random access memory (RAM), read-only memory (ROM), hard disk drives, and removable memory devices, which may include memory sticks, memory cards, flash drives, external hard disk drives, etc.

[0116] Similarly, the term PLC or automation controller as used herein can encompass functionality that can be shared across multiple components, systems, and / or networks. As an example, one or more PLCs or automation controllers can communicate and collaborate with various networked devices across a network. This can include virtually any type of controller, communication module, computer, input / output (I / O) device, sensor, actuator, and human-machine interface (HMI) communicating via a network, including control networks, automation networks, and / or public networks. PLCs or automation controllers can also communicate with and control a variety of other devices, such as standard or safety-grade I / O modules (e.g., analog modules, digital modules, programmable / intelligent I / O modules), other programmable controllers, communication modules, sensors, actuators, output devices, etc.

[0117] Networks can include public networks such as the Internet and intranets, as well as automation networks such as Control and Information Protocol (CIP) networks, including device networks, control networks, security networks, and Ethernet / IP. Other networks include Ethernet, DH / DH+, remote I / O, fieldbus, Modbus, process fieldbus, CAN, wireless networks, serial protocols, etc. Additionally, network devices can include a wide range of possibilities (hardware and / or software components). These include components such as switches with Virtual Local Area Network (VLAN) capabilities, LANs, WANs, agents, gateways, routers, firewalls, Virtual Private Network (VPN) devices, servers, clients, computers, configuration tools, monitoring tools, and / or other devices.

[0118] In order to provide context for the various aspects of the disclosed topic, Figure 11 and Figure 12 The following discussion is intended to provide a brief, general description of suitable environments in which the various aspects of the disclosed subject matter can be implemented. Although the various embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the various embodiments may also be implemented in combination with other program modules and / or implemented as a combination of hardware and software.

[0119] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the method of the present invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each of which can be operatively coupled to one or more associated devices.

[0120] The implementations shown herein can also be practiced in a distributed computing environment, where certain tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0121] Computing devices typically include various media, which can include computer-readable storage media, machine-readable storage media, and / or communication media, these terms being used herein to distinguish themselves from each other. A computer-readable storage medium or a machine-readable storage medium can be any available storage medium accessible by a computer, and includes both volatile and non-volatile media, and both removable and non-removable media. By way of example and not limitation, a computer-readable storage medium or a machine-readable storage medium can be implemented in conjunction with any method or technique for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0122] Computer-readable storage media may include, but are not limited to: random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc (BD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” as used herein for storage devices, memories, or computer-readable media should be understood as modifiers that exclude only the propagation of transient signals themselves, and do not waive the rights to all standard storage devices, memories, or computer-readable media that do not merely propagate transient signals themselves.

[0123] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example via access requests, queries or other data retrieval protocols, to perform various operations on the information stored on the media.

[0124] Communication media typically represent computer-readable instructions, data structures, program modules, or other structured or unstructured data as data signals (such as modulated data signals, carrier waves, or other transmission mechanisms), and include any information transmission or delivery medium. The term "modulated data signal" or signal refers to a signal that sets or alters one or more characteristics of its properties in a manner that encodes information in one or more signals. By way of example and not limitation, communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0125] Refer again Figure 11 Example environment 1100 for implementing various embodiments of the aspects described herein includes a computer 1102, which includes a processing unit 1104, system memory 1006, and a system bus 1108. The system bus 1108 couples system components, including but not limited to system memory 1106, to the processing unit 1104. The processing unit 1104 can be any processor from a variety of commercially available processors. Dual-microprocessor and other multiprocessor architectures may also be used as the processing unit 1104.

[0126] System bus 1108 can be any of several types of bus architectures, and can also use any of a variety of commercially available bus architectures to interconnect to memory buses (with or without memory controllers), peripheral buses, and local buses. System memory 1106 includes ROM 1110 and RAM 1112. The Basic Input / Output System (BIOS) can be stored in non-volatile memory such as ROM, erasable programmable read-only memory (EPROM), or EEPROM, wherein the BIOS contains basic routines that facilitate the transfer of information between elements within computer 1102, for example, during startup. RAM 1112 may also include high-speed RAM such as static RAM for caching data.

[0127] Computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA), one or more external storage devices 1116 (e.g., floppy disk drive (FDD) 1116, memory stick or flash drive reader, memory card reader, etc.), and an optical disc drive 1120 (e.g., capable of reading from or writing to CD-ROMs, DVDs, BDs, etc.). Although the internal HDD 1114 is shown as being located within computer 1102, the internal HDD 1114 can also be configured for external use in a suitable rack (not shown). Additionally, although not shown in environment 1100, a solid-state drive (SSD) may be used in addition to HDD 1114, or an SSD may be used in place of HDD 1114. HDD 1114, external storage device 1116, and optical disc drive 1120 can be connected to system bus 1108 via HDD interface 1124, external storage interface 1126, and optical drive interface 1128, respectively. Interface 1124 for the external drive implementation may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are considered in the embodiments described herein.

[0128] The drive and its associated computer-readable storage medium provide non-volatile storage of data, data structures, computer-executable instructions, etc. For computer 1102, the drive and storage medium are adapted to store any data in a suitable digital format. Although the above description of computer-readable storage media refers to a corresponding type of storage device, those skilled in the art will understand that other types of computer-readable storage media may also be used in the example operating environment, whether such storage media are currently existing or to be developed in the future, and furthermore, any such storage medium may contain computer-executable instructions for performing the methods described herein.

[0129] Multiple program modules can be stored in the drive and RAM 1112, including the operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or part of the operating system, applications, modules, and / or data can also be cached in RAM 1112. The systems and methods described herein can be implemented using a variety of commercially available operating systems or combinations of operating systems.

[0130] Computer 1102 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate a hardware environment for operating system 1130, and the emulated hardware may optionally be compatible with... Figure 12The hardware shown is different. In such an implementation, the operating system 1130 may include one of a plurality of virtual machines (VMs) hosted at the computer 1102. Furthermore, the operating system 1130 may provide a runtime environment for the application 1132, such as the Java Runtime Environment or the .NET Framework. A runtime environment is a consistent execution environment that enables the application 1132 to run on any operating system that includes that runtime environment. Similarly, the operating system 1130 may support containers, and the application 1132 may be in the form of a container, which is a lightweight, stand-alone, executable software package that includes, for example, the application's code, runtime, system tools, system libraries, and setup.

[0131] Furthermore, computer 1102 can be equipped with security modules such as Trusted Processing Modules (TPMs). For example, using a TPM, the boot component hashes the boot component that is immediately following it in time and waits for the result to match a security value before loading the next boot component. This process can occur at any layer of the computer 1102's code execution stack, for example, it can be applied at the application execution level or the operating system (OS) kernel level, thereby achieving security at any code execution level.

[0132] Users can input commands and information into computer 1102 through one or more wired / wireless input devices such as keyboard 1138, touchscreen 1140, and pointing devices such as mouse 1118. Other input devices (not shown) may include microphones, infrared (IR) remote controls, radio frequency (RF) remote controls, or other remote controls, joysticks, virtual reality controllers and / or virtual reality headsets, gaming pads, styluses, image input devices (e.g., camera devices), gesture sensor input devices, visual motion sensor input devices, emotion or face detection devices, biometric input devices (e.g., fingerprint or iris scanners), etc. These and other input devices are typically connected to processing unit 1104 via input device interface 1144, which can be coupled to system bus 1108, but may also be connected via other interfaces such as parallel ports, IEEE 1394 serial ports, gaming ports, USB ports, IR interfaces, BLUETOOTH® interfaces, etc.

[0133] Monitor 1144 or other types of display devices may also be connected to system bus 1108 via an interface such as video adapter 1146. In addition to monitor 1144, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

[0134] Computer 1102 can operate in a networked environment using logical connections via wired and / or wireless communications to one or more remote computers, such as remote computer 1148. Remote computer 1148 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer-to-peer device, or other general-purpose network node, and typically includes many or all of the elements described with respect to computer 1102, although only memory / storage device 1150 is shown for simplicity. The depicted logical connections include wired / wireless connections to a local area network (LAN) 1152 and / or a larger network (e.g., a wide area network (WAN) 1154). Such LAN and WAN network environments are common in offices and companies and facilitate enterprise-wide computer networks such as intranets, all of which can be connected to global communication networks such as the Internet.

[0135] When used in a LAN network environment, computer 1102 can connect to local area network 1152 via a wired and / or wireless communication network interface or adapter 1156. Adapter 1156 can facilitate wired or wireless communication to LAN 1152, and LAN 1152 may also include a wireless access point (AP) disposed thereon for communication with adapter 1156 in wireless mode.

[0136] When used in a WAN network environment, computer 1102 may include modem 1158, or may be connected to a communication server on WAN 1154 via other means for establishing communication over WAN 1154, such as via the Internet. Modem 1158, which may be internal or external and may be a wired or wireless device, may be connected to system bus 1108 via input device interface 1142. In a networked environment, program modules described with respect to computer 1102 or parts thereof may be stored in remote memory / storage device 1150. It should be understood that the network connection shown is an example, and other means of establishing communication links between computers may be used.

[0137] When used in a LAN or WAN network environment, in addition to the external storage device 1116 described above, computer 1102 can also access cloud storage systems or other network-based storage systems, or alternatively, instead of the external storage device 1116 described above, computer 1102 can access cloud storage systems or other network-based storage systems. Typically, the connection between computer 1102 and the cloud storage system can be established, for example, via adapter 1156 or modem 1158 through LAN 1152 or WAN 1154. When computer 1102 is connected to the associated cloud storage system, external storage interface 1126 can manage the storage provided by the cloud storage system, as if it were any other type of external storage, with the help of adapter 1156 and / or modem 1158. For example, external storage interface 1126 can be configured to provide access to cloud storage sources as if these sources were physically connected to computer 1102.

[0138] Computer 1102 is operable to communicate with any wireless device or entity operatively arranged in wireless communication, such as a printer, scanner, desktop and / or portable computer, portable data assistant, communication satellite, any equipment or location associated with a wirelessly detectable tag (e.g., kiosk, newsstand, shop shelf, etc.), and telephone. This may include Wi-Fi and BLUETOOTH® wireless technologies. Therefore, communication can be a predefined structure like a conventional network or simply self-organizing communication between at least two devices.

[0139] Figure 12This is a schematic block diagram of a sample computing environment 1200 with which the disclosed subject matter can interact. The sample computing environment 1200 includes one or more clients 1202. Clients 1202 can be hardware and / or software (e.g., threads, processes, computing devices). The sample computing environment 1200 also includes one or more servers 1204. Servers 1204 can also be hardware and / or software (e.g., threads, processes, computing devices). Servers 1204 can accommodate threads to perform transformations by employing one or more implementations, such as those described herein. One possible communication between clients 1202 and servers 1204 can be in the form of data packets suitable for transmission between two or more computer processes. The sample computing environment 1200 includes a communication framework 1206 that can be used to facilitate communication between clients 1202 and servers 1204. Clients 1202 are operatively connected to one or more client data storage devices 1208 that can be used to store local information of clients 1202. Similarly, server 1204 is operatively connected to one or more server data storage devices 1210 that can be used to store local information of server 1204.

[0140] The foregoing description includes examples of the invention. It is certainly impossible to describe every conceivable combination of components or methods in order to describe the disclosed subject matter, but those skilled in the art will recognize that many other combinations and arrangements of the invention are possible. Therefore, the disclosed subject matter is intended to cover all such changes, modifications, and variations that fall within the spirit and scope of the appended claims.

[0141] In particular, with respect to the various functions performed by the components, devices, circuits, systems, etc., described above, unless otherwise indicated, the terminology used to describe such components (including references to "means") is intended to correspond to any component that performs the specified function of the described component (e.g., any functionally equivalent component), even if it is not structurally equivalent to the disclosed structure, which performs the functions of the exemplary aspects of the disclosed subject matter shown herein. In this regard, it should also be recognized that the disclosed subject matter includes computer-readable media and systems having computer-executable instructions for actions and / or events of various methods for performing the disclosed subject matter.

[0142] Furthermore, while a particular feature of the disclosed subject matter may be disclosed only for one of several implementations, such features may be combined with one or more other features of other implementations, which may be desirable and advantageous for any given or particular application. Moreover, with regard to the use of the terms "includes" and "including" and their variations in the specification or claims, these terms are intended to be included in a manner similar to the term "comprising."

[0143] In this application, the word "exemplary" is used to indicate that it is used as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the word "exemplary" is intended to present the concept in a specific manner.

[0144] The various aspects or features described herein can be implemented as methods, apparatus, or articles of art using standard programming and / or engineering techniques. As used herein, the term "article of art" is intended to encompass any computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes...), optical discs [e.g., compact discs (CDs), digital versatile discs (DVDs)...], smart cards, and flash memory devices (e.g., cards, sticks, key drives...).

Claims

1. A system, comprising: A memory that stores executable components; And A processor operably coupled to the memory, the processor executing the executable components, the executable components including: A human-machine interface (HMI) deployment component configured to deploy an HMI application to an HMI terminal for execution, wherein the HMI application presents a display screen on the HMI terminal that includes graphical objects presenting data generated by an industrial automation system; A user interface component configured to receive natural language input describing a new display screen to be added to the HMI application via interaction with the HMI application; A generative artificial intelligence (AI) component configured to formulate a prompt for a generative AI model in response to receiving the natural language input, the prompt being designed to obtain a response from the generative AI model that includes information for the generative AI component to generate the new display screen, wherein the prompt is generated based on an analysis of the natural language input and one or more custom models trained with training data; and An HMI generation component configured to add the new display screen to the HMI application and present the new display screen on the HMI terminal.

2. The system according to claim 1, wherein, The generative AI component is configured to: in response to determining that the natural language input describes a request to display an interlock associated with the state of a machine of the industrial automation system, Identify the interlock associated with the state of the machine, Identify a data source indicating the current state of the interlock, and Generate the new display screen to include a graphical representation of the interlock linked to the data source.

3. The system according to claim 1, wherein, The generative AI component is configured to: in response to determining that the natural language input describes a request to display an active or historical alarm condition corresponding to at least one of a specified aspect or a specified time period of the industrial automation system, Identify the alarm corresponding to the request, Identify a data source indicating the current state of the alarm, and Generate the new display screen to include a graphical representation of the alarm linked to the data source.

4. The system according to claim 1, wherein, The generative AI component is configured to: in response to determining that the natural language input describes a request to view information available on a data source not currently accessed by the HMI application, establish a link between the HMI application and the data source and present information from the data source on the new display screen.

5. The system according to claim 4, wherein, The data source is a work order system, and The information includes work order information related to open or closed work orders stored in the work order system.

6. The system according to claim 1, wherein, The generative AI component is further configured to in response to detecting the occurrence of an alarm condition on the industrial automation system: Monitor and record a sequence of user interactions with the HMI application and the industrial automation system, In response to determining that the alarm condition has been cleared, a query is presented via the HMI application regarding whether the user interaction sequence will be added to the standard operating procedure of the industrial automation system; as well as In response to receiving a response to the query indicating that the user interaction sequence will be added to the standard operating procedure, a record of the user interaction sequence associated with the alarm status is created.

7. The system according to claim 6, wherein, The user interface component is also configured to, in response to the detection of a recurrence of the alarm condition, present information about the user interaction sequence on the HMI application based on the record.

8. The system according to claim 1, wherein, The training data includes at least one of the following: information defining industry standards, technical details for the corresponding type of industrial control application, knowledge of different industrial verticals, information describing industry best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for the corresponding type of control application, or customer-specific training data describing internal HMI design preferences.

9. The system according to claim 1, wherein, The generative AI component is further configured to: in response to receiving another natural language input requesting a summary of a portion of a data log generated and maintained by the HMI application via interaction with the HMI application, generate a natural language summary of the portion of the data log based on analysis of: the data log; the other natural language input; and at least one of a response prompted by the generative AI model or the content of one or more custom models, and The user interface component is also configured to present the natural language summary via the HMI application.

10. The system according to claim 1, wherein, The natural language input describes the information about the industrial automation system to be presented on the new display screen, as well as the display format or layout for the information. The generative AI component is configured to generate the new display screen according to the display format or layout to display information about the industrial automation system.

11. A method comprising: A human-machine interface (HMI) application is deployed to an HMI terminal for execution by a system including a processor, wherein the HMI application presents a display screen on the HMI terminal including graphical objects that present data generated by the industrial automation system; The system receives natural language input describing a new display screen to be added to the HMI application via interaction with the HMI application; In response to receiving the natural language input, the system formulates a prompt for a generative artificial intelligence (AI) model, the prompt being designed to derive a response from the generative AI model including information used by the system to generate the new display screen, wherein the prompt is generated based on analysis of the natural language input and one or more custom models trained with training data; and The system modifies the HMI application to add the new display screen to the HMI application.

12. The method according to claim 11, wherein, The receiving includes: receiving a request, as input in the natural language, to display an interlock associated with the state of the machine in the industrial automation system, and The method further includes: responding to receiving the request: The system identifies the interlocks associated with the state of the machine. The system identifies the data source indicating the corresponding current state of the interlock, and The system generates the new display screen to include a graphical representation of the interlock linked to the data source.

13. The method according to claim 11, wherein, The receiving includes: receiving a request, as input in natural language, to display an activity or historical alarm status corresponding to at least one specified aspect or time period of the industrial automation system, and The method further includes: responding to receiving the request. The system identifies the alarm identified by the request. The system identifies the data source that indicates the corresponding current state of the alarm, and The system generates the new display screen to include a graphical representation of the alarm linked to the data source.

14. The method of claim 11, further comprising: In response to the determination that the natural language input describes a request to view information available on a data source not currently accessed by the HMI application, the system establishes a link between the HMI application and the data source, and presents the information from the data source on the new display screen.

15. The method according to claim 14, wherein, The data source is a work order system, and The information includes work order information related to open or closed work orders stored in the work order system.

16. The method of claim 11, further comprising: In response to the detection of an alarm condition on the industrial automation system: The system monitors and records the sequence of user interactions with the HMI application and the industrial automation system until the alarm condition is cleared. In response to the detection that the alarm status has been cleared, the system presents a query via the HMI application regarding whether the user interaction sequence will be added to the standard operating procedures of the industrial automation system; as well as In response to receiving a query indicating that the user interaction sequence will be added to the standard operating procedure, the system creates a record of the user interaction sequence associated with the alarm status.

17. The method of claim 16, further comprising: In response to the detection of a recurrence of the alarm condition, the system presents information about the user interaction sequence on the HMI application based on the records.

18. The method according to claim 11, wherein, The training data includes at least one of the following: information defining industry standards, technical details for the corresponding type of industrial control application, knowledge of different industrial verticals, information describing industry best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for the corresponding type of control application, or customer-specific training data describing internal HMI design preferences.

19. A non-transitory computer-readable medium having instructions stored thereon, the instructions being responsive to execution to cause a human-machine interface (HMI) development system including a processor to perform operations, the operations including: Deploying a human-machine interface (HMI) application to an HMI terminal for execution, wherein the HMI application presents a display screen on the HMI terminal including graphical objects that present data generated by the industrial automation system; Receive natural language input describing a new display screen to be added to the HMI application through interaction with the HMI application; In response to receiving the natural language input, a prompt is formulated for a generative artificial intelligence (AI) model, the prompt being designed to derive a response from the generative AI model including information used by the system to generate the new display screen, wherein the prompt is generated based on analysis of the natural language input and one or more custom models trained with training data; and Modify the HMI application to add the new display screen to the HMI application.

20. The non-transitory computer-readable medium of claim 19, further comprising: In response to the determination that the natural language input describes a request to view information available on a data source not currently accessed by the HMI application, the system establishes a link between the HMI application and the data source, and presents the information from the data source on the new display screen.

Images