Generative ai industry automation event-based cue engineering

Abstract

Generative AI industrial automation event-based prompt engineering is provided. A human-machine interface (HMI) development system utilizes a generative AI model to assist in the development of an HMI project based on specified functional requirements that can be provided to the development system as intuitive natural language speech or written text. The system can formulate and implement HMI project edits during design time based on analysis of the natural language design input. The system can also receive and process natural language requests to modify a runtime HMI according to described modification criteria after the HMI project has been deployed as a runtime application.

Description

Technical Field

[0001] The topics disclosed in this article generally relate to industrial automation systems, and for example, to the development and deployment of industrial human-machine interfaces (HMIs). Background Technology

[0002] Industrial human-machine interfaces (HMIs) include a computer terminal with display capabilities that executes an HMI runtime application. This application defines the display screens presented to the operator of the industrial automation system, navigation structures for navigating between screens, and data links or bindings between graphical elements and corresponding data labels in the controller's data tables. HMI developers typically use HMI development platforms to design these aspects of the HMI. These platforms support graphical and menu-driven development workflows, where developers select graphical display and control elements from a library to include on each display interface, and manipulate these selected elements on a prototype of the display interface, for example, via drag-and-drop interaction, to produce the desired layout. Summary of the Invention

[0003] The following is a simplified overview to provide a basic understanding of some of the aspects described in this paper. This overview is neither a comprehensive review nor intended to identify key / important elements or define the scope of the various aspects described herein. Its sole purpose is to present some ideas in a simplified form as a prelude to the more detailed descriptions that follow.

[0004] In one or more embodiments, a system is provided, comprising: a human-machine interface (HMI) deployment component configured to deploy an HMI application to an HMI terminal for execution, wherein the HMI application presents a display screen on the HMI terminal including graphical objects representing data generated by an industrial automation system; a user interface component configured to receive natural language input describing an event-driven action to be performed by the HMI application via interaction with the HMI application; a generative artificial intelligence (AI) component configured to: in response to receiving the natural language input, formulate a prompt for a generative AI model and formulate an edit configuring the HMI application to perform the event-driven action, the prompt being designed to obtain a response including information from the generative AI model, the information being used by the generative AI component to determine the event-driven action based on the natural language input, wherein the prompt is generated based on analysis of the natural language input and one or more custom models trained with training data; and an HMI generation component configured to apply the edit to the HMI application.

[0005] Furthermore, one or more embodiments provide a method comprising: deploying a human-machine interface (HMI) application to an HMI terminal for execution by a system including a processor, wherein the HMI application presents a display screen on the HMI terminal including graphical objects representing data generated by an industrial automation system; receiving natural language input describing an event-driven action to be performed by the HMI application via interaction with the HMI application; in response to receiving the natural language input, formulating a prompt for a generative artificial intelligence (AI) model, the prompt being designed to obtain a response from the generative AI model including information used by the system to determine the event-driven action, wherein the prompt is generated based on analysis of the natural language input and one or more custom models trained with training data; formulating an edit by the system to configure the HMI application to perform the event-driven action, wherein the formulation includes formulating the edit based on the natural language input, one or more custom models, and the response from the generative AI model; and applying the edit to the HMI application by the system.

[0006] Furthermore, according to one or more embodiments, a non-transitory computer-readable medium having instructions stored thereon, the instructions being responsive to execution to cause a system to perform operations, the operations including: deploying a human-machine interface (HMI) application to an HMI terminal for execution, wherein the HMI application presents a display screen on the HMI terminal including graphical objects representing data generated by an industrial automation system; receiving natural language input describing an event-driven action to be performed by the HMI application via interaction with the HMI application; formulating a prompt for a generative artificial intelligence (AI) model in response to receiving the natural language input, the prompt being designed to obtain a response from the generative AI model including information used by the system to determine the event-driven action, wherein the prompt is generated based on analysis of the natural language input and one or more custom models trained with training data; and configuring the HMI application to perform the event-driven action based on the natural language input, one or more custom models, and the response from the generative AI model.

[0007] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various modes that can be practiced, all of which are intended to be covered herein. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a block diagram of an example industrial control environment.

[0009] Figure 2 It is a diagram of a general architecture that includes industrial controllers and human-machine interfaces (HMIs).

[0010] Figure 3 This is a block diagram of an example HMI development system.

[0011] Figure 4 This is a diagram illustrating a sample data flow associated with creating an HMI project using an HMI development system.

[0012] Figure 5 This is a graph showing the training of a custom model.

[0013] Figure 6 This diagram illustrates the commissioning of an HMI application to an HMI terminal.

[0014] Figure 7 This diagram illustrates how a HMI application can be dynamically edited using natural language prompts during runtime when developing an HMI system.

[0015] Figure 8a This is a flowchart of the first part of an example method for configuring an HMI runtime application to perform event-driven actions using natural language prompts.

[0016] Figure 8b This is a flowchart of the second part of an example method for configuring an HMI runtime application to perform event-driven actions using natural language prompts.

[0017] Figure 9 This is an example computing environment.

[0018] Figure 10 This is an example of a networked environment. Detailed Implementation

[0019] This disclosure will now be described with reference to the accompanying drawings, wherein similar reference numerals are used to refer to similar elements. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding thereof. However, it may be apparent that this disclosure can be practiced without these specific details. In other instances, well-known structures and apparatuses are shown in block diagram form to facilitate their description.

[0020] As used herein, the terms “component,” “system,” “platform,” “layer,” “controller,” “terminal,” “station,” “node,” and “interface” are intended to refer to a computer-related entity or an entity associated with or part of an operating device having one or more specific functions, wherein such an entity may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to: a process running on a processor; a processor; a hard disk drive; multiple storage drives (optical or magnetic storage media) including fixed (e.g., secured with screws or bolts) or removable fixed solid-state storage drives; an object; an executable file; an executing thread; a computer-executable program, and / or a computer. By way of illustration, both an application running on a server and the server itself can be components. One or more components may reside within a process and / or an executing thread, and components may reside on one computer and / or be distributed among two or more computers. Furthermore, components as described herein may be executed from various computer-readable storage media on which various data structures are stored. Components can communicate, for example, via local and / or remote processing based on signals having one or more data packets (e.g., data from a component interacting with another component in a local system, a distributed system, and / or interacting with other systems via signals through a network such as the Internet). As another example, a component can be a device having specific functions provided by mechanical components operated by electrical or electronic circuitry, which is operated by software or firmware applications executed by a processor, wherein the processor may be internal or external to the device and executes at least a portion of the software or firmware application. As yet another example, a component can be a device providing specific functions through electronic components without mechanical parts, the electronic components including a processor to execute software or firmware that at least partially provides the functions of the electronic components. As yet another example, an interface can include input / output (I / O) components and associated processors, applications, or application programming interface (API) components. While the foregoing examples relate to aspects of components, the illustrated aspects or features also apply to systems, platforms, interfaces, layers, controllers, terminals, etc.

[0021] As used herein, the terms “inference” and “inference” generally refer to the process of reasoning or inferring the state of a system, environment, and / or user based on a set of observations captured via events and / or data. For example, inference can be used to identify specific contexts or actions, or it can generate probability distributions of states. Inference can be probabilistic—that is, calculating the probability distribution of states of interest based on considerations of data and events. Inference can also refer to techniques used to construct higher-level events from a set of events and / or data. Such inference enables the construction of new events or actions from a set of observed events and / or stored event data, regardless of whether the events are closely related in time or whether the events and data originate from one or more event and data sources.

[0022] Furthermore, the term "or" is intended to mean inclusive "or" rather than exclusive "or". That is, unless otherwise stated or clearly indicated from the context, the phrase "X adopts A or B" is intended to mean any natural inclusive arrangement. That is, any of the following instances satisfy the phrase "X adopts A or B": X adopts A; X adopts B; or X adopts both A and B. Additionally, unless otherwise stated or clearly indicated from the context, the articles "a" and "an" used in this application and the appended claims should generally be interpreted as meaning "one or more".

[0023] Furthermore, as used herein, the term "set" does not include an empty set; for example, a set containing no elements. Therefore, "set" as used in this disclosure includes one or more elements or entities. For illustration, a set of controllers includes one or more controllers; a set of data resources includes one or more data resources, and so on. Similarly, as used herein, the term "group" refers to a collection of one or more entities; for example, a group of nodes refers to one or more nodes.

[0024] Various aspects or features will be presented according to the system, which may include multiple devices, components, modules, etc. It should be understood and recognized that various systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc., discussed in conjunction with the accompanying drawings. Combinations of these methods may also be used.

[0025] Figure 1This is a block diagram of an example industrial control environment 100. In this example, multiple industrial controllers 118 are deployed throughout the industrial plant environment to monitor and control corresponding industrial systems or processes related to product manufacturing, machining, motion control, batch processing, material handling, or other such industrial functions. The industrial controllers 118 typically execute corresponding control programs to monitor and control industrial units 120 that constitute controlled industrial assets or systems (e.g., industrial machines). One or more industrial controllers 118 may also include software controllers executing on a personal computer or other hardware platform or on a cloud platform. Some hybrid devices may also combine controller functionality with other functions (e.g., visualization). The control programs executed by the industrial controllers 118 may include any conceivable type of code for processing input signals read from the industrial unit 120 and controlling output signals generated by the industrial controllers, including but not limited to ladder logic, sequential function charts, function block diagrams, or structured text.

[0026] Industrial device 120 may include both input devices that provide data related to the controlled industrial system to industrial controller 118 and output devices that respond to control signals generated by industrial controller 118 to control the industrial system. Example input devices may include telemetry devices (e.g., temperature sensors, flow meters, level sensors, pressure sensors, etc.), manual operator control devices (e.g., pushbuttons, selector switches, etc.), safety monitoring devices (e.g., safety mats, safety ropes, light curtains, etc.), and other such devices. Output devices may include motor drivers, pneumatic actuators, signaling devices, robot control inputs, valves, etc. Some industrial devices, such as industrial device 120... M It can operate autonomously on the factory network 116 without being controlled by the industrial controller 118.

[0027] Industrial controller 118 can communicatively interface with industrial device 120 via hardwired or network connections. For example, industrial controller 118 may be equipped with native hardwired inputs and outputs that communicate with industrial device 120 to control the device. Native controller I / O may include digital I / O that sends discrete voltage signals to and receives discrete voltage signals from field devices, or analog I / O that sends analog voltage or current signals to and receives analog voltage or current signals from devices. Controller I / O may communicate with the controller's processor via a backplane, allowing digital and analog signals to be read into and controlled by the control program. Industrial controller 118 may also communicate with industrial device 120 via factory network 116 using, for example, communication modules or integrated network ports. Exemplary networks may include the Internet, intranet, Ethernet, DeviceNet, ControlNet, data highways and data highway enhancements (DH / DH+), remote I / O, fieldbus, Modbus, PROFIBUS, wireless networks, serial protocols, etc. The industrial controller 118 may also store persistent data values ​​that can be referenced by the control program and used for control decisions, including but not limited to measured or calculated values ​​representing the operating status of the controlled machine or process (e.g., tank level, position, alarms, etc.) or captured time-series data collected during the operation of the automated system (e.g., status information at multiple time points, diagnostic events, etc.). Similarly, some intelligent devices—including but not limited to motor drives, instruments, or condition monitoring modules—may store data values ​​used for control and / or visualization of operating status. Such devices may also capture time-series data or events in a log for later retrieval and viewing.

[0028] Industrial automation systems typically include one or more personal machine interface (HMI) terminals 114, enabling factory personnel to view telemetry and status data associated with the automation system and to control aspects of system operation. The HMI terminal 114 can communicate with one or more industrial controllers 118 via a factory network 116 and exchange data with them to facilitate visualization of information related to the controlled industrial process on one or more pre-developed operator interface screens. The HMI terminal 114 can also be configured to allow operators to submit data to designated data tags or memory addresses on the industrial controller 118, providing operators with a means to issue commands to the controlled system (e.g., cycle start commands, device actuation commands, etc.), modify setpoints, etc. The HMI terminal 114 executes an HMI runtime application that generates one or more display screens through which operators interact with the industrial controller 118, and thus with the controlled process and system. Example display screens can use graphical representations of processes displaying measured or calculated values ​​to visualize the current status of an industrial system or its associated devices, employing color or positional animations based on the status, presenting alarm notifications, or other such techniques for presenting relevant data to operators. Data presented in this manner is read from the industrial controller 118 by the HMI terminal 114 and displayed on one or more display screens according to a display format selected by the HMI developer. The HMI terminal 114 may include a fixed-position or mobile device with a user-installed or pre-installed operating system and user-installed or pre-installed graphical application software.

[0029] Some industrial environments may also include other systems or devices related to specific aspects of the controlled industrial system. These may include, for example, one or more data history databases 110 that aggregate and store production information collected from industrial controller 118 and other industrial devices.

[0030] Industrial units 120, industrial controllers 118, HMI terminals 114, associated controlled industrial assets, and other plant floor systems (such as data history databases 110, vision systems, and other such systems) operate at the operational technology (OT) level within the industrial environment. Higher-level analytics and reporting systems can operate at a higher enterprise level within the information technology (IT) domain of the industrial environment; for example, on an office network 108 or a cloud platform 122. These higher-level systems may include, for example, an Enterprise Resource Planning (ERP) system 104 that integrates and collectively manages advanced business operations such as finance, sales, order management, marketing, human resources, or other business functions. A Manufacturing Execution System (MES) 102 can monitor and manage control operations at the control level based on higher-level business considerations, thereby driving those control-level operations toward results that meet defined business objectives (e.g., order fulfillment, resource tracking and management, asset utilization tracking, etc.). A reporting system 106 can collect operational data from industrial units on the plant floor and generate daily or shift reports summarizing operational statistics for controlled industrial assets.

[0031] Figure 2 This is a diagram of a general architecture including industrial controller 118 and HMI 114. Industrial facilities may include one or more controlled processes 2101 to 210 associated with product manufacturing, machining, motion control, batch processing, material handling, or other such industrial functions. N As described above, processes 2101 to 210 are executed. N Devices and machines (e.g., Figure 1 The device 120 and its associated machines can be monitored and controlled by an industrial controller 118, which executes a control program 204 to facilitate controlled processing 2101 to 210. N The monitoring and control of the controller 118. The control program 204 can be virtually any type of code used to process input signals read into the controller 118 and control output signals from the controller 118, including but not limited to ladder logic programming, sequential function charts, function block diagrams, or structured text. Data read into or generated by the controller 118 can be stored in data table 206 within the controller's memory.

[0032] Controller 118 can be connected to controlled processors 2101 to 210 via factory network 116 or via another hardwired or network connection. NThe controller exchanges data between its input and output devices. For example, the controller 118 may be equipped with native hardwired input and output points that exchange digital and analog signals with field devices to control the devices. Native controller I / O may include digital I / O that sends discrete voltage signals to and receives discrete voltage signals from field devices, or analog I / O that sends analog voltage or current signals to and receives analog voltage or current signals from devices. The controller 118 converts input signals from the controlled process 210 into digital and analog data values ​​stored in the controller's data table 206. The control program 204 processes these input data values ​​according to a user-defined control algorithm and sets the values ​​of the controller's digital and analog output signals based on this processing. The values ​​of the output signals, as well as any other values ​​calculated by the control program 204, are stored in the data table 206.

[0033] HMI 114 uses data stored in data table 206 of the controller to connect with controlled processes 2101 to 210 N The relevant information is visualized as graphics and alphanumeric data. For this purpose, HMI 114 communicates with controller 118 via factory network 116 or via a direct connection, and reads and writes data to data table 206 through this connection. HMI 114 renders controlled processes 2101 to 210. N A navigable interface display screen showing the current operation or status information. In some embodiments, the display screen may present controlled processes 2101 to 210 being performed. N The machine is a graphical representation, and these graphical representations can be animated based on the current state of the corresponding machine, as determined by data values ​​contained in the controller's data table 206. These animations may include, for example, setting the color of graphical elements based on the state of the corresponding machine component, changing the height of the filling graphic based on the corresponding filling level of the tank, setting the position or orientation of graphical elements based on the corresponding position or orientation of the machine component, displaying alphanumeric text conveying measurement values ​​(e.g., temperature, pressure, flow rate, etc.), or other such animations.

[0034] The operator can also interact with the HMI display screen to send changes to controlled processes 2101 to 210 to the controller 118. NThe commands issued by the HMI 114 may include, for example, changing control setpoints, initiating start or stop commands, changing the operating mode of the machine or process, clearing alarm messages presented by the HMI 114, or other such commands. To provide a means of issuing these commands, the display screen may include interactive graphical controls—such as graphical buttons, data input fields, or other such controls—linked to corresponding data tags defined in datasheet 206. Through interaction with these controls, the operator can write digital or analog values ​​to these data tags, and these values ​​are controlled by the control program 204 in conjunction with the control of industrial processes 2101 to 210. N To handle it.

[0035] Typically, HMI 114 includes a display-capable computer terminal that executes HMI runtime application 202. HMI runtime application 202 defines the display screens presented to the operator (including the definitions of graphical elements and controls included on each display screen and the arrangement of these elements), navigation structures for navigating between display screens, and data links or bindings between graphical elements and corresponding data labels in the controller's data table 206. HMI developers typically use an HMI development platform to design these aspects of the HMI, which compiles the design into the HMI runtime application 202 that can be downloaded to and executed on the HMI terminal. These HMI development platforms typically support graphical and menu-driven development workflows, where developers select graphical display and control elements from a library to include on each display screen, and manipulate these selected elements on a prototype of the display screen, for example, via drag-and-drop interaction, to produce the desired layout. For elements whose appearance or behavior depends on the values ​​of data labels defined in the controller's data table 206, developers typically define bindings to appropriate data labels by invoking the element's properties window and specifying the data label in the appropriate property field of the window. Similarly, for elements designed to write data to controller 118—such as graphical buttons and data input fields—users typically configure these elements to write their data to data labels via interaction with the element's properties window. This graphical development approach can also be cumbersome and time-consuming.

[0036] As an alternative to this graphical HMI development method, one or more embodiments described herein provide an HMI development system that supports the use of generative artificial intelligence (AI) to assist in the development and use of industrial HMI applications. In one or more embodiments, the HMI development system may support industry-specific cue engineering services that use natural language cueing describing the functional and visual requirements of the HMI to assist users in developing industrial visualization projects. To this end, the HMI development system can utilize generative AI models and associated neural networks to generate parts of the HMI project—including display screen content and layout, screen navigation structure, links between animated graphics and data sources such as controller data labels, alarm definitions, color settings, and other such aspects—based on functional requirements provided to the HMI development system as intuitive natural language input (e.g., spoken or written natural language text). The HMI development system may include a dedicated cue engineering layer and associated custom models (trained using training data such as knowledge of various types of industrial control applications, knowledge of specific types of industrial assets, vertical-domain specific industry standards and best practices, sample HMI layouts, and other such training data) that generate cueing or meta-cueing based on the user's natural language input to be submitted to the generative AI model, such as a Large Language Model (LLM).

[0037] Figure 3 This is a block diagram of an example HMI development system 302 according to one or more embodiments of this disclosure. Aspects of the systems, apparatus, or processes described in this disclosure can be embodied within one or more machines, for example, in one or more computer-readable media (or media) associated with one or more machines. Such components, when executed by one or more machines, such as one or more computers, one or more computing devices, one or more automation devices, one or more virtual machines, etc., can cause one or more machines to perform the described operations.

[0038] HMI development system 302 may include a user interface component 304, an HMI generation component 306, an HMI deployment component 308, a generative AI component 310, a training component 312, one or more processors 318, and memory 320. In various embodiments, one or more of the user interface component 304, HMI generation component 306, HMI deployment component 308, generative AI component 310, training component 312, one or more processors 318, and memory 320 may be electrically coupled and / or communicatively coupled to each other to perform one or more functions of the HMI development system 302. In some embodiments, components 304, 306, 308, 310, and 312 may include software instructions stored on memory 320 and executed by one or more processors 318. HMI development system 302 may also be compatible with… Figure 3 It may interact with other hardware and / or software components not described herein. For example, processor 318 may interact with one or more external user interface devices such as a keyboard, mouse, display monitor, touchscreen, or other such interface devices.

[0039] User interface component 304 can be configured to receive user input and present output to the user in any suitable format (e.g., visual, audio, haptic, etc.). In some embodiments, user interface component 304 can be configured to generate interface displays and provide them to client devices (e.g., laptops, tablets, smartphones, etc.) and exchange data via these interface displays. Input data that can be received via various embodiments of user interface component 304 may include, but is not limited to, natural language chat input or prompts, interactions with the HMI development interface (e.g., selection input, drag-and-drop input, alphanumeric input, etc.) or other such inputs. Output data presented by various embodiments of user interface component 304 may include natural language responses to chat input or prompts, HMI displays in both design mode and runtime mode, answers to user-submitted questions or requests, HMI development suggestions, or other such outputs.

[0040] HMI generation component 306 can be configured to create HMI projects based on design input (including natural language requests or functional descriptions) received via user interface component 304. HMI deployment component 308 can be configured to debug the HMI application created by HMI generation component 306 onto an appropriate execution platform, such as an HMI terminal.

[0041] Generative AI component 310 can be configured to use generative AI to assist HMI generation component 306 in generating or analyzing portions of the HMI project—including generating, formatting, and configuring the HMI display screen. To this end, generative AI component 310 can use associated custom models 322 trained with domain-specific industrial training data to implement prompting engineering functions. Generative AI component 310 can generate prompts or meta-prompts and submit them to one or more generative AI models and associated neural networks, where these prompts are generated based on natural language requests or queries submitted by the designer and domain-specific information contained in the custom models 322. Depending on the nature of the designer's request or query, the response returned by the generative AI model in response to the prompt can be used by HMI generation component 306 or user interface component 304 to generate portions of the HMI project, presenting answers to questions from the designer regarding a portion of the HMI project or regarding the design platform itself. Training component 312 can be configured to train one or more custom models 322 with various types of relevant training data. The system 302 uses these custom models 322 by performing the following operations: generating and customizing parts of the HMI application, and generating appropriate hints for the generative AI model as needed to assist in the development of the HMI.

[0042] One or more processors 318 may perform one or more of the functions described in the systems and / or methods disclosed herein. Memory 320 may be a computer-readable storage medium storing computer-executable instructions and / or information for performing the functions described in the systems and / or methods disclosed herein.

[0043] Figure 4 This is a diagram illustrating example data flows associated with creating an HMI project 402 using the HMI development system 302, according to one or more implementations. Some implementations of the HMI development system 302 can be implemented on a cloud platform, enabling access for multiple industrial customers with authorized access to services using the HMI development system. Cloud-based implementations can also facilitate collaborative project development, whereby multiple developers contribute design and programming input to a public HMI project 402. Alternatively, some implementations of the HMI development system 302 can be executed at least partially on local client devices, while accessing remote services and repositories as needed.

[0044] A client device 404 (e.g., a laptop, tablet, desktop, mobile device, wearable AR / VR device, etc.) owned by a user with appropriate authentication credentials can access the project development tools of the HMI development system and utilize these tools to create HMI projects 402—including display screen definitions, layouts of graphical elements on the display screen, screen navigation structures, animation definitions of graphical elements, links to data labels for their value-driven animations, or other such aspects of HMI project 402—for visualizing the status and operational information of the industrial automation system. Through interaction with a development interface generated by the system's user interface component 304 and delivered to the client device 404, developers can submit design inputs 412 to the HMI development system 302 in various supported formats. Design inputs 412 can include explicit HMI design inputs, such as menu-driven screen creation, selection and placement of graphical elements (e.g., via drag-and-drop interaction with the development interface), manual definition of animation attributes and data links, and other such inputs 412.

[0045] Additionally, the development services of the HMI system may include a chat-based interface and associated natural language processing services that utilize generative AI to assist users in creating, editing, or customizing HMI projects 402 for industrial applications, as well as searching for answers to specific questions related to HMI project 402 or the HMI development platform itself. To this end, the HMI development system 302 may include a generative AI component 310 that responds to natural language prompts submitted by the user as part of design input 412. These prompts may include, for example, desired visual layouts, screen navigation definitions, graphic content to be included on the display screen and the screen location where the content should reside, alarm requests, data sources or labels for controlling the animation properties of selected graphic objects, questions about development tools supported by the HMI development system 302, natural language descriptions of requests for HMI design recommendations, or other such prompts. Depending on the nature of the prompts, the generative AI component 310 can create or edit portions of HMI project 402, generate development recommendations for users to consider, answer questions about HMI project 402 or about development tools available as part of the development platform of the HMI development system, or other such content designed to assist users in developing HMI project 402.

[0046] Generative AI component 310 can use an associated custom model 322 trained with domain-specific industrial training data to implement prompting engineering functions, and can interface with generative AI model 406 (e.g., LLM or another type of model) and associated neural network. Figure 5This is a diagram illustrating the training of a custom model 322 used by the generative AI component 310. In some embodiments, the generative AI model 406 may reside and execute outside the HMI development system 302, and the generative AI component 310 may include suitable connectivity tools and protocols, application programming interfaces (APIs), or other services that enable the generative AI component 310 to exchange prompts and responses with the generative AI model 406. The custom model 322 may be trained using a training dataset 502 representing a set of domain-specific industry knowledge, which may assist the generative AI component 310 in generating or modifying portions of the HMI project 402 in a manner that satisfies the user's natural language design description with high probability and meets any application-specific or vertical domain-specific requirements. Example training data 502 that can be used to train the custom model 322 includes, but is not limited to, information defining industry standards (e.g., global or vertical-specific safety standards, food and drug standards, design standards such as ISA-88 standards, etc.), technical details or design standards for various types of industrial control applications (e.g., batch control processing, die casting, valve control, agitator control, etc.), knowledge of specific industrial verticals (e.g., automotive, food and beverage, pharmaceutical, oil and gas, textile, mining, etc.), knowledge of industrial best practices, technical specifications for various types of industrial installations or assets (e.g., industrial controllers, motor drives such as variable frequency drives, sensors, etc.), control design rules, sample HMI display layouts for various types of control applications or use cases, customer-specific training data describing internal HMI design preferences or standards (e.g., preferred screen layout format, preferred graphics type, preferred text font, etc.), or other such training data.

[0047] During project development, the generative AI component 310 can generate prompts 504 as needed and submit them to the generative AI model 406. These prompts 504 are designed to elicit responses 506 that assist in HMI project development tasks. These prompts 504 are generated based on the user's natural language input and industry knowledge and reference data encoded in a trained custom model 322. The generative AI component 310 can, as needed, refer to the custom model 322, combine this with processing the user's natural language queries or requests (which can be submitted as design input 412), and prompt the generative AI model 406 to obtain responses 506 from the auxiliary user interface component 304 and the HMI generation component 306 in handling these requests and queries.

[0048] Return to Figure 4Based on user design input 412, user interface component 304 can present design feedback 418 designed to assist developers in developing HMI project 402. At least some of this design feedback 418 may include natural language chat prompts generated by generative AI component 310 requesting specific information items from the user (e.g., as part of an iterative dialogue with the user aimed at determining design requirements), HMI design recommendations (e.g., recommendations for HMI screen layouts or navigation controls), design views of the HMI project 402 under development, responses to natural language queries submitted by the user regarding HMI project 402 or the system's HMI development tools, or other such feedback.

[0049] Implementations of the HMI development system 302 may use prompting engineering services to process natural language design input 412 submitted by users via user interface components 304 (e.g., via a spoken word interface or a text-based chatbot). These prompting engineering services may leverage industry knowledge encoded in a custom model 322 (such as that learned from training data 502) and responses 506 prompted by generative AI models 406 to accurately determine the designer's design requirements and generate a portion of the HMI project 402 to address those requirements, or to provide refined answers to design queries.

[0050] When a user submits natural language design input 412 describing an HMI design request to the HMI development system 302, the generative AI component 310 analyzes the input 412 based on domain-specific industry knowledge and design rules encoded in a custom model 322 (i.e., knowledge and rules represented by training data 502). Based on this analysis and depending on the nature of the design input 412, the generative AI component 310 creates or modifies a portion of the HMI project 402 in a manner that meets the design requirements described by the design input 412. In cases where the design input 412 includes a natural language query about the HMI project 402 being edited or about the system 302 itself, the generative AI component 310 generates and returns a response to the query as design feedback 418. This response may, for example, be an answer to a question about the HMI project 402 being viewed, guidance on appropriate development tools supported by the system 302 that can be used to solve the HMI design problem specified by the query, or other such responses.

[0051] In addition to referencing the information contained in the custom model 322, the generative AI component 310 can also prompt the generative AI model 406 as needed to obtain a response 506 (see [link]). Figure 5The response 506 helps to formulate an appropriate HMI design (or natural language response) in response to the user's natural language design input 412. For example, in response to receiving a natural language design request, the generative AI component 310 may determine whether sufficiently accurate HMI design actions can be performed (e.g., creating HMI project 402, creating or modifying a display screen defined by project 402, placing appropriately configured graphical objects on the display screen, removing graphical objects from the display screen, configuring animation data links for animate graphical objects, etc.) solely based on relevant information contained in the custom model 322, or alternatively, whether supplementary information from the generative AI model 406 is needed to determine an appropriate HMI design action with a sufficiently high probability of satisfying the design request described by the natural language design input 412. If the supplementary information from the generative AI model 406 is deemed necessary, the generative AI component 310 may formulate a prompt 504 based on an analysis of the design input 412 and a relevant subset (or a corresponding subset of the training data 502) of industry knowledge encoded in the custom model 322. These prompts 504 are designed to obtain responses 506 from the generative AI model 406, which can be used to generate or modify a portion of the HMI project 402 in a manner that satisfies the user's natural language design requests.

[0052] In developing a response to a user’s question about HMI project 402 or about design tools supported by HMI development system 302, generative AI component 310 may aggregate information determined to be relevant to the query from custom model 322 (e.g., knowledge of common design standards for the type of industrial control application of interest, technical or specification data of industrial assets or devices, industrial design standard information, vertical industry-specific industrial standards, knowledge of the development platform of the IDE system, etc.) with language-specific combinations or grammatical information obtained from generative AI model 406 as a response 506 to develop a natural language answer to the user’s query.

[0053] System 302 can also guide iterative natural language chat exchanges with the user when necessary to determine the user's HMI design request or increase the likelihood that the generative AI component 310 will formulate HMI design actions that meet the user's design request. For example, the generative AI component 310 can parse the initial natural language design input 412 to determine the type of design requirement being described by the input 412 and refine and contextualize the initial input 412 in a manner that is expected to help the customization model 322 or the generative AI model 406 quickly and accurately achieve the desired HMI design solution. If the generative AI component 310 determines that additional information from the user will produce a design solution with a higher probability (i.e., a probability exceeding a defined threshold) of meeting the user's initial request, the generative AI component 310 can formulate and present one or more query responses as design feedback 418, which prompt the user to provide more refined information that will enable the generative AI component 310 to provide a more complete or more accurate HMI design solution (i.e., a solution estimated to have an accuracy exceeding a defined threshold) for the user's request. Through iterative chat exchanges, the generative AI component 310 can collaborate with the user to explore potential HMI content and configuration changes that may meet the user's needs. The generative AI component 310 can guide these natural language conversations with the user, partly based on learned knowledge of the types of questions that need to be answered, to generate an HMI item 402 or a portion thereof that meets the user's needs.

[0054] Users can interact with system 302 using such iterative natural language dialogues to facilitate incremental design and refinement of HMI project 402. For example, using a natural language design request submitted as design input 412, users can initiate the creation of a new HMI project 402 and continue to refine project 402 by adding new display screens (e.g., “create a panel with two instant buttons, gauges, drop-down lists, and a rotation box”), adjusting color schemes (e.g., “change the background color of station #6 screen to gray”), adding or removing processing stations or other graphic elements (e.g., “add a redundant standby pump in parallel with pump #6”), repositioning or resizing graphic objects (e.g., “move the leak test station to the top of the screen”, “make the tank smaller”, etc.), or performing other such refinements. As needed, the generative AI component 310 can guide the refinement process by presenting natural language prompts as design feedback 418. These natural language prompts request additional information from the user that may help converge a suitable HMI design for the user's control system, or provide suggestions for improving or optimizing HMI project 402 based on the current design state of project 402 (e.g., recommendations for rearranging or resizing graphical objects to make more efficient use of space, recommendations for modifying the display screen navigation structure for project 402, etc.).

[0055] In addition to enabling users to submit free-form natural language design input 412, some implementations of the user interface component 304 may present users with pre-written or pre-loaded prompts for selection and submission to the generative AI component 310. These pre-written prompts may be stored in a prompt repository 408 of the HMI development system 302 and represent questions or development tasks typically submitted by users of the HMI development system 302. In an example scenario, the user interface component 304 may present a list of 10 of the most common questions or requests submitted by users of the system 302 as selectable natural language prompts, where selecting a prompt from the list allows submission to the generative AI component 310 for processing. Where appropriate, the user interface component 304 may enable users to customize one or more parameters of the selected pre-loaded prompt to suit specific needs (e.g., by indicating the specific display screen, graphical object, data label, controller definition, program instruction, or industrial asset targeted by the selected pre-written prompt).

[0056] In another example, the generative AI component 310 can determine the general design theme targeted by the user's initial design input 412 (e.g., creating a suitable layout for graphical objects on a display screen, designing a suitable screen navigation structure for HMI project 402, etc.) and select a subset of pre-written tips from the tip repository 408 that are determined to be helpful to the design theme. The user interface component 304 can present these tips on the system's development interface for the user to select as needed. The selection of a pre-written tip causes the selected tip to be submitted to the generative AI component 310 for processing as design input 412.

[0057] Generative AI component 310 can use a range of methods to process the natural language design input 412 submitted by the user and formulate prompts 504 for the generative AI model 406, which are designed to produce responses 506 that assist the user's design requests. According to the example method, generative AI component 310 can access archives of chat communications between itself and other users of system 302 and identify chat sessions initiated by user queries that share similarities with the initial design input 412 submitted by the current user. Upon recognizing these archived chat sessions, the generative AI component 310 can analyze these past chat sessions to determine the type of design action that the system 302 ultimately performs on the HMI project 402 as a result of these sessions (e.g., creating or configuring a display screen with content based on specific keywords from a user's query, adding a specific type of graphical object to the display screen, configuring data links for controlling the animation properties of graphical objects, adding a controller definition to the HMI project 402 that defines the controller with which the project 402 exchanges data, etc.), and update the HMI project 402 based on the results of these past chat sessions and as appropriate for the user's initial request or if necessary.

[0058] Analysis of these archived chat sessions, as well as any other relevant industry knowledge or expertise encoded in the custom model 322, can also assist the generative AI component 310 in inferring user needs from the initially vaguely worded natural language design input 412 and modifying the design of the HMI project 402 in a way that addresses those needs. If the generative AI component 310 determines that supplementary information from the generative AI model 406 is necessary to determine a design modification of the HMI project 402 with a sufficiently high probability of satisfying the user's request, the generative AI component 310 can also formulate a prompt 504, designed to prompt the generative AI model 406 to obtain at least a portion of the information inferred to be of interest to the user. This could include, for example, formulating a prompt 504 to request a specific type of information from the generative AI model 406 that may not have been specified in the user's design input 412 but has been determined by the generative AI component 310 to be information that will address the user's needs. In this way, the generative AI component 310 and its associated custom model 322 can proactively frame the user's natural language design input 412 in a way that quickly and accurately guides the generative AI model 406 to the user's desired HMI design solution (e.g., generating an HMI project 402 that meets the design requirements implied by the natural language design input 412).

[0059] In another example approach, the generative AI component 310 may enhance the user's natural language design input 412 with additional contextualized information from the custom model 322, and integrate this additional information with the user's request to generate a prompt 504 submitted to the generative AI model 406. The type of additional contextual information added to the design input 412 may depend on the nature of the design request and may include, but is not limited to, information obtained from a supplier knowledge base known to be relevant to the user's design request or from device documentation for an industrial installation.

[0060] Various example HMI design actions, such as HMI project creation and editing, can now be performed by system 302 based on the processing of user natural language design input 412. Typically, system 302 can perform virtually any type of HMI project creation, development, and editing action based on and according to a natural language request or design description submitted as design input 412. For example, in response to receiving a natural language request to create a new HMI project 402 (e.g., “Create new project”), HMI generation component 306 (with the assistance of generative AI component 310 as needed) can initiate a new HMI project 402 within the system's development environment. As described above, the natural language design input 412 can be submitted to system 302 as typed or spoken natural language text, which can be input via a chatbot interface or another type of natural language dialogue interface presented by user interface component 304.

[0061] As part of the initial request to create HMI project 402 or via a subsequent separate natural language design request, the user can submit natural language design requirements or configuration requests for the newly created HMI project 402. Example HMI design actions that can be performed by system 302 based on such natural language design input 412 may include, but are not limited to: creating HMI project 402, defining the display screens constituting HMI project 402, defining navigation modes or controls for navigating between display screens (including adding graphical navigation controls to selected display screens and defining which other display screens these navigation controls will invoke), selecting both static and animated graphical objects and adding them to the display screens, defining animation controls for animated graphical elements (including specifying data labels for industrial controllers or another data source for controlling the animation state of graphical objects), and specifying the graphical objects on the display screen. Define the layout or position on the display screen, define the color scheme for the display screen, define the conditions for triggering alarms (e.g., upper or lower limits of specified data labels for triggering high-level or low-level alarms), define the text of the corresponding alarm, define multiple versions of a given HMI item 402 to be used under different environmental conditions (e.g., strong light conditions, outdoor use, high particle environment, etc.), define the script to be triggered in response to specified conditions detected by the HMI during runtime, name the display screen or graphical object, add controller definitions to the HMI item 402 to define the industrial controllers with which the HMI will exchange data values, or define other such attributes of the HMI item 402.

[0062] For example, as sample natural language design input 412, a user could submit "Create a new project where navigation is on the left, help and login are in the upper right, alert summary is centered at the top, and the company logo is on the left. Create three sub-areas across all units: alert history and top ten alerts." The generative AI component 310 can analyze this input 412 using industry or application knowledge encoded in the custom model 322 and responses 506 prompted by the generative AI model 406 as needed to determine or infer the user's HMI design requirements and instruct the HMI generation component 306 to create or edit the HMI project 402 in a way that meets those requirements. In this example, the HMI generation component 306 would define the HMI project 402, which includes the display screen, add navigation button controls to the left side of the screen, add help and login button controls to the upper right corner of the screen, center the alert summary near the top of the screen, and add the company logo to the left side of the screen.

[0063] The user's natural language design input 412 may include explicit references to portions or elements of HMI item 402 as needed to describe the HMI configuration actions to be performed. These references may include display screen names (e.g., "Add a button for navigating to the overview screen to the lower left of the line 1 screen"), names or identifiers of graphical elements or their attributes (e.g., "Set the open state color of the #1 valve graphic to green and link it to the valve open label of controller 1."), or other such explicit references.

[0064] In some scenarios, users can submit natural language design input 412 describing the visualization requirements of the industrial automation system that needs HMI project 402 in more general terms (i.e., without references to specific elements of HMI project 402), rather than submitting natural language design input 412 describing HMI configuration actions with explicit references to elements or configuration functions of HMI project 402 itself. According to this method, users can describe, via natural language design input 412, the industrial assets, devices, or machines (e.g., tanks, valves, presses, conveyors, processing stations, etc.) that are part of the automation system and want to be visualized on the HMI, the functional relationships between assets (e.g., "I need a tank called tank #1 to feed material to another tank called tank #2 via an inlet valve called valve #1"), and information about the assets the user wishes to visualize ("Show me the status of the valves and the tank's fill level"). Users can also provide other details about the assets or applications, such as tank capacity, maximum flow rates between tanks, or other relevant information about the assets. HMI generation component 306, assisted by generative AI component 310, can process these functional descriptions to determine the user's visualization requirements and develop HMI project 402 to meet those requirements. Development actions performed by system 302 in response to the natural language design input 412 may include adding appropriate graphical objects to the HMI display screen, arranging these graphical objects in a manner that reflects the functional relationships between assets represented by the graphical objects, determining and configuring data links between the animation attributes of these objects and their corresponding controller data labels, or other such HMI development functions.

[0065] In other example scenarios, the user's natural language design input 412 can describe the type of industrial automation system or control application to be visualized, as well as application-specific details of the user system that can be used by system 302 to create and customize suitable HMI projects 402 for the automation system. For example, the user can specify via natural language design input 412 the batch processing required for producing a specified type of material (e.g., plastics, chemicals, pharmaceuticals, etc.). This design input 412 can describe any functional details that can be used by generative AI component 310 and HMI generation component 306 to determine suitable graphical content, layout, and animations for visualizing the control application.

[0066] The generative AI component 310 can formulate appropriate HMI configuration actions in response to any of the various types of natural language design inputs 412 described above, based in part on industry-specific knowledge encoded in the custom model 322 (or underlying training data 502). The industry knowledge encoded in the custom model 322 can assist the generative AI component 310 and the HMI generation component 306 in determining the preferred or compliant configuration for the HMI project 402, which also meets the user design requirements specified in the natural language design inputs 412. This encoded knowledge reduces the burden on the user to provide high-granularity details of the control application or automation system in which the HMI is designed, because the system 302 can leverage the domain-specific knowledge recorded in the custom model 322 to infer user needs and formulate an appropriate HMI design that meets those inferred needs.

[0067] For example, some of the training data in training data 502 used to train the custom model 322 may include knowledge of standard or general system designs for corresponding different types of industrial control applications (e.g., batch processing applications, die casting applications, material handling applications, machining applications, etc.). This knowledge may include: the type, number, and configuration of the corresponding types of industrial assets (e.g., machines, monitoring and control devices, conveyors, processing stations, barrels or tanks, pipes, ovens, presses, etc.) typically used in such applications; the functional relationships between these assets (e.g., the arrangement of assets, the direction and sequence of parts or materials flowing through assets, etc.); or other such domain-specific knowledge. In response to receiving natural language design input 412 from the user describing the functional requirements for visualizing an automated system designed to implement a specific type of industrial control application, the generative AI component 310 can access relevant domain-specific information about the specified type of control application from the custom model 322 and use this information in conjunction with aspects of developing an HMI design with a high probability of satisfying the requirements described by the user's natural language design input 412. This can include, for example (based on the type of control application or industrial vertical described by the user's design input 412), inferring a set of industrial assets that are highly likely to be part of the automation system to be visualized, and generating an HMI project 402 with a display screen having a preliminary layout of graphical objects representing these assets. In such a scenario, even if the user's natural language design input 412 does not specify the identifiers or types of assets that will constitute the automation system, the generative AI component 310 and the HMI generation component 306 can infer a suitable set of graphical asset representations and the arrangement of these graphical representations.

[0068] In some implementations, the custom model 322 can be trained using information about a specific custom machine (e.g., a machine or production line built by an OEM for a customer), or the custom model 322 can include a pre-trained model provided by the OEM. The machine-specific training data 502 can include, for example, the names and functions of workstations constituting the machine or production line (e.g., tool stations, quality inspection stations, ovens, etc.), the identification and arrangement of components constituting the machine (e.g., valves, barrels, pumps, industrial robots, actuators, stoppers, etc.), machine specification data, or other such information. The generative AI component 310 can utilize the machine-specific information used to train these models 322 to appropriately configure the HMI project 402 to visualize the custom machine based on user design input 412. The custom model 322, trained using machine-specific training data 502 for machines known to be deployed at a customer's facility, can assist the HMI development system 302 in generating an HMI project 402 that accurately reflects the customer's equipment with relatively little design input 412 from the user.

[0069] By supporting HMI development through the use of natural language design inputs 412 that describe the functional requirements of HMI project 402 at varying degrees of granularity or specificity, HMI development system 302 enables users in various roles within an industrial enterprise to contribute to the design of HMI project 402. These roles include plant engineers or control engineers designing the corresponding automation system, maintenance personnel who will maintain and troubleshoot the automation system, and machine operators who will operate the automation system. These diverse user roles may require different information presentations from HMI project 402, and members of these roles can ensure that HMI project 402 meets their specific needs by providing natural language design inputs 412 that describe their role-specific requirements at various levels of specificity. For example, a control engineer can submit natural language design input 412 that describes visual requirements based on specific device or apparatus names, clearly identified controller data labels for animation attributes driving graphical objects, or other technical languages ​​and references that machine operators may not be familiar with. Additionally, machine operators can be invited to contribute natural language design input 412, which describes their HMI requirements based on the information they wish to see and how they want that information to be organized (e.g., "I want to see the tank fill level and outlet flow rate on the same screen."). In all these scenarios, system 302 can process these natural language inputs as needed, utilizing domain-specific knowledge encoded in a custom model 322 and responses from the generative AI model 406, to implement development actions on HMI project 402 that are intended to meet the requirements of these diverse users.

[0070] System 302 can support multi-user collaborative development of HMI project 402, receiving natural language design input 412 from users with different roles. In some such collaborative development scenarios where users with different roles contribute to the development of HMI project 402, some implementations of system 302 can process the design input 412 for HMI project 402 according to the role of the user from whom the design input 412 is received. For example, some implementations of HMI development system 302 can create multiple versions of HMI project 402 for the development of a given automation system, wherein each version of project 402 is created using design input 412 received from a user with a single general role (e.g., machine or production line operator, plant engineer, maintenance personnel, etc.). This produces multiple versions of HMI project 402 specific to the corresponding different user roles, which can be invoked by users with these roles during runtime. In this way, system 302 enables users with different roles to create versions of HMI project 402 that meet the specific requirements of their roles.

[0071] Alternatively, the HMI generation component 306 can integrate design inputs 412 from multiple users with different roles into a single HMI project 402 that satisfies all the requirements specified by the multi-user design inputs 412. In some such implementations, if requests from these roles conflict with each other, the system 302 can prioritize the design request from one role over the design request from another. For example, if the system 302 determines that design inputs 412 received from the plant engineer conflict with design inputs 412 submitted by the machine operator (e.g., in terms of the placement or orientation of graphic objects, color schemes, content to be included in or omitted from the HMI), the system 302 can choose to implement the plant engineer's design inputs 412 in the HMI project 402 and override the design inputs 412 received from the machine operator.

[0072] In this regard, the generative AI component 310 can infer the user's level of expertise based on the wording of the natural language design input 412, and formulate design feedback 418 based on this inferred level of expertise (e.g., answers to follow-up questions about the user's design request, questions about HMI design, or questions about development tools provided by system 302). The user's level of expertise can be inferred, for example, based on determining whether the words, phrases, or terms used in the design input 412 are likely to be used by a relatively advanced professional who can be expected to understand the technically more sophisticated design feedback 418, or alternatively, by an external user who might need to include more basic information in the feedback 418.

[0073] For users at these different skill levels, the generative AI component 310 can express any design feedback 418 or response to design input 412 at a level of assumed understanding that is appropriate for the user's inference, including prompting the user to provide additional information to assist the system 302 in determining appropriate HMI development actions that have the potential to meet the user's request. This can influence the choice of words used in the feedback 418 and the granularity of the response content.

[0074] Some implementations of the HMI development system 302 can also be configured to process design input 412, which includes various types of design documents generated as part of the design process for the automation system to be visualized. For example, a user can submit digital design drawings (e.g., computer-aided design (CAD) drawings) of the automation system or machine requiring the HMI project 402 as design input 412. Example CAD drawings that can be submitted as design input 412 may include electrical drawings, I / O drawings, mechanical drawings, panel layout drawings, or other such documents. With the assistance of the custom model 322 or responses prompted by the generative AI model 406, the generative AI component 310 can determine the appropriate content and format for the corresponding HMI project 402 based on the analysis of these drawings, and generate the HMI project 402 with that content and format.

[0075] In the example scenario, generative AI component 310 can identify the types of machines or devices included in the automation system, and the communication or functional relationships between these machines and devices, based on the analysis of I / O drawings, mechanical drawings, or electrical drawings of the automation system. Generative AI component 310 can also infer the type of industrial processing performed by the automation system based on the analysis of these drawings. Furthermore, based on knowledge of information types of the inferred industrial application type that the operator is typically interested in (such as that encoded in custom model 322, or based on the analysis of other archived HMI applications of similar types of industrial applications), generative AI component 310 can instruct HMI generation component 306 to generate HMI project 402, which contains graphical objects representing a subset of known machines or devices of typical interest (e.g., tanks, valves, motors, conveyors, etc.), and is organized to represent the functional relationships between the machines or devices. HMI generation component 306 can also define the animation attributes of these graphical objects and the data links (e.g., as determined from the I / O drawings) between them and their corresponding data tags or addresses in the industrial controller that will monitor and control the automation system.

[0076] Some implementations of the HMI development system also allow users to submit or import previously developed code snippets or portions of previously developed HMI visualizations (e.g., pre-developed HMI displays or completed HMI applications or projects 402), and to submit requests for natural language summaries of these pre-developed components. For example, a developer might have an HMI project 402 or other type of industrial visualization application (or part of such an application) developed by another developer, and might have specific questions about the functionality of an unfamiliar HMI. The developer can submit the pre-developed project 402 along with requests for summaries of the HMI's functionality or specific questions about the HMI's design to system 302. Example questions that can be submitted by the user for the submitted HMI project 402 may include, but are not limited to: questions about the type of industrial application targeted by the submitted project 402; questions about the animation or data linking properties of the graphical objects included in the HMI project 402 (e.g., “What controls the state of the Vat#3 graph?”, “What data label does the start button link to?”, etc.); questions about the computing resources required to execute the HMI project 402 (e.g., “How much memory is needed to run this?”); questions about the type of information that a given display screen is intended to convey; questions about the functionality of the control graphics included on the display screen (e.g., “What does the slide control graphic do?”); or other such questions. In response to these submissions, the generative AI component 310 can generate a natural language answer to the user’s question based on an analysis of both the user’s question and the submitted HMI project 402, and present the answer as design feedback 418. As in the previous example, the generative AI component 310 can utilize industry knowledge contained in the custom model 322 and responses 506 prompted as needed from the generative AI model 406 to determine and formulate answers to the user’s questions about the functionality of the HMI project.

[0077] In some scenarios, when an HMI project 402 is submitted to system 302 for analysis, system 302 can provide a general natural language summary of the purpose and function of the HMI project, rather than submitting questions specific to an unfamiliar HMI project 402. This general summary may specify, for example, the overall purpose of the HMI (e.g., the type of machine or industrial process the HMI is designed to visualize), the type of information provided by each display screen in the HMI, the computational requirements for performing the HMI project (e.g., memory and processing power requirements), or other such information.

[0078] Some implementations of system 302 can also support similar reverse hinting engineering methods for analyzing and summarizing unfamiliar control code submitted to system 302. Similar to the reverse hinting engineering analysis of unfamiliar HMI items 402 or visualization components, generative AI component 310 can generate general natural language functional summaries of control code segments submitted to system 302 by the user, or it can generate natural language answers to specific questions about the submitted code. Example summaries or answers may include: a description of the control function or machine to which the code is designed to perform (e.g., batch control process, sheet metal stamping machine control, sheet metal tension control, control loops to keep process variables within target ranges, etc.), a description of the language in which the code is written, an identification of the hardware platform capable of executing the code, I / O requirements for executing the code (e.g., the number of analog inputs, digital inputs, analog outputs, and digital outputs required to interface the code with the automation system that the code is to monitor and control), or other such descriptions. Generative AI component 310 can also determine possible edits to unfamiliar code based on analysis, to improve code efficiency or readability, or reduce the total number of lines of code without changing its intended function, and generate natural language recommendations describing these edits. Examples of edits that can be recommended in this way include, but are not limited to: removing redundant code, merging duplicate code sections into a single routine (which is referenced within the code as needed), modifying variable naming terminology in the code, or other such edits. In any of these reverse-hint engineering scenarios, generative AI component 310 can, as needed, leverage relevant content from custom model 322 and responses 506 hinted at from generative AI model 406, combined with analysis of the submitted HMI project or code, and generate a natural language summary or answer.

[0079] Once development on HMI project 402 is complete and HMI generation component 306 is completed, HMI project 402 can be deployed to HMI terminal 114 or other computing platforms with display capabilities for execution. Figure 6This diagram illustrates the debugging of HMI project 402 onto HMI terminal 114. The system's HMI deployment component 308 can communicatively connect to HMI terminal 114 via a direct connection between HMI development system 302 and terminal 114, or via a remote connection to terminal 114 in the case of an implementation where system 302 is executed on a cloud platform or other remote platform. When executed on HMI terminal 114, the compiled HMI application 602 presents an HMI with a display screen and associated graphical elements and behaviors defined by design input 412 and generated and configured at least in part based on the processing of the developer's natural language prompts. Where HMI project 402 also defines communication parameters for HMI terminal 114 itself, the execution of HMI application 602 will configure the terminal's networking or communication settings according to the communication settings defined in HMI project 402. This may include, for example, setting the terminal's networking parameters to operate on the factory network 116, and setting the HMI's communication settings so that the terminal 114 will communicate with industrial devices (e.g., controller 118) with which the HMI terminal 114 will exchange data.

[0080] In some implementations, when a new HMI project 402 has been created, the user can also request a test script via natural language input 412. This test script is designed to test the interaction between the HMI application 602 and the industrial controller 118 with which the application 602 will communicate. Based on this request, the generative AI component 306 can generate such a test script based on any available information about the HMI terminal 114, the industrial controller 118, and the network 116, as well as the communication requirements defined by the HMI project 402 itself, and bind the test script to the HMI application 602. After the HMI application 602 is deployed, the test script can be executed on the HMI terminal 114 to test the communication link between animated graphical objects on the HMI application's display screen and the corresponding data tags in the controller 118. If errors are found based on the execution of these test scripts—such as improper data tag linking, improper or unconfigured graphical objects, or other such errors—information about these errors can be returned to the system 302, which can then use the generative AI component 310 to determine debugging actions to correct these errors. System 302 can then implement these corrections in HMI project 402, redeploy application 602 to HMI terminal 114, and re-execute the test scripts to ensure that all errors have been corrected.

[0081] After deploying HMI project 402 as runtime application 602, some implementations of HMI development system 302 can also support dynamic runtime modifications to the resulting HMI application 602 using natural language prompts. Figure 7This diagram illustrates the use of natural language prompts 706 during runtime of the HMI development system 302 to implement dynamic editing 704 of the HMI application 602. At any time during the runtime operation of the HMI application 602, an operator or other user can submit natural language prompts 706 to the application 602, requesting modifications to the visual aspects of the HMI application. These natural language prompts 706 can be submitted via a text-based or verbal chat interface integrated into the HMI application 602 and are then routed to the HMI development system 302 for processing. The generative AI component 310 can process these natural language prompts 706 to determine the nature of the modifications requested by the user and return HMI edits 704 that implement the requested modifications on the HMI application 602.

[0082] In an example scenario, a user can indicate via natural language prompt 706 that the display screen presented by HMI application 602 is difficult to see due to the environmental conditions of the area where the HMI is being used (e.g., bright ambient light or relative darkness). Prompt 706 can specify the nature of the visual difficulty, such as, "I can't see the screen in this bright light," or "This control room is dark; could you brighten the display?" Based on an evaluation of prompt 706, generative AI component 310 can adjust application 602 or HMI terminal 114 itself to correct the problem (e.g., by adjusting brightness or contrast levels to counteract the bright environmental conditions). Another example prompt 706 can request countermeasures for visual impairments (e.g., "I can't see blue in this environment; can we correct that?" or "I can't see yellow very clearly; could you choose an alternative color?"), and in response to such prompt 706, generative AI component 310 can determine and implement appropriate countermeasures to compensate for the indicated impairment (e.g., adjusting the brightness, contrast, or color selection on the HMI display).

[0083] Natural language prompt 706 can also be used to change the content of a display rendered by an HMI application. For example, a user can submit natural language prompt 706 requesting the addition or removal of graphical elements, such as animated graphical objects or numbers, from the currently rendered display. In response to receiving and processing the prompt, generative AI component 310 will return an HMI edit 704 that implements the indicated modifications on HMI application 602.

[0084] Users can also dynamically create new custom display screens using natural language prompts 706 as needed. For example, a given machine operator might want to aggregate graphical objects currently distributed across multiple different screens onto a single new display screen. To achieve this, the operator can submit a natural language prompt 706 describing what the operator wants to see (e.g., “Get me the valve from monitor 2, get me the pump from monitor 3, and build a new display for me.”), and based on the translation of that prompt 706, the generative AI component will return an HMI editor 704 that creates a screen with the described content and adds it to the HMI application 602.

[0085] System 302 can also process natural language prompts 706 that describe the functionality the operator wants to see from a more abstract perspective, rather than explicitly identifying what the operator wants to see with specific graphical objects. For example, natural language prompt 706 might indicate that the operator wants to see a display of the open and closed status of an auxiliary calibration valve (e.g., “Give me the screen for calibration valve #2”). Based on the analysis of this prompt 706, industry knowledge encoded in the custom model 322 (which may include knowledge of the types of information known to be helpful in performing such calibrations), and analysis of the HMI application 602 itself, the generative AI component 310 can determine the appropriate content and associated data links expected to be useful for performing valve calibration (i.e., links to appropriate controller data labels or other sources of the required data), and return an edit 704 that creates a new screen including that content. In some cases, the content of the new screen can be extracted from other screens already defined in the HMI application 602.

[0086] In another scenario, an operator might want to view the identifiers and statuses of interlocks or permissions for a specific machine state or control action of interest. Typically, interlocks or permissions for a given machine state or control action are a set of conditions that must be true before the control system allows the machine to be placed in that machine state or for the control action to be executed. In the example scenario, the operator might want to view the conditions that prevent the machine from being placed in automatic mode. Even if a dedicated display screen for showing the status of these interlocks is not currently defined as part of the HMI runtime application 602, the operator can submit a natural language prompt 706 requesting to view the interlocks for the machine state of interest (e.g., “Show me the interlocks for automatic mode for this machine”). In response to the request, based on analysis of the industrial control program executed on the controller 118 that monitors and controls the machine (e.g., by identifying program conditions that allow the machine to be placed in automatic mode, and descriptions of these conditions such as those inferred from corresponding comments or I / O addresses), any available design documents accessible to system 302 (from which the generative AI component 310 can determine any safety devices or other control devices that act as permissions or interlocks for machine states or control actions), and (if necessary) knowledge encoded in the custom model 322 (e.g., proprietary information about the machine's control design previously submitted by plant engineers, industry knowledge of interlocks known to be necessary for the type of control process performed by the machine), and response 506 prompted by the generative AI model 406, the generative AI component 310 can identify interlocks of interest. Based on the results of this analysis, the HMI generation component 306 can return to an edit 704 to create a new interlock screen that presents the interlocks as graphical or alphanumeric indicators, and displays the current status of each interlock. An example interlocking screen created in this way can display each interlock as a color-coded indicator containing an alphanumeric description of the interlock (e.g., “Automatic mode switch,” “Safety door closed,” “Clamper in initial position,” “Pusher retracted,” “Light curtain cleared,” etc.), where the color of the indicator conveys the current state of the interlock (e.g., green for satisfied and red for unsatisfied).

[0087] In addition to creating the content and layout of the interlocking screen, editor 704 can also configure the animation properties of the interlocking for each graphical interlocking indicator, such that the color state of the interlocking indicator is associated with the appropriate data source (e.g., controller data labels) representing the current state of the interlocking. To this end, generative AI component 310 can determine which data labels in the control program executed on controller 118 represent the corresponding current state of the interlocking based on analysis of the control program or other information sources, and associate the animation properties of the graphical interlocking indicator with these data labels.

[0088] An operator can use a similar approach to create dynamic alarm screens that meet the content and formatting preferences specified by the operator's natural language prompt 706. For example, an operator might submit natural language prompt 706 requesting to view alarms associated with a specified machine station and also requesting a preferred display format for the alarms (e.g., "Show me the grid of alarms for the pick-up and place station"). Based on the analysis of this prompt 706, the generative AI component 310 can determine the identification and status of alarm conditions defined for the specified station and return an edit 704 to create a new alarm screen that presents the grid of identified alarms and their respective active or inactive states (e.g., in color-coded animation). As in the previous example, the generative AI component 310 can utilize any necessary information related to identifying the relevant alarms and their current status, including but not limited to: control procedures for monitoring and controlling the station of interest, proprietary design information or industry knowledge stored in the custom model 322, responses 506 from prompts in the generative AI model 406, or other such information. The operator can also use the natural language prompt 706 to instruct the system 302 to filter the view of alarms as desired. For example, a natural language prompt 706 stating "Show me all priority 1 alarms for the shampoo production line" can enable the generative AI component 310 to identify a subset of active machine alarm conditions associated with the specified production line and having an importance level of priority 1, and return to an edit 704 to create a new display screen to present the identifiers of these alarms.

[0089] This method can also be used to create and filter presentations of other types of information on HMI runtime application 602. For example, an operator can submit a natural language prompt 706 requesting to view selected historical operational data, where prompt 706 describes the desired data based on the machine or device of interest, the specific metric to view, the time range of interest, or other such indicators. Example prompt 706 could display "Show me the temperature and pressure of the standby feedwater pump in Unit 7 between midnight and 2 a.m." Prompt 706 or subsequent prompts can also specify the desired format for the data, such as a time series line graph, pie chart, data table, or other such format. Based on the analysis of the prompt 706, generative AI component 310 can determine the identifier and source of the requested data and return to edit 704 to create a new data presentation screen and present the requested data on the new screen. The presentation format is based on the format specified in prompt 706, or, if no format is specified, on the format determined to be suitable for the type of requested data.

[0090] Some implementations of system 302 can also dynamically construct a display screen based on a user's natural language prompt 706, which accesses and displays information from new data sources not currently accessed by the HMI runtime application 602. These data sources may include, for example, a work order management system that stores open and closed maintenance work orders for the plant, a cloud-based industrial analytics system that collects operational data from automation systems on the plant floor and generates operational statistics based on analysis of that data, or other such systems. In an example scenario, a user might submit a natural language prompt 706 requesting the construction of a new display screen linked to the work order management system and presenting specified information from that system; for example, "Construct a screen showing open work orders for the press." Based on the analysis of this prompt 706, the generative AI component 310 can identify the work order management system containing the desired information, retrieve a subset of work order data from that system that satisfies the user's request (e.g., a subset of open work orders for maintenance operations to be performed on the press), and modify the runtime application 602 to create and display a new display screen presenting the open work order information, including a description and status of open maintenance activities. The work order management system can be a cloud-based system accessible by the HMI development system 302 or a local system residing in the factory workshop. The HMI development system 302 can retrieve relevant work order information from the system to display on the runtime application 602.

[0091] System 302 can also dynamically link the HMI runtime application 602 to other products or services based on the user's natural language prompts 706. For example, a user can submit natural language prompts 706 requesting information about products or services the factory might need to purchase (e.g., replacement parts or devices, engineering services, etc.). Based on the analysis of the prompts 706, the generative AI component 310 can identify suitable product or service providers and link the HMI runtime application 602 to the customer service or technical support portal associated with the provider. Depending on the nature of the user's request, system 302 can present information retrieved from the provider's website about the requested product or service (e.g., pricing and availability) on a new display screen, establish real-time communication links with customer support experts associated with the product or service provider, or otherwise dynamically attach the HMI runtime application 602 to the desired product or service platform as needed by the user.

[0092] In a similar manner, system 302 may respond to natural language prompts 706 that request assistance with operational problems observed by the machine or system monitored by the HMI runtime application 602 or requests for assistance with the functionality of the HMI system itself, by creating a new display screen and linking it to a remote technical support expert or knowledge base article.

[0093] In a design-time scenario, system 302 can also utilize pre-loaded or pre-written prompts during the runtime of HMI application 602. For example, generative AI component 310 can retrieve selected pre-written prompts from prompt repository 408 when appropriate and present these pre-written prompts to the user via HMI application 602 for selection and processing by generative AI component 310. Generative AI component 310 can select an appropriate subset of pre-written prompts to present to the user based on the current state of HMI application 602 or its associated automation system, or based on the user's interaction pattern with HMI application 602. For example, generative AI component 310 can determine that the user is attempting to perform a specific operational task based on the user's interaction with HMI application 602. Based on this determination, generative AI component 310 can select and present a subset of pre-written prompts determined to be relevant to the operational task from prompt repository 408. The user's selection of one of these pre-written prompts allows the selected prompt to be processed by generative AI component 310.

[0094] In some implementations, system 302 may enable a user to submit a natural language prompt 706 defining an event-driven action to be performed by the HMI runtime application 602 or system 302 itself in response to the occurrence of a specified event. Generative AI component 310 can process these prompts 706 to determine the event-driven action to be performed and configure the HMI application 602 to perform the specified action in response to the detection of the defined event. For example, a user may submit a natural language prompt 706 stating, “When valve #3 fails, create a record of the time.” Based on the analysis of this prompt 706, generative AI component 310 can identify the event that triggers the action (in this example, valve #3 failing) and the event-driven action to be performed (in this example, creating a record of the time of the failure), and formulate an edit 704 configuring the HMI application 602 to perform the specified action whenever the HMI detects the occurrence of the specified event. Based on the nature of the event-driven action specified by prompt 706, generative AI component 310 can identify relevant data tags or other data sources (e.g., controller data tags indicating an abnormal state of a valve) that communicate when the specified event occurred, and configure HMI application 602 to monitor these data tags to determine when the event occurred. Generative AI component 310 can also identify any relevant data tags associated with the event-driven action (e.g., the source of any data to be logged in response to the event, data tags or registers to be written by the HMI application in response to the event, etc.), and configure the event-driven action to utilize these data tags as needed. Once HMI runtime application 602 has been configured to execute event-driven actions according to prompt 706, application 602 will execute the defined action in response to each detected event occurrence, according to prompt 706.

[0095] Other example event-driven actions that can be configured in this manner may include, but are not limited to: changing the setpoint of the monitored industrial process or otherwise writing specified data values ​​to controller data tags; changing the current operating mode of a machine or process; recording the value of a specified key performance indicator or a description of an event; navigating the HMI interface to a specified display screen; presenting a customized message based on the event; changing the properties of a specified graphical object (e.g., changing the object's color, size, or position); or other such actions. Events that can be defined as triggering such actions may include, but are not limited to: a machine or device transitioning to a specified state (e.g., an abnormal state); a specified key performance indicator of a controlled automated system or process transitioning outside a defined value range (e.g., temperature, pressure, speed, etc.); meeting defined production targets (e.g., the total number of products manufactured reaches a specified quantity, production rate or product quality metric reaches a specified threshold, etc.); a specified machine malfunction occurring; a tripping of an industrial safety device; or other such events.

[0096] Although the methods for defining and configuring event-driven actions have been described above in the context of HMI runtime application 602, a similar approach can be used during design time to configure HMI project 402 to perform desired event-driven actions before project 402 is deployed as runtime application 602. Furthermore, in some implementations, system 302 can retrain the custom model 322 or generative AI model 406 after event-driven actions have been configured based on prompts 706, edits 704, and any additional information from user prompts related to configuring event-driven actions.

[0097] In some implementations, after system 302 has implemented dynamic runtime modifications to HMI application 602 based on user natural language prompts 706, the user can submit a request to save the modified version of HMI application 602 as a template 702 for storage along with the HMI project 402 from which the runtime HMI application 602 was compiled. Each template 702 represents a version of the underlying HMI project 402 that has been modified based on a given set of natural language prompts 706 as described above, and can be given a name or other type of identifier to distinguish template 702 from other templates 702 associated with project 402. The user can invoke the selected template via interaction with HMI application 602 (e.g., by requesting a selected template via natural language prompts 706). When the selected template 702 is invoked, user interface component 304 can modify the HMI application 602 currently running on HMI terminal 114 according to the selected template 702. In another example scenario, template 702 can record dynamic modifications made to HMI application 602 to make HMI more suitable for a given environment, context, or user. These may include modifications to adapt the HMI to bright light environments, dark environments, visually impaired users, or other such contextual considerations. System 302 may invoke one of these templates 702 in response to an explicit request to adjust the HMI application 602 according to the selected template 702, or automatically in response to system 302 determining that the contextual conditions for creating the template 702 are currently active.

[0098] As another runtime feature, the HMI development system 302 can also handle natural language prompts 706, which include questions about the current operation of the HMI application 602 itself or about the automation system visualized by the application 602. For example, a user can submit a natural language prompt 706 requesting assistance in diagnosing performance problems of the visualized automation system or activity alerts generated by the HMI application 602. Such a prompt 706 could, for example, request suggested countermeasures to resolve current alert conditions reported by the HMI or performance problems observed by the operator and described by the prompt 706. In response to such a prompt 706, the generative AI component 310 can formulate a natural language response 708 to the prompt 706 based on the nature of the problem described by the prompt 706, knowledge of the specific automation system and its components encoded in the custom model 322 (e.g., if one of the custom models 322 is a machine-specific model, such as a model provided by an OEM), more general knowledge encoded in the custom model 322 regarding the type of industrial application performed by the automation system, or one or more of the responses 506 from the prompt 406 of the generative AI model 406. The system 302 can present the resulting natural language responses 708 on the HMI via a chat interface used to submit the prompt 706. Depending on the nature of the user query, these responses 708 may describe a recommended workflow for resolving an alarm condition or performance problem, answer questions about the nature of the alarm or problem, or other such information. The ability to process chat-based queries about alarm conditions or performance problems can simplify diagnostic processing and assist machine operators or other plant personnel in quickly resolving performance issues.

[0099] In some implementations, system 302 can monitor the workflow or HMI interaction patterns of machine operators or engineers in response to specific alarm conditions or performance issues, and further refine the diagnostic process by training one or more custom models 322 based on these observed interaction patterns and the learned workflow knowledge known to have successfully resolved these performance issues. Generative AI component 310 can then use these trained models 322 to generate suggestions as a natural language response 708 for resolving similar types of alarms or performance issues. For example, generative AI component 310 can determine that when a given alarm condition is triggered on the HMI, the corresponding alarm is resolved most quickly or efficiently when the operator performs a specific set of interactions with the HMI and automation system (e.g., navigating to a specific display screen, changing the status of a binary tag or the value of a setpoint via HMI interaction, performing a series of control panel or HMI interactions, removing parts from a workstation in the automation system, etc.). Therefore, training component 312 can utilize this learned optimal interaction pattern to train one or more custom models 322. If an alarm is triggered in a subsequent operational plan, the generative AI component 310 can utilize these updated trained models 322 to generate and present a natural language response 708, which describes the action to be taken by the operator to resolve the alarm situation. This guidance can be presented in response to the detection of an alarm situation or in response to a request from the operator for assistance in resolving the alarm (submitted as a natural language prompt 706). In this way, the system 302 can dynamically generate and maintain up-to-date standard operating procedures (SOPs) to resolve alarm situations or operational anomalies.

[0100] In some implementations, in addition to creating SOPs for alarm solution workflows based on observations of interaction patterns changing over time, as discussed above, or as an alternative, system 302 can also update the SOP more promptly after an operator has resolved a given alarm condition. For example, after an alarm condition or anomaly has occurred on an automated system and has been resolved by a machine or production line operator, system 302 can present a natural language prompt on the HMI asking the operator if they wish to update the automated system's SOP to record the sequence of operator actions as the preferred response to the alarm condition. If the operator responds by requesting an update (or creation) of the SOP to record the workflow, the generative AI component 310 can create a record of the sequence of actions the operator took to resolve the problem and associate that record with the alarm condition resolved using that operator sequence. As discussed above, this SOP information can be recorded as part of a customized model 322 so that when the alarm condition recurs later, system 302 can present a description of these actions via the HMI runtime application 302 to provide guidance to other operators.

[0101] Because system 302 can track and record interaction patterns of different operators during runtime, system 302 can present these past user interaction patterns in response to the user's natural language prompt 706. For example, if the automated system experiences an alarm condition or performance problem that the current machine operator is unfamiliar with, the operator can submit a natural language prompt 706 requesting assistance in resolving the problem by explicitly requesting a previous interaction pattern that was successfully resolved by another operator, or by more generally requesting assistance in resolving the problem. In response to such a prompt 706, generative AI component 310 can retrieve information about one or more past interaction sequences that are known to have successfully mitigated the alarm condition or performance problem, or relevant portions of a Standard Operating Procedure (SOP) recording previous sequences of actions used to mitigate the alarm condition, and present this information on the HMI. The interaction sequence may include an ordered sequence of actions performed by a previous operator to resolve the problem. These actions may include interactions with the HMI (e.g., navigating to a specific display screen, interacting with specific HMI controls such as graphical buttons or data input fields), interactions with the control panel of the automation system (e.g., pressing a button, setting the position of a selector switch, placing the automation system in a specific operating mode, manually moving machine components of the automation system to their original positions, etc.), or other interactions with the automation system that can be detected by system 302 (e.g., removing a part or debris from a location on a conveyor of the automation system monitored by presence sensors). In response to requests for these past interactions, the system's user interface component 304 may present information about these past interactions on the HMI in any suitable format, including but not limited to natural language descriptions of these interactions or graphical demonstrations that visualize the interactions.

[0102] In another example, a user can submit a natural language prompt 706 that requests information about all interactions performed on the HMI and its associated automation systems within a given timeframe (e.g., interactions for previous shifts, interactions for the past six hours, etc.), rather than requesting a recommended set of interactions to resolve a specific alarm or performance issue, and system 302 can formulate and present descriptions of these interactions via the HMI.

[0103] In some scenarios, HMI runtime application 602 can be configured to automatically generate data logs that record details of operational or technical events related to HMI application 602, HMI terminal 114, or the HMI runtime application 602 itself. Such data logs may also record user interactions with the HMI, diagnostic or health status data for HMI terminal 114 or HMI runtime application 602, or other such information. A given data log may include one or more files (e.g., text files or other formats) that record a timestamped list of detected events or states, where each item in the list includes a description of the detected activity and the date and time the activity was detected. HMI runtime application 602 may generate and store these data logs locally on HMI terminal 114, or send these data logs to system 302 for storage.

[0104] If desired, a user can submit a request for a natural language summary of a data log or a portion thereof via interaction with the HMI runtime application 602. For example, a user can submit a natural language prompt 706 requesting a summary of activities within a specified time frame and / or a subset of activities falling into a specified activity category or related to a specific aspect of the automation system or machine (e.g., “What does the data log say about the power outage that occurred this morning in the feeder system?”). Based on the analysis and interpretation of this request, the generative AI component 310 can identify a relevant subset of information recorded in the data log (e.g., a subset of information falling within a specified time frame and corresponding to the category or type of desired information), generate a natural language summary of the selected data log information, and present the summary as a natural language response 708. As in the previous example, the generative AI component 310 can formulate these data log summaries as needed, utilizing information contained in the custom model 322 and responses 504 prompted from the generative AI model 406. An example summary developed in this manner may describe the sequence of events preceding an abnormal event that occurred on a machine on the production line, the subsequent actions taken by the operator in response to the abnormal event, the total amount of time the machine was offline due to the abnormal event, the inferred root cause of the abnormal event, or other such overviews.

[0105] Figures 8a to 8bMethods according to one or more embodiments of this application are illustrated. Although the methods shown herein are illustrated and described as a series of actions for the purpose of simplicity, it should be understood and recognized that the invention is not limited to the order of actions, as some actions may occur in a different order than those shown and described herein and / or occur simultaneously with other actions according to the invention. For example, those skilled in the art will understand and recognize that the method may alternatively be represented as a series of interrelated states or events, such as in a state diagram. Furthermore, not all actions shown are necessary for implementing the method according to the invention. In addition, when different entities formulate different parts of the method, one or more interaction diagrams may represent the methodology or approach according to this disclosure. Furthermore, two or more of the disclosed example methods may be implemented in combination with each other to achieve one or more features or advantages described herein.

[0106] Figure 8a The first part of an example method 800a for configuring an HMI runtime application to perform event-driven actions using natural language prompts is shown. First, at 802, a natural language request describing an event-based action to be performed by the HMI application in response to a specified event is received via a chat interface or another type of natural language interface supported by the HMI application. The chat interface can be invoked via appropriate interaction with the HMI application. The natural language request can describe virtually any type of event-driven action that the user wants the HMI application to perform. The request can describe the action to be performed and the event that triggers the execution of that action. For example, a user could submit a natural language prompt stating, "Record the tank pressure value when the batch temperature exceeds 115 degrees."

[0107] At step 804, a trained custom model or generative AI is used to analyze the request received in step 802 to determine whether sufficient information can be inferred from the request to determine the HMI configuration edit required to satisfy it; that is, whether the request contains enough information to identify the event, the corresponding action, and the data source that will be needed to detect the event and perform the action. The custom model can be trained using a training dataset representing a range of domain-specific industry knowledge. Example training data that can be used to train custom models includes, but is not limited to: information defining industry standards (e.g., global or vertical-specific safety standards, food and pharmaceutical standards, design standards such as ISA-88, etc.), technical details or design standards for various types of industrial control applications (e.g., batch control processes, die casting, valve control, agitator control, etc.), knowledge of specific industrial verticals (e.g., automotive, food and beverage, pharmaceutical, oil and gas, textile, mining, etc.), knowledge of industrial best practices, technical specifications for various types of industrial equipment or assets (e.g., industrial controllers, motor drives such as variable frequency drives, sensors, etc.), control design rules, customer-specific training data describing internal HMI design preferences or standards (e.g., preferred screen layout formats, preferred graphics types, preferred text fonts, etc.), design documents for control systems monitored by HMI applications, or other such training data. As part of the analysis, the system can also generate prompts and submit them to the generative AI model, and, where necessary, combine the analysis of the user's request with the generated natural language response tailored to the user, using the content of the generative AI model's response.

[0108] At 806, it is determined whether more information from the user is needed to determine the event-driven action requested by the user, and appropriate edits are applied to the HMI application that will configure the event-driven action. If additional information is needed (yes at step 806), the method proceeds to step 808, where the HMI system determines the required additional information and presents a natural language prompt designed to guide the user to provide the additional information. In determining the nature of the necessary additional information, the system may refer to industry knowledge encoded in the training model and responses from prompts from the generative AI model. At 810, a response to the prompt generated in step 808 is received via a chat interface.

[0109] Steps 806 to 810 are repeated as a natural language dialogue with the user until sufficient information has been obtained that can be translated into event-driven HMI actions. When no further information from the user is needed (no at step 806), the method proceeds to... Figure 8bThe second part, 800b, is shown. At 812, the HMI system determines an event-driven action requested by the user based on analysis of at least one of the user's natural language request and response obtained in steps 802 and 810, the content of the trained custom model, or the response prompted by a generative AI model. At 814, the HMI application is configured to perform an action in response to the detection of an event, based on the user's natural language request and any subsequent responses.

[0110] The embodiments, systems, and components described herein, as well as control systems and automation environments, that can implement the various aspects set forth in this specification may include computer or network components, such as those capable of interacting across networks: servers, clients, programmable logic controllers (PLCs), automation controllers, communication modules, mobile computers, onboard computers for mobile vehicles, wireless components, control components, etc. Computers and servers include one or more processors (electronic integrated circuits that perform logic operations using electrical signals), said one or more processors being configured to execute instructions stored in media such as random access memory (RAM), read-only memory (ROM), hard disk drives, and removable memory devices, said removable memory devices may include memory sticks, memory cards, flash drives, external hard disk drives, etc.

[0111] Similarly, the term PLC or automation controller as used herein can encompass functionality that can be shared across multiple components, systems, and / or networks. As an example, one or more PLCs or automation controllers can communicate and collaborate with various networked devices across a network. This can include virtually any type of controller, communication module, computer, input / output (I / O) device, sensor, actuator, and human-machine interface (HMI) communicating via a network, including control networks, automation networks, and / or public networks. PLCs or automation controllers can also communicate with and control a variety of other devices, including, for example, standard or safety-grade I / O modules such as analog modules, digital modules, programmable / intelligent I / O modules, other programmable controllers, communication modules, sensors, actuators, output devices, etc.

[0112] Networks can include public networks such as the Internet, intranets, and automation networks such as Control and Information Protocol (CIP) networks, which include device networks, control networks, security networks, and Ethernet / IP. Other networks include Ethernet, DH / DH+, remote I / O, fieldbus, Modbus, process fieldbus, CAN, wireless networks, serial protocols, etc. Additionally, network devices can include a wide variety of possibilities (hardware and / or software components). These include components such as switches with Virtual Local Area Network (VLAN) capabilities, LANs, WANs, agents, gateways, routers, firewalls, Virtual Private Network (VPN) devices, servers, clients, computers, configuration tools, monitoring tools, and / or other devices.

[0113] In order to provide context for the various aspects of the disclosed topic, Figure 9 and Figure 10 The following discussion is intended to provide a brief, general description of suitable environments in which the various aspects of the disclosed subject matter can be implemented. Although the various embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the various embodiments may also be implemented in combination with other program modules and / or implemented as a combination of hardware and software.

[0114] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the methods of this invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each of which can be operatively coupled to one or more associated devices.

[0115] The implementations shown herein can also be practiced in a distributed computing environment, where certain tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0116] Computing devices typically include various media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, these terms being used herein to distinguish themselves from each other. A computer-readable storage medium or a machine-readable storage medium can be any available storage medium accessible by a computer, and includes both volatile and non-volatile media, and both removable and non-removable media. By way of example and not limitation, a computer-readable storage medium or a machine-readable storage medium can be implemented using any method or technique for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0117] Computer-readable storage media may include, but are not limited to: random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc (BD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” as used herein for storage devices, memories, or computer-readable media should be understood as modifiers that exclude only the propagation of transient signals themselves, and do not waive the rights to all standard storage devices, memories, or computer-readable media that do not merely propagate transient signals themselves.

[0118] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example via access requests, queries or other data retrieval protocols, to perform various operations on the information stored on the media.

[0119] Communication media typically represent computer-readable instructions, data structures, program modules, or other structured or unstructured data as data signals (such as modulated data signals, carrier waves, or other transmission mechanisms), and include any information transmission or delivery medium. The term "modulated data signal" or signal refers to a signal that sets or alters one or more characteristics of its properties in a manner that encodes information in one or more signals. By way of example and not limitation, communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0120] Refer again Figure 9An example environment 900 for implementing various embodiments of the aspects described herein includes a computer 902, which includes a processing unit 904, system memory 906, and a system bus 908. The system bus 908 couples system components, including but not limited to the system memory 906, to the processing unit 904. The processing unit 904 can be any processor among various commercially available processors. A dual-microprocessor or other multiprocessor architecture may also be used as the processing unit 904.

[0121] System bus 908 can be any of several types of bus architectures, which can also use any of a variety of commercially available bus architectures to interconnect to memory buses (with or without memory controllers), peripheral buses, and local buses. System memory 906 includes ROM 910 and RAM 912. The Basic Input / Output System (BIOS) can be stored in non-volatile memory such as ROM, erasable programmable read-only memory (EPROM), or EEPROM, where the BIOS contains basic routines that facilitate the transfer of information between components within computer 902, for example, during startup. RAM 912 may also include high-speed RAM such as static RAM for caching data.

[0122] Computer 902 also includes an internal hard disk drive (HDD) 914 (e.g., EIDE, SATA), one or more external storage devices 916 (e.g., floppy disk drive (FDD) 916, memory stick or flash drive reader, memory card reader, etc.), and an optical disc drive 920 (e.g., capable of reading from or writing to CD-ROMs, DVDs, BDs, etc.). Although the internal HDD 914 is shown as being located within computer 902, it can also be configured for external use in a suitable rack (not shown). Additionally, although not shown in environment 900, a solid-state drive (SSD) may be used in addition to HDD 914, or an SSD may be used instead of HDD 914. HDD 914, external storage devices 916, and optical disc drive 920 can be connected to system bus 908 via HDD interface 924, external storage interface 926, and optical drive interface 928, respectively. The interface 924 for the external driver implementation may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external driver connectivity technologies are considered in the implementations described herein.

[0123] The drive and its associated computer-readable storage medium provide non-volatile storage of data, data structures, computer-executable instructions, etc. For computer 902, the drive and storage medium are adapted to store any data in a suitable digital format. Although the above description of computer-readable storage media refers to a corresponding type of storage device, those skilled in the art will understand that other types of computer-readable storage media may also be used in the example operating environment, whether such storage media are currently existing or to be developed in the future, and furthermore, any such storage medium may contain computer-executable instructions for performing the methods described herein.

[0124] Multiple program modules can be stored in the drive and RAM 912, including an operating system 930, one or more application programs 932, other program modules 934, and program data 936. All or part of the operating system, applications, modules, and / or data can also be cached in RAM 912. The systems and methods described herein can be implemented using various commercially available operating systems or combinations of operating systems.

[0125] Computer 902 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate a hardware environment for operating system 930, and the emulated hardware may optionally be compatible with... Figure 9 The hardware shown is different. In such an implementation, the operating system 930 may include one of a plurality of virtual machines (VMs) hosted at the computer 902. Furthermore, the operating system 930 may provide a runtime environment for the application 932, such as the Java Runtime Environment or the .NET Framework. A runtime environment is a consistent execution environment that enables the application 932 to run on any operating system that includes that runtime environment. Similarly, the operating system 930 may support containers, and the application 932 may be in the form of a container, which is a lightweight, standalone, executable software package that includes, for example, the application's code, runtime, system tools, system libraries, and setup.

[0126] Furthermore, the computer 902 can implement a security module such as a Trusted Processing Module (TPM). For example, using a TPM, the boot component hashes the boot component that is immediately following it in time and waits for the result to match a security value before loading the next boot component. This process can occur at any layer of the computer 902's code execution stack, for example, it can be applied at the application execution level or the operating system (OS) kernel level, thereby achieving security at any level of code execution.

[0127] Users can input commands and information into computer 902 through one or more wired / wireless input devices such as keyboard 938, touchscreen 940, and pointing devices such as mouse 918. Other input devices (not shown) may include microphones, infrared (IR) remote controls, radio frequency (RF) remote controls, or other remote controls, joysticks, virtual reality controllers and / or virtual reality headsets, gaming pads, styluses, image input devices (e.g., camera devices), gesture sensor input devices, visual motion sensor input devices, emotion or face detection devices, biometric input devices (e.g., fingerprint or iris scanners), etc. These and other input devices are typically connected to processing unit 904 via input device interface 942, which can be coupled to system bus 908, but may also be connected via other interfaces such as parallel ports, IEEE 1394 serial ports, gaming ports, USB ports, IR interfaces, BLUETOOTH® interfaces, etc.

[0128] The monitor 944 or other types of display devices can also be connected to the system bus 908 via an interface such as the video adapter 946. In addition to the monitor 944, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

[0129] Computer 902 can operate in a networked environment using logical connections via wired and / or wireless communications to one or more remote computers (e.g., remote computer 948). Remote computer 948 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer-to-peer device, or other general-purpose network node, and typically includes many or all of the elements described with respect to computer 902, although only memory / storage device 950 is shown for simplicity. The depicted logical connections include wired / wireless connections to a local area network (LAN) 952 and / or a larger network (e.g., a wide area network (WAN) 954). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks such as intranets, all of which can connect to global communication networks such as the Internet.

[0130] When used in a LAN networking environment, computer 902 can connect to local area network 952 via a wired and / or wireless communication network interface or adapter 956. Adapter 956 can facilitate wired or wireless communication to LAN 952, which may also include a wireless access point (AP) disposed thereon for communication with adapter 956 in wireless mode.

[0131] When used in a WAN networking environment, computer 902 may include modem 958, or may be connected to a communication server on WAN 954 via other means for establishing communication over WAN 954, such as via the Internet. Modem 958, which may be internal or external and may be a wired or wireless device, may be connected to system bus 908 via input device interface 942. In a networking environment, program modules described with respect to computer 902 or parts thereof may be stored in remote memory / storage device 950. It should be understood that the network connection shown is an example, and other means of establishing communication links between computers may be used.

[0132] When used in a LAN or WAN networking environment, in addition to the external storage device 916 described above, computer 902 can also access cloud storage systems or other network-based storage systems, or alternatively, computer 902 can access cloud storage systems or other network-based storage systems instead of the external storage device 916 described above. Typically, the connection between computer 902 and the cloud storage system can be established, for example, via adapter 956 or modem 958 through LAN 952 or WAN 954. When connecting computer 902 to the associated cloud storage system, external storage interface 926 can manage the storage provided by the cloud storage system, as if it were any other type of external storage, with the help of adapter 956 and / or modem 958. For example, external storage interface 926 can be configured to provide access to cloud storage sources as if these sources were physically connected to computer 902.

[0133] Computer 902 is operable to communicate with any wireless device or entity operatively arranged in wireless communication, such as a printer, scanner, desktop and / or portable computer, portable data assistant, communication satellite, any equipment or location associated with a wirelessly detectable tag (e.g., kiosk, newsstand, store shelf, etc.), and telephone. This can include Wi-Fi and BLUETOOTH® wireless technologies. Therefore, communication can be a predefined structure like a conventional network or simply self-organizing communication between at least two devices.

[0134] Figure 10This is a schematic block diagram of a sample computing environment 1000 with which the disclosed subject matter can interact. The sample computing environment 1000 includes one or more clients 1002. Clients 1002 can be hardware and / or software (e.g., threads, processes, computing devices). The sample computing environment 1000 also includes one or more servers 1004. Servers 1004 can also be hardware and / or software (e.g., threads, processes, computing devices). Servers 1004 can accommodate threads to perform transformations by employing one or more implementations, such as those described herein. One possible communication between clients 1002 and servers 1004 can be in the form of data packets suitable for transmission between two or more computer processes. The sample computing environment 1000 includes a communication framework 1006 that can be used to facilitate communication between clients 1002 and servers 1004. Clients 1002 are operatively connected to one or more client data storage devices 1008 that can be used to store local information of clients 1002. Similarly, server 1004 is operatively connected to one or more server data storage devices 1010 that can be used to store local information of server 1004.

[0135] The foregoing description includes examples of the invention. It is certainly impossible to describe every conceivable combination of components or methods for the purpose of describing the disclosed subject matter, but those skilled in the art will recognize that many other combinations and arrangements of the invention are possible. Therefore, the disclosed subject matter is intended to cover all such changes, modifications, and variations that fall within the spirit and scope of the appended claims.

[0136] In particular, with respect to the various functions performed by the components, devices, circuits, systems, etc., described above, unless otherwise indicated, the terminology used to describe such components (including references to "means") is intended to correspond to any component that performs the specified function of the described component (e.g., any functionally equivalent component), even if it is not structurally equivalent to the disclosed structure, which performs the functions of the exemplary aspects of the disclosed subject matter shown herein. In this regard, it will also be appreciated that the disclosed subject matter includes computer-readable media and systems having computer-executable instructions for actions and / or events of various methods for performing the disclosed subject matter.

[0137] Furthermore, while a particular feature of the disclosed subject matter may be disclosed only for one of several implementations, such features may be combined with one or more other features of other implementations, which may be desirable and advantageous for any given or particular application. Moreover, with regard to the use of the terms "includes" and "including" and their variations in the specification or claims, these terms are intended to be inclusive in a manner similar to the term "comprising."

[0138] In this application, the word "exemplary" is used to mean something used as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the word "exemplary" is intended to present the concept in a specific manner.

[0139] The various aspects or features described herein can be implemented as methods, apparatus, or articles of art using standard programming and / or engineering techniques. As used herein, the term "article of art" is intended to encompass any computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes…), optical discs [e.g., compact discs (CDs), digital versatile discs (DVDs)…], smart cards, and flash memory devices (e.g., cards, sticks, key drives…).

Claims

1. A system comprising: Memory that stores executable components; as well as A processor operatively coupled to the memory, the processor executing the executable component, the executable component comprising: A human-machine interface (HMI) deployment component is configured to deploy an HMI application to an HMI terminal for execution, wherein the HMI application presents a display screen including graphical objects on the HMI terminal, the graphical objects presenting data generated by an industrial automation system; A user interface component configured to receive natural language input describing an event-driven action to be performed by the HMI application via interaction with the HMI application; A generative artificial intelligence (AI) component configured to, in response to receiving the natural language input, formulate prompts for a generative AI model and formulate edits configuring the HMI application to perform the event-driven action, the prompts being designed to obtain responses from the generative AI model including information used by the generative AI component to determine the event-driven action based on the natural language input, wherein the prompts are generated based on analysis of the natural language input and one or more custom models trained with training data; and An HMI generation component is configured to apply the edits to the HMI application.

2. The system according to claim 1, wherein, The event-driven action is at least one of the following: changing the setpoint of the monitored industrial process; writing a specified data value to a controller data tag; changing the current operating mode of the machine or process; recording the value of a specified key performance indicator; or recording a description of the event that triggered the event-driven action. Present the designated display screen of the HMI application; Present a customized message; or change the properties of a specified graphic object within the graphic object.

3. The system according to claim 1, wherein, The natural language input describes the event used to trigger the event-driven action, and the event is at least one of the following: a machine or device transitions to a specified state; a specified key performance indicator of a monitored industrial process transitions outside a defined range; a defined production target is met; a specified machine malfunction occurs; or an industrial safety device trips.

4. The system according to claim 1, wherein, The generative AI component is configured to respond to determining the natural language input describing the event-driven action: Determine the event specified by the natural language input; Identify one or more data sources that indicate whether the event has occurred; as well as The editor is configured to configure the HMI application to monitor one or more data sources and to perform the event-driven action in response to an indication from one or more data sources that an event has occurred.

5. The system according to claim 1, wherein, The natural language input is the first natural language input. The user interface component is also configured to receive second natural language input via interaction with the HMI application, the second natural language input describing a request to display activity or historical alarm status corresponding to at least one of a specified aspect or a specified time period of the industrial automation system. The generative AI component is also configured to respond to receiving the second natural language input: Identify the alarm corresponding to the request; Identify the data source that indicates the corresponding current state of the alarm; and A new display screen is generated, which includes a graphical representation of the alarm linked to the data source.

6. The system according to claim 1, wherein, The natural language input is the first natural language input. The user interface component is also configured to receive second natural language input via interaction with the HMI application, the second natural language input describing a request to display a selected subset of historical operational data collected from the industrial automation system, and The generative AI component is also configured to respond to receiving the second natural language input: Identify the historical operation data corresponding to the request; Identify the data source corresponding to the historical operation data; as well as A new display screen is generated, which presents a selected subset of the historical operation data retrieved from the data source.

7. The system according to claim 6, wherein, The second natural language input also describes the format in which a subset of the historical operation data is to be presented, and The generative AI component is configured to format a selected subset of the historical operation data on a new display screen according to the format indicated by the second natural language input.

8. The system according to claim 1, wherein, The training data includes at least one of the following: information defining industry standards; technical details for the corresponding type of industrial control application; knowledge of different industrial verticals; information describing industry best practices; technical specifications for different types of industrial devices or machines; control design rules; sample HMI display layouts for the corresponding type of control application; customer-specific training data describing internal HMI design preferences; or design documents for the industrial automation system.

9. The system according to claim 1, wherein, The generative AI component is also configured to determine additional information based on the analysis of the natural language input, which would enable the generative AI component to determine the event-driven action: Generate a natural language response that prompts the additional information; The natural language response is presented via the user interface component; and The editing process is further developed based on the analysis of the additional information.

10. The system according to claim 9, wherein, The generative AI component is also configured to retrain one or more custom models based on the edits, the natural language input, and the additional information.

11. A method comprising: A human-machine interface (HMI) application is deployed to an HMI terminal for execution by a system including a processor, wherein the HMI application presents a display screen including graphical objects on the HMI terminal, the graphical objects presenting data generated by the industrial automation system; The system receives natural language input describing the event-driven actions to be performed by the HMI application via interaction with the HMI application; In response to receiving the natural language input, the system formulates a prompt for a generative artificial intelligence (AI) model, the prompt being designed to obtain a response from the generative AI model including information used by the system to determine the event-driven action, wherein the prompt is generated based on an analysis of the natural language input and one or more custom models trained with training data; The system formulates an edit to configure the HMI application to perform the event-driven action, wherein the formulation includes: formulating the edit based on the natural language input, the one or more customized models, and the response from the generative AI model; and The system applies the edits to the HMI application.

12. The method according to claim 11, wherein, The event-driven action is at least one of the following: changing the setpoint of the monitored industrial process; writing a specified data value to a controller data tag; changing the current operating mode of the machine or process; recording the value of a specified key performance indicator; or recording a description of the event that triggered the event-driven action. Present the designated display screen of the HMI application; Present a customized message; or change the properties of a specified graphic object within the graphic object.

13. The method according to claim 11, wherein, The natural language input describes the event used to trigger the event-driven action, and the event is at least one of the following: a machine or device transitions to a specified state; a specified key performance indicator of a monitored industrial process transitions outside a defined range; a defined production target is met; a specified machine malfunction occurs; or an industrial safety device trips.

14. The method according to claim 11, wherein, The editing process includes responding to determining the natural language input describing the event-driven action: The system infers the event specified by the natural language input; The system infers one or more data sources indicating whether the event has occurred; as well as The system formulates the editor, which is used to configure the HMI application to monitor one or more data sources and to perform the event-driven action in response to indications from one or more data sources that an event has occurred.

15. The method according to claim 11, wherein, The natural language input is the first natural language input, and The method further includes: The system receives second natural language input via interaction with the HMI application. This second natural language input describes a request to display activity or historical alarm status corresponding to at least one of a specified aspect or a specified time period of the industrial automation system. In response to receiving the second natural language input: The system identifies the alarm corresponding to the request; The system identifies the data source that indicates the corresponding current state of the alarm; and The system generates a new display screen, which includes a graphical representation of the alarm linked to the data source.

16. The method according to claim 11, wherein, The natural language input is the first natural language input, and The method further includes: The system receives a second natural language input via interaction with the HMI application, the second natural language input describing a request to display a selected subset of historical operational data collected from the industrial automation system, and In response to receiving the second natural language input: The system identifies the historical operation data corresponding to the request; The system identifies the data source corresponding to the historical operation data; and The system generates a new display screen that presents a selected subset of the historical operation data retrieved from the data source.

17. The method according to claim 16, wherein, The second natural language input also describes the format in which a subset of the historical operation data is to be presented, and The generation of the new display screen includes: formatting a selected subset of the historical operation data on the new display screen according to the format indicated by the second natural language input.

18. A non-transitory computer-readable medium having instructions stored thereon, the instructions being responsive to execution to cause a system including a processor to perform operations, the operations comprising: The human-machine interface (HMI) application is deployed to an HMI terminal for execution, wherein the HMI application presents a display screen including graphical objects on the HMI terminal, and the graphical objects present data generated by the industrial automation system; Through interaction with the HMI application, natural language input describing the event-driven actions to be performed by the HMI application is received; In response to receiving the natural language input, a prompt is formulated for a generative artificial intelligence (AI) model, the prompt being designed to obtain a response from the generative AI model including information used by the system to determine the event-driven action, wherein the prompt is generated based on an analysis of the natural language input and one or more custom models trained with training data; Based on the natural language input, the one or more custom models, and the response from the generative AI model, the HMI application is configured to perform the event-driven action.

19. The non-transitory computer-readable medium according to claim 18, wherein, The event-driven action is at least one of the following: changing the setpoint of the monitored industrial process; writing a specified data value to a controller data tag; changing the current operating mode of the machine or process; recording the value of a specified key performance indicator; or recording a description of the event that triggered the event-driven action. Present the designated display screen of the HMI application; Present a customized message; or change the properties of a specified graphic object within the graphic object.

20. The non-transitory computer-readable medium according to claim 18, wherein, The natural language input describes the event used to trigger the event-driven action, and the event is at least one of the following: a machine or device transitions to a specified state; a specified key performance indicator of a monitored industrial process transitions outside a defined range; a defined production target is met; a specified machine malfunction occurs; or an industrial safety device trips.

Images