Contextualized HMI synthesis using hydrated large language models in process automation

Abstract

Methods for generating a human-machine interface (HMI) (208) and corresponding systems (100) and computer-readable mediums (126). A method includes receiving (310) a piping and instrumentation diagram (P&ID) (414) for which an HMI is to be generated. The method includes performing HMI synthesis (314) based on the P&ID (414) and a hydrated large language model (LLM) (412). The method includes producing (316) the HMI (208) corresponding to the P&ID (414). The method includes storing (320) the HMI (208) for controlling a physical system represented by the P&ID (414).

Description

202410332 CONTEXTUALIZEDHMI SYNTHESISUSINGHYDRATEDLARGELANGUAGEMODELS INPROCESSAUTOMATIONTECHNICAL FIELD

[0001] The present disclosure is directed, in general, to systems and methods for generating Human-Machine Interfaces (HMI) to allow users to interact with a machine, system, or device, in particular but not limited to HMI in process automation systems. BACKGROUND OF THE DISCLOSURE

[0002] Design of effective and efficient HMI can be crucial for providing a way for operators and other users to work with automation systems and other systems. HMI synthesis refers to the process of designing, developing, and generating interfaces that allow humans to interact with machines or systems effectively. This process can involve creating graphical user interfaces, dashboards, control panels, and other interaction modalities that enable users to operate, monitor, and control machines or systems.

[0003] One of the key challenges when configuring, assembling, or repurposing automation system equipment is programming the low-level controls, e.g., sensors and actuators, of the machines to accomplish the processing goal. The process of breaking down requirements to setting up a process plan, programming, testing and validation of automation system requires a certain level of expertise and can also take significant amount of time depending on its complexity. Current approaches rely on manual programming of individual devices and systems. Improved systems are desirable.202410332 SUMMARY OF THE DISCLOSURE

[0004] Various disclosed embodiments include methods for generating a human-machine interface (HMI) and corresponding systems and computer-readable mediums. A method includes receiving a piping and instrumentation diagram (P&ID) for which an HMI is to be generated. The method includes performing HMI synthesis based on the P&ID and a hydrated large language model (LLM). The method includes producing the HMI corresponding to the P&ID. The method includes storing the HMI for controlling a physical system represented by the P&ID.

[0005] Various embodiments also include receiving an initial LLM, training the initial LLM with P&ID symbology, producing a trained LLM based on the initial LLM and the training, and hydrating the trained LLM to produce the hydrated LLM.

[0006] In various embodiments, the HMI synthesis includes employing multimodal prompts to generate a unified task description for the HMI. In various embodiments, the HMI synthesis includes creating layouts, visual elements, and controls that users interact with on screens or panels. In various embodiments, the HMI synthesis includes evaluating and customizing user interface (UI) and user experience (UX) considerations. In various embodiments, the HMI synthesis includes functionality integration to incorporate the necessary functionalities and features into the HMI to support the operations and control of a system represented by the P&ID. In various embodiments, the HMI synthesis includes generating executable code and display code for the HMI.

[0007] In various embodiments, the HMI synthesis includes interpreting a user prompt using the hydrated LLM, extracting geometric parameters using the P&ID and the hydrated LLM, and generating GUI objects and underlying code for the HMI using the hydrated LLM.

[0008] Various embodiment also include determining whether to revise the HMI; and when the computer system determines that the HMI should be revised, then receiving user prompts to refine the hydrated LLM and the HMI, using the received user prompts to re- hydrate the LLM, and repeating the performing the HMI synthesis.202410332

[0009] Disclosed embodiments also include a computer system comprising a processor and an accessible memory, particularly configured to perform processes as disclosed herein. Disclosed embodiments also include a non-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more computer systems to perform processes as disclosed herein.

[0010] The foregoing has outlined rather broadly the features and technical advantages of the present disclosure so that those skilled in the art may better understand the detailed description that follows. Additional features and advantages of the disclosure will be described hereinafter that form the subject of the claims. Those skilled in the art will appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the art will also realize that such equivalent constructions do not depart from the spirit and scope of the disclosure in its broadest form.

[0011] Before undertaking the DETAILED DESCRIPTION below, it may be advantageous to set forth definitions of certain words or phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or” is inclusive, meaning and / or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, whether such a device is implemented in hardware, firmware, software or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words and phrases. While some terms may include a wide variety of embodiments, the appended claims may expressly limit these terms to specific embodiments.202410332 BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a more complete understanding of the present disclosure, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like numbers designate like objects, and in which:

[0013] FIG. 1 illustrates a block diagram of a computer system in which an embodiment can be implemented;

[0014] FIG. 2 illustrates an example of HMI synthesis using P&ID and a contextualized LLM instance in accordance with disclosed embodiments;

[0015] FIG. 3 illustrates a process in accordance with disclosed embodiments; and

[0016] FIG. 4 illustrates an example of a process for HMI synthesis in accordance with disclosed embodiments.202410332 DETAILED DESCRIPTION

[0017] FIGS.1 through 4, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged device. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

[0018] Disclosed embodiments enable the constrained generation of HMI for plant automation and other systems using technical data to generate an HMI using generative artificial intelligence (AI) and a large language model (LLM). In a specific example, disclosed embodiments can use a Piping and Instrumentation Diagram (P&ID), which is a graphical representation of a process automation system. Disclosed embodiments can use the P&ID of a physical plant to define the boundary conditions of the generative model.

[0019] HMI synthesis typically involves user-centered design that focuses on the needs, preferences, and capabilities of the end-users to ensure the interface is intuitive and easy to use, visualization that creates visual representations of data and system states, such as charts, graphs, and indicators, to facilitate understanding and decision-making, and interaction design that defines how users will interact with the system through buttons, touch screens, voice commands, and other input methods. HMI synthesis can also include usability testing to evaluate the interface with real users to identify and address usability issues, ensuring the interface is effective and efficient. The goal of HMI synthesis is to create interfaces that are user-friendly, informative, and responsive, enhancing the overall user experience and improving the efficiency and safety of human-machine interactions.

[0020] HMI Synthesis is a common problem in plant design and automation. However, HMI design optimization for productivity improvement, as with any user interface (IU) or user experience (UX) optimization problem, is described by more obscure criteria. Regardless of what the specifics are, engineers require an intuitive way to solve both problems in a complementary way especially for HMI synthesis in process automation.202410332

[0021] HMI synthesis for plant automation is often ad-hoc, but almost always depends on a process automation engineer’s background in factory automation and the complexity of the P&ID on which the plant is based. In some cases, the HMI display mirrors the P&ID, and the process engineer uses a software tool to adopt the geometry of the control elements in the P&ID as UI / UX elements on the HMI screen as buttons, indicators etc.

[0022] Disclosed HMI synthesis processes can recognize, encapsulate, and process polygons, conic sections, and line synthesis, and other elements in two dimensions (2D) and their corresponding analogs in three dimensions (3D). Constraints can imposed on the emerging geometric shape according to the particular industrial context, which can also informs factors such as the physics of the environment, including flow physics, condense matter physics, thermodynamics, the measurements required for automation within the industrial process like fluid volume within the realized geometric shapes, viscosity, density, pressure, flow rate, etc. These factors than are used to identify the apparatus required to acquire such measurements, for example, flow meter, pressure transducer, etc., all of which matters greatly to the control problem managed by an industrial HMI system.

[0023] Disclosed embodiments enable the constrained generation of HMI for plant automation using the plant P&ID to define the boundary conditions of the generative model. The “plant” can represent any physical facility for which a P&ID is available. The actual geometry of the UI elements within the target HMI, in various embodiments, is defined by prompts to the LLM. In this model, the bi-directional interaction between the LLM and the user bi-directional interaction is also initialized by the context provided by the specialized instance of the LLM. The system can then automatically create the HMI based on the LLM, the P&ID, and the user inputs.

[0024] FIG. 1 illustrates a block diagram of a computer system in which an embodiment can be implemented, for example as a computer system particularly configured by software or otherwise to perform the processes as described herein, and in particular as each one of a plurality of interconnected and communicating systems as described herein. The computer system depicted includes a processor 102 connected to a level two cache / bridge 104, which is connected in turn to a local system bus 106. Local system bus 106 may be,202410332 for example, a peripheral component interconnect (PCI) architecture bus. Also connected to local system bus in the depicted example are a main memory 108 and a graphics adapter 110. The graphics adapter 110 may be connected to display 111.

[0025] Other peripherals, such as local area network (LAN) / Wide Area Network / Wireless (e.g. WiFi) adapter 112, may also be connected to local system bus 106. Expansion bus interface 114 connects local system bus 106 to input / output (I / O) bus 116. I / O bus 116 is connected to keyboard / mouse adapter 118, disk controller 120, and I / O adapter 122. Disk controller 120 can be connected to a storage 126, which can be any suitable machine usable or machine readable storage medium, including but not limited to nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), magnetic tape storage, and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs), and other known optical, electrical, or magnetic storage devices.

[0026] Storage 126 can store any data that may be necessary or useful for performing processes as described herein, including (but not limited to) executable code 152, AI system(s) 154, LLM(s) 156, Piping and Instrumentation Diagrams 158, HMIs 160 (including any supporting executable, display, PLC, or interface code), user inputs and prompts 162, training information and materials 164, and other data 166.

[0027] Also connected to I / O bus 116 in the example shown is audio adapter 124, to which speakers (not shown) may be connected for playing sounds. Keyboard / mouse adapter 118 provides a connection for a pointing device (not shown), such as a mouse, trackball, trackpointer, touchscreen, etc.

[0028] Those of ordinary skill in the art will appreciate that the hardware depicted in FIG. 1 may vary for particular implementations. For example, other peripheral devices, such as an optical disk drive and the like, also may be used in addition or in place of the hardware depicted. The depicted example is provided for the purpose of explanation only and is not meant to imply architectural limitations with respect to the present disclosure.202410332

[0029] A computer system in accordance with an embodiment of the present disclosure includes an operating system employing a graphical user interface. The operating system permits multiple display windows to be presented in the graphical user interface simultaneously, with each display window providing an interface to a different application or to a different instance of the same application. A cursor in the graphical user interface may be manipulated by a user through the pointing device. The position of the cursor may be changed and / or an event, such as clicking a mouse button, generated to actuate a desired response.

[0030] One of various commercial operating systems, such as a version of Microsoft Windows™, a product of Microsoft Corporation located in Redmond, Wash. may be employed if suitably modified. The operating system is modified or created in accordance with the present disclosure as described.

[0031] LAN / WAN / Wireless adapter 112 can be connected to a network 130 (not a part of computer system 100), which can be any public or private computer system network or combination of networks, as known to those of skill in the art, including the Internet. Computer system 100 can communicate over network 130 with server system 140, which is also not part of computer system 100, but can be implemented, for example, as a separate computer system 100.

[0032] FIG. 2 illustrates an example of HMI synthesis using P&ID and a contextualized LLM instance in accordance with disclosed embodiments. Here, the LLM employs multimodal prompts to generate a unified task description. In this example case, a user 204 uploads a P&ID diagram 202 to a fine-tuned LLM 206 with the capacity for image synthesis as shown in FIG 2, which produces the HMI interface 208 corresponding to P&ID diagram 202.

[0033] Stated differently, a computer system 100 that implements an LLM 206 receives a P&ID diagram 202. Based on the P&ID diagram 202 and the LLM 206, the computer system 100 generates an HMI interface 208 using techniques as disclosed herein.202410332

[0034] In this example, the computer system 100 operates LLM 206 on the assumption that the control elements on the P&ID are of three broad classes, including indicators (which can include any status device such as a lamp, a gauge, or otherwise), buttons (including similar control elements such as dials, sliders, switched, etc.), and exclusions.

[0035] An “exclusion list” can be received by the computer system 100 as a text prompt from a user, alongside other functional metadata to “hydrate” the two-dimensional (2D) 2- D P&ID image. In particular, the computer system can identify elements of the P&ID as indicators and buttons, and exclude elements that are on the exclusion list. Then the system can hydrate the P&ID image to apply appropriate metadata to the indicators and buttons for generating appropriate UI / UX interface code for each of these.

[0036] In the context of creating an LLM or for operational initialization, “hydration” generally refers to the process of loading or initializing the model with the necessary data and parameters it requires to function. This can include loading pre-trained weights to transferring learned parameters from storage (e.g., disk) into memory, setting up the environment to initialize any required libraries, dependencies, or computational resources needed for the model to operate, and feeding initial data so that the LLM has the initial data or context needed to start processing. Hydration ensures that the model is fully operational and ready to perform tasks such as generating text, answering questions, or engaging in conversation.

[0037] Hydration for an existing LLM refers to the process of enriching an LLM with specific context or additional data to improve its performance for particular tasks, sometimes called contextual enrichment. This can involve contextual data injection, which provides the model with relevant information or context to tailor its responses. For instance, giving background information about a specific topic before querying the model. Hydration can also involve fine-tuning to train the model further on a specialized dataset to enhance its understanding and performance in a specific domain. Hydration can also involve memory augmentation to implement mechanisms to allow the model to recall previous interactions or facts, thereby making its responses more coherent and contextually202410332 appropriate. Hydration can make the LLM more accurate, relevant, and useful for the given application or interaction.

[0038] The hydration process as used in disclosed embodiments offers a simplification that allows the LLM to simply solve a binary classification problem.

[0039] FIG.3 illustrates a process 300 in accordance with disclosed embodiments that can be performed, for example, by one or more computer systems 100 (referred to generically as the “system” below).

[0040] At 302, the system receives an initial large language model with image synthesis. “Receiving,” as used herein, can include loading from storage, receiving from another device or process, receiving via an interaction with a user, or otherwise. In particular, 302 can include creating an LLM with image synthesis if one is not already created.

[0041] At 304, the system can train the initial LLM, if necessary, with P&ID symbology, control function details, and other information about the elements represented in a P&ID, how and in what ways each element can be controlled, what code, signals, and interfaces are required to control each element, how each element can interact with each others, which elements are “inert” in terms of control, which elements cannot be controlled but can deliver information, and other relevant information.

[0042] At 306, the system can fine-tune the initial model with respect to specific project- or plant-relevant data for the specific environment, systems, or other details that may be represented in the particular P&IDs being processed. This can be done, for example, using the COMOS family of software products by Siemens.

[0043] At 308, the system produces a trained LLM with multimodality, for example based on the initial LLM, the training at 304, and / or the fine-tuning at 306. The multimodal LLM, in this context, means that the LLM can “understand” and process at least both textual prompts and inputs and image-based prompts and inputs, including in particular P&IDs which may be in image form (including portable document (PDF) format, image formats, and others).202410332

[0044] At 310, the system receives a P&ID for which an HMI is to be generated.

[0045] At 312, the system hydrates the trained LLM. This can include hydrating the trained LLM with the P&ID, particularly on an initial pass. This can include hydrating the trained LLM with any other information that is helpful for interpreting the specific P&ID or for producing an HMI corresponding to the P&ID, including user prompts, system requirements, interface requirements, and other requirements, preferences, or information.

[0046] At 314, the system performs HMI synthesis based on the P&ID and the hydrated trained LLM. As part of 314, the system can employ multimodal prompts to generate a unified task description for the HMI to be produced. Step 314 can include designing the HMI that corresponds to the P&ID, which may include creating layouts, visual elements, and controls that users interact with on screens or panels. Step 314 can include evaluating and customizing UI / UX considerations, such as ensuring the interface is intuitive, easy to understand, and efficient for users to accomplish their tasks. Step 314 can include functionality integration to incorporate the necessary functionalities and features into the interface design to support the operations and control of the plant, process control system, industrial system, or other system represented by the P&ID. Step 314 can include generating any necessary executable or display code necessary or useful for implementing the HMI.

[0047] The HMI synthesis can include recognizing each element present in the P&ID and determining whether any control element or other element presented is not on the exclusion list and is a button, an indicator, or otherwise. As part of 314, the system can use Chain of Thought (CoT) reasoning to perform such processes as solving color scheme problems using internalized information of P&ID symbology and the associated function alongside the functional information provided in the metadata during the hydration step to create local indicator and control color groups.

[0048] HMI synthesis can include generation of GUI objects (and underlying code), where elements are drawn from a distribution defined by a set of examples. In various embodiments, the elements that are drawn are geometric shapes used as symbols for control or display within an HMI. The manner in which these elements are drawn can include the202410332 use of a generative model, such as a Variational Auto Encoder (VAE) or a Large Language Model (LLM). A VAE can have both an encoder and a decoder. The encoder can include full connected layers compressing the geometric vertices into latent variables that will later be used by the decoder to reconstruct the geometric object.

[0049] FIG. 4 illustrates an example of a process for HMI synthesis in accordance with disclosed embodiments, usable in particular for unit symbol synthesis.

[0050] At 402, the system interprets the user prompt 410 using LLM 412. Here, the system uses LLM 412 to interpret the input prompt and identify the type of geometric object and relevant parameters. The system can assign a prompt to every element in an associated P&ID 414 for the plant that the HMI is intended / designed to control.

[0051] At 404, the system extracts geometric parameters using the P&ID 414 and the LLM 412.

[0052] For example, in a 2D case, the geometric parameter extraction can include: • For polygons, extracting the number of sides and possibly vertex coordinates. • For conic sections, extracting parameters such as the center, axes lengths, or coefficients. • For lines, extracting the slope, intercept, or coefficients in the standard form.

[0053] In various embodiments, a P&ID 414 can use mathematical definitions as disclosed herein. For example, the components of a P&ID can be represented as follows: • Pipes and connections can be represented as lines or polylines. • Instruments and control devices can be represented as symbols or shapes (e.g., circles, rectangles). • Valves can be represented as various symbols, often as geometric shapes like diamonds or ellipses.202410332 • Process equipment can be represented as complex shapes (e.g., tanks, pumps) often composed of basic geometric shapes. • Text annotations can be represented as text with specific positions.

[0054] Each of these can have a corresponding mathematical representation. For example, the each of the components can be represented as follows: • For pipes and connections (lines / polylines): oRepresent pipes as line segments: ^^ = ^(^^^, ^^^), (^^^, ^^^)^o Polylines are sequences of connected line segments: ^^= ^(^^^, ^^^), (^^^, ^^^), ... , (^^^, ^^^)^• For instruments and control devices (symbols / shapes): oRepresent as circles: ^^ = ^(^^, ^^), ^^^o Represent as rectangles as grouped line segments. • For valves (geometric shapes): oRepresent as diamonds: ^^ = ^(^^, ^^), ^^, ℎ^^o Represent as ellipses: ^ = ^(^^, ^^), ^^^, ^^^• For process equipment (complex shapes): o Tanks: represent as combinations of rectangles and ellipses. o Pumps: represent as combinations of circles and rectangles. • For text annotations: oRepresent as text with position: ^^ = ^^^^^^^, (^^, ^^)^202410332

[0055] The entire P&ID can be represented as a graph where nodes represent components (instruments, valves, equipment) and edges represent pipes. For such a graph representation, the system can use: •^ = (^, ^) where ^ is the set of vertices (components) and ^ is the set of edges(pipes). •Each vertex ^ ∈ ^ has associated geometric properties.

[0056] For vertex properties, each vertex v can be represented as: •^ = (^^!^, !^^^"^^^^#)• For example, an instrument might be ^ = ($^^$%^, (^, ^, ^)), and a valve might be^ = (&^^"'(&, (^, ^, ^, ℎ)).

[0057] For edge properties, each edge ^ ∈ ^ can be represented as a connection betweentwo vertices: •^ = )^^ , ^*+• can be represented as a polyline ^,= ^(^,^, ^,^), (^,^, ^,^), ... , (^,^, ^,^)^

[0058] As one non-limiting example, extracting a component from the P&ID for HMI synthesis could use prompts like the following: • Instrument: Circle at (2, 3) with radius 1 •EXTRACTED: ^^ = ^(2,3), 1^• Pipe: Line from (2, 3) to (5, 6) •EXTRACTED: ^^ = ^(2,3), (5,6)^202410332

[0059] At 406, the system generates GUI objects and underlying code using the LLM 412. The extracted geometric components of the P&ID can then be used to generate the geometric shapes on the HMI layout. Object generation can include using mathematical formulations to generate the corresponding objects, as described above. These can include: • For polygons, computing vertex coordinates to form a closed path. • For conic sections, generating points satisfying the conic section equations. • For lines, computing points lying on the line using the slope-intercept form or standard form.

[0060] The system can then modify these geometric GUI shapes in the HMI in terms of color, text annotation, or other control or display augmentation.

[0061] Returning to the process of FIG. 3, at 316, the system produces the HMI corresponding to the P&ID, which can include the visible GUI objects and any necessary executable or display code, including but not limited to programmable logic controller (PLC) code. This can include actually displaying the HMI to a user for testing and interaction and can include executing the HMI in either a real-world or virtual environment to ensure that the HMI can correctly interact with the system represented by the P&ID.

[0062] At 318, the system can determine whether to revise the HMI. This can include receiving user feedback regarding the HMI or can be based on the testing, interaction, or execution that has been performed.

[0063] If the HMI does not need to be revised, at 320, the system can store the HMI, execute and display it for operation to control the system represented by the P&ID, transmit it to another device or process, or otherwise.

[0064] If the HMI should be revised, at 322, the system can receive user prompts or other inputs to refine the trained LLM and the desired HMI. The inputs can include user prompts, whether textual or in some other form supported by the multimodal LLM, images, specifications, or other data regarding the P&ID or the system represented by the P&ID,202410332 information regarding the requirements of the HMI, or other information. For example, the system can use the received user text prompts to re-hydrate the LLM and also provide specific instruction on the UI / UX modification.

[0065] The system can then return to 312 to further hydrate the trained model using the received inputs. This hydration improves general design principles and corrects wrong global assumptions whereas specific instructions address specialized UI / UX element rendering issues.

[0066] Of course, those of skill in the art will recognize that, unless specifically indicated or required by the sequence of operations, certain steps in the processes described above may be omitted, performed concurrently or sequentially, or performed in a different order. In particular, as described herein, the process can include automatically generating and simulating the machine program based on the user prompt, so that the user can review the simulation(s) directly rather than providing human feedback at each step.

[0067] Those skilled in the art will recognize that, for simplicity and clarity, the full structure and operation of all computer systems suitable for use with the present disclosure is not being depicted or described herein. Instead, only so much of a computer system as is unique to the present disclosure or necessary for an understanding of the present disclosure is depicted and described. The remainder of the construction and operation of computer system 100 may conform to any of the various current implementations and practices known in the art.

[0068] It is important to note that while the disclosure includes a description in the context of a fully functional system, those skilled in the art will appreciate that at least portions of the mechanism of the present disclosure are capable of being distributed in the form of instructions contained within a machine-usable, computer-usable, or computer-readable medium in any of a variety of forms, and that the present disclosure applies equally regardless of the particular type of instruction or signal bearing medium or storage medium utilized to actually carry out the distribution. Examples of machine usable / readable or computer usable / readable mediums include: nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories202410332 (EEPROMs), and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs).

[0069] Although an exemplary embodiment of the present disclosure has been described in detail, those skilled in the art will understand that various changes, substitutions, variations, and improvements disclosed herein may be made without departing from the spirit and scope of the disclosure in its broadest form.

[0070] None of the description in the present application should be read as implying that any particular element, step, or function is an essential element which must be included in the claim scope: the scope of patented subject matter is defined only by the allowed claims. Moreover, none of these claims are intended to invoke 35 USC §112(f) unless the exact words “means for” are followed by a participle. The use of terms such as (but not limited to) “mechanism,” “module,” “device,” “unit,” “component,” “element,” “member,” “apparatus,” “machine,” “system,” “processor,” or “controller,” within a claim is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or enhanced by the features of the claims themselves, and is not intended to invoke 35 U.S.C. §112(f).

Claims

202410332 WHAT IS CLAIMED IS:

1. A method (300) performed by a computer system (100), comprising: receiving (310) a piping and instrumentation diagram (P&ID) (414) for which a human-machine interface (HMI) (208) is to be generated; performing HMI synthesis (314) based on the P&ID (414) and a hydrated large language model (LLM) (412); producing (316) the HMI (208) corresponding to the P&ID (414); and storing (320) the HMI (208) for controlling a physical system represented by the P&ID (414).

2. The method of claim 1, further comprising: receiving (302) an initial LLM (412); training (304) the initial LLM (412) with P&ID symbology; producing (308) a trained LLM (412) based on the initial LLM (412) and the training (304); and hydrating (312) the trained LLM (412) to produce the hydrated LLM (412).

3. The method of claim 1, wherein the HMI synthesis (314) includes employing multimodal prompts to generate a unified task description for the HMI (208).

4. The method of claim 1, wherein the HMI synthesis (314) includes creating layouts, visual elements, and controls that users interact with on screens or panels.

5. The method of claim 1, wherein the HMI synthesis (314) includes evaluating and customizing user interface (UI) and user experience (UX) considerations.

6. The method of claim 1, wherein the HMI synthesis (314) includes functionality integration to incorporate the necessary functionalities and features into the HMI202410332 (208) to support the operations and control of a system represented by the P&ID (414).

7. The method of claim 1, wherein the HMI synthesis (314) includes generating executable code and display code for the HMI (208).

8. The method of claim 1, wherein the HMI synthesis (314) includes: interpreting a user prompt using the hydrated LLM (412); extracting geometric parameters using the P&ID (414) and the hydrated LLM (412); and generating GUI objects and underlying code for the HMI using the hydrated LLM (412).

9. The method of claim 1, further comprising: determining whether to revise the HMI (208); and when the computer system determines that the HMI (208) should be revised, then receiving user prompts (322) to refine the hydrated LLM (412) and the HMI (208), using the received user prompts to re-hydrate the LLM (412), and repeating the performing the HMI synthesis (314).

10. A computer system (100) comprising a processor (102) and an accessible memory (108), the computer system (100) particularly configured to perform a process as in any of claims 1-9.

11. A non-transitory computer-readable medium (126) encoded with executable instructions that, when executed, cause one or more computer systems (100) to perform a process as in any of claims 1-9.

Images