Method for industrial plant operation

Abstract

Embodiments of the present disclosure relate to methods for industrial plant operation. The present disclosure provides a computer-implemented method for industrial plant operation, the method comprising obtaining image data comprising a plurality of frames of a plant monitoring and control graphical user interface, GUI, for an industrial plant, inputting the image data into a multi-modal neural generative model, the multi-modal neural generative model generating a first output based on the image data, the first output comprising an analysis of changes to the industrial plant operation derived from the plurality of frames, including a current frame and a previous frame, thereby providing an operational context for a current state of the industrial plant, the multi-modal neural generative model generating a second output based on the first output and optionally the image data, the second output comprising an analysis of the current state of the industrial plant taking into account the operational context.

Description

Technical Field

[0001] This disclosure provides a computer-implemented method for operating an industrial plant, a corresponding data processing system, a computer program product, and a computer-readable medium. Background Technology

[0002] Monitoring the technical equipment in industrial plants is often partially performed by the user. For example, there may be situations in industrial plant operations where problems cannot be resolved automatically. In these cases, the user (also known as the operator) will be prompted to intervene. Typically, a graphical user interface (GUI) is presented and output to the user, showing process and equipment status, such as signal levels, and is configured to also indicate alarms.

[0003] In many cases, industrial plants are complex and may issue several alerts simultaneously. Some alerts may be more urgent than others and require more immediate user attention; however, this may not be immediately apparent from the graphical user interface. Some users may be able to address this to some extent based on knowledge and experience. However, this is unreliable, making it impossible to reliably ensure monitoring of plant operations. Therefore, plant operations are likely to be suboptimal and potentially risky.

[0004] Therefore, the purpose of this disclosure is to provide a method and system that allows for the resolution of these challenges, and in particular, allows for more reliable and / or improved operation of industrial plants. Summary of the Invention

[0005] This disclosure provides methods, data processing systems, computer program products, and computer-readable media, as stated in the independent statement.

[0006] This disclosure provides a computer-implemented method for operating an industrial plant.

[0007] The method includes acquiring image data of one or more frames, particularly multiple frames, of a graphical user interface (GUI) for factory monitoring and control in industrial plants.

[0008] In this disclosure, a graphical user interface (GUI) for monitoring and controlling industrial plants will be broadly understood as, for example, a GUI that provides data on the status of the industrial plant and / or allows interaction with the industrial plant, particularly with its automated systems.

[0009] This method involves inputting image data into a multimodal neural generative model, such as a multimodal LLM.

[0010] The method includes a multimodal neural generative model generating a first output based on image data, the first output including an analysis of changes in the operation of an industrial plant derived from multiple frames, including at least the current frame and previous frames, thereby providing an operational context for the current state of the industrial plant.

[0011] The method includes a multimodal neural generative model generating a second output based on a first output and optionally image data, the second output including an analysis of the current state of the industrial plant taking into account the operating context.

[0012] The method includes a multimodal neural generative model generating user-aided output, which includes a representation of the analysis results of an analysis of changes associated with plant operation and / or a current state and / or suggested user actions and / or query responses in response to user queries associated with plant operation. Alternatively or additionally, the method includes a multimodal neural generative model generating control output, which is configured to control automated actions associated with plant operation.

[0013] The generation of user-aided outputs and / or control outputs employs a retrieval-enhanced generative algorithm (RAG) that utilizes a knowledge base associated with the plant's operation. This knowledge base may, for example, include a vector repository and / or a knowledge graph.

[0014] It will be understood that the above allows for the resolution of the challenges initially described, and, for example, improves plant operation and makes it more reliable.

[0015] User-assisted outputs ensure that users address the most urgent matters with the highest priority, without relying on user experience or knowledge. Control outputs can even allow for the resolution of some matters without any user interaction.

[0016] It should be understood that the GUI will typically be generated, at least in part, based on operational data received from operating and monitoring equipment in an industrial plant, such as operating parameters and sensor data from which alarms may be generated. While the possibility of reaching user auxiliary outputs or control outputs may vary or the processing of operational data may be improved, this disclosure uses GUI frames to acquire user auxiliary outputs and / or control outputs.

[0017] This makes the methods and systems disclosed herein more flexible and applicable to a variety of systems, as it does not require knowledge of engineering data related to the industrial plant at hand or specific programming dependent on the specific control system of the industrial plant. The method operates reliably while remaining agnostic to anything beyond the GUI frame. In addition to its high flexibility, the methods and systems also allow for increased data security because engineering and operational data do not need to be provided to the neural generative model. The neural generative model can simply receive frames (i.e., image data) and user and / or system prompts for the first output. For the second output, which depends on von RAG, the information can be general technical information (e.g., specifications of components in the industrial plant) or more specific, depending on the application at hand, such as available data, whether the instance is running locally or on a trusted server, etc.

[0018] No matter how specific the information in the knowledge base is, the challenges described in the background section will be resolved in any situation.

[0019] In this disclosure, the term industrial plant operation can refer to the operation of multiple pieces of equipment (also called components) within an industrial plant. This operation can involve operating these multiple pieces of equipment, such as starting them (e.g., opening and closing valves) and / or starting them and / or stopping them and / or operating them, such as using operating parameters, and / or setting operating parameters and / or changing operating parameters and / or monitoring (e.g., via sensors) the multiple pieces of equipment, and / or controlling any of the above via control equipment. Users can interact with the industrial plant operation, for example, by performing user intervention on automated operation. For example, a user can interact with the industrial plant operation via user input to a computing system, which can then be used to generate control data, and / or through direct interaction with the multiple pieces of equipment, such as manually operating valves.

[0020] Image data can include one or more images. Specifically, image data can include multiple frames of a graphical user interface, such as screenshots. A frame can be considered an image in a sequence of images, such as a sequence of screenshots of a user interface. Multiple frames of a graphical user interface can be acquired by automatically capturing screenshots of the graphical user interface, particularly at multiple points in time. This can be a screenshot of what is visible on the screen, and it can also optionally include methods provided by the operating system's application for obtaining bitmaps of the rendered GUI (e.g., even if it is not shown within the currently visible window area). In this way, for a plant control system GUI with multiple process graphics, the tool can capture all process graphics even if the user is only specifically viewing one.

[0021] The current frame can be the frame displayed to the user at the current time. The previous frame can be a frame captured at a time earlier than the current frame. It does not have to be a frame that comes before the current frame. A single previous frame or multiple previous frames can be used.

[0022] A plant monitoring and control graphical user interface (GUI) can be a graphical interface that displays the current operating status of an industrial system and / or process. For example, a GUI may display multiple pieces of equipment (also called components) of an industrial system, their interrelationships, their current operating status, operating parameter settings, data based on sensor data from sensors monitoring the operation of the industrial system, signal levels of the multiple pieces of equipment, and / or alarms. The GUI may include user interaction elements, such as control elements. User interaction elements may include, for example, buttons, text input elements, user selection elements, and / or navigation elements. At least some of the user interaction elements may be configured to allow users to view and / or intervene in the operation of the industrial plant, such as by selecting and / or confirming actions to be performed automatically (e.g., controlled via a plant control system).

[0023] Multimodal large language models and neural generative models can be any suitable neural generative model that can handle inputs including image and text data. For example, a neural generative model can be a pre-trained generative transformer. As mentioned above, neural generative models can be, for example, large language models or LLMs.

[0024] Changes in the operation of an industrial plant can include, for example, changes in the operating parameters of the industrial system and / or sensor data from sensors that monitor the operation of the industrial system. For example, changes can involve changes in temperature, pressure, tank levels, resource consumption (e.g., power consumption) of a piece of equipment, and / or the output of a piece of equipment.

[0025] In this disclosure, the operating context may refer to, for example, one or more of the following: alarm and / or warning status of the plant, sensor readings (particularly those most important for operation), alarms, warnings and rates of change of sensor readings, unusual combinations of the above elements, basic judgments about the good condition of the plant, acceptable and unacceptable conditions, basic information about the consequences if the situation continues, basic information about the impact of modifications to alarms, warnings, and sensor readings, and the state of the plant having these.

[0026] User-assisted output can be graphical output, particularly output incorporated into a graphical user interface. For example, it can include symbols, text, highlights, etc. User-assisted output can optionally be at least partially interactive. User-assisted output can include, in particular, tactile and / or acoustic output in addition to visual output.

[0027] User-assisted output may include proactive user-assisted output, i.e., output of user prompts that have not been requested. Alternatively or additionally, user-assisted output may include reactive user-assisted output, i.e., output provided in response to a user request or query. For example, reactive user-assisted output may be a response to user prompts input into a neural generative model.

[0028] User-assisted output may include a representation of the results of the analysis of changes. For example, such a representation may include visual indicator elements, such as highlights and / or symbols on a graphical user interface, and / or textual information representing the analysis results, such as in the form of a response in a chat window and / or as an information box. For example, the textual information may describe how the state of an industrial plant has changed and, optionally, what this might mean in the current context.

[0029] User-aided output may include a representation of the current state of the analysis results. For example, this representation may include visual indicator elements, such as highlights and / or symbols on a graphical user interface, and / or textual information representing the analysis results, such as in the form of a response in a chat window and / or as an information box. For instance, the textual information may describe the current state of an industrial plant and, optionally, what this might mean in the current context.

[0030] User-assisted outputs may include representations of suggested user actions in relation to plant operations. For example, text outputs may be provided to users, such as suggestions for manual intervention and / or, for example, triggering automated interventions via a GUI.

[0031] User-assisted output may include query responses to user queries related to plant operation. For example, a user query may include one or more descriptions and / or questions, such as the plant's status, changes in the plant's status, what this means, and / or what actions to take. User queries may be entered, for example, via a chat interface of a neural generative model, which may optionally be part of a GUI.

[0032] Control outputs configured to control automated actions associated with plant operation can, for example, output instructions to control equipment in an industrial plant to control plant operation in a manner specified in the control output, such as automatically changing the settings of multiple pieces of equipment, opening or closing valves, starting or shutting down multiple pieces of equipment, etc.

[0033] Known techniques for retrieving enhanced generated RAGs can be employed. This can include, but is not limited to, interaction with vector repositories, graph databases, agent / tool ​​concepts, text-to-query methods for constructing SQL / whisper and / or other retrieval commands, and / or vision-based retrieval techniques (see ColPali).

[0034] The knowledge base associated with factory operations can be represented, for example, by vectors provided in a vector repository, and / or can be provided as a knowledge graph. Further details are provided below. This can include applying methods to generate a joint embedding space—this can occur at the text level (see word2vec) and document level (latent Dirichlet assignment, latent semantic analysis) using methods such as pre-trained language models (e.g., BERT, T5). In particular, image embeddings (e.g., via ImageNet), multimodal embeddings, graph embeddings, and structured data embeddings can also be employed.

[0035] In some aspects of this disclosure, the first output is generated solely based on the visual stream of the GUI, particularly image data. Specifically, for example, no operational or engineering data can be used to generate the first output. This further enhances the aforementioned flexibility and data protection potential.

[0036] In this context, the visual stream of a GUI may include image data of multiple frames of the GUI, as well as metadata, such as timestamps, that may be associated with it.

[0037] In some aspects of this disclosure, one or more local instances of a multimodal neural generative model are used to generate a first output, a second output, and / or an auxiliary output and / or a control output.

[0038] In some aspects of this disclosure, image data is processed only at a local instance of the multimodal neural generative model, and optionally, other data, including the first output data, besides image data, is processed at the local instance of the multimodal neural generative model and / or at a cloud-based instance of the multimodal neural generative model.

[0039] Local instances of neural generative models have the advantages mentioned above that can further improve data protection.

[0040] Image data reveals a wealth of information about industrial plants and processes; therefore, from a data protection perspective, processing image data at a local instance of a neural generative model may be advantageous.

[0041] In some examples, local instances of multimodal neural generative models can be used to generate secondary and / or auxiliary outputs. Since the secondary output employs a RAG (Research Aggregator), which can also use sensitive data without the risk of data exposure, the outputs can be made more accurate for specific industrial plants because both general-purpose and plant-specific data can be used without risk, while maintaining data security.

[0042] Furthermore, as initially explained, rapid response is sometimes important, such as in the case of alerts. Relying on local instances allows for control over the availability of access to instances of the neural generative model and also avoids reliance on data transfer, thus allowing for better assurance of short response times.

[0043] In some aspects of this disclosure, the method includes generating control outputs and also includes controlling one or more actions associated with plant operation based on the control outputs, particularly without requiring prior user confirmation.

[0044] Problems that trigger warnings or alarms may arise during factory operation. While some may indeed require human intervention, analysis using neural generative models could lead to situations where automated intervention is sufficient. Generating control outputs and controlling actions associated with factory operation based on these outputs reduces some of the burden on users, as they may not need to intervene at all—that is, the actions can be performed entirely automatically, or they may only need to acknowledge that the action can be performed automatically.

[0045] It will be understood that by automatically resolving certain situations based on control outputs, users have greater ability to solve problems that actually require user intervention, even without the knowledge or experience to identify these problems, which makes the whole process more reliable.

[0046] In some aspects of this disclosure, control outputs include process state change requests, and methods include automatically issuing process state change requests to control components that control plant operation.

[0047] Process status change requests may include, for example, interactions with plant assets, such as modifying setpoints to change the valve status of a tank or the RPM of a motor. Alternatively or additionally, it may include, for example, switching to a redundant system, requesting to shut down an asset, and / or restart an asset. Requests may have different time dimensions, such as "immediately," "within the next hour," "during a shift," or "soon."

[0048] It should be understood that these tasks can be performed automatically, and therefore the claimed method improves reliability by automatically enabling the handling of the task via control output through the processing of the GUI by a neural generative model. This is because the method does not rely on the user's judgment, experience, and knowledge, and can be executed without delay compared to waiting for the user to take action.

[0049] In some aspects of this disclosure, the method includes generating and / or updating runtime context data, particularly context objects, based on a first output and optionally a second output and / or a knowledge base, and providing the context data as input for a multimodal neural generative model and / or outputting a representation of the context data to a user, particularly as part of a query response and / or automatically independent of the user query.

[0050] For example, operational context data may include data on past states, current states, state changes, user actions, and / or conclusions drawn from one or more of them. Context can be a (particularly unified) set of asset states and their interrelationships. Context may be based on event / alarm data, telemetry data (sensor readings), and optionally combined with basic judgments of normal and abnormal behavior, knowledge of potential actions and their consequences for the system, derived from engineering documents, operator manuals, and / or other plant-related and process-operation-related information sources (e.g., shift records, operator logs, etc.).

[0051] In some aspects of this disclosure, the method includes configuring and outputting a user interface to a user based on a first output and / or a second output and / or an auxiliary output.

[0052] In some aspects, the configuration and output user interface may include generating suggested user actions, particularly prompts for users to confirm requests to control components controlling plant operation for process status changes. Alternatively or additionally, the configuration and output user interface may include generating query responses related to plant operation in response to user queries. Alternatively or additionally, the configuration and output user interface may include modifying the plant monitoring graphical user interface (GUI) to provide and / or identify information potentially relevant to user actions.

[0053] For example, the user interface could be a modified factory monitoring and control graphical user interface (GUI) where parts are highlighted and / or hidden and / or rearranged and / or supplemented with labels or symbols and / or where alarms are removed and / or highlighted, and / or where the user is prompted to confirm automated actions and / or where the user is prompted to perform certain actions.

[0054] As another example, the user interface may include a chat window that allows users to interact with the neural generative model, i.e., to enter prompts and view the model's output. The user interface, such as the chat window, may be a standalone output or part of the output of a graphical user interface (GUI) for factory monitoring and control. The chat window may allow iterative interaction, where the user can provide additional input to the neural generative model, for example, in the form of prompts, and in response receive new outputs from the model.

[0055] It should be noted that providing the above-mentioned user interface improves the overall reliability of plant operation because the user's appropriate actions rely less on the user's knowledge and experience.

[0056] In some aspects of this disclosure, the method includes automatically outputting auxiliary outputs to a user and / or outputting auxiliary outputs to a user in response to a user query input into a multimodal neural generative model.

[0057] Automatic output can be any output not specifically requested by the user, such as via prompts. Automatic output can take the form of, for example, form notifications and / or pop-ups and / or interactive icons, which the user can choose to access more information. Alternatively or additionally, automatic output may include automatic highlighting of portions of the GUI and / or the display of symbols, which may optionally be user-interactive icons, such as indicating that warnings and / or additional information are available.

[0058] Specifically, user queries can be user input entered into the neural generative model, for example, via a chat window. User queries can also take the form of prompts.

[0059] Output in response to a user query may be provided, for example, in a chat that allows the user to interact with the neural generative model, specifically by providing input to the model. Alternatively or additionally, one or more of the other outputs mentioned above may be output in response to a user query, such as pop-ups, highlights, symbols, etc.

[0060] Optionally, there may be settings that determine whether to enable automatic output and / or under what circumstances automatic output is allowed.

[0061] In some aspects of this disclosure, the knowledge base includes shift logs, operator actions and / or work procedures from audit trails.

[0062] Alternatively or additionally, in some aspects of this disclosure, the knowledge base includes the technical specifications of one or more pieces of equipment in an industrial plant. For example, such specifications may include equipment type, equipment characteristics, etc. This type of information may, for example, originate from parts lists and / or specifications from the equipment manufacturer and / or engineering data from the industrial plant.

[0063] Alternatively or additionally, in some aspects of this disclosure, the knowledge base includes operating specifications for one or more pieces of equipment in an industrial plant. For example, operating specifications may include ranges of operating parameters, alarm levels, signal levels, etc. This type of information may, for example, originate from parts lists and / or specifications from the equipment manufacturer and / or engineering data from the industrial plant.

[0064] Alternatively or additionally, in some aspects of this disclosure, the knowledge base includes specifications of relationships and / or interactions between multiple pieces of plant equipment in an industrial plant. For example, such relationships may specify components expected to be connected to each other or interact in a particular manner. For instance, relationships and / or interactions may specify valves, pipes, or pumps that interact with specific equipment and / or specific monitoring devices for monitoring specific equipment. This type of information may, for example, originate from parts lists and / or specifications from equipment manufacturers and / or engineering data from industrial plants or other industrial plants and / or be based on domain knowledge.

[0065] Alternatively or additionally, in some aspects of this disclosure, the knowledge base includes technical specifications for one or more processes of an industrial plant.

[0066] This type of information can, for example, originate from engineering data and / or domain knowledge from industrial plants and processes or other industrial plants and processes.

[0067] Alternatively or additionally, in some aspects of this disclosure, the knowledge base includes specifications for one or more potential alarms associated with the operation of an industrial plant, particularly alarms associated with the operation of one or more pieces of equipment in the industrial plant. For example, there may be specific alarm levels associated with multiple pieces of equipment, and alarms may be issued based on these alarm levels.

[0068] This type of information may, for example, originate from parts lists and / or specifications from equipment manufacturers and / or from industrial plants, or from engineering data and / or domain knowledge used in other industrial plants.

[0069] In some aspects of this disclosure, the multimodal neural generative model is a transformer-based model, particularly a transformer-based model that does not require domain-specific training. Specifically, a generatively pre-trained transformer model can be employed. More specifically, the multimodal neural generative model can be configured to process inputs including image data and text data, and can be configured to output at least text data and / or control data.

[0070] The above is advantageous because the method can be applied in a general way and deployed quickly, without requiring significant technical effort to train the model. Furthermore, providing user-aided output is a language-related task, and neural generative models are powerful tools that allow users to interact with the model intuitively. This also increases reliability, as users, even without extensive experience, can obtain the information needed for the correct user intervention in a highly targeted manner (e.g., in contrast to searches in specifications or online).

[0071] Alternatively, a generative model that allows the generation of image data, such as a diffusion-based model, can be used to provide auxiliary outputs that include the generated image data, such that instructions can be provided not only in text form but also supplemented with graphical information.

[0072] This disclosure also provides a data processing system including one or more processing components configured to perform the methods of this disclosure, particularly as described above.

[0073] This disclosure also provides a computer program product including instructions that, when executed by a computer, cause the computer to perform the methods of this disclosure, particularly as described above.

[0074] This disclosure also provides a computer-readable medium including instructions that, when executed by a computer, cause the computer to perform the methods of this disclosure, particularly as described above.

[0075] The features and advantages outlined above in the context of the method are similarly applicable to the processing systems, computer program products, and computer-readable media described herein.

[0076] Further features, examples, and advantages will become clear from the detailed description of the accompanying drawings. Attached Figure Description

[0077] In the attached diagram,

[0078] Figure 1 This is a flowchart illustrating the method according to this disclosure;

[0079] Figure 2 A schematic diagram of a data processing system according to this disclosure is shown;

[0080] Figure 3 This is an example of the "operator in the loop" configuration according to this disclosure;

[0081] Figure 4 This is an example of a "no operator in the loop" configuration based on this disclosure;

[0082] Figure 5 This is an example of an "operator in the loop" configuration that only has a local instance of the neural generative model;

[0083] Figure 6 This is an example demonstrating the compatibility of this method with configurations having multiple different process controls;

[0084] Figure 7 These are exemplary workflow steps illustrating the method according to this disclosure;

[0085] Figures 8a to 8dThe inputs are shown, such as screenshots of the GUI at different times, and user outputs, such as text-based active outputs and reactive outputs. Detailed Implementation

[0086] Figure 1 This is a flowchart illustrating a computer-implemented method for operating an industrial plant according to the present disclosure.

[0087] The method includes, in step S11, acquiring image data of one or more frames, particularly multiple frames, of a graphical user interface (GUI) for factory monitoring and control in an industrial plant;

[0088] The method includes, in step S12, inputting image data into a neural generative model, such as a multimodal large language model (LLM). The multimodal neural generative model can be, for example, a transformer-based model, particularly a transformer-based model that does not require domain-specific training.

[0089] The method includes, in step S13, generating a first output based on image data. This first output includes an analysis of changes in the operation of an industrial plant derived from multiple frames, including at least the current frame and previous frames, thereby providing an operational context for the current state of the industrial plant. In some examples, the first output is generated solely based on the visual flow of the GUI, specifically image data.

[0090] The method includes, in step S14, a multimodal neural generative model generating a second output based on a first output and optionally image data, the second output including an analysis of the current state of the industrial plant taking into account the operating context.

[0091] The method includes, in step S15, a multimodal neural generative model generating output.

[0092] For example, the method may include, in step S15a, a multimodal neural generative model generating user-aided output, which includes a representation of the analysis results of an analysis of changes associated with plant operation and / or the current state and / or suggested user actions and / or a query response in response to a user query associated with plant operation. In some examples, the method includes automatically outputting aided output to the user and / or outputting aided output to the user in response to a user query input into the multimodal neural generative model.

[0093] Alternatively or additionally, the method may include, in step S15b, a multimodal neural generative model generating a control output configured to control automated actions associated with plant operation.

[0094] When the method includes generating control outputs, in step S16b, the method may optionally also include controlling one or more actions associated with plant operation based on the control outputs, particularly without requiring prior user confirmation. The control outputs may include process state change requests, and the method includes automatically issuing process state change requests to control components that control plant operation.

[0095] In some examples, one or more local instances of the multimodal neural generative model are used to generate a first output, a second output, and / or auxiliary outputs and / or control outputs. In the examples, image data is processed only at a local instance of the multimodal neural generative model, and optionally, other data besides image data, including the first output data, is processed at the local instance of the multimodal neural generative model and / or at a cloud-based instance of the multimodal neural generative model.

[0096] The generation of user-aided outputs and / or control outputs employs a retrieval-enhanced generation RAG, which utilizes a knowledge base associated with plant operations. This knowledge base may include, for example, a vector repository and / or a knowledge graph. The knowledge base may include one or more of the following: shift logs, operator actions from audit trails, work processes; technical specifications of one or more pieces of equipment in the industrial plant; operating specifications of one or more pieces of equipment in the industrial plant; specifications of relationships and / or interactions between multiple pieces of plant equipment in the industrial plant; technical specifications of one or more processes in the industrial plant; and specifications of one or more potential alarms associated with the operation of the industrial plant, particularly alarms associated with the operation of one or more pieces of equipment in the industrial plant.

[0097] The method may include, in optional step S17, configuring and outputting a user interface based on a first output and / or a second output and / or an auxiliary output. Configuring and outputting the user interface may include generating suggested user actions, particularly prompts for the user to confirm the issuance of a process state change request to the control components controlling the plant's operation, and / or

[0098] In response to user queries, generate query responses related to plant operations, and / or modify the plant monitoring graphical user interface (GUI) to provide and / or identify information potentially relevant to user actions.

[0099] In optional step S18, the method may include generating and / or updating runtime context data, particularly context objects, based on a first output and optionally a second output and / or a knowledge base, and providing the context data as input for a multimodal neural generative model and / or outputting a representation of the context data to the user, particularly as part of a query response and / or automatically independent of the user query.

[0100] Figure 2A data processing system 1 according to the present disclosure is shown, which includes components for performing the methods of the present disclosure (e.g., one of the methods described above), such as a processing component, such as a processor 1a. A memory 1b may be provided as part of the processing system and / or connected to the processing system.

[0101] Optionally, system 1 may include or be connected to user interface device 2, which may be configured to output a user interface and optionally to receive user input, such as a display device.

[0102] The system can be connected to external computing resources, such as external processing and / or data storage resources, such as cloud-based resources.

[0103] An exemplary data connection 4 is shown, which may be wired or wireless, for example.

[0104] For the sake of explanation, Figure 2 A local instance 5 of the neural generative model and a cloud-based instance 6 of the neural generative model are shown.

[0105] The following will refer to Figures 3 to 8a to Figure 8d Further features and advantages are described below. In the following text, the Large Language Model (LLM) is specifically referenced as an example of a neural generative model; however, these considerations and features are similarly applicable to other neural generative models.

[0106] The methods and systems disclosed herein address challenges such as monitoring complex technical equipment via a user interface, or monitoring an entire plant via an operator's workplace, which can be very challenging and complex, especially when it involves judging changes. For example, the user (also known as the operator) needs to decide whether and / or which of several potential concurrent scenarios, such as two new priority 2 alarms and a dropping tank level, is indeed problematic and requires intervention. Reliability may be compromised, for example, because it depends on the user's knowledge and experience.

[0107] It should be understood that the methods and systems disclosed herein allow for effective context monitoring of technical equipment using multimodal neural generative models, such as large language models, and context-based information retrieval.

[0108] This method and system allow for proactive and reactive support for users, such as operators. It is necessary to monitor complex technical equipment, systems, or plants based on a user interface. For example, this method and system can provide proactive and reactive information about operating technical equipment, taking into account real-time status and knowledge bases. This method and system utilize multimodal large language models and knowledge bases, for example, to generate user-specific support.

[0109] Technical equipment can be monitored using a technical equipment monitor based on the visual flow of the system's GUI, particularly the GUI frames. By focusing (specifically in some examples) on the visual flow / GUI frames, the system can be easily integrated with various systems. For example, integration can occur only at the UI level. These methods and systems are agnostic to the specific control systems of industrial plants, and therefore may not require any engineering or data integration.

[0110] Optionally, in some respects, queries for the knowledge base and optionally other services can be more targeted by continuously updating the context object based on the latest UI changes.

[0111] LLM allows outputting text descriptions to support users, optionally in response to specific user queries. This can be helpful for any user who is not very familiar with industrial plants and / or GUIs, such as new or infrequent users or maintenance personnel. This method and system allow them to quickly understand the status of the plant, even if they are unfamiliar with a particular GUI screen in that particular installation.

[0112] Note that using LLM also allows output to be provided in any suitable language or several languages, which can increase reliability. For example, users can query the output in their native language. This avoids errors based on a lack of language skills. Alternatively, LLM output can be automatically translated using appropriate software tools.

[0113] For example, for users who may better understand languages ​​different from those used for common GUI elements (e.g., operator displays) and / or documentation, the text describing the situation and / or how to solve the problem can be provided in other natural languages ​​and / or translated into other natural languages.

[0114] The output generated by LLM can allow the GUI to be modified, for example, to specifically draw the user’s attention to important things such as anomalies, state changes, and process variables beyond expected values.

[0115] Figures 3 to 6 Different examples of the methods of this disclosure are shown in the figure.

[0116] Figure 3 This can be considered an example of an "operator in a loop." The LLM receives a stream of GUI images from the operator's workplace and optionally receives user prompts, such as questions, via a chat interface. In return, the LLM provides proactive suggestions and / or responses to the prompts, such as answers to questions. In this example, two instances are shown for illustrative purposes only, one local and one cloud-based. Other configurations, such as having only local instances or only cloud-based instances, are conceivable.

[0117] Figure 4 This can be considered an example of a "no operator in the loop." As will be seen here, the LLM creates and proactively provides state change requests to the control system. No user intervention is required. Other configurations, such as having only local or only cloud-based instances, are conceivable.

[0118] It should be understood that Figure 3 and Figure 4 An example of this combination is conceivable: a user can interact with the LLM and optionally receive proactive recommendations, and the LLM can also issue proactive state change requests to the control system.

[0119] Figure 5 An example of an "operator in a loop" is shown, consisting only of a local instance of an LLM.

[0120] Figure 6 This demonstrates the ability to use multiple different process controls integrated with LLM. That is, the method and system can be applied very flexibly and are independent of specific process control configurations.

[0121] Figure 7 These are exemplary workflow steps illustrating the method according to this disclosure.

[0122] As shown on the left, an image of the GUI interface at the current time (“now”) is provided to an image storage device, which also includes images of past GUI interfaces. Optionally, the user can provide user input, such as questions or prompts about the industrial system and processes, such as what an alarm means or what process state a piece of equipment is in.

[0123] Based on images from image storage, changes can be determined from GUI images representing the current time ("now") and the past (e.g., "now minus 30 seconds"). The LLM can then determine information about these changes and the current situation. Using plant context, such as information about the current state and changes, along with information from a knowledge base, the LLM can determine supporting actions that can be output to the user (active and / or reactive) and used to control the operation of an industrial plant.

[0124] Figure 8a and Figure 8b The GUI is depicted at two different times. Figure 8cThe diagram illustrates, for example, (exemplarily proactive) outputs from an LLM that can be provided to the user. The outputs provide interpretations of different state changes, such as a decrease in pressure and yield in reactor R11, an increase in temperature in reactor R11, and an increase in pressure and temperature in reactor R12. Thus, without requiring any extensive investigation, the user is informed that a decrease in pressure and yield may indicate a controlled shutdown, an increase in temperature may be related to adjustments in the process, or an increase in pressure and temperature may suggest an increase in activity or a modification of operating parameters.

[0125] Figure 8d An example of the response output is depicted, where the chat window illustrates user input, such as user prompts, which are presented here in the form of a description of what the user observes (current productivity, current temperature, an alarm with priority 2 activated and indicating a long response time) and what the user should do to correct the situation. The figure also depicts the responsive LLM output, suggesting steps the user should take to correct the situation (e.g., verify the output temperature, initiate a transition step, open the PID panel, confirm the long response alarm).

[0126] Therefore, from this illustrative example, it can be understood that the method of this disclosure makes the process more efficient and reliable because it does not rely on the user's knowledge or research to perform appropriate user interventions. The method assists users in performing technical tasks through a continuous and / or guided human-computer interaction process, particularly independent of user preferences, because the output is based on additional information, context, and insights that the user would not otherwise have access to.

[0127] The output in the chat window can be one way to provide this guidance, which may be advantageous given that this is usually a feature already provided by LLM-based software solutions, making it easy and efficient to incorporate, but other rendering methods achieve similar technical effects.

[0128] While this disclosure has been detailed and described in the accompanying drawings and foregoing description, such description and illustration are to be considered exemplary rather than limiting. This disclosure is not limited to the disclosed embodiments. In view of the foregoing description and drawings, it will be apparent to those skilled in the art that various modifications can be made within the scope of this disclosure as defined by the claims.

Claims

1. A computer-implemented method for operating an industrial plant, the method comprising: Acquire (S11) image data including multiple frames of a graphical user interface (GUI) for factory monitoring and control in industrial plants; The image data is input (S12) into the multimodal neural network generation model; The multimodal neural generative model generates a first output (S13) based on the image data. The first output includes an analysis of changes in the operation of the industrial plant derived from the plurality of frames, which include at least the current frame and previous frames, thereby providing an operational context for the current state of the industrial plant. The multimodal neural generative model generates a second output (S14) based on the first output and optionally the image data, the second output including an analysis of the current state of the industrial plant taking into account the operating context; The multimodal neural network generation model generates (S15a, S15b): User-assisted output, the user-assisted output including: The representation of the analysis results of the analysis of the changes associated with the plant operation and / or the current status and / or suggested user actions and / or query responses to user queries associated with the plant operation, and / or Control outputs, configured to control automated actions associated with the operation of the factory. The generation of the user-aided output and / or the control output employs a retrieval-enhanced generation (RAG) that utilizes a knowledge base associated with the plant operation.

2. The method of claim 1, wherein the first output is generated solely based on the visual flow of the GUI, particularly the image data.

3. The method according to claim 1 or 2, wherein one or more local instances of the multimodal neural generation model are used to generate the first output, the second output, and / or auxiliary output and / or control output.

4. The method of claim 3, wherein the image data is processed only at a local instance of the multimodal neural generative model, and optionally, wherein other data, including the first output data, besides the image data, is processed at the local instance of the multimodal neural generative model and / or at a cloud-based instance of the multimodal neural generative model.

5. The method according to any one of the preceding claims, wherein the method comprises: The control output is generated, and the control output is also used to control one or more actions associated with the operation of the plant, in particular without requiring prior user confirmation.

6. The method according to claim 5, wherein, The control output includes a process state change request, and the method includes: automatically issuing the process state change request to a control component that controls the operation of the plant.

7. The method according to any one of the preceding claims, comprising: Based on the first output and optionally the second output and / or the knowledge base, runtime context data, in particular context objects, is generated and / or updated, and the context data is provided as input for the multimodal neural generative model and / or a representation of the context data is output to the user, in particular as part of a query response and / or automatically independent of the user query.

8. The method according to any one of the preceding claims, comprising: Configure and output the user interface to the user based on the first output and / or the second output and / or the auxiliary output.

9. The method of any one of claims 8, wherein configuring and outputting the user interface comprises: Generate suggested user actions, specifically prompts for the user to confirm a request to the control components that control the operation of the plant to issue a process state change request, and / or In response to a user query, generate a query response associated with the operation of the factory, and / or Modify the factory monitoring graphical user interface (GUI) to provide and / or identify information that is potentially relevant to user actions.

10. The method according to any one of the preceding claims, wherein the method comprises: The auxiliary output is automatically output to the user, and / or the auxiliary output is output to the user in response to a user query input into the multimodal neural generative model.

11. The method according to any one of the preceding claims, wherein the knowledge base comprises one or more of the following: Shift logs, operator actions, such as those from audit trails, and / or work procedures, The technical specifications of one or more pieces of equipment in the industrial plant. The operating specifications of one or more pieces of equipment in the industrial plant. The specifications governing the relationships and / or interactions between the plant equipment in the industrial plant. Technical specifications for one or more processes in the industrial plant. The specification of one or more potential alarms associated with the operation of the industrial plant, particularly alarms associated with the operation of one or more pieces of equipment in the industrial plant.

12. The method according to any one of the preceding claims, wherein the multimodal neural generative model is a transformer-based model, particularly a transformer-based model that does not require domain-specific training, and / or wherein the multimodal neural generative model is a large language model (LLM).

13. A data processing system (1) comprising one or more processing components (1a) configured to perform the method according to any one of claims 1 to 12.

14. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

15. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Images