Method and system for automatically generating optimized human machine interface (HMI) screen templates

Abstract

A computer-implemented method and system (102) for automating the generation of Human-Machine Interface (HMI) screen templates are disclosed. The method involves extracting design parameters, equipment status, and key performance indicators (KPIs) from an industrial automation system that models interdependencies using knowledge graphs, and resolving ambiguities by sending hints to a large language model trained in the domain. The optimized HMI screen templates are generated based on contextual information and are adapted to improve operator responsiveness and usability. The system achieves real-time adaptability by updating the screen templates in response to plant changes or operator feedback. Validation and simulation modules ensure functionality by simulating operator interactions. A continuously learning and updating machine learning model refines the design based on feedback and evolving requirements. The deployed templates enhance plant operations by improving usability and aligning with operational objectives.

Description

Technical Field

[0001] This disclosure relates to the generation and optimization of human-machine interface (HMI) screen templates for industrial automation systems. Specifically, it focuses on leveraging artificial intelligence, including natural language processing (NLP) and machine learning, to automatically create, deploy, and iteratively improve HMI screen templates, enhancing usability, operator responsiveness, and operational efficiency in factory environments. Background Technology

[0002] Human-Machine Interface (HMI) screens play a crucial role in industrial automation systems, serving as the primary medium through which operators and technicians monitor, control, and optimize manufacturing processes. In many industrial environments, Siemens' Total Integrated Automation (TIA) portal acts as a centralized platform for configuring and managing these systems, including the design and deployment of HMI screens. While the TIA portal provides a robust set of tools, the process of creating HMI screens has traditionally relied heavily on manual work, requiring professionals to design and configure each screen from scratch. This conventional approach is not only labor-intensive but also prone to human error, often leading to increased project costs and extended timelines.

[0003] The traditional approach to creating HMI screens within a TIA portal involves manually designing the layout, logic, and functionality of each screen. Engineers must painstakingly arrange visual elements—such as graphics, indicators, and input fields—and map these elements to specific devices and parameters. Furthermore, ensuring consistency in style, configuration, and interdependencies across multiple screens requires significant expertise and attention to detail. Organizations frequently rely on highly skilled labor to meet these needs, which increases costs and the likelihood of errors or inconsistencies in the final design.

[0004] To mitigate these challenges, some workflows incorporate pre-designed screen templates or libraries as an alternative to entirely manual development. These templates provide users with pre-configured layouts or widgets that can be adapted to project-specific requirements. While this approach streamlines certain aspects of the design process, it still requires considerable customization to meet the unique needs of each industrial application. Users must expend significant effort to ensure the templates align with system requirements, accommodate equipment interdependencies, and adhere to operating standards. Furthermore, pre-designed templates often fail to address complex industrial parameters, such as tolerance thresholds and intricate relationships between plant equipment, leaving many original challenges unresolved.

[0005] While pre-designed libraries and templates offer partial solutions, they do not fully automate the HMI screen creation process or provide guidance on key parameters such as equipment interdependencies, material tolerances, or system KPIs. As industrial processes evolve, static templates struggle to adapt, increasing the likelihood of production inefficiencies, ignored warnings, and suboptimal layouts. The need for continuous updates and adjustments exacerbates the reliance on skilled operators, leading to increased time consumption, design errors, and high labor costs.

[0006] In this context, there is an urgent need for an advanced, automated approach to HMI screen creation. Summary of the Invention

[0007] Therefore, the object of the present invention is to provide a method and system for automatically generating optimized human-machine interface (HMI) screen templates, which improves the availability, efficiency and operator responsiveness of industrial automation systems.

[0008] The purpose of this disclosure is achieved through a computer-implemented method for automatically generating optimized human-machine interface (HMI) screen templates, which improves the availability, efficiency, and operator responsiveness of industrial automation systems. HMI screen templates refer to visually appealing and interactively structured templates that enable operators to monitor and control industrial processes, displaying elements such as equipment controls, key performance indicators (KPIs), alarms, and real-time equipment status.

[0009] This method involves extracting multiple design parameters from user prompts by applying Natural Language Processing (NLP). The user prompt provides input from the operator or system designer, specifying requirements for the HMI screen template. For example, a prompt such as "Generate a screen template for monitoring temperature and pressure with alarm thresholds and equipment control ranges" informs the system about desired functionality. The extracted design parameters include elements such as the operating range of equipment control, system performance metrics (KPIs), and alarm thresholds, along with layout characteristics such as color schemes and font styles.

[0010] The method further includes generating a knowledge graph that represents multiple interdependencies among extracted design parameters, KPIs, and real-time device states. The knowledge graph is constructed using a graph neural network (GNN) to model relationships between entities, such as device control, performance metrics, and alarm conditions. For example, the graph captures dependencies such as the impact of temperature settings on energy efficiency or alarm triggering based on abnormal pressure levels. The knowledge graph serves as the basis for understanding operational dynamics and optimizing screen templates.

[0011] The method further includes extracting multiple design guidelines from a knowledge graph using a supervised learning algorithm trained on historical design data and multiple design parameters. These design guidelines provide actionable recommendations for visual and interactive layouts, such as optimal placement of navigation menus, high-contrast color schemes for critical warnings, and user-friendly layouts tailored to specific industrial environments. The extraction process incorporates preprocessed data, including KPI measurements, equipment specifications, and tolerance levels, as well as insights from historical user preferences and common design patterns derived from existing HMI screen templates.

[0012] The method further includes transmitting cues to a large language model to generate at least one HMI screen template. The transmitted cues include information indicating design guidelines and additional contextual factors, such as operator skill level and plant layout. For example, if the operator is novice, the cues emphasize simplified navigation and prominent visual cues. The large language model processes the cues to produce an optimized HMI screen template aligned with user needs and plant-specific configuration.

[0013] The method further includes deploying the generated HMI screen templates into industrial automation systems to improve operator responsiveness. Deployment involves configuring the screen templates within the system, mapping interactive elements to corresponding devices, and ensuring compatibility with factory hardware and software platforms. For example, control buttons are linked to device actuators, and dashboards displaying KPIs are integrated with monitoring systems. The deployed screen templates enhance operator interaction by providing real-time feedback, intuitive control, and a clear visualization of factory operations.

[0014] The method further includes automatically suggesting layout modifications, color schemes, and font styles for the generated HMI screen template based on usability metrics. Usability metrics derived from simulated interactions and operator feedback assess factors such as accessibility, visual clarity, and intuitiveness of interaction. For example, the method might suggest repositioning frequently used controls to the center of the screen or replacing low-contrast text with high-contrast alternative text to improve readability.

[0015] The method further includes providing operators with a user interface to preview and customize the generated HMI screen templates before deployment. Operators can adjust elements such as layout structure, color coding, and font style to ensure that the screen templates meet specific operational and aesthetic preferences.

[0016] The method further includes detecting operator feedback associated with interactions with the generated HMI screen template. Feedback captured through logs, surveys, or sensors provides insights into areas of usability challenges and improvements. For example, recurring errors in activation controls or delayed responses to alarms indicate that design elements need refinement.

[0017] The method further includes updating the machine learning (ML) model based on detected operator feedback. The update process involves retraining or fine-tuning the ML model using augmented training data that incorporates insights from operator interactions. For example, the ML model learns to prioritize layouts that improve navigation efficiency or enhance the visibility of critical warnings.

[0018] This method further includes applying reinforcement learning to iteratively improve the ML model based on operational feedback and evolving plant requirements. Reinforcement learning uses reward signals derived from availability metrics and system performance data to guide the ML model's learning process, ensuring continuous adaptation to the dynamic industrial environment.

[0019] The method further includes analyzing operator interaction patterns with HMI screen templates to identify areas for usability improvement. Interaction patterns, such as frequent navigation delays or ignored warnings, are analyzed to reveal inefficiencies and inform targeted design modifications.

[0020] The object of the present invention is also achieved by a programming system capable of automatically generating optimized human-machine interface (HMI) screen templates. This programming system includes one or more processors and memory coupled to the processors. The memory includes an automation module stored in the form of machine-readable instructions executable by the processor. The automation module is configured to perform the methods described above.

[0021] The object of the present invention is also achieved by a computer program product having machine-readable instructions stored therein, which, when executed by a processing unit, are configured to perform the method steps described above. Attached Figure Description

[0022] The present disclosure is further described below with reference to the illustrated embodiments shown in the accompanying drawings, wherein: Figure 1 This is a block diagram of an environment for automatically generating HMI screen templates according to embodiments of the present disclosure; Figure 2 Such as Figure 1 The diagram shows a block diagram of a programming system, such as a programming system, in which embodiments of the present disclosure can be implemented; Figure 3 Such as Figure 2 The diagram shows an automation module, such as an automation module, in which embodiments of this disclosure can be implemented; and Figure 4 This is a flowchart illustrating an exemplary method for testing and executing an automated user interface according to an embodiment of the present disclosure. Detailed Implementation

[0023] Various embodiments are described with reference to the accompanying drawings, wherein similar reference numerals are used to refer to the drawings and similar elements throughout. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of one or more embodiments. It may be apparent that such embodiments can be practiced without these specific details.

[0024] Figure 1 This is a block diagram of a factory environment 100 for automatically generating optimized human-machine interface (HMI) screen templates according to embodiments of the present disclosure. Figure 1 In this context, factory environment 100 includes programming system 102, user interface 106, and one or more client devices 120A-N. As used herein, "factory environment" refers to a processing environment that includes configurable physical and logical computing resources such as networks, servers, storage devices, applications, and services distributed on platforms such as cloud computing platforms. Factory environment 100 provides on-demand network access to a shared pool of configurable computing resources. Programming system 102 is connected to the user interface via network 104 (such as a local area network (LAN), wide area network (WAN), Wi-Fi, the Internet, or other communication protocols). Programming system 102 is also connected to one or more client devices 120A-N via network 104.

[0025] User interface 106 includes one or more screen elements 108A-N. Examples of the one or more screen elements 108A-N include, for example, device controls, alarm indicators, dashboards, sliders, and drop-down menus. User interface 106 is designed for managing and monitoring industrial automation systems. The one or more screen elements 108A-N are operated by executing multiple software modules in a controller device of user interface 106. The controller device is configured to execute the software modules and control one or more screen elements 108A-N to achieve specific operational objectives in user interface 106. In one example, the controller device may be a cloud-based server or an industrial control system.

[0026] Each of the one or more screen elements 108A-N has multiple properties described in technical documents such as HMI design guidelines, interaction flowcharts, and user manuals. These properties include, but are not limited to, compatibility information, display settings, operational scope, interaction states, and lifecycle management details. The multiple documents further include process-related documents such as user stories, use cases, and design specifications, as well as non-process documents such as user manuals, style guidelines, and compliance documents. Furthermore, the multiple documents include information relating to the interdependencies between the properties of the screen elements.

[0027] The programming system 102 is connected via network 104 to one or more screen elements 108A-N in the user interface 106. The one or more screen elements 108A-N can also be connected to other industrial devices and sensors via physical and non-physical connections (such as IoT networks and 5G communication protocols). Figure 1 (Not shown in the image). Although Figure 1 The illustration shows a programming system 102 connected to a user interface 106, but those skilled in the art can imagine that the programming system 102 can be used to generate and deploy a large number of HMI screen templates across multiple factory environments.

[0028] One or more client devices 120A-N may include desktop computers, laptops, tablets, smartphones, and similar computing devices. Each of the one or more client devices 120A-N is provided with design tools 122A-N for generating and / or editing the user interface 106. For example, one or more client devices 120A-N may access programming system 102 to automatically generate at least one HMI screen template. One or more client devices 120A-N may also access cloud applications (such as visualizing factory performance metrics) via a web browser. Throughout this specification, the terms "client device" and "user device" are used interchangeably.

[0029] Programming system 102 may be a standalone server deployed at a control station or a remote server hosted on a cloud computing platform. In a preferred embodiment, programming system 102 is a cloud-based programming system capable of delivering applications for managing user interfaces 106 including one or more screen elements 108A-N. Programming system 102 may include platform 110 (such as a cloud computing platform), automation module 112, server 114, network interface 116, and database 118. Server 114 may include hardware resources such as memory units and processors for storing machine-readable instructions and applications. Platform 110 uses the hardware resources of server 114 to enable functions such as data reception, processing, rendering, and communication. Automation module 112 is configured to generate optimized HMI screen templates, such as... Figure 3 As described in [the text].

[0030] Database 118 stores information related to user interface 106 and one or more client devices 120A-N. Database 118 can be structured as an SQL or NoSQL data store and can reside locally or in the cloud. Examples of stored information include design guidelines, user interaction logs, simulation results, screen element configurations, and operational metrics such as KPIs.

[0031] Figure 2 Such as Figure 1The diagram shows a block diagram of a programming system 102, such as the programming system shown, in which embodiments of the present disclosure can be implemented. Figure 2 In the programming system 102, there are processing unit 202, memory 204, storage unit 206, communication interface 208, input / output unit 210, network interface 212 and bus 214.

[0032] Processing unit 202 can be any type of computing circuit, such as a microprocessor, microcontroller, or graphics processing unit. Memory 204 can include volatile and non-volatile memory for storing machine-readable instructions that can execute automated processes, such as generating HMI screen templates. Storage unit 206 can store a database containing design data and user feedback, such as database 118.

[0033] Communication interface 208 enables communication between programming system 102 and client devices 120A-N, allowing engineers to interact with generated HMI screen templates. Input / output unit 210 may include devices such as a touchscreen or keyboard for providing user input. Network interface 212 manages network connectivity, such as LAN, WAN, or wireless communication.

[0034] Those skilled in the art will understand that Figure 2 The hardware depicted may vary depending on the specific implementation. The examples depicted are provided for illustrative purposes and are not intended to imply architectural limitations.

[0035] Those skilled in the art will recognize that, for the sake of simplicity and clarity, the complete structure and operation of all data processing systems applicable to this disclosure are not depicted or described herein. Instead, only those aspects of programming system 102 that are unique to or necessary for understanding this disclosure are depicted and described.

[0036] Note that user interface 106 refers to visual and interactive components that allow users to interact with the application. User interface 106 includes multiple user-interactive visual elements, such as buttons, sliders, and readers, which facilitate control of the application.

[0037] When executed by processing unit 202, automation module 112 enables processing unit 202 to automatically generate optimized human-machine interface (HMI) screen templates. Automation module 112 enables processing unit 202 to extract multiple design parameters from user prompts using natural language processing (NLP). The HMI screen templates serve as predefined layouts or structures, facilitating visualization and interaction between operators and industrial automation systems. HMI screen templates organize key elements, such as equipment controls, process data, alarms, and key performance indicators (KPIs), in an intuitive and accessible manner to ensure seamless operation within the plant setup. Optimized HMI screen templates refer to screen templates customized to enhance usability, efficiency, and accuracy while minimizing the risk of operational errors. Optimization ensures that the HMI screen template aligns with best practices within the screen template, caters to specific user needs, and conforms to parameters such as tolerance levels and equipment constraints.

[0038] Multiple design parameters affect various aspects of the screen template, contributing to its functionality and visual layout. Examples of these parameters include a first parameter for specifying the scope of operation of device controls, a second parameter for highlighting KPIs (Key Performance Indicators) of system performance, a third parameter for configuring alarms to indicate critical thresholds, and layout elements such as color schemes and font styles. Each design parameter plays a role in ensuring that the generated HMI screen template effectively meets operational and monitoring requirements.

[0039] User prompts refer to input provided by operators or system designers to guide the automated generation of human-machine interface screen templates. User prompts can consist of specific instructions, queries, or preferences that indicate the desired characteristics or focus areas of the screen template. For example, a user prompt could be "Generate a screen template for monitoring temperature and pressure with alarm thresholds and equipment control ranges." Such prompts inform the system operator of their requirements and serve as the basis for extracting multiple design parameters.

[0040] The automation module 112 further enables the processing unit 202 to apply Natural Language Processing (NLP) to interpret, analyze, and extract relevant information from user prompts. NLP facilitates the conversion of human-readable instructions into structured data, which can be processed by the programming system 102 to generate human-machine interface screen templates. Examples of NLP techniques involve intent recognition that identifies operator goals, entity extraction that isolates specific design parameters (such as KPIs or equipment controls), and contextual analysis that ensures the generated screen templates are aligned with operator preferences. The purpose of natural language processing is to enable seamless communication between the operator and the automated screen generation system, thereby ensuring that the human-machine interface screen templates accurately and efficiently meet the operator's requirements.

[0041] The automation module 112 further enables the processing unit 202 to categorize the extracted design parameters into multiple functional groups, including equipment control, KPI, and alarm groups, to streamline the generation of at least one human-machine interface (HMI) screen template. The categorization of multiple design parameters is essential for systematically organizing information and ensuring that the generated HMI screen templates are both intuitive and functional. By grouping them based on their roles within the industrial automation system, the categorization process simplifies screen template generation, allowing the programming system 102 to focus on the specific operational requirements of each group.

[0042] Functional groups represent different categories within multiple design parameters, each defined by its sole purpose contributing to the functionality of the human-machine interface screen template. The equipment control group contains design parameters associated with the manipulation and control of industrial equipment. Examples of parameters within the equipment control group include operating range, tolerance levels, and control buttons, which are essential for operators to safely and accurately adjust equipment settings.

[0043] The Key Performance Indicators (KPIs) set contains design parameters relevant to monitoring and evaluating the performance of industrial systems. Parameters within the KPI set include metrics such as productivity, energy consumption, and system efficiency, which serve as key indicators for assessing the operational health of the plant. The KPI set ensures that human-machine interface screen templates display this performance data in a way that allows operators to make informed decisions.

[0044] Alarm groups contain design parameters configured to warn operators of critical conditions or anomalies in the industrial system. Parameters within an alarm group include thresholds, visual indicators, and audible signals that notify the operator when predefined limits are exceeded or when a fault is detected. Alarm groups ensure that the human-machine interface screen template provides timely and effective notifications to prevent operational disruptions or hazards.

[0045] By categorizing multiple design parameters into equipment control groups, KPI groups, and alarm groups, the automation module 112 further enables the processing unit 202 to ensure that the generated human-machine interface screen templates are organized, user-friendly, and tailored to address the specific functional requirements of the factory environment 100.

[0046] Automation module 112 further enables processing unit 202 to utilize graph neural networks (GNNs) to analyze and model multiple interdependencies between design parameters, key performance indicators (KPIs), and one or more device states. Graph neural networks provide a powerful computational framework for analyzing and modeling complex relationships within interconnected data points. The use of graph neural networks involves representing multiple interdependencies among design parameters, KPIs, and device states as nodes and edges within a graph structure. The graph neural network processes this graph structure to reveal patterns, relationships, and dependencies that inform the generation of optimized human-machine interface screen templates.

[0047] Multiple interdependencies refer to the relationships and influences among multiple design parameters, key performance indicators, and equipment states that collectively determine the operational dynamics of a plant. Examples of interdependencies include the connection between equipment control range and tolerance levels, where exceeding the tolerance level affects the corresponding key performance indicator related to system efficiency. Another example of interdependency is the relationship between alarm thresholds and equipment states, where abnormal equipment states trigger alarms to notify operators of potential problems. Modeling interdependencies is crucial for ensuring that human-machine interface screen templates reflect the interconnected nature of industrial operations.

[0048] Key performance indicators (KPIs) represent measurable metrics for assessing the effectiveness, efficiency, and overall health of industrial processes. Examples of KPIs include production output and energy consumption; production output measures the quantity of goods produced within a specified time frame, and energy consumption assesses the amount of energy used during operation. KPIs provide operators with actionable insights into plant performance and are integrated into the design of human-machine interface (HMI) screen templates.

[0049] Equipment status indicates the real-time condition and operational status of industrial equipment within a plant. Examples of equipment status include machine temperature readings indicating whether equipment is operating within safety limits, and pressure levels in piping indicating the stability of fluid transport systems. Equipment status is critical for monitoring and maintaining the operational integrity of industrial systems and plays a key role in determining the parameters displayed on human-machine interface (HMI) screen templates.

[0050] The automation module 112 further enables the processing unit 202 to generate a knowledge graph that includes multiple extracted design parameters, multiple key performance indicators, and multiple interdependencies among one or more equipment states. The knowledge graph is generated using a graph neural network, a high-level machine learning algorithm specifically designed to process and analyze graph-structured data. Graph neural networks enable the representation and modeling of complex relationships by capturing the dependencies and interactions between nodes and edges within the graph. The graph neural network analyzes the input data to construct a knowledge graph that accurately reflects the dynamics of factory operations.

[0051] Knowledge graphs represent structured graphical models that organize information in the form of nodes and edges. Nodes represent entities such as design parameters, key performance indicators, and equipment status, while edges represent relationships or dependencies between these entities. Knowledge graphs provide a comprehensive visual and computational framework for storing, analyzing, and inferring relationships within interconnected factory elements. In the context of human-machine interface screen generation, knowledge graphs serve as a fundamental tool for understanding and leveraging multiple interdependencies to optimize screen templates.

[0052] The automation module 112 further enables the processing unit 202 to transform multiple interdependencies into a knowledge graph by analyzing the relationships between extracted design parameters, key performance indicators, and equipment states, and mapping these relationships to edges on the nodes of a connection graph. For example, the interdependency between equipment control parameters (such as temperature settings) and key performance indicators (such as energy efficiency) is represented as an edge connecting the node for temperature control parameters to the node for energy efficiency. Another example of interdependency transformation is the relationship between equipment states (such as pressure levels) and alarm design parameters, where exceeding a pressure threshold triggers an alarm. This relationship is transformed into a knowledge graph as an edge connecting the pressure state node to the alarm parameter node.

[0053] Automation module 112 also enables processing unit 202 to perform feature extraction on preprocessed data derived from at least one of multiple key performance indicator measurements, one or more equipment specifications, and one or more tolerance levels. Preprocessed data refers to data that has undergone a series of transformations and refinements to ensure that the data is clean, consistent, and in a format suitable for further analysis and computational processing. Preprocessing may include operations such as removing noise from the raw input data, normalizing values ​​to a standard scale, handling missing data through interpolation or imputation, and constructing the data into a format that conforms to the requirements of the feature extraction algorithm. For example, raw temperature readings from the machine may be cleaned to remove outliers and normalized to conform to a predefined operating range before being classified as preprocessed data.

[0054] Feature extraction is performed using machine learning algorithms designed to identify and isolate meaningful attributes or features from preprocessed data, essential for understanding patterns and relationships within the data. The machine learning algorithms for feature extraction are trained using labeled training data corresponding to historical measurements, equipment specifications, tolerance ranges, and predefined operating states. The training data is typically derived from historical plant operations, simulation data, or domain-specific benchmarks. For example, a machine learning model can be trained on a dataset containing variations in energy consumption patterns at different productivity levels, equipment tolerance levels under normal and stress conditions, and alarm trigger thresholds for specific machine malfunctions.

[0055] The input to machine learning algorithms includes preprocessed data representing multiple key performance indicator measurements, one or more equipment specifications, and one or more tolerance levels. For example, input may include normalized values ​​of KPIs, such as production output; equipment characteristics, such as operating temperature range; and tolerance values, such as acceptable pressure deviations. Machine learning algorithms process these inputs to identify relevance, anomalies, and critical attributes relevant to the feature extraction process.

[0056] The output generated by the machine learning algorithm includes extracted features representing the most significant attributes contributing to the plant's operational dynamics. Examples of the output include dimensionality reduction of principal components isolating energy consumption trends, spatial features identifying vibration modes in machinery, and statistical thresholds classifying alarm triggering conditions. These outputs are constructed into a feature set, which serves as the basis for further analysis and modeling.

[0057] The automation module 112 also enables the processing unit 202 to use the output of machine learning algorithms during feature extraction to isolate and optimize attributes directly related to the generation of the human-machine interface screen template. For example, extracted KPI-related features, such as productivity and energy efficiency, can inform the visual elements of the screen template, while features derived from equipment specifications, such as operating range, can guide the layout and function of control elements.

[0058] Automation module 112 also enables processing unit 202 to identify one or more historical user preferences and one or more general design patterns from existing human-machine interface (HMI) screen templates. Historical user preferences refer to patterns, tendencies, or choices made by users when interacting with the HMI screen template in previous scenarios. Examples of historical user preferences include frequently placing control buttons on the right side of the screen for easy access, or repeatedly using color coding, such as green for normal operation and red for alarms, to enhance rapid visual recognition. Historical user preferences are typically derived from user behavior data, feedback logs, and interaction history collected from HMI systems deployed in a factory environment.

[0059] General design patterns refer to recurring structural or stylistic elements in human-machine interface (HMI) screen templates that are widely adopted due to their effectiveness and usability. Examples of general design patterns include using tabbed navigation menus to organize different parts of the interface, including summary dashboards to quickly display aggregated key performance indicators, and using graphical representations such as bar charts or line charts to visualize trends in device status. General design patterns are derived from industry standards, best practices, and design conventions that have been validated through repeated use in industry environments.

[0060] Machine learning algorithms are used to identify historical user preferences and common design patterns. These algorithms are designed to analyze existing human-computer interface screen templates and extract meaningful insights. Algorithms such as clustering techniques that group similar design elements based on user interaction data and natural language processing models that process textual feedback associated with screen template usage are used to identify patterns and preferences. For example, clustering algorithms can identify users who consistently interact with navigation menus located at the top of the screen, while natural language processing models can detect user feedback indicating a preference for larger font sizes in alert notifications.

[0061] Automation module 112 also enables processing unit 202 to receive existing HMI screen templates from various sources, including previously deployed factory systems, design archives maintained by industrial automation companies, and libraries containing industry-specific standardized templates. These existing HMI screen templates serve as input data for the identification process, providing historical context and design examples needed to extract user preferences and design patterns. For example, templates used in plants specializing in chemical processing can be analyzed to identify preferences for multi-layered dashboards that simultaneously display equipment status and tolerance levels.

[0062] The automation module 112 also enables the processing unit 202 to extract historical design data from features, identified historical user preferences, and common design patterns. Historical design data refers to archived or previously implemented design elements, configurations, and structures that have been utilized in human-machine interface screen templates over time. This historical design data encapsulates accumulated design knowledge and practices that have proven effective in industrial environments and is stored in a format suitable for analysis and reuse in the generation of new screen templates.

[0063] Examples of historical design data include specific layout configurations, such as a grid-based arrangement for organizing control buttons, proportional spacing between visual elements to ensure clarity, and the standard placement of the navigation menu at the top of the screen. Additional examples include color schemes for visually distinguishing device status, such as green for operating devices, yellow for devices approaching tolerance limits, and red for malfunctioning devices. Historical design data may also include frequently used font types and sizes in alert notifications to ensure readability during critical operations, or the consistent use of icons and graphic symbols to represent devices and key performance indicators.

[0064] The automation module 112 also enables the processing unit 202 to analyze historical design data by employing machine learning algorithms, such as pattern recognition models and rule-based systems. These algorithms identify recurring design elements within extracted features and correlate them with historical user preferences and common design patterns. For example, pattern recognition models can detect recurring structures in screen templates, such as dashboards with clustered key performance indicators, while rule-based systems can compile best practices, such as maintaining minimal contrast between text and background colors for optimal readability.

[0065] The automation module 112 also enables the processing unit 202 to utilize the interaction between features, historical user preferences, and general design patterns to isolate functionally effective and user-friendly design elements. For example, if an extracted feature indicates frequent user interaction with a control button located in the lower right corner of a screen template, and historical user preferences confirm this trend, the machine learning algorithm extracts that design element as part of the historical design data. Similarly, if a general design pattern indicates that graphical representations such as pie charts are widely used to visualize energy consumption metrics, the machine learning algorithm incorporates this insight into the historical design data.

[0066] Automation module 112 further enables processing unit 202 to extract multiple design guidelines from a knowledge graph by applying supervised learning algorithms to historical design data and multiple design parameters. Design guidelines refer to structured rules, principles, or recommendations that specify the optimal arrangement, visualization, and interaction elements within a human-machine interface screen template to ensure usability, efficiency, and operational compliance. Design guidelines serve as actionable instructions, telling screen elements (such as control buttons, dashboards, notifications, and device indicators) how they should be organized and presented to the operator.

[0067] Examples of design guidelines include principles such as maintaining minimal spacing between interactive elements to prevent accidental activation; ensuring critical alerts are highlighted in bold red for immediate visibility; and placing frequently used controls in locations that minimize operator hand movement. Other examples include color scheme recommendations, such as avoiding combinations that may cause eye strain, and guidelines for visual hierarchy, such as placing summary dashboards at the top of the screen for quick access to aggregated information. The design guidelines also include device-specific, customized rules, such as using graphical indicators for temperature levels in thermal machines or providing slider controls for adjustable tolerance ranges.

[0068] A supervised learning algorithm for extracting multiple design guidelines is trained using training data that includes historical design data, multiple design parameters, and labeled training data with known guidelines from industry standards or expert input. The training data includes examples of screen templates annotated with corresponding design guidelines, such as labels indicating preferred locations of control buttons, recommended font sizes for device status, or acceptable color combinations for visual clarity. For example, the training dataset could include screen templates used in chemical processing plants, annotated with guidelines to highlight temperature thresholds due to their importance.

[0069] The automation module 112 also enables the processing unit 202 to analyze the inputs of the supervised learning algorithm, including design parameters in the knowledge graph, extracted features (such as recurring placement patterns of navigation menus), and relationships between historical data indicating user preferences for the graphical dashboard. The algorithm detects relevance, dependencies, and trends consistent with existing design guidelines.

[0070] The output generated by the supervised learning algorithm includes extracted design guidelines that provide actionable insights for creating optimized screen templates. For example, the output might include rules specifying that navigation menus should be located at the top of the screen to enhance accessibility, or guidelines suggesting the use of contrasting colors to display key performance indicators to ensure clarity. The output also captures device-specific guidelines, such as indicators for displaying tolerance levels next to pressure readings in machines prone to overload.

[0071] The automation module 112 also causes the processing unit 202 to send prompts to a large language model to generate at least one human-machine interface screen template, wherein the sent prompts include information indicating multiple design guidelines. The prompts include additional contextual information, such as operator skill level and plant layout, to tailor the generated screen template to specific user needs.

[0072] The automation module 112 also enables the processing unit 202 to generate prompts using prompt templates selected from a large number of predefined templates based on criteria that conform to the requirements for generating human-machine interface screen templates. These criteria include compatibility with multiple design guidelines, relevance to plant-specific parameters, and suitability for addressing specific user needs, such as operator skill levels and plant layout. For example, if the plant layout includes multiple areas with different operational roles, the automation module 112 selects a prompt template designed to accommodate area-specific design considerations. Similarly, if the operator's skill level is novice, the automation module 112 selects a prompt template constructed to emphasize simplicity and clarity in the generated screen template.

[0073] The automation module 112 also enables the processing unit 202 to populate the selected prompt template with specific information required to generate the prompt. Multiple design guidelines are integrated into the selected prompt template to ensure that the generated screen template adheres to established principles and rules. For example, the selected prompt template may include placeholders for layout rules, color scheme preferences, and interaction principles derived from multiple design guidelines. The automation module 112 ensures that these placeholders are populated with specific guidelines, such as recommendations to place control buttons in accessible areas or to use bold red text for alert notifications.

[0074] The automation module 112 also enables the processing unit 202 to incorporate additional contextual information, such as operator skill level or plant layout, into the selected prompt template. For example, if the operator's skill level is novice, the automation module 112 integrates contextual information that suggests a simplified navigation structure and highlighted visual cues for key performance indicators. Similarly, if the plant layout comprises multiple areas, the automation module 112 integrates contextual information, such as the spatial arrangement of equipment or the interdependencies between different areas, to ensure that the generated screen template reflects the plant's operational structure.

[0075] Automation module 112 also enables processing unit 202 to deploy the generated human-machine interface screen template to the industrial automation system to improve operator responsiveness. The deployed screen template is validated against the software requirements and equipment compatibility of the industrial automation system. Validation ensures that graphical elements are correctly rendered on the display device and that interactive components respond effectively to input commands. The deployment process involves mapping interactive elements (such as control buttons and sliders) to corresponding devices and sensors in the plant, and then loading the screen template onto the operator console or control panel.

[0076] By deploying human-machine interface screen templates into industrial automation systems, automation module 112 further enables processing unit 202 to enhance operator responsiveness through intuitive visualization and interactive control. For example, a graphical dashboard displaying aggregated KPIs allows operators to monitor trends and take corrective actions, such as adjusting machine speed or reallocating resources, to maintain optimal performance.

[0077] The automation module 112 also enables the processing unit 202 to automatically suggest multiple layout modifications, color schemes, and font styles for the generated human-machine interface screen template based on usability metrics. Suggestions are generated by applying a machine learning algorithm trained to analyze usability metrics, historical design data, and contextual information associated with the generated screen template. The machine learning algorithm evaluates the usability of the screen template according to predefined criteria and outputs actionable suggestions for design improvement.

[0078] For example, machine learning algorithms can suggest layout modifications, such as repositioning control buttons to more accessible areas of the screen based on historical user preferences. If usability metrics indicate that operators frequently miss alerts located in the top left corner of the screen, a machine learning algorithm can suggest repositioning the alert to the center of the screen to ensure greater visibility. Similarly, machine learning algorithms might recommend adjusting color schemes, such as replacing low-contrast combinations with higher-contrast alternatives to enhance readability. For example, the algorithm might suggest using white text as a status indicator on a dark blue background instead of gray text on a light background. Font style suggestions might include recommendations such as switching to a sans-serif font to improve readability or increasing the font size of key information, such as alert notifications.

[0079] The automation module 112 also enables the processing unit 202 to generate recommendations using several inputs. These inputs include initial human-machine interface screen templates received from a large language model, historical design data capturing user interaction trends and preferences, and contextual information such as operator skill levels and factory layout. Furthermore, machine learning algorithms evaluate usability metrics derived from assessments of the screen templates' effectiveness and user-friendliness. For example, usability metrics may include measures such as the average time an operator spends locating a specific control, the frequency of errors in activating interactive elements, and the clarity of visual information displayed on the screen.

[0080] The output generated by the machine learning algorithm includes several actionable suggestions for layout modifications, color schemes, and font styles. These suggestions are structured into specific improvements that can be applied to the human-machine interface screen template. For example, the output might specify that the navigation menu should be repositioned to the top center of the screen, propose using a green background for normal operation indicators, and recommend bold fonts for alert messages. The algorithm outputs these suggestions in a format that can be applied directly during the screen template customization process or further refined.

[0081] Automation module 112 also enables processing unit 202 to calculate usability metrics, which are quantitative and qualitative measures that evaluate the effectiveness, efficiency, and satisfaction associated with using the human-machine interface screen template. Usability metrics are calculated using data collected from simulated interactions, operator feedback, and historical user behavior analysis. For example, calculating usability metrics might involve conducting simulations where operators use the screen template to perform specific tasks, such as activating controls or responding to alarms, and measuring the time spent completing these tasks. Usability metrics might also involve analyzing error rates, such as the frequency of accidental control activation or missed alarms.

[0082] Automation module 112 also enables processing unit 202 to incorporate factors such as visual clarity, layout accessibility, and interactive intuitiveness into the calculation of usability metrics. For example, visual clarity can be evaluated by analyzing the contrast between text and background colors, while layout accessibility can be evaluated by measuring the proximity of frequently used controls to the operator's field of vision. Interactive intuitiveness can be calculated by analyzing how quickly and accurately the operator can navigate through the screen template to access desired functions. By generating recommendations based on usability metrics, automation module 112 ensures that the human-machine interface screen template meets operator needs and improves operational efficiency in factory environment 100.

[0083] The automation module 112 also enables the processing unit 202 to detect operator feedback associated with interactions with the generated human-machine interface screen template. Detecting operator feedback involves observing, capturing, and analyzing the responses, behaviors, and inputs provided by the operator when using the screen template. Operator feedback provides valuable insights into the usability, effectiveness, and user satisfaction of the screen template, enabling improvements and refinements to better meet operator needs and increase operational efficiency.

[0084] Operator feedback is detected through mechanisms integrated into human-machine interface screen templates and industrial automation systems. For example, screen templates may include a logging system that records operator actions, such as the frequency of activation of specific control buttons or the order in which menu options are navigated. If an operator repeatedly interacts with a particular control button, the feedback indicates its frequent use and potential importance, suggesting that the button's layout should prioritize accessibility. Conversely, if certain interactive elements are rarely used, operator feedback may suggest redesigning or reassigning these elements to optimize screen space.

[0085] Operator feedback is also detected through direct input mechanisms, such as surveys or rating systems embedded in screen templates. For example, after completing a task, operators might be prompted to rate their experience on a scale of one to five, or to comment on the clarity of the instructions displayed on the screen. If operators consistently rate navigation menus as difficult to use, the feedback will highlight areas where improvements are needed in layout or labeling.

[0086] Furthermore, advanced sensors integrated into industrial automation systems are used to detect operator feedback through physiological and behavioral observations. For example, eye-tracking sensors can detect where the operator's attention is focused on a screen template, providing feedback on which areas are visually prominent and which areas might be overlooked. If eye-tracking data shows that the operator rarely looks at a warning indicator located in the lower left corner of the screen, the feedback suggests repositioning the indicator to a more central location to improve visibility.

[0087] Another example of operator feedback is detected through error logs. If an operator frequently activates incorrect controls or fails to respond to alerts in a timely manner, the feedback will reveal potential problems with the layout, color scheme, or interaction design of the screen template. For example, if an operator accidentally activates the close button because it is too close to a frequently used control, the feedback indicates that spacing needs to be increased or the button repositioned to prevent errors.

[0088] The automation module 112 also enables the processing unit 202 to analyze operator feedback associated with the interaction to continuously refine the human-machine interface screen template based on detected usability challenges. By leveraging operator feedback, the automation module 112 ensures that the screen template evolves in line with operator preferences, operational requirements, and industry standards, resulting in a more efficient and user-friendly interface.

[0089] The automation module 112 also enables the processing unit 202 to update the machine learning model based on detected operator feedback. Updating the machine learning model involves modifying its parameters, retraining its architecture, or augmenting its training data to incorporate insights derived from operator feedback. This process ensures that the machine learning model evolves to reflect real-world usage patterns, operator preferences, and usability challenges, thereby improving its ability to generate future human-machine interface screen templates that meet operational requirements and user needs.

[0090] Operator feedback serves as a crucial input for updating the machine learning model. For example, if operator feedback indicates frequent difficulty locating specific interactive elements on a screen template, the machine learning model is updated to prioritize layout templates that place key controls in highly visible and accessible areas. Feedback can include data such as error logs showing repeated activation of incorrect controls, eye-tracking information indicating neglected areas of the screen, or survey findings highlighting dissatisfaction with navigation complexity. This feedback is analyzed and transformed into actionable insights that guide modifications to the machine learning model.

[0091] The automation module 112 also enables the processing unit 202 to extract features from operator feedback, which are used as the basis for updating the machine learning model. For example, if operators consistently rate certain fonts as difficult to read, the feature extraction process identifies font-related usability metrics such as font size, contrast, and style preference. These features are added to the training data used by the machine learning model, enabling it to learn patterns consistent with operator preferences. The training data can also be augmented with additional examples reflecting the detected feedback, such as screen templates displaying improved layouts, color scheme annotations, or interactive templates based on operator input.

[0092] The automation module 112 also enables the processing unit 202 to retrain the machine learning model using augmented training data to incorporate insights from operator feedback. During retraining, the machine learning model adjusts its weights, biases, and decision-making algorithms to optimize its ability to generate human-machine interface screen templates that address identified usability challenges. For example, if operator feedback emphasizes the need for a simplified navigation structure for novice users, the retraining process ensures that the machine learning model learns to prioritize screen templates with streamlined menus and intuitive labeling.

[0093] The update process may also involve fine-tuning the machine learning model to improve its accuracy and adaptability without requiring a complete retraining. Fine-tuning involves making small adjustments to the model parameters based on specific operator feedback. For example, if feedback indicates that alarm indicators should consistently use bold red font, the machine learning model is fine-tuned to incorporate that design guideline into its future outputs.

[0094] The updated machine learning model generates improved human-machine interface screen templates that better meet operator needs and enhance usability. For example, based on updated insights, the machine learning model might suggest repositioning frequently used controls to the center of the screen or recommending high-contrast color schemes for key information. The updated model continuously incorporates operator feedback into its decision-making process, ensuring that its output evolves over time to reflect changing user requirements and operating conditions.

[0095] The automation module 112 also enables the processing unit 202 to modify the generated human-machine interface screen template using an updated machine learning model to enhance its usability for operators. Modifying the screen template involves applying updated insights, recommendations, and design principles learned by the machine learning model to refine the screen template's structure, visual elements, and interactive components. The modification process ensures that the screen template conforms to operator preferences, operational requirements, and usability standards derived from operator feedback.

[0096] The modification process begins by analyzing the output of the updated machine learning model to identify specific areas for improvement in the human-machine interface screen template. For example, if the updated machine learning model emphasizes the need to improve the visibility of critical alerts, the modification process involves enhancing the color scheme and font style of the alert indicators. Automation module 112 modifies the screen template by replacing low-contrast color schemes with high-contrast alternatives, such as using a bright red background for alerts and bold white font for alert messages.

[0097] The automation module 112 also enables the processing unit 202 to apply layout modifications suggested by the updated machine learning model to optimize the arrangement of interactive elements on the screen template. For example, if the updated machine learning model recommends repositioning frequently used controls closer to the operator's field of vision, the modification process involves repositioning such controls to the center of the screen. The automation module 112 ensures that the modified layout enhances accessibility and reduces the time spent by the operator locating and activating controls.

[0098] The modification process incorporates usability-centric adjustments to interactive elements such as sliders, buttons, and input fields. For example, if an updated machine learning model suggests increasing the size of control buttons to accommodate operators using touch interfaces, automation module 112 modifies the screen template by resizing the buttons and ensuring sufficient spacing between interactive elements. Similarly, if the machine learning model recommends introducing tooltips for novice operators, the modification process involves embedding descriptive labels or instructions that appear when the operator hovers over or clicks a specific control.

[0099] The automation module 112 also enables the processing unit 202 to refine the navigation path based on recommendations from the updated machine learning model. For example, if the updated machine learning model identifies inefficiencies in navigating between different screens, the modification process introduces shortcuts, streamlined transitions, or breadcrumb navigation elements. These modifications ensure that operators can seamlessly access the required functions without encountering unnecessary delays or confusion.

[0100] The modification process incorporates visual aesthetic improvements to enhance operator engagement and satisfaction. For example, if an updated machine learning model suggests an ergonomic color scheme, the automation module 112 modifies the screen template by implementing calming colors for background elements and vibrant colors for operable controls. The automation module 112 ensures that all visual adjustments adhere to usability guidelines while maintaining design consistency.

[0101] Automation module 112 also enables processing unit 202 to apply reinforcement learning to iteratively improve the machine learning model based on operational feedback and evolving factory requirements. Applying reinforcement learning involves training the machine learning model through a continuous loop of trial, evaluation, and adjustment, where the machine learning model learns the optimal behavior and design strategies for generating human-machine interface screen templates. The reinforcement learning process uses operational feedback as a reward signal to guide the learning process of the machine learning model and adapts to evolving factory requirements by incorporating dynamic data from the real-world industrial environment.

[0102] The reinforcement learning process begins by defining objectives aligned with usability and operational efficiency goals. For example, a reinforcement learning algorithm might aim to maximize operator satisfaction, minimize response time to alarms, or optimize task completion rates using human-machine interface screen templates. The algorithm assigns reward values ​​to actions taken by the machine learning model based on the degree to which these actions achieve the defined objectives. For instance, if the machine learning model generates screen templates that reduce the average time operators spend locating control buttons, the reinforcement learning algorithm assigns high reward values ​​to the corresponding design parameters.

[0103] During the iterative improvement process, operational feedback serves as the primary input to the reinforcement learning algorithm. Operational feedback includes data such as operator interaction logs, error rates, task completion times, and system performance metrics. For example, if operators frequently fail to respond to alarms due to low visibility, operational feedback might highlight the need for design modifications, such as increasing the salience of alarm indicators. The reinforcement learning algorithm uses operational feedback to evaluate the effectiveness of design choices made by the machine learning model and adjusts its parameters to improve future outputs.

[0104] Automation module 112 also enables processing unit 202 to iteratively adapt the machine learning model to evolving factory requirements. Factory requirements may include changes in production workflows, the introduction of new machines, or changes in operator skill levels. For example, if the factory introduces a new set of sensors requiring additional monitoring and control, the reinforcement learning algorithm incorporates the new requirements into the training process of the machine learning model, ensuring that future screen templates adapt to the added functionality. This algorithm continuously adapts to changes in the factory environment 100, ensuring that the machine learning model remains relevant and effective over time.

[0105] Automation module 112 also enables processing unit 202 to analyze operator interaction patterns with the human-machine interface screen template to identify areas for usability improvement. Analyzing operator interaction patterns involves capturing and evaluating data related to how operators engage with interactive elements of the screen template during plant operations. This analysis process identifies trends, inefficiencies, and challenges in operator behavior, enabling targeted modifications to enhance usability.

[0106] Operator interaction patterns are captured using logging mechanisms embedded in screen templates and industrial automation systems. For example, analytics might reveal that operators frequently navigate across multiple screens to access critical controls, indicating a need for streamlined navigation paths. Interaction patterns, such as repeated activation of specific buttons, prolonged pauses during decision-making, or frequent errors in task completion, are evaluated to reveal usability challenges. For instance, if operators consistently activate incorrect controls due to unclear labeling, the analytics process will identify areas for improvement in label design or repositioning of controls.

[0107] The automation module 112 also enables the processing unit 202 to use statistical and machine learning techniques to identify correlations and causal relationships in operator interaction patterns. For example, data showing a high error rate in certain tasks may be related to the low visibility of relevant controls on a screen template. The analysis process identifies actionable insights, such as modifying the layout to place key controls in more accessible locations or introducing guiding tooltips to guide the operator through the task.

[0108] By analyzing operator interaction patterns with HMI screen templates, automation module 112 ensures that usability improvements are data-driven and aligned with real-world operational needs. Incorporating reinforcement learning, the iterative improvement process continuously refines the machine learning model and HMI screen templates, thereby enhancing operator responsiveness and overall plant performance.

[0109] Automation module 112 also enables processing unit 202 to implement an iterative refinement strategy to ensure that the human-machine interface screen templates evolve based on dynamic factory conditions and operator behavior. The iterative refinement process involves integrating newly identified usability insights, operational feedback, and evolving factory requirements into the machine learning model and screen template design. This continuous improvement cycle enhances the system's adaptability and effectiveness over time.

[0110] Automation module 112 also enables processing unit 202 to evaluate human-machine interface screen templates deployed in real-world operational settings to monitor their performance and effectiveness. Real-world evaluation involves capturing detailed data on operator interaction, task completion rates, error frequency, and system response during actual factory operations. For example, automation module 112 can monitor how quickly operators respond to alarms using screen templates, or how efficiently they adjust equipment settings during production. Metrics such as average task completion time, operator satisfaction, and system override frequency are analyzed to assess whether the screen templates meet their intended usability and operational objectives.

[0111] The automation module 112 also enables the processing unit 202 to detect areas for improvement during live evaluation by analyzing the discrepancies between expected and actual operator performance. For example, if operators take longer than expected to locate certain controls, this indicates a need to adjust the layout or enhance visual cues. Similarly, if alarms are frequently ignored or missed, screen templates may need to be modified to improve the prominence and accessibility of alarm indicators. This real-time evaluation process ensures the system can meet operational challenges and continuously adapt to plant conditions.

[0112] The automation module 112 also enables the processing unit 202 to integrate operational data collected during live evaluation into the machine learning model to refine its design recommendations. Operational data serves as valuable input for retraining and fine-tuning the machine learning model, enabling it to learn from real-world scenarios and generate more effective screen templates in future iterations. For example, data showing frequent operator errors when activating controls can prompt the machine learning model to prioritize screen templates with enhanced labels and larger interactive elements.

[0113] The automation module 112 also enables the processing unit 202 to generate updated human-machine interface screen templates based on the refined output of the machine learning model. These updated templates incorporate the latest usability insights, design optimizations, and operator feedback to address previously detected challenges. For example, updated templates might feature redesigned navigation menus for faster access to key controls, an improved alert system with bold visual indicators, or a more intuitively reorganized dashboard displaying aggregated KPIs.

[0114] Automation module 112 also enables processing unit 202 to redeploy updated human-machine interface screen templates to the industrial automation system. Redeployment involves validating the updated templates to ensure compatibility with plant hardware and software requirements, and then replacing or enhancing existing screen templates on operator consoles and control panels. For example, updated screen templates can be loaded onto touchscreen interfaces used by production line operators or supervisory control terminals used by plant administrators. The redeployment process ensures that operators have access to the latest interface templates optimized for availability and efficiency.

[0115] Automation module 112 also enables processing unit 202 to continuously monitor the performance of redeployed screen templates to verify their effectiveness and inform further refinement. Monitoring involves collecting data on operator interaction, system response, and plant performance metrics to assess whether the updated templates have achieved the intended operational objectives. For example, if updated screen templates reduce the average response time to alarms or improve operator satisfaction, this confirms the success of the implemented changes. Conversely, if new usability challenges arise, automation module 112 initiates additional refinement cycles to address these issues.

[0116] Automation module 112 also enables processing unit 202 to apply reinforcement learning to incorporate long-term trends and evolving plant requirements into a machine learning model. Reinforcement learning algorithms analyze long-term collected operational data to identify recurring patterns, emerging challenges, and opportunities for further optimization. For example, if long-term data reveals increasing complexity in production workflows, the machine learning model can adapt to generate screen templates with advanced organizational structures, such as multi-level dashboards or dedicated navigation paths for different plant areas.

[0117] Automation module 112 also enables processing unit 202 to integrate evolving plant requirements into its machine learning model by continuously updating the training dataset with new operational insights, equipment specifications, and industry standards. For example, the introduction of new machines requiring additional monitoring and control or updated regulatory guidelines for plant operation prompts the machine learning model to incorporate these changes into its design logic. This ensures that future screen templates remain compatible with the latest plant configurations and conform to industry best practices.

[0118] Automation module 112 also enables processing unit 202 to maintain a feedback loop between the machine learning model and the industrial automation system to ensure continuous adaptability. The feedback loop allows the machine learning model to learn from each iteration of screen template deployment and refinement, thereby creating a self-improving system that responds to dynamic plant conditions. For example, data from operator interaction with updated templates informs the machine learning model about successful design strategies, which are reinforced in subsequent outputs. Similarly, new usability challenges detected during live operation prompt the machine learning model to adjust its design recommendations accordingly.

[0119] By implementing an iterative refinement strategy, deploying updated screen templates, and leveraging reinforcement learning, automation module 112 ensures that human-machine interface screen templates continuously evolve to meet operator needs and factory requirements. This approach improves operational efficiency, reduces usability challenges, and ensures that industrial automation systems remain adaptable and effective over time.

[0120] Figure 3 This is a block diagram of automation module 112, such as... Figure 2 Those shown, in which embodiments of the invention can be implemented. Figure 3 In the process, the automation module 112 includes a request processor 302, a parameter extraction module 304, a dependency analysis module 306, a natural language processing (NLP) module 308, a knowledge graph database 310, a verification and simulation module 312, and a deployment module 314. Combined with... Figure 1 and Figure 2 explain Figure 3 .

[0121] Request processor 302 is configured to receive requests to initiate the automatic generation or refinement of human-machine interface screen templates for factory environment 100. For example, the request can be received from one or more operators or factory administrators via network 104. In alternative embodiments, the request can be automatically triggered by an industrial automation system based on detected changes in factory configuration or equipment status. For example, a factory administrator may request a screen redesign to accommodate new machines, or an automatic request may be triggered by a system upgrade.

[0122] The parameter extraction module 304 is configured to identify and extract multiple design parameters, equipment status, and key performance indicators (KPIs) from plant data sources, such as one or more devices 108A-N. For example, the parameter extraction module 304 can identify temperature control parameters and extract their operating range and current status.

[0123] The dependency analysis module 306 is configured to analyze extracted design parameters, equipment status, and KPIs to determine the interdependencies and correlations between these elements. For example, the dependency analysis module 306 can map the relationship between equipment control parameters such as speed and KPIs such as productivity. Alternative embodiments may include using graph neural networks or other machine learning algorithms to improve the accuracy of dependency analysis.

[0124] Natural Language Processing (NLP) module 308 is configured to extract and interpret contextual information from operator preferences, historical design data, and industry standards. For example, it can extract textual descriptions or interpretations of operator feedback or design guidelines related to alarm visualizations.

[0125] The knowledge graph database 310 is configured to store multiple interdependencies between extracted design parameters, equipment states, and KPIs. For example, the knowledge graph database 310 can store relationships between equipment control settings and their impact on energy efficiency. In an alternative embodiment, the database can be integrated with a distributed cloud system for scalable and collaborative data management across factory areas.

[0126] The verification and simulation module 312 is configured to generate simulated instances of the proposed human-machine interface screen template and simulate operator interaction in a virtual environment. For example, the verification and simulation module 312 may simulate an operator responding to an alarm or adjusting device settings based on visual elements in the screen template. In an alternative embodiment, the simulation may include stress test scenarios, such as evaluating operator performance under high-pressure operating conditions.

[0127] Deployment module 314 is configured to deploy the generated screen templates and execute them in the actual factory environment 100. For example, deployment module 314 automates the process of integrating the screen templates into the operator console, testing their compatibility with factory equipment, and validating their functionality under real-world conditions.

[0128] The automation module 112 for automatically generating and optimizing Human-Machine Interface (HMI) screen templates offers numerous advantages by leveraging advanced computing and adaptive technologies. A key advantage is the ability to dynamically adapt HMI screen templates to real-time operator needs and industry requirements. This is achieved through a parameter extraction module (304) that identifies key design parameters, equipment status, and key performance indicators (KPIs), and a knowledge graph database (310) that models the interdependencies between these elements using advanced correlation algorithms. Furthermore, the validation and simulation module (312) enhances usability by simulating operator interaction with the proposed HMI screen templates, ensuring that functionality aligns with operational objectives prior to deployment.

[0129] Another advantage is the system's ability to adapt to changes within the factory environment (such as new machines or updated operator preferences). This adaptability is achieved through screen templates generated in real time based on new data, facilitated by the automation module (112), which continuously processes screenshots and operator feedback. Machine learning models and reinforcement learning techniques further refine the templates over time, achieving long-term optimization by learning from operator interaction patterns and feedback, ensuring improved usability. Furthermore, the system's deployment module (314) streamlines the integration of proven HMI screen templates with the factory environment, ensuring compatibility with operator consoles and control panels.

[0130] System 102 also improves the efficiency and accuracy of design generation by automating processes that traditionally require manual intervention. The use of graph neural networks in the creation of the knowledge graph allows for highly accurate modeling of interdependencies, while the Natural Language Processing (NLP) module (308) extracts contextual information from prompts and design guidelines to ensure alignment with operator needs. By suggesting layout modifications, color schemes, and font styles based on usability metrics, the system further ensures that HMI screen templates are intuitive and operator-friendly, thereby reducing cognitive load and improving responsiveness during high-pressure scenarios.

[0131] Finally, the system achieves continuous improvement through its feedback loop, where the verification and simulation module (312) detects operator feedback, and the machine learning model is updated accordingly. This feedback loop enables iterative refinement of the screen template, ensuring that the design evolves to meet changing plant requirements and operator preferences. The combination of advanced image processing, NLP, machine learning, and reinforcement learning provides a robust, scalable, and adaptive solution for optimizing HMI screen templates in industrial automation systems. By leveraging these modules and features, the disclosed systems and methods achieve significant improvements in usability, adaptability, efficiency, and operator responsiveness, accurately and reliably addressing the challenges of the plant environment.

[0132] Figure 4 This is a process flowchart illustrating an exemplary method for generating and validating automated human-machine interface screen templates according to embodiments of the present disclosure.

[0133] In step 402, by applying a data extraction algorithm to the input data collected from one or more devices 108A-N and sensors, one or more design parameters, device status, and key performance indicators (KPIs) associated with the plant environment 100 are identified. The design parameters, device status, and KPIs represent operational metrics and controls related to plant processes.

[0134] In step 404, a correlation algorithm is used to establish interdependencies among the identified design parameters, equipment states, and KPIs. In the knowledge graph, interdependencies are represented as edges connecting nodes. For example, the correlation algorithm can identify the relationship between temperature control parameters and energy efficiency KPIs.

[0135] In step 406, based on the analysis of the knowledge graph, operator preferences, and historical design data, potential ambiguities associated with the established interdependencies are identified. For example, semantic ambiguity may arise if multiple temperature thresholds are associated with conflicting alarm settings.

[0136] In step 408, a prompt is sent to a large language model to generate contextual information for resolving the detected ambiguity. The large language model is trained on domain-specific data, such as industry standards for factory automation and operator interaction guidelines. For example, the prompt might request clarification on the most effective alarm visualization format for temperature threshold violations.

[0137] In step 410, based on the established interdependencies and contextual information provided by the large language model, one or more optimized human-machine interface screen templates are generated. The generated templates are adapted to operator preferences, equipment specifications, and usability guidelines. For example, the screen templates may feature a dashboard displaying aggregated KPIs and interactive controls for adjusting equipment settings.

[0138] In step 412, the generated human-machine interface screen template is validated and deployed in factory environment 100. Validation includes simulating operator interaction to ensure usability and functionality, while deployment involves integrating the screen template into operator consoles and control panels. The deployed template is tested for compatibility with factory equipment and operational requirements, and a usability report is generated to highlight areas for further refinement.

[0139] This invention can take the form of a computer program product comprising program modules accessible from a computer-usable or computer-readable medium storing program code used by or in connection with one or more computers, processors, or instruction execution systems. For the purposes of this specification, a computer-usable or computer-readable medium can be any means capable of containing, storing, communicating, propagating, or transmitting a program used by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium that is not included in the definition of a physical computer-readable medium as a signal carrier, including semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), hard disks, and optical disks, such as optical disc read-only memory (CD-ROM), read / write optical discs, and DVDs. As those skilled in the art will appreciate, both the processor and the program code used to implement each aspect of this technology can be centralized or distributed (or a combination thereof).

[0140] While the invention has been described in detail with reference to certain embodiments, it should be understood that the invention is not limited to these embodiments. In view of this disclosure, many modifications and variations exist on their own without departing from the scope of the various embodiments of the invention as described herein. Therefore, the scope of the invention is defined by the following claims rather than the foregoing description. All changes, modifications, and variations within the meaning and scope of equivalents of the claims are considered to be within its scope. All advantageous embodiments claimed in the method claims can also be applied to the system / apparatus claims.

Claims

1. A method for automatically generating optimized human-machine interface (HMI) screen templates, comprising: Use Natural Language Processing (NLP) to extract multiple design parameters from user prompts; Generate a knowledge graph that includes multiple interdependencies between extracted design parameters, multiple key performance indicators (KPIs), and real-time device status; By applying supervised learning algorithms to historical design data and multiple design parameters, multiple design guidelines are extracted from the knowledge graph. Send prompts to a large language model to generate at least one HMI screen template, wherein the prompts sent include information indicating multiple design guidelines; and Deploy at least one generated HMI screen template into an industrial automation system to improve operator responsiveness.

2. The method according to claim 1, wherein, Generating knowledge graphs also includes: The graph neural network (GNN) is used to analyze and model multiple interdependencies between multiple design parameters, multiple key performance indicators (KPIs) and equipment status.

3. The method according to claim 1, further comprising: The extracted design parameters are categorized into functional groups, such as device controls, KPIs, and alarms, to streamline the generation of at least one HMI screen template.

4. The method of claim 1, wherein extracting multiple design guidelines further comprises: Feature extraction is performed on preprocessed data derived from at least one of multiple key performance indicator measurements, equipment specifications, and tolerance levels.

5. The method of claim 1, wherein extracting multiple design guidelines further comprises: Identify historical user preferences and common design patterns from existing HMI screen templates.

6. The method of claim 1, wherein the sent prompt further comprises: Additional contextual information, including operator skill level and plant layout, is included to tailor at least one generated HMI screen template to specific user needs.

7. The method of claim 1, further comprising: Based on usability metrics, multiple layout modifications, color schemes, and font styles are automatically suggested for at least one generated HMI screen template.

8. The method of claim 1, further comprising: Provide operators with a user interface to preview and customize the generated HMI screen templates before deployment.

9. The method of claim 1, further comprising: Detect operator feedback associated with the interaction with the generated human-machine interface (HMI) screen template; The machine learning (ML) model is updated based on the detected operator feedback; and The generated HMI screen templates are modified using an updated ML model to enhance their usability for operators.

10. The method of claim 9, further comprising: Based on operational feedback and evolving factory requirements, reinforcement learning is applied to iteratively improve machine learning (ML) models.

11. The method of claim 9, wherein updating the ML model further comprises: Analyze operator interaction patterns with HMI screen templates to identify areas for usability improvement.

12. A programming system (102) for automatically generating optimized human-machine interface (HMI) screen templates, comprising: One or more processors; and A memory coupled to the one or more processors, wherein the memory includes an automation module stored in the form of machine-readable instructions executable by the one or more processors, wherein the automation module is configured to perform the method according to any one of claims 1-11.

13. A computer program product storing machine-readable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-11.

Images