System and method for generating HMI graphics

Abstract

The present disclosure relates to systems and methods for generating HMI graphics. The present disclosure relates to computer-implemented methods and arrangements for generating human-machine interface HMI graphics associated with an industrial automation system that implements machine learning. The method includes receiving (S402) a migration HMI graphic. The migration HMI graphic is obtained from an old HMI graphic, and the migration HMI graphic includes multiple industrial objects. In addition, the method includes identifying (S404) missing industrial objects in the above-mentioned migration HMI graphic using a historical tag data set, the historical tag data set including multiple tagged HMI graphics of industrial objects, the industrial objects including graphic objects and data objects. Once the missing industrial object is identified, the migration HMI graphic is corrected (S406) to include the identified missing industrial object, and a corrected HMI graphic is generated (S408). In addition, the method includes transmitting (S408) to a computing device a notification indicating that the HMI graphic has been corrected.

Description

Technical Field

[0001] The present disclosure relates generally to industrial automation systems and more particularly to systems and methods for generating human-machine interface (HMI) graphics associated with an industrial automation system implementing machine learning. Background Art

[0002] Industrial automation systems include many sensors, actuators, and controllers for monitoring and controlling operations and processes in industry. Such controllers include applications with graphical screens for operators to monitor the operation of the industrial automation system. The graphical screens are provided on a computing device using a human-machine interface (HMI).

[0003] HMI is a common connection tool to interface with controllers. As needed, HMI exchanges information between operators and machines to coordinate multiple industrial processes. The goal of HMI design is to enable operators to manage manufacturing processes through improved decision-making while preventing failure situations to reduce downtime. Therefore, the use of context / situational data to perceive changes in information and information flows provides an effective operator HMI design practice. The migration or upgrade strategy of an existing HMI design to the latest version depends on multiple factors, such as improved user interfaces, added features, and efficient communication. The main challenge in this article is to design and debug the target HMI graphics as efficiently and economically as possible.

[0004] There are a variety of HMI migration tools; however, these migration tools are not 100% accurate and require manual intervention to produce the target HMI graphics. The manual verification task consumes a lot of time. More specifically, when 300-400 HMIs need to be intervened, manual intervention for each HMI will consume a lot of time and be prone to errors. In addition, for complex HMI objects, there are a large number of objects of different sizes and shapes, and the manual verification of these objects will be very cumbersome, thereby reducing the HMI migration efficiency. Therefore, this manual solution may not be suitable in the HMI migration process, and an improved HMI migration arrangement is needed to improve the HMI migration efficiency and accuracy. Summary of the invention

[0005] The inventors of the present invention, after creative and insightful reasoning, realized that there are foreseen problems as more and more HMI graphic transformations need to be implemented in future markets as discussed above and below.

[0006] The present disclosure seeks to disclose a system and method to infer and provide an efficient / automatic way to generate HMI graphics and / or convert HMI graphics from one version to another, for example, from a legacy HMI to a migration HMI.

[0007] The present disclosure provides a solution that aims to mitigate, alleviate or eliminate one or more of the above-mentioned deficiencies and disadvantages in the art, alone or in any combination, and provides a solution of an automatic / efficient mechanism to provide improved HMI graphics during HMI migration.

[0008] A first aspect is a computer-implemented method for generating an HMI graphic associated with an industrial automation system. The method includes receiving a migration HMI graphic. The migration HMI graphic is obtained from an old HMI graphic, and the migration HMI graphic includes multiple industrial objects. In addition, the method includes identifying lost industrial objects in the above-mentioned migration HMI graphic using a historical tag data set, the historical tag data set including multiple tagged HMI graphics of industrial objects, the industrial objects including graphic objects and data objects. Once the lost industrial object is identified, the migration HMI graphic is corrected to include the identified lost industrial object, and a corrected HMI graphic is generated and output to the system. In some aspects, a notification indicating that the corrected HMI graphic is transmitted to a computing device or any other portable device of an operator operating an industrial operating system or any related individual.

[0009] In some embodiments, the industrial objects in the legacy HMI graphic are associated with an older version of the industrial automation system / industrial device, and the industrial object(s) in the migration HMI graphic are associated with a newer version of the industrial automation system / industrial device.

[0010] In some embodiments, the at least one missing industrial object indicates that the at least one industrial object is missing in the context of the migrated HMI graphic (118).

[0011] In some embodiments, a method for identifying missing objects in a migration HMI graphic includes identifying industrial objects in both the migration HMI graphic and the legacy HMI graphic using object recognition techniques. The method also includes determining differences between the identified industrial objects in the migration HMI graphic and the legacy HMI graphic, and identifying the missing industrial objects in the migration HMI graphic based on the determined differences and using a historical HMI tag data set for at least one industrial object.

[0012] In some embodiments, the accuracy of the corrected HMI graphics including the industrial objects is evaluated using a historical HMI tag dataset including a plurality of tagged HMI graphics (or images) of the industrial objects. In certain aspects, the historical HMI tag dataset may be acquired based on domain knowledge / domain experts.

[0013] In some embodiments, a method for identifying missing objects in a migration HMI graphic includes acquiring placeholders in the migration HMI graphic, detecting locations of missing industrial objects in the migration HMI graphic based on the placeholders, and creating a bounding box around the identified placeholders.

[0014] In some embodiments, the bounding box is created using template matching and image processing techniques.

[0015] In some embodiments, the method further comprises evaluating the identified missing industrial object using a one-to-one mapping between a plurality of industrial objects. In some aspects, the one-to-one mapping model utilizes image processing techniques based on a historical HMI tag dataset comprising a plurality of tagged HMI graphics of industrial objects / equipment.

[0016] The second aspect is an industrial automation system that implements machine learning to generate HMI graphics. The industrial automation system includes a processor, a memory, and a graphic object detection model coupled to the processor and the memory. The graphic object detection model is configured to receive a migration HMI graphic. The migration HMI graphic is obtained from an old HMI graphic, and the migration HMI graphic includes multiple industrial objects. In addition, the graphic object detection model is configured to use a historical tag data set to identify lost industrial objects in the above-mentioned migration HMI graphics, and the historical tag data set includes multiple tagged HMI graphics of industrial objects, and the industrial objects include graphic objects and data objects. Once the lost industrial object is identified, the migration HMI graphic is corrected to include the identified lost industrial object, and the corrected HMI graphic is generated and output. In some aspects, a notification indicating that the corrected HMI graphic is also transmitted to a computing device or any other portable device of an operator operating an industrial operating system or any related individual.

[0017] Advantageously, some embodiments provide for automatic correction of HMI graphics (ie, legacy HMI) that would otherwise require manual effort to correct the HMI graphics (ie, migration HMI graphics) during the HMI migration process.

[0018] Advantageously, some embodiments improve the efficiency and accuracy of HMI migration by implementing machine learning and image processing techniques.

[0019] Advantageously, some embodiments leverage industry domain knowledge and machine vision-based intelligence for automatic validation and correction of HMI graphics during HMI migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The foregoing will become apparent from the following more particular description of exemplary embodiments, as illustrated in the accompanying drawings, in which like reference numerals refer to the same parts throughout the different views. The drawings are not necessarily drawn to scale, emphasis instead being placed upon illustrating exemplary embodiments.

[0021] Figure 1A and 1B is a block diagram illustrating an embodiment of an industrial automation system;

[0022] Figure 2 It shows the realization Figure 1A A block diagram of the computing environment of an industrial automation system;

[0023] Figure 3 According to some example embodiments Figure 1A Example configuration of a graphical object detection model for an industrial automation system;

[0024] Figure 4 is a flowchart schematically depicting a proposed method according to some example embodiments;

[0025] Figure 5 is a flow chart schematically depicting a process for identifying missing industrial objects and restoring them onto a migration HMI graphic according to some example embodiments; and

[0026] 6-8 are example scenarios of HMI graphics migration from a legacy HMI graphic to a migration HMI graphic and further to a corrected HMI graphic, according to some example embodiments. DETAILED DESCRIPTION

[0027] Aspects of the present disclosure will be described more fully below with reference to the accompanying drawings. However, the systems and methods disclosed herein can be implemented in many different forms and should not be construed as limited to the aspects set forth herein. The same numbers in the accompanying drawings always refer to the same elements.

[0028] The terms used herein are only used to describe specific aspects of the present disclosure and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0029] Some example embodiments presented herein relate to systems and methods for generating HMI graphics associated with an industrial automation system implementing machine learning.As part of the development of the example embodiments presented herein, a problem will first be identified and discussed.

[0030] HMI migration tools are used to convert / migrate HMI graphics from one version to another. However, these HMI migration tools are not 100% accurate and require manual intervention to produce the target HMI graphics. In addition, the target HMI graphics are manually corrected so that they look aesthetically pleasing and have proper alignment, including each graphic element and its resolution, maintaining accurate conversion of static and dynamic information, etc. This formatted HMI needs to be manually checked and any differences corrected, which reduces the efficiency of HMI migration.

[0031] Advantageously, some embodiments are directed to improving the efficiency and accuracy of HMI graphics migration by enabling automated intervention of migrating HMI graphics that automatically verifies and corrects differences in the HMI graphics.

[0032] Advantageously, some embodiments provide adequate support for migration of complex legacy graphic objects, reduced migration effort and cost, more automation and less manual engineering effort, and improved consistency between HMI graphics.

[0033] Advantageously, some embodiments enable the utilization of industrial domain knowledge during HMI conversion and machine vision-based intelligence to automatically validate and correct HMI graphics.

[0034] Advantageously, some embodiments provide a benchmarking platform for comparing HMI migration tools. That is, a comparison function between any two arbitrary HMI migration tools is provided.

[0035] As shown in FIG1 , an industrial automation system 100 is used to control a plurality of industrial devices / components that are responsible for handling different processes and machines in a plurality of industrial plants (or any technical installations). For example, process control when manufacturing a product (i.e., an insulator in an insulator manufacturing plant) requires continuous monitoring of industrial devices / components (e.g., pumps, valves, motors, boilers, turbines, and generators (BTGs)) that are required to be used when processing raw materials (e.g., clay, porcelain, etc.) in the process of manufacturing the insulators. The industrial automation system 100 facilitates such continuous monitoring of industrial devices / components in such a plurality of industrial plants. Each industrial plant represents / indicates one or more manufacturing facilities for producing at least one product (i.e., an insulator) or other material.

[0036] The industrial automation system 100 includes a plurality of industrial devices 102 (or industrial objects) responsible for controlling / monitoring and / or manufacturing products (e.g., insulators) or materials. The industrial automation system 100 also includes one or more sensors 104a and one or more actuators 104b. The sensors 104a and the actuators 104b are communicatively coupled to the plurality of industrial devices 102. In some aspects, the sensor 104a can measure / monitor the configuration and / or operational readings of the industrial device 102. Configuration and / or operational readings (e.g., temperature, pressure, current flow rate), and further trigger an alarm in the event of any abnormality when reading / monitoring such configuration and / or operational readings. In some aspects, the actuator 104b is responsible for driving the operational functions of the industrial device 102. For example, in the event that the sensor 104a issues an alarm, the actuator 104b can be responsible for closing the valve. Therefore, each actuator 104b is responsible for operating one or more conditions of the industrial device 102 in the industrial automation system 100.

[0037] In some aspects, the connection between the sensor 104a / actuator 104b and the plurality of industrial devices 102 is via a network interface 106, such as an Ethernet network, an electrical signal network, Bluetooth, or any other network or type of network(s). In some other aspects, the network interface 106 between the sensor 104a and the actuator 104b facilitates the transmission of data between the sensor 104a and the actuator 104b.

[0038] All measurements from the sensors 104a and actuators 104b are transmitted to the industrial controller 108 via the network interface 106. In some aspects, the industrial controller 108 may include, for example, a distributed control system (DCS), a programmable logic controller (PLC), a proportional integral derivative (PID) controller, etc. In one example, each controller 108 may also represent a computing device. The controller 108 is responsible for controlling the operation of the sensors 104a and actuators 104b. For example, the controller 108 may operate the actuator 104b using data obtained from the sensor 104a. In some other aspects, the controller 108 may be used to control / create / modify the configuration of the sensors 104a and actuators 104b. In some other aspects, the controller 108 may be configured to create and control the logical connections between each sensor 104a and each actuator 104b. In addition, the controller 108 may control the configuration and / or connection between each sensor 104a / actuator 104b and the industrial component 102.

[0039] Each controller 108 may also be responsible for controlling one or more aspects of an industrial process, such as controlling the operation of a boiler used to process raw materials, such as clay used to make insulators.

[0040] An operator of the industrial automation system 100 can access and interact with the controller 108 using a human machine interface (HMI) 110. In some aspects, the controller 108 can have an HMI embedded / installed (not shown) within the controller 108. Access and interaction with the controller 108 can be performed via a computing device 112. In some aspects, access to the controller 108 can include, for example, access to the connection and configuration of sensors 104a, actuators 104b, and some industrial devices 102. In addition, in some aspects, interaction with the controller 108 can include, for example, identifying different states of the industrial process, such as alarm states, operating values ​​of various industrial devices 102, etc.

[0041] In some aspects, the computing device 112 may include, for example, any computer, such as a stand-alone computer, a laptop computer, a personal computer (PC), multiple display screens, a tablet computer, a portable device, etc. The computing device 112 may include a display unit and a graphics processing unit (GPU), the GPU being configured to process and display graphical images (e.g., HMI graphics 114) of the industrial devices 102 and the connections between each industrial component 102. In one example, the HMI graphic 114 indicates a graphical form of the connection between each industrial object 102 and each sensor 104a / actuator 104b, also displayed by the computing device 112. In addition, the HMI graphic 114 indicates process lines representing process connections, instrument lines indicating process recordings or measurement points, control connections, etc. Each industrial component 102 may be uniquely represented / graphically displayed on the display unit of the computing device 112 using the HMI. For example, each actuator 104b may be graphically represented as a valve, while each sensor 104a may be represented as an instrument (e.g., Figures 7A-7C shown).

[0042] As described above, each industrial component 102 can be uniquely represented / graphically displayed on the display unit of the computing device 112, and thus similarly, each controller 108 (or each version of operating firmware of the controller 108) includes its own HMI graphics library that is different from other controllers or one version of operating firmware of other controllers. Therefore, whenever a version of the controller 108 is migrated to other controllers or to a newer / updated version of operating firmware, migration of the HMI graphics is always required. In some aspects, the HMI graphics conversion model / HMI migration tool 116 can be used to convert or migrate the old HMI graphics 114 to the new HMI graphics (i.e., migrate the HMI graphics 118, such as Figure 1B ) to provide an enhanced user interface experience.

[0043] The migration or upgrade strategy of the old HMI graphics 114 design to the migration HMI graphics 118 depends on many factors, such as improved user interfaces, added features, and efficient communication. The main challenge in this article is to design and debug the target HMI graphics (final HMI graphics) as efficiently and economically as possible. Existing / traditional HMI graphic conversion tools migrate graphic data from one HMI to another HMI. However, due to the dynamic nature of functional specifications in new applications, existing HMI graphic migration tools may not have all the functions of migrating from old HMI graphics to new HMI graphics. Most HMI graphic migration tools are still inaccurate due to incorrect symbol or label mapping. Therefore, manual intervention is required to fix inaccuracies (which increases the chance of error) when formatting the target HMI graphics. This manual work may be expensive and does not allow users to obtain the full benefits of automatic conversion.

[0044] Unlike conventional mechanisms for migrating HMI graphics, the present invention seeks to implement / incorporate machine learning mechanisms to automatically identify inaccuracies (ie, missing industrial graphic objects) in migrating HMI graphics during the HMI conversion process.

[0045] In order to achieve the desired technical effect of the present invention, namely, automatically identifying inaccuracies in the migrated human-machine interface graphics during the HMI conversion process, the present invention attempts to provide a graphic object detection model 120 (interchangeably used as an HMI graphic object detection model 120) configured to perform the proposed method. In some aspects, the graphic object detection model 120 is configured to obtain the migrated HMI graphic 118 (the output of the HMI migration tool 116). The graphic object detection model 120 is then configured to automatically identify inaccuracies in the migrated HMI graphic 118 and correct the inaccuracies in the migrated HMI graphic 118. Once corrected, the graphic object detection model 120 can be configured to output a corrected HMI graphic 122 (i.e., including all industrial graphic objects and text information present in the old HMI graphic 114), such as Figure 1B As shown, the accuracy and efficiency of HMI migration are improved.

[0046] In some aspects, the graphical object detection model 120 can be hosted on a cloud computing server and can be accessed by an operator / individual using any suitable network interface. In some other aspects, the graphical object detection model 120 can be part of the HMI migration tool 116.

[0047] In some aspects, the present invention provides a benchmark platform for comparing an HMI conversion tool (i.e., an HMI migration tool 116) (integrated with a graphic object detection model 120) with any other HMI migration tool (not shown). For example, a comparison function between any two arbitrary HMI conversion tools is constructed in the following manner: a) using the HMI conversion tool 116 to identify missing graphic objects and text information on the migration HMI graphic 118, b) repeating step a using other HMI migration tools). In addition, the missing information is checked (in step c)) on the display unit of the computing device 112 to compare the HMI migration tool 116 and the other migration tools. The dashboard gives a detailed analysis of the computing device 112 to display the legacy HMI graphics 114 from the legacy system, and the option to choose between the two HMI migration tools. The output of the comparison after performing the above steps a, b, and c is displayed on the display unit of the computing device 112. Therefore, the benchmark platform indicates a comparison between HMI migration tools that can be effectively used by developers, designers, and operators to analyze and compare across HMI migration tools.

[0048] refer to Figure 2 The computing environment 200 may include a processing unit 201, one or more memory devices 202 (referred to herein as memory 202), a storage unit 203, an input unit 204, and an output unit 205. The computing environment 200 may also include one or more buses 206 that functionally couple the various components of the computing environment 200.

[0049] Memory 202 may include volatile memory (memory that retains its state when powered on) such as random access memory (RAM) and / or non-volatile memory (memory that retains its state even when not powered on) such as read-only memory (ROM), flash memory, ferroelectric RAM (FRAM), etc. Persistent data storage (as that term is used herein) may include non-volatile memory. In some example embodiments, volatile memory may enable faster read / write access than non-volatile memory. However, in some other example embodiments, certain types of non-volatile memory (e.g., FRAM) may enable faster read / write access than certain types of volatile memory. In some examples, storage unit 203 may be equivalent to memory 202. In various implementations, memory 202 may include a variety of different types of memory, such as various types of static random access memory (SRAM), various types of dynamic random access memory (DRAM), various types of unchangeable ROM, and / or writable variants of ROM, such as electrically erasable programmable read-only memory (EEPROM), flash memory, etc. The memory 202 may include main memory and various forms of cache memories, such as instruction cache(s), data cache(s), and translation lookaside buffer(s) (TLBs), etc. In addition, cache memories such as data caches may be multi-level caches organized as a hierarchy of one or more cache levels (L1, L2, etc.).

[0050] Storage unit 203 may include removable storage devices and / or non-removable storage devices, including but not limited to magnetic storage devices, optical disk storage devices, and / or tape storage devices. Storage unit 203 may provide non-volatile storage of computer executable instructions and other data. Removable and / or non-removable storage unit 203 and memory 202 are examples of computer readable storage media (CRSM).

[0051] In some aspects, the storage unit 203 stores a historical HMI tag dataset 207. The historical HMI tag dataset 207 includes a plurality of tag HMI graphics of the industrial objects 102. The tag HMI graphics are based on industrial domain knowledge (obtained from various sources, such as portals of industrial domain experts, online industrial domain portals, industry-specific domain portals, etc.). For example, the industrial domain knowledge may include all information about missing graphic objects in the migration HMI graphics 114 and their corresponding areas on the migration HMI graphics 114.

[0052] Therefore, the present invention utilizes industrial domain knowledge (ie, historical HMI tag dataset 207) during HMI graphic conversion and machine learning (or machine vision) based intelligence to perform automatic validation and correction of HMI graphics.

[0053] Storage unit 203 may store computer executable code, instructions, etc. that may be loaded into memory 202 and executable by processing unit 201 to cause processing unit 201 to perform or initiate various operations. Storage unit 203 may additionally store data that may be copied to memory 202 during execution of the computer executable instructions for use by processing unit 201. Furthermore, output data generated as a result of execution of the computer executable instructions by processing unit 201 may initially be stored in memory 202 and may ultimately be copied to storage unit 203 for non-volatile storage.

[0054] More specifically, the storage unit 203 may store an operating system (O / S); a database 102 configured to access the memory 202; and one or more program modules, applications, engines, managers, computer executable codes, scripts, etc., such as various modules of the graphic object detection model 120. Any component described as being stored in the storage unit 203 may include any combination of software, firmware, and / or hardware. The software and / or firmware may include computer executable instructions (e.g., computer executable program code) that may be loaded into the memory 202 for execution by one or more of the processing units 201 to perform any of the corresponding operations described previously.

[0055] The processing unit 201 may be configured to access the memory 202 and execute computer executable instructions loaded therein. For example, the processing unit 201 may be configured to execute computer executable instructions of various program modules, applications, engines, managers, etc. of the graphic object detection model 120 to cause or facilitate various operations according to one or more embodiments of the present disclosure to be performed. The processing unit 201 may include any suitable processing unit capable of accepting data as input, processing the input data according to the stored computer executable instructions, and generating output data. The processing unit 201 may include any type of suitable processing unit, including but not limited to a central processing unit, a microprocessor, a reduced instruction set computer (RISC) microprocessor, a complex instruction set computer (CISC) microprocessor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SoC), a digital signal processor (DSP), etc. In addition, the processing unit 201 may have any suitable micro-architecture design, which includes any number of constituent components, such as registers, multiplexers, arithmetic logic units, a cache controller for controlling read / write operations to a cache memory, a branch predictor, etc. The micro-architecture design of the processing unit 201 may be capable of supporting any one of a variety of instruction sets.

[0056] The input unit 204 and the output unit 205 may facilitate the receipt of input information from one or more I / O devices by the graphic object detection model 120 and the output of information from the graphic object detection model 120 to one or more I / O devices. The I / O device may include any of a variety of components, such as a display or screen having a touch surface or a touch screen; an audio output device for producing sound, such as a speaker; an audio capture device, such as a microphone; an image and / or video capture device, such as a camera; a haptic unit; and the like. Any of these components may be integrated into the components of the computing environment 200 or only into the industrial automation system 100, or may be separate. The I / O device may also include, for example, any number of peripheral devices, such as data storage devices, printing devices, and the like. The input unit 204 and the output unit 205 may also include an I / O interface for external peripheral device connection, such as a universal serial bus (USB), FireWire, Thunderbolt, an Ethernet port, or other connection protocols that can be connected to one or more networks. .

[0057] The I / O interface(s) may also include one or more connection ports to connect to one or more controllers 108, interfaces for network cables, and to connect to other computing devices (similar to computing device 112). Connections to the ports may be via electrical signal cables, HMI cables, etc. In addition, the I / O interface(s) may also include one or more antennas to connect to one or more networks via a local area network (LAN) or, alternatively, via a wireless local area network (WLAN) (e.g., Wi-Fi) wireless, Bluetooth, and / or wireless networking radio.

[0058] The bus(es) 206 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may allow information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between the various components of the visual perception module 103. The bus(es) 206 may include, but are not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, etc.

[0059] refer to Figure 3 , the graphic object detection model 120 may include an image processing unit 302 , a text mining unit 304 , a machine learning (ML) unit 306 , and a graphic object generation unit 308 .

[0060] The graphic object detection model 120 may be configured to receive a migration HMI graphic 118 from the HMI migration tool 116 using the image processing unit 302. The migration HMI graphic 118 is obtained from the old HMI graphic 114. That is, the migration HMI graphic 118 is an output HMI graphic from the HMI migration tool 116 (responsible for migrating the old HMI graphic 114). In some aspects, the migration HMI graphic 118 includes one or more industrial objects 102 in the form of graphic objects or data objects. The graphic object may include, for example, a sensor 104a, an actuator 104b, an industrial object 102 (e.g., a boiler, a pump, etc.), and a graphical representation of a connector (or an indicator, such as an arrow indicator) between the industrial object 102 and between the sensor 104a and the actuator 104b. In addition, the data object may include, for example, text or any number associated with each graphical representation of the sensor 104a, the actuator 104b, and the industrial object 102. This is shown, for example, in the form of an old HMI graphic 114 and a corrected HMI graphic 122.

[0061] Additionally, the HMI graphic object detection model 120 may be configured to identify one or more missing industrial objects in the migration HMI graphic 118. Components of the graphic object detection model 120 (e.g., the image processing unit 302, the text mining unit 304, and the machine learning (ML) unit 306) communicate with the storage unit 203 to utilize the historical HMI tag dataset 207 during identification of missing industrial objects in the migration HMI graphic 118 as described above.

[0062] In some aspects, the image processing unit 302 can be configured to identify the missing industrial objects and the corresponding locations of the missing industrial objects in the migration HMI graphic 118 based on object recognition techniques that identify HMI graphic objects in both the legacy HMI graphic 114 and the migration HMI graphic 118. According to one aspect of the invention, in order to find the missing industrial objects in the migration HMI graphic 118, the image processing unit 302 can be configured to determine the differences between the identified industrial objects in the legacy HMI graphic 114 and the migration HMI graphic 118 using image mapping.

[0063] In certain aspects, the location of the missing industrial objects in the migration HMI graphic 118 can be obtained by detecting the presence of placeholders / default objects in the migration HMI graphic 118. The placeholders / default objects in the migration HMI graphic 118 are obtained during the conversion of the legacy HMI graphic 114. That is, when the legacy HMI graphic 114 is converted using the HMI migration tool 116, the converted HMI graphic (i.e., the migration HMI graphic 118) includes one or more placeholders / default objects that indicate the missing industrial objects.

[0064] In some aspects, the image processing unit 302 can be configured to use a combination of template matching and image processing methods / models to obtain a bounding box around the placeholder / default object. The bounding box indicates the location of the missing industrial object in the migration HMI graphic 118 and can be used as an input to the ML unit 306. For example, the bounding box is placed around the placeholder and these are potential areas (inputs) for the ML unit 306 to identify the missing industrial object. In some aspects, the pixel input associated with the old HMI graphic 114 in the area surrounded by the bounding box is obtained by the ML unit 306 for detecting possible industrial objects lost in the migration HMI graphic 118.

[0065] In some aspects, the ML unit 306 implements a convolutional neural network (CNN) model to identify relationships between industrial objects 102 of the industrial automation system 100 using the historical HMI tag dataset 207. The image processing unit 302 communicates with the ML unit 306 implementing the NN. The ML unit 306 can be configured to classify the industrial objects 102 (identified from the image processing unit 302) in both the legacy HMI graphics 114 and the migration HMI graphics 118 using the historical HMI tag dataset 207. In one example, semantic information of the industrial objects 102 is determined from the historical HMI tag dataset 207, wherein the semantic information includes relationships between the industrial objects 102, contextual connections between the industrial objects 102, and the like. In some embodiments, the ML unit 306 implementing the CNN model is trained using the historical HMI tag dataset 207. In some other embodiments, the ML unit 306 implementing the CNN model can be trained using historical data of automation HMI objects with associated tags, and cross-trained using historical tag data associated with the automation HMI objects. For example, labeled data associated with the automated HMI object may also be generated by user input based on at least one feedback stimulus. In yet another embodiment, the ML unit 306 implementing the CNN model may be trained using unlabeled historical data of the automated HMI object and cross-trained using labeled historical data associated with the automated HMI object.

[0066] In addition to the semantic mapping of the industrial objects 102 and their corresponding regions / locations on the migration HMI graphic 118, the HMI graphic object detection model 120 utilizes a one-to-one mapping to identify the correct industrial objects to be placed in the migration HMI graphic 118. That is, the HMI graphic object detection model 120 replaces the placeholders (indicated by the bounding boxes) with the correct industrial objects in the migration HMI graphic 118.

[0067] Therefore, lost industrial objects in the designated area of ​​the migration HMI graphic 118 can be automatically restored, thereby reducing manual effort or manual intervention. Therefore, a lot of time can be saved by reducing manual work.

[0068] In some aspects, the text mining unit 304 can be configured to identify missing text or data content and its bounding box in the migration HMI graphic 118. For example, the text mining unit 304 implements a scene text recognition technique / model to identify missing text or data content in the migration HMI graphic 118.

[0069] In view of the above, Figure 4 An example method 400 implemented by the HMI graphic object detection model 120 is shown. According to the method 400, a migration HMI graphic 118 is obtained from a legacy HMI graphic 114 (as detailed above). The HMI graphic object detection model 120 then receives the migration HMI graphic 118 (block S402). The migration HMI graphic 118 includes at least one industrial object.

[0070] According to the method 400, at least one missing industrial object is identified in the migration HMI graphic 118 using the historical HMI tag dataset 207 for at least one industrial object (block S404). The at least one industrial object includes a graphical object and a data / text object.

[0071] According to the method 400, the migration HMI graphic 118 is corrected to include the at least one missing industrial object (block S406). Additionally, according to the method 400, a corrected HMI graphic 122 is generated, wherein the corrected HMI graphic 122 incorporates the at least one missing industrial object (block S408).

[0072] According to the method 400, a notification is transmitted to the computing device 112 indicating that the HMI graphic 122 has been corrected (block S410). The notification may include, for example, an alarm configured by the computing device 112 or a user desired configuration, an audible notification such as a beep, etc.

[0073] In view of the above, Figure 5 An example method 500 is shown implemented by the HMI graphic object detection model 120 to identify and repair missing industrial objects in the migration HMI graphic 118. According to the method 500, at least one industrial object is identified 102 in the migration HMI graphic 118 and the legacy HMI graphic 114 (block S502). An object recognition model can be used for such identification of the industrial object 102.

[0074] Furthermore, according to the method 500 , differences between the identified industrial objects in the legacy HMI graphic 114 and the migration HMI graphic 118 are determined (block S504 ).

[0075] Furthermore, according to the method 500, the presence of placeholder / default objects in the migration HMI graphic 118 is identified (block S506). The placeholder / default objects can be used to identify the location of the missing industrial object in the migration HMI graphic 118. In some aspects, the placeholder / default objects are directly obtained from the migration HMI graphic 118, which also indicates that one or more industrial objects are missing (in a particular area) compared to one or more industrial objects in the legacy HMI graphic 114.

[0076] Furthermore, according to the method 500, at least one bounding box is created around the identified at least one placeholder (block S508). To obtain the bounding box of the default object or placeholder, the HMI graphic object detection model 120 utilizes a combination of template matching and image processing techniques.

[0077] In addition, according to the method 500, information about both the missing industrial object and its corresponding region is identified using the bounding box on the migration HMI graphic 118 (block S510). This can be achieved by using the ML unit 306, which utilizes a one-to-one mapping to identify (using the historical HMI tag dataset 207) the correct industrial object to be placed in the migration HMI graphic 118.

[0078] Furthermore, according to the method 500 , the placeholders are replaced with the correct industrial objects on the migration HMI graphic 118 (block S512 ).

[0079] refer to Fig. 6A , the legacy HMI graphic 114 shows a graphical representation of an industrial object (i.e., the main pump 602) on the computing device 112 of the industrial automation system 100. The legacy HMI graphic 114 also shows graphical representations of other industrial objects, including process lines, connection lines, and any text data associated with the industrial objects on the computing device 112 of the industrial automation system 100.

[0080] In some aspects, for the purpose of HMI migration, the legacy HMI graphic 114 is transmitted (as input) to an HMI migration tool 116. The output of the HMI migration tool 116 may be represented as a migration HMI graphic 118 (eg, Figure 6B shown).

[0081] refer to Figure 6B , the migration HMI graphic 118 shows the migration graphical form of the industrial object 102. As mentioned above, these HMI migration tools are not 100% accurate and require human intervention to generate the target HMI graphic. Figure 6BAs shown, the industrial object (i.e., the primary pump 602) is missing in the migration HMI graphic 118. Therefore, according to the legacy method, a manual check is performed on the migration HMI graphic 118 to identify the missing industrial object, i.e., the primary pump 602. This manual review / check of the formatted HMI, and further manual correction of the discrepancy (missing industrial object) reduces the efficiency of the HMI migration.

[0082] Unlike such conventional methods, the proposed method utilizes historical HMI tagged images (of industrial objects) to train the HMI graphic object detection model 120 using machine learning techniques to automatically verify and correct any discrepancies in the migrated HMI graphic 118. In addition, the proposed method also utilizes the object recognition model to resolve conflicts caused by ambiguous or almost similar-looking industrial objects in the migrated HMI graphic 118.

[0083] Reference again Figure 6B , the HMI graphic object detection model 120 creates a bounding box 604 in place of the default object / placeholder to indicate the missing primary pump 602. The HMI graphic object detection model 120 then performs a one-to-one mapping to identify the correct object (including multiple HMI tag images of industrial objects) to be placed in the migration HMI graphic 118 using machine learning techniques and the historical HMI tag dataset 207. The placeholder in this article indicates the location of the missing industrial object.

[0084] Once the correct industrial object to be placed on the migration HMI graphic 118 is identified, the HMI graphic object detection model 120 replaces / restores the placeholder with the correct industrial object on the migration HMI graphic 118. Figure 6C , the missing primary pump 606 replaces the placeholder / default object of the bounding box 602. The corrected HMI graphic 122 is then transmitted for further addition / review (to meet additional customer requirements) and to constitute the target HMI graphic.

[0085] Similar to Fig. 6A , Figure 7B The legacy HMI graphic 114 is shown including a graphical representation of an industrial object (i.e., tank 702) on the computing device 112 of the industrial automation system 100. The legacy HMI graphic 114 also shows graphical representations of other industrial objects, including process lines 704-706 with arrows (downward), process lines, connection lines, and any text data associated with the industrial objects on the computing device 112 of the industrial automation system 100.

[0086] For the purpose of HMI migration, the legacy HMI graphic 114 is transferred (as input) to an HMI migration tool 116. The output from the HMI migration tool 116 may be represented as a migration HMI graphic 118 (eg, Figure 7Bshown).

[0087] refer to Figure 7B , the migration HMI graphic 118 shows a migration graphical version of the industrial object 102. As described above, the industrial object (ie, the tank 702) is missing, and there are differences in the arrowed process lines 704-706 (upward) in the migration HMI graphic 118.

[0088] As described above, the HMI graphic object detection model 120 creates bounding boxes 708 and 710 in place of the default objects / placeholders to indicate the difference in the missing tank 702 and the arrowed process lines 704-706. The HMI graphic object detection model 120 then performs a one-to-one mapping to identify the correct objects (including multiple HMI tag images of industrial objects) to be placed in the migration HMI graphic 118 using machine learning techniques and the historical HMI tag dataset 207.

[0089] Once the correct industrial object to be placed on the migration HMI graphic 118 is identified, the HMI graphic object detection model 120 replaces / restores the placeholder with the correct industrial object on the migration HMI graphic 118. Figure 7C , the missing tank 702 and the differences in the arrowed process lines 704-706 replace the placeholder / default objects of the bounding boxes 708 and 710. The corrected HMI graphic 122 is then transmitted for further addition / review (to meet additional customer requirements) and to constitute the target HMI graphic.

[0090] exist Figure 6A-6C Similar operations performed in are repeated to correct Figure 8A-8C HMI graphics shown. Reference Fig. 8A , a legacy HMI graphic 114 is displayed on the computing device 112 of the industrial automation system 100, the legacy HMI graphic 114 including a graphic form of an industrial object (ie, control valve 802 & KV-1804A and MOL.SIEVE CONTROL) having text buttons and their values.

[0091] For the purpose of HMI migration, the legacy HMI graphic 114 is transmitted (as input) to an HMI migration tool 116. The output of the HMI migration tool 116 may be represented as a migration HMI graphic 118 (eg, Figure 8B shown).

[0092] refer to Figure 8B , the transition HMI graphic 118 shows the transition graphic form of the control valve 802 , and the text button along with its value is missing in the transition HMI graphic 118 .

[0093] As described above, the HMI graphic object detection model 120 creates bounding boxes 804 in place of default objects / placeholders to indicate the missing control valve 802 and text buttons and their values ​​in the migration HMI graphic 118. The HMI graphic object detection model 120 then performs a one-to-one mapping to identify the correct objects to be placed in the migration HMI graphic 118 using machine learning techniques and the historical HMI tag dataset 207 (including multiple HMI tag images of industrial objects).

[0094] Once the correct industrial object to be placed on the migration HMI graphic 118 is identified, the HMI graphic object detection model 120 replaces / restores the placeholder with the correct industrial object on the migration HMI graphic 118. Figure 8C , the missing control valve 802 and text button along with their values ​​replace the placeholder / default objects of the bounding box 804. The corrected HMI graphic 122 is then transferred for further addition / review (to meet additional customer requirements) and to compose the target HMI graphic.

[0095] Thus, as discussed above in FIGS. 6-8 , the present invention provides automatic verification and correction of any discrepancies in the migrated HMI graphics 118 , which significantly saves production / developer time by reducing the manual effort to correct the HMI graphics.

[0096] Aspects of the present disclosure are described with reference to the accompanying drawings (e.g., block diagrams and / or flow charts). It is understood that several entities in the drawings (e.g., blocks of the block diagrams) and combinations of entities in the drawings may be implemented by computer program instructions, which may be stored in a computer-readable memory or loaded onto a computer or other programmable data processing device. Such computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, and / or other programmable data processing device to produce a machine, so that instructions executed by a processor of a computer and / or other programmable data processing device create a device for implementing the functions / actions specified in the block diagrams and / or flow charts.

[0097] In some implementations and according to some aspects of the present disclosure, the functions or steps marked in the box may not occur in the order marked in the operation diagram. For example, depending on the functions / actions involved, two boxes shown in succession may actually be executed substantially simultaneously, or sometimes the boxes may be executed in reverse order. In addition, according to some aspects of the present disclosure, the functions or steps noted in the box may be executed continuously in a loop.

[0098] In the drawings and specification, exemplary aspects of the present disclosure have been disclosed. However, many changes and modifications may be made to these aspects without departing substantially from the principles of the present disclosure. Therefore, the present disclosure should be regarded as illustrative rather than restrictive, and is not limited to the specific aspects discussed above. Therefore, although specific terms are used, they are used only in a general and descriptive sense and not for the purpose of limitation.

[0099] The description of the example embodiments provided herein is presented for the purpose of illustration. The description is not intended to be exhaustive or to limit the example embodiments to the precise form disclosed, and modifications and variations may be made in accordance with the above teachings, or modifications and variations may be obtained from various alternative practices of the embodiments provided. The examples discussed herein are selected and described to explain the principles and properties of the various example embodiments and their practical applications, so that those skilled in the art can utilize the example embodiments in various ways and various modifications suitable for the specific purposes under consideration. The features of the embodiments described herein may be combined in all possible combinations of methods, devices, modules, systems, and computer program products. It should be understood that the example embodiments presented herein may be practiced in any combination with each other.

[0100] It should be noted that the word "comprising" does not necessarily exclude the presence of other elements or steps than those listed, and the word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. It should also be noted that any reference numerals do not limit the scope of the claims, that example embodiments may be implemented at least in part by hardware and software, and that several "means", "units" or "devices" may be represented by the same item of hardware.

[0101] The various example embodiments described herein are described in the general context of method steps or processes, which can be implemented on the one hand by a computer program product embodied in a computer-readable medium, the computer program product including computer-executable instructions, such as program code, which are executed by a computer in a network environment. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), etc. In general, program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer-executable instructions, related data structures, and program modules represent examples of program codes for performing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps or processes.

[0102] In the drawings and the specification, exemplary embodiments have been disclosed. However, many changes and modifications may be made to these embodiments. Therefore, although specific terms are used, they are used only in a general and descriptive sense and not for the purpose of limitation, and the scope of the embodiments is defined by the following claims.

Claims

1. A computer-implemented method for generating a human-machine interface (HMI) graphic, the HMI graphic being associated with an industrial automation system (100) implementing machine learning, the method being performed by a graphical object detection model (120), the method comprising: Receiving ( S402 ) a migration HMI graphic ( 118 ), wherein the migration HMI graphic ( 118 ) is obtained from a legacy HMI graphic ( 114 ), and wherein the migration HMI graphic ( 118 ) includes a plurality of industrial objects ( 102 ); identifying ( S404 ) at least one missing industrial object in the migration HMI graphic ( 118 ) using a historical HMI tag dataset ( 207 ) of at least one industrial object ( 102 ), wherein the at least one industrial object ( 102 ) includes a graphic object and a data object; correcting ( S406 ) the migrated HMI graphic ( 118 ) to include the at least one missing industrial object; generating ( S408 ) a corrected HMI graphic ( 122 ) including the at least one missing industrial object; and transmitting ( S410 ) a notification indicating the corrected HMI graphic ( 122 ) to a computing device; The identification (S404) and the correction (S406) further include: Identifying the presence of a placeholder in the migration HMI graphic (118); identifying a location of the at least one missing industrial object in the migration HMI graphic (118) using the placeholder; utilizing a one-to-one mapping of the at least one industrial object and its location with the historical HMI tag data set (207) to identify a correct industrial object to be placed in the migration HMI graphic (118); and Replace the placeholders with the correct industrial objects.

2. The computer-implemented method of claim 1 , wherein identifying (S404) the at least one missing industrial object in the migration HMI graphic (118) comprises: identifying ( S502 ) the at least one industrial object ( 102 ) in the migration HMI graphic ( 118 ) and the legacy HMI graphic ( 114 ) using object recognition technology; Determining ( S504 ) differences between the identified industrial objects in the legacy HMI graphic ( 114 ) and the identified industrial objects in the migration HMI graphic ( 118 ); as well as The at least one missing industrial object is identified (S506) in the migration HMI graphic (118) based on the determined differences and using the historical HMI tag data set (207) of the at least one industrial object (102).

3. The computer-implemented method of any one of claims 1 and 2, wherein identifying the at least one missing industrial object in the migration HMI graphic (118) comprises: At least one bounding box is created ( S508 ) around the identified at least one placeholder. The computer-implemented method of claim 3 , wherein the at least one bounding box is created using template matching and image processing.

5. The computer-implemented method of any one of claims 1 to 2, wherein scene text recognition technology is used to identify the lost industrial object.

6. The computer-implemented method of claim 1, wherein the notification comprises at least one of a text notification, an audio notification, and a graphical notification, and wherein the historical HMI tag dataset (207) comprises a plurality of HMI tag images of the at least one industrial object (102).

7. The computer-implemented method of claim 1, wherein the at least one missing industrial object indicates that at least one industrial object is missing in a context of the migration HMI graphic (118).

8. An industrial automation system (100) implementing machine learning to generate human-machine interface (HMI) graphics, wherein the industrial automation system (100) comprises: Processing unit (201); A memory (202), communicatively coupled to the processing unit (201); as well as A graphic object detection model (120) is communicatively coupled to the processing unit (201) and the memory (202), wherein the graphic object detection model (120) is configured to: receiving a migration HMI graphic (118), wherein the migration HMI graphic (118) is obtained from a legacy HMI graphic (114) and wherein the migration HMI graphic (118) includes at least one industrial object (102), identifying at least one missing industrial object in the migration HMI graphic (118) using a historical HMI tag dataset (207) of the at least one industrial object (102), wherein the at least one industrial object (102) includes a graphic object and a data object, correcting the migration HMI graphic (118) to include the at least one missing industrial object, generating a corrected HMI graphic (122) including the at least one missing industrial object, and transmitting a notification indicating the corrected HMI graphic (122) to a computing device; The graphic object detection model (120) is further configured to: Identifying the presence of a placeholder in the migration HMI graphic (118); identifying a location of the at least one missing industrial object in the migration HMI graphic (118) using the placeholder; utilizing a one-to-one mapping of the at least one industrial object and its location with the historical HMI tag data set (207) to identify a correct industrial object to be placed in the migration HMI graphic (118); as well as Replace the placeholders with the correct industrial objects.

9. The industrial automation system (100) of claim 8, wherein the graphical object detection model (120) configured to identify the at least one missing industrial object in the migration HMI graphic (118) comprises: identifying the at least one industrial object (102) in the migration HMI graphic (118) and the legacy HMI graphic (114) using object recognition techniques; determining differences between the identified industrial objects in the legacy HMI graphic (114) and the identified industrial objects in the migration HMI graphic (118); as well as The at least one missing industrial object is identified in the migration HMI graphic (118) based on the determined differences and using the historical HMI tag data set (207) for the at least one industrial object.

10. The industrial automation system (100) according to any one of claims 8 and 9, wherein the graphic object detection model (120) configured to identify the at least one missing industrial object in the migration HMI graphic (118) comprises: At least one bounding box is created around the identified at least one placeholder.

11. The industrial automation system (100) according to claim 8, wherein scene text recognition technology is used to identify the lost industrial object.

12. The industrial automation system (100) of claim 8, wherein the notification comprises at least one of a text notification, an audio notification, and a graphical notification, and wherein the historical HMI tag dataset (207) comprises a plurality of HMI tag images of the at least one industrial object (102).

13. A non-transitory computer readable medium having a computer program including program instructions thereon, the computer program being loadable into a graphical object detection model (120) and configured to cause execution of the method according to any one of claims 1 to 7 when the computer program is run by the graphical object detection model (120).

Images