Automated extraction of asset data from engineering data sources

Abstract

A system and method for controlling an industrial plant includes a process for inputting an engineering diagram of a unit of the industrial plant, the engineering diagram including symbols representing assets of the industrial plant; a process for extracting one or more assets from the engineering diagram using machine learning to recognize the one or more assets, the one or more assets being equipment, meters, connectors, and lines, the lines relating the equipment, meters, and connectors to one another; a process for using machine learning to determine one or more relationships among the equipment, meters, connectors, and lines to recognize the one or more relationships; and a process for creating a flow graph from the equipment, meters, connectors, and lines and the relationships among the equipment, meters, connectors, and lines.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This patent application claims the benefit of priority to U.S. Provisional Application No. 62 / 823,469, entitled "Systems and Methods for Performing Industrial Plant Diagnostics and Operations," filed March 25, 2019, U.S. Provisional Application No. 62 / 842,929, entitled "Systems and Methods for Performing Industrial Plant Diagnostics," filed May 3, 2019, and U.S. Provisional Application No. 62 / 823,377, entitled "Systems and Methods for Detecting and Predicting Faults in an Industrial Process Automation System," filed March 25, 2019, which are incorporated herein by reference. This application also relates to the subject matter of U.S. Nonprovisional Application No. 16 / 021,867, entitled "Machine Learning Analysis of Piping and Instrumentation Diagrams," filed June 28, 2018, which application is incorporated herein by reference.

[0002] Aspects of the present disclosure generally relate to industrial process automation and control systems. More particularly, aspects of the present disclosure relate to systems and methods for extracting plant assets from engineering diagrams and other engineering data sources, creating a plant asset hierarchy from the plant assets, building an ontology knowledge base from the plant asset hierarchy and asset association relationships with other assets, and providing a human machine interface ("HMI") for controlling the industrial process automation and control system based on the plant asset hierarchy and the ontology knowledge base. [Background technology]

[0003] Plant operators are tasked with ensuring that industrial plants operate properly. This requires, for example, monitoring and addressing alarms triggered by plant components (e.g., pumps, valves, tanks, sensors), performing actions on plant components (e.g., stopping components, starting components), and generally overseeing the proper operation of the plant. To perform these tasks, plant operators may use HMIs that provide the plant operator with a visual representation of the plant components and data collected from the plant components. The HMIs may enable the plant operator to interact with the plant components (e.g., to perform actions such as shutting down a component).

[0004] An HMI may be designed for a particular plant to capture the components within that plant, display the data used to operate the plant, and provide an interface for plant operators to initiate any desired actions. Designing an HMI can be expensive, time consuming, and may require plant and process engineering expertise. Today, plant HMIs are created using a process-centric approach that focuses on the process definition. Meanwhile, plant operators have a more asset-centric view of the plant (e.g., plants, areas, units, equipment, devices, set points for specific equipment, etc.) that focuses on the automation control definition. Thus, plant operators may find their current plant HMIs unintuitive and difficult to use. Summary of the Invention [Problem to be solved by the invention]

[0005] Thus, improvements are needed in the area of ​​industrial plant commissioning and operation. [Means for solving the problem]

[0006] Embodiments of the present disclosure provide a system and method for controlling an industrial process automation and control system. The method and system automatically extracts plant assets from engineering diagrams and other plant engineering data sources by using machine learning (ML) models and algorithms, creates a plant asset registry, establishes asset relationships to build an asset hierarchy from the plant assets, generates an ontology knowledge base from the plant asset hierarchy, and provides an HMI for controlling the industrial process based on the plant asset hierarchy and the ontology knowledge base.

[0007] In general, in one aspect, embodiments of the present disclosure relate to a control system for an industrial plant and a corresponding method therefor. The control system (and method therefor) comprises, among other things, one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions that, when executed by the one or more processors, cause the control system to input an engineering diagram for a unit of the industrial plant, the engineering diagram including symbols representing assets of the industrial plant, and to extract one or more assets from the engineering diagram using machine learning and to recognize the one or more assets. The one or more assets include equipment, instruments, connectors, and lines, the lines relating the equipment, instruments, and connectors to one another. The processor-executable instructions also cause the control system to use machine learning to determine one or more relationships between the equipment, instruments, connectors, and lines and to recognize the one or more relationships. The processor-executable instructions also cause the control system to create a flow graph from the equipment, instruments, connectors, and lines and the relationships between the equipment, instruments, connectors, and lines.

[0008] According to any one or more of the preceding embodiments, the process of extracting one or more assets from the engineering diagram includes a geometry-based extraction process and a machine learning-based classification process. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to extract one or more control loops from the engineering diagram based on one or more of equipment, instruments, connectors, and lines. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to extract a unit identifier from the engineering diagram, the unit identifier uniquely identifying a particular unit among a plurality of units in the industrial plant, and / or to further execute a process of extracting a drawing number and a revision number from the engineering diagram, the drawing number uniquely identifying the engineering diagram among a plurality of engineering diagrams of the industrial plant, and the revision number indicating a revision of the engineering diagram that is incremented whenever there is a change in the engineering diagram. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to assign a unique identifier to each equipment, instrument, connector, and line of the engineering diagram using the unit identifier from the engineering diagram. According to any one or more of the preceding embodiments, the process of determining one or more relationships between the equipment, instruments, connectors, and lines includes a process of generating a line-line graph, a process of generating an equipment-line graph, a process of generating a connector-line graph, and a process of generating an instrument-line graph. According to any one or more of the preceding embodiments, the processor-executable instructions further cause the control system to perform a process of merging the line-line graph, the equipment-line graph, the connector-line graph, and the instrument-line graph with each other to generate a flow graph for the engineering diagram, and / or a process of merging flow graphs from multiple engineering diagrams for the industrial plant to create a flow graph for the industrial plant.According to any one or more of the aforementioned embodiments, the processor-executable instructions further cause the control system to display the equipment, gauges, connectors, and lines, and the relationships between the equipment, gauges, connectors, and lines, in an asset hierarchy.

[0009] In general, in another aspect, embodiments of the present disclosure relate to a control system for an industrial plant and a corresponding method therefor. The control system (and method therefor) comprises, among other things, one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions that, when executed by the one or more processors, cause the control system to input data, including structured data and unstructured data, from a plurality of engineering data sources for the industrial plant and extract one or more domain entities from the structured data and unstructured data. The processor-executable instructions also cause the control system to extract instances of the one or more domain entities from the structured data and unstructured data and receive a semantic model constructed based on the one or more domain entities and the instances of the one or more domain entities. The processor-executable instructions further cause the control system to create and store a knowledge graph based on the semantic model and extract information from the knowledge graph to construct a machine interface (HMI) display and control application for the industrial plant.

[0010] According to any one or more of the aforementioned embodiments, the processor executable instructions further cause the control system to use machine learning to extract one or more assets from the unstructured data, recognize the one or more assets, and / or store one or more assets of the industrial plant in an asset hierarchy using a namespace that uniquely identifies each asset. According to any one or more of the aforementioned embodiments, the processor executable instructions further cause the control system to provide a programming interface of an application (API) of the asset registry to allow users and external systems to access and use the asset registry. According to any one or more of the aforementioned embodiments, the processor executable instructions further cause the control system to input data from multiple engineering data sources by inputting one or more of alarm data, cabinet load reports, control databases, cross wiring reports, field wiring indexes, historical data, instrument indexes, asset hierarchies, nested load reports, plant asset indexes, process control narratives, and HMI specifications. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to display the knowledge base as a plurality of interlinked, labeled property nodes, each node representing a domain entity and / or displaying the properties and relationships of each node when the node is selected by a user. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to enable a user to perform a search of the knowledge base using a natural language query, display the nodes responsive to the search along with assets the nodes share in common, and / or display a color-based legend of the nodes.

[0011] In general, in yet another aspect, embodiments of the present disclosure relate to a control system for an industrial plant and a corresponding method therefor. The control system (and method therefor) comprises, among other things, one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions that, when executed by the one or more processors, cause the control system to input data from multiple engineering data sources for the industrial plant, extract one or more assets of the industrial plant from the data using machine learning, and identify the one or more assets. The processor-executable instructions also cause the control system to create an HMI (human machine interface) asset model from the assets, the HMI asset model arranging the assets in a hierarchical structure, and generate an HMI display from the HMI asset model, the HMI display displaying symbols representing one or more assets of the industrial plant and lines representing processes, connections, and data links between the assets. The processor-executable instructions also cause the control system to display the HMI display to a user, and to enable the user to navigate vertically and horizontally along the HMI display, the HMI display dynamically changing which assets are displayed based on changes in the user's position on the HMI display.

[0012] According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to identify all currently occurring alarms for the industrial plant and assets corresponding to the alarms and display the alarms on an HMI display and / or display an HMI display at a plant level, the plant level displaying all currently occurring alarms identified for the industrial plant and assets corresponding to the alarms. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to determine and display on the HMI display assets that are potential root causes of an alarm when a user selects an asset corresponding to an alarm, the control system determining the assets that are potential root causes by finding all assets connected to each other that also have alarms, and / or allowing a user to manually correct the root cause of an alarm on the HMI display and providing the root cause from the user as feedback to the control system. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to automatically zoom in or out on the assets displayed on the HMI display based on a change in the user's position on the HMI screen. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to display the assets on the HMI display as a two-dimensional view, the HMI display allowing a user to navigate horizontally and vertically within the two-dimensional view. According to any one or more of the preceding embodiments, the processor executable instructions further cause the control system to dynamically change the assets displayed on the HMI display based on alarms occurring on the assets, and / or cause the HMI display to display assets upon execution based on one of a static weight assigned to the asset based on its position in an asset hierarchy and a dynamic weight assigned to the asset based on alarms occurring on the asset.

[0013] In general, in yet another aspect, embodiments of the present disclosure relate to a computer-readable medium storing computer-readable instructions for causing one or more processors to perform a method according to any one or more of the preceding embodiments.

[0014] A more particular description of the present disclosure briefly summarized above can be made by reference to various embodiments, some of which are illustrated in the accompanying drawings, which illustrate selected embodiments of the disclosure, but which should not be considered as limiting its scope, in which other equally effective embodiments may be recognized. [Brief description of the drawings]

[0015] [Figure 1] 1 illustrates an example plant asset hierarchy according to an embodiment of the present disclosure. [Diagram 2] 1 illustrates an example engineering diagram that can be used to build a plant asset hierarchy in accordance with an embodiment of the present disclosure. [Diagram 3] 1 illustrates an exemplary industrial plant control system that may be used in a process according to an embodiment of the present disclosure. [Figure 4] 1 illustrates an exemplary intelligent processing module that may be used with an industrial plant control system in accordance with an embodiment of the present disclosure. [Diagram 5] 1 illustrates an example HMI application that may be used with an industrial plant control system in accordance with an embodiment of the present disclosure. [Figure 6] 1 illustrates an exemplary ML-based asset extraction and relationship building according to an embodiment of the present disclosure. [Figure 7A] 1 illustrates an exemplary arrowhead extraction according to an embodiment of the present disclosure. [Figure 7B] 1 illustrates an exemplary arrowhead extraction according to an embodiment of the present disclosure. [Figure 8] 1 illustrates an exemplary pump extraction according to an embodiment of the present disclosure. [Figure 9] 1 illustrates an exemplary connector extraction according to an embodiment of the present disclosure. [Figure 9A] 1 illustrates an exemplary connector extraction according to an embodiment of the present disclosure. [Figure 9B] 1 illustrates an exemplary connector extraction according to an embodiment of the present disclosure. [Figure 9C] 1 illustrates an exemplary connector extraction according to an embodiment of the present disclosure. [Figure 10A] 1 illustrates an exemplary line-line graph according to an embodiment of the present disclosure. [Figure 10B] 1 illustrates an exemplary line-line graph according to an embodiment of the present disclosure. [Figure 11] 1 illustrates an exemplary instrument extraction according to an embodiment of the present disclosure. [Figure 12A] 1 illustrates an exemplary instrument line graph according to an embodiment of the present disclosure. [Figure 12B] 1 illustrates an exemplary instrument line graph according to an embodiment of the present disclosure. [Figure 13A] 1 illustrates an exemplary connector line graph according to an embodiment of the present disclosure. [Figure 13B] 1 illustrates an exemplary connector line graph according to an embodiment of the present disclosure. [Figure 14A] 1 illustrates an exemplary instrument line graph according to an embodiment of the present disclosure. [Figure 14B] 1 illustrates an exemplary instrument line graph according to an embodiment of the present disclosure. [Figure 15A] 1 illustrates a PI&D flow graph according to an embodiment of the present disclosure. [Figure 15B] 1 illustrates an exemplary line graph with devices removed in accordance with an embodiment of the present disclosure. [Figure 16A] 1 illustrates an exemplary line-line graph with devices removed in accordance with an embodiment of the present disclosure. [Figure 16B] 1 illustrates a PI&D flow graph according to an embodiment of the present disclosure. [Figure 17] 1 illustrates an example control loop according to an embodiment of the present disclosure. [Figure 18] 1 illustrates an exemplary method for building a plant engineering domain ontology, according to an embodiment of the present disclosure. [Figure 19] 1 illustrates exemplary alarm data according to an embodiment of the present disclosure. [Figure 20] 1 illustrates an example cabinet load report according to an embodiment of the present disclosure. [Figure 21] 1 illustrates an exemplary control database according to an embodiment of the present disclosure. [Figure 22] 1 illustrates an example cross-wire report according to an embodiment of the present disclosure. [Diagram 23] 4 illustrates an example field wiring index according to an embodiment of the present disclosure. [Figure 24] 1 illustrates exemplary historical data according to an embodiment of the present disclosure. [Diagram 25] 1 illustrates an example instrument index according to an embodiment of the present disclosure. [Figure 26] 1 illustrates an example asset hierarchy according to an embodiment of the present disclosure. [Figure 27] 1 illustrates an exemplary nest load report according to an embodiment of the present disclosure. [Figure 28] 1 illustrates an example plant asset index according to an embodiment of the present disclosure. [Figure 29] 1 illustrates an exemplary process control narrative according to an embodiment of the present disclosure. [Diagram 30] 1 illustrates an exemplary tabular format for data extraction, according to an embodiment of the present disclosure. [Diagram 31] 1 illustrates an exemplary user interface for building a semantic model using entity types, according to an embodiment of the present disclosure. [Diagram 32] 1 illustrates an example HMI display screen showing a knowledge graph according to an embodiment of the present disclosure. [Diagram 33] 1 illustrates an exemplary HMI display screen showing user query results, according to an embodiment of the present disclosure. [Diagram 34] 1 illustrates an exemplary HMI design specification according to an embodiment of the present disclosure. [Diagram 35] 1 illustrates an example input for building an HMI model according to an embodiment of the present disclosure. [Diagram 36] 1 illustrates an example HMI screen showing root cause determination of an alarm according to an embodiment of the present disclosure. [Figure 37] 1 illustrates an example HMI screen showing aggregation of plant level alarms according to an embodiment of the present disclosure. [Figure 38] 1 illustrates an exemplary dynamic zooming according to a user position on an HMI screen according to an embodiment of the present disclosure. [Figure 39] 1 illustrates an exemplary HMI screen showing dynamic zooming in accordance with an embodiment of the present disclosure. [Figure 40A] 1 illustrates an example plant-level alarm aggregation by an HMI according to an embodiment of the present disclosure. [Figure 40B] 1 illustrates an example plant-level alarm aggregation by an HMI according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Wherever possible, the same reference numbers have been used to indicate identical elements common to the figures, however, elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.

[0017] This description and the accompanying drawings illustrate exemplary embodiments of the present disclosure and should not be construed as limiting the claims that define the scope of the present disclosure, including equivalents. Various mechanical, configurational, structural, electrical, and operational changes may be made, including equivalents, without departing from the scope of this description and claims. In some cases, well-known structures and techniques have not been shown or described in detail so as not to obscure the present disclosure. Furthermore, elements and their associated aspects described in detail with reference to one embodiment may, whenever practical, be included in other embodiments in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and not described with reference to a second embodiment, the element may nevertheless be claimed as being included in the second embodiment.

[0018] Referring now to FIG. 1, an example of the type of industrial plant asset hierarchy 100 that can be created in accordance with an embodiment of the present disclosure is shown. As used herein, the term "plant asset" generally refers to any plant equipment, instruments, groups thereof, and their associated devices that are commonly used in industrial plants. At the bottom of the asset hierarchy 100 are equipment (O) 102, such as sensors, monitors, actuators, etc. Just above the equipment 102 are equipment 104, such as vessels, heaters, exchangers, pumps, motors, mixers, etc. One or more instruments 102 can be connected to one or more instruments 104 to form either a closed or open control loop 106 that performs some sub-process in the plant process, such as mixing, heating, boiling, etc. Several types of equipment 104 and instruments 102 can be combined to form a unit (U) 108 that performs some function in the plant process, such as a gas synthesizer unit. A plurality of such units 108 constitute an area (A) 110 , a plurality of areas 110 constitute a plant site (S) 112 , and a plurality of sites 112 form an enterprise (N) 114 .

[0019] Furthermore, in general, each node in the hierarchy 100 can have children and parents, but each child can typically only have one parent. However, assets such as pipes and wires can be treated as equipment, so a pipe carrying material to / from two assets can be logically split into two pipes. A device or instrument can only connect to one piece of equipment. Such a hierarchy 100 arranges and organizes plant assets in a manner that is logical and very easy to understand and use for both the plant control system and the plant operators.

[0020] FIG. 2 illustrates an exemplary engineering drawing or diagram 200 that can be used to build an asset hierarchy 100 (or a portion thereof) according to an embodiment of the present disclosure. The engineering diagram 200 may be any diagram, such as a process flow diagram (PFD) or a piping and instrumentation diagram (P&ID), that graphically depicts how various instruments 102 (e.g., sensors) and equipment 104 (e.g., vessels) in a plant are connected to one another. The exemplary diagram 200 in this example is a P&ID, where each P&ID displays multiple symbols representing devices or instruments, equipment, and other components typically required to implement one unit 108. Connectors 202 located throughout the diagram 200 specify how individual PI&Ds (i.e., units) are connected to form areas, sites, etc. Each diagram 200 also typically has an information block 204, usually in the lower right corner, that displays a drawing identifier 206, a unit identifier, and other information (e.g., revision number) that helps to uniquely identify the PI&D. Other well-known portions of PI&D not specifically mentioned herein include tag numbers, lines (eg, process, electrical, data links, etc.), loops, etc.

[0021] 3 illustrates an example industrial plant control system 300 that can be used to process engineering diagrams 200 and other plant engineering data sources in accordance with an embodiment of the present disclosure. As can be seen, the example system 300 includes one or more processors 302, internal input and / or output ("I / O") interfaces 304, and memory 306, all communicatively coupled and / or electrically connected to one another. These components enable the system 300 to, among other things, extract various instruments 102, equipment 104, control loops 106, units 108, areas 110, and sites 112 from the engineering diagrams 200, lines (e.g., process, electrical, data links, etc.), establish relationships between assets, build a plant asset hierarchy 100, generate an ontology knowledge base from the plant asset hierarchy 100, and provide an HMI for controlling the plant ontology knowledge base.

[0022] The system 300 may be communicatively coupled and / or electrically connected to an external I / O component 308. The external I / O component 308 allows the system 300 to interact and communicate with users and external systems. Communication may be accomplished through one or more communication networks (not explicitly shown) connected to the I / O component 308. The communication networks may include wide area networks and / or local area networks that may be connected to other telecommunication networks, including other WANs, LANs, and / or portions of the Internet or an intranet. The communication network may be any telecommunication network that facilitates the exchange of data, such as, for example, those that operate according to the IEEE 802.3 (e.g., Ethernet) and / or IEEE 802.11 (e.g., Wi-Fi) protocols. In another embodiment, the communication network is any medium that allows data to be physically transferred over a serial or parallel communication channel (e.g., copper wire, optical fiber, computer bus, wireless communication channel, etc.).

[0023] The operation of the various components of the system 300 are generally known in the art and therefore will only be briefly mentioned herein. The processor 302 may be configured to execute processor-executable instructions stored in the memory 306. The memory 306 may be configured to provide the processor-executable instructions to the processor 302 upon request. Among the instructions stored in the memory 306 is an industrial plant control application 310. The industrial plant control application 310 may include several functional modules that cooperate to, among other things, extract assets, establish asset relationships, build asset hierarchies, generate a knowledge base, and provide an HMI. These functional modules may include, for example, an image converter 312, a filtering algorithm 314, a symbol extractor module 316, a tag pre-processing algorithm 318, a tag extractor module 320, a rules engine 322, a line extractor module 324, an intelligent processing module 326, and an HMI application 328.

[0024] The image converter 312 is configured to convert the engineering diagram 200 to an image format. In some embodiments, the image converter 312 takes the diagram 200 in a portable document file (PDF) or other electronic data format and converts the diagram to another image format, such as Portable Network Graphics (PNG), Joint Photographic Experts Group (JPEG), Graphics Interchange Format (GIF), etc. In some embodiments, the image converter 312 creates two image files, one for display and one for computation (e.g., by the filtering algorithms 314, the symbol extractor 316, the tag pre-processing 318, the tag extractor 320, the rules engine 322, the line extractor 324, and the intelligent processing 326).

[0025] The filtering algorithm 314 is configured to process the computational image to obtain an approximate size of the symbol therein. Exemplary symbols include, but are not limited to, those conforming to the International Society of Automation (ISA) standard for instruments, control / display elements, programmable logic controllers (PLCs), valves, pumps, and the like. In some embodiments, the symbol includes an identification character (e.g., "FIC") and a tag number (e.g., "123"). Obtaining the approximate size of the symbol helps normalize the symbol for machine learning purposes (via intelligent processing 326) as described later herein (e.g., to avoid generating training data for symbols of different sizes).

[0026] The symbol extractor 316 is configured to detect symbols extracted from the image. In some embodiments, the symbol extractor 316 applies image processing algorithms to identify possible regions of symbols in the image and then detects symbol types and locations within the image via gross symbol identification techniques. The symbol extractor 316 tracks the number of symbols detected and maintains a running count of newly detected symbols to determine if a new symbol has been detected during a given execution cycle.

[0027] The tag preprocessing 318 is configured to remove the symbol lines from the detected symbols in the computed image, leaving only the tag components. In some embodiments, this involves centering the symbol and then removing the symbol lines from the symbol, leaving only the tag components. The connected pixels are clustered and all those smaller than the standard text size and those larger than the standard text size are removed. Each cluster of pixels is assigned a bounding box that defines a boundary around the cluster for processing purposes. The tag preprocessing 210 then finds bounding boxes at the same level vertically, in left-to-right order. This allows the tag preprocessing 210 to remove non-tag pixels and noise.

[0028] The tag extractor 320 is configured to extract tag components of symbols in the computational image, such as tag names and tag numbers. In some cases, adjacent characters in the tag name and / or tag number are joined together and need to be separated. In these cases, the tag extractor 320 checks for vertical gaps in the characters of the tag, segments the characters, and then performs character recognition using machine learning techniques (via intelligent processing 326). If no vertical gaps exist, the tag extractor 320 determines whether the width-to-height ratio of a given character set is greater than a predefined threshold (e.g., 0.6, etc.). If the width-to-height ratio is greater than the predefined threshold, the tag extractor 320 applies segmentation using vertical pixel density. Areas that show peaks of white pixels are potential areas that are segmented into concatenated characters. The tag extractor 320 then performs character recognition using machine learning techniques.

[0029] The rules engine 322 is configured to validate tags extracted from the computed image based on one or more rules. In some embodiments, the rules are based on the ISA symbol standard and are divided into two categories: major compliance checks (e.g., red category) and minor compliance checks (e.g., orange category). Exemplary major compliance checks include, but are not limited to, verifying that the symbol is one of the valid types (e.g., field device, control room display, etc.) and verifying that the tag name has one or more identifying characters. Exemplary minor compliance checks include, but are not limited to, verifying that the identifying characters in the tag name do not include any numeric characters and that the tag number in the tag name does not include any non-terminal alphabetic characters.

[0030] The line extractor 324 is configured to extract lines between symbols in the computed image. In some embodiments, the extracted lines include piping and connection symbols such as piping, process connections, electrical signals, pneumatic signals, data links, capillary tubes for filling systems, hydraulic signal lines, and guided electromagnetic or sonic signals. As will be appreciated by those skilled in the art, lines are extracted from the image using a geometric line fitting algorithm. Once the lines are extracted, portions of the lines are subjected to one or more machine learning models (via intelligent processing 326) to obtain the type of line as described above. Further details regarding the operation of modules 312-324 can be found in the above-mentioned U.S. Non-Provisional Application No. 16 / 021,867, which is incorporated herein by reference. Intelligent processing 326 and HMI application 328 are further described below.

[0031] 4 illustrates an exemplary implementation of intelligent processing 326 according to an embodiment of the present disclosure. As described above, this module 326 implements machine learning models and algorithms to identify symbols in a computational image. In some embodiments, intelligent processing 326 applies a deep neural network to specific areas of an image obtained by advancing a sliding window across the image. In the example of FIG. 4, intelligent processing 326 includes a data input component 402, one or more machine learning models 404, an automatic feedback / correction component 406, a user application 408, a manual feedback / correction component 410, and an analyzer 412.

[0032] In general operation, the data input component 402 receives data representing diagrams (e.g., P&ID, PFD, etc.) and, after appropriate pre-processing, feeds the data to one or more machine learning models 404. The machine learning models 404 use machine learning and image processing techniques to extract relevant information, such as names, numbers, symbols, lines, loops, etc., from the input data. The machine learning models 404 may also use image processing and / or geometric algorithms to reduce noise and increase accuracy. The automatic feedback / correction component 406 applies rules and algorithms configured to detect errors in the output received from the machine learning models 404. These errors are used to automatically correct the model output and are fed back to the machine learning models 404 via the analyzer 412, thereby updating the learning of the machine learning models 404. The processed output from the automatic feedback / correction component 406 is then displayed to a user for validation via a user application 408 (e.g., HMI application 328). Corrections made by the user are captured by the manual feedback / correction component 410 and fed back to the machine learning model 404 via the analyzer 412 to update the learning of the machine learning model 404. In this manner, the intelligent processing 326 can continually develop and improve its evaluation of the input data and its extraction of relevant information therefrom.

[0033] As described above, the intelligent processing 326, in some embodiments, applies a deep neural network to specific areas of the resulting image. The deep neural network processing provides multi-class classification of symbol candidates. In some embodiments, the symbols are classified by ISA symbology. Exemplary symbols include, but are not limited to, instruments, control / display elements, programmable logic controllers (PLCs), valves, pumps, etc. In some embodiments, the intelligent processing 326 utilizes at least three types of convolutional neural networks to recognize various symbol candidates. The three types include a decider network that determines whether the input is a single or multiple characters, a single character network that recognizes single alphabets and numbers, and a multiple character network that recognizes multiple characters or words. In some embodiments, the intelligent processing 326 also utilizes context-based prediction to distinguish between similar characters, e.g., to distinguish the capital letter "I" from the number "1" and the number "0" from the letter "O".

[0034] FIG. 5 illustrates an exemplary implementation of an HMI application 328 according to an embodiment of the present disclosure. The HMI application 328 generally begins at block 504 with inputting data from various plant engineering data sources, including plant engineering diagrams (e.g., PI&D, PFD, etc.) into the industrial plant control system 300. At block 506, the data is processed to extract relevant information and assets from the data, such as numbers, digits, symbols, lines, loops, etc. Asset extraction is performed using machine learning via the intelligent process 316 and ML models therein. Processing data using the ML-based asset extraction described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system). The processing system and method described herein may improve the process of HMI design by allowing image recognition models that identify plant components in diagrams to perform more efficiently and accurately. This can be achieved by inputting images that are pre-determined by a pre-processing module that are more likely to lead to accurate identification by the image recognition ML model. This can also allow for better training of the model and therefore better accuracy of the model.

[0035] In block 508, the extracted assets and other related information are used as input to an asset relationship establishment process to build an asset hierarchy, as described in more detail later herein. Establishing asset relationships is also performed using intelligent processes 316 and ML models therein. Building asset hierarchies using the ML-based asset relationship building process described herein is not well understood, routine, or conventional in the field of HMI design. Building asset hierarchies using the methods and systems described herein constitutes a practical application (e.g., designing HMIs for industrial plants using specialized automated design systems).

[0036] At block 510, the asset hierarchy can be used to create a knowledge graph based on a semantic model, as described in more detail later herein. Creating a knowledge graph using the methods and systems described herein is not well understood, routine, or conventional in the field of HMI design. Creating a knowledge graph using the methods and systems described herein constitutes a practical application (e.g., designing HMIs for industrial plants using specialized automated design systems).

[0037] At block 512, the knowledge graph can be used to generate (e.g., automatically using a computer) an HMI asset model, as described in more detail later herein. Generating knowledge graphs using the methods and systems described herein is not well understood, routine, or conventional in the field of HMI design. Generating knowledge graphs using the methods and systems described herein constitutes a practical application (e.g., designing HMIs for industrial plants using specialized automated design systems).

[0038] At block 514, the HMI asset model can be used to provide or construct (e.g., automatically using a computer) an HMI, as described in more detail herein below. Generating an HMI using the methods and systems described herein is not well understood, routine, or conventional in the field of HMI design. Providing an HMI using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0039] FIG. 6 illustrates the exemplary ML-based asset extraction process 506 and the ML-based asset relationship building process 508 from FIG. 5 in more detail.

[0040] The ML-based asset extraction 506 generally begins at block 602 where engineering diagrams (e.g., PI&D, PFD, etc.) are converted from PDF to image format. At block 604, a user enters information identifying the plant, site and area where each diagram is used. At block 606, all text within the diagram is found, and at block 608, the unit identifier for the diagram is found.

[0041] In some embodiments, finding the unit identifier involves searching the information block of the diagram or other predefined portion of the diagram for a particular keyword, such as "drawing" or "unit" or "section" or variations thereof. The predefined portion may be automatically identified based on an analysis of previous diagrams (e.g., the rightmost area having a width of 20% of the width of the P&ID and PFD, and the lowest area having a height of 20% of the height of the P&ID and PFD), or the portion may be specified by the user. Upon finding a keyword (e.g., "unit"), the system checks for text related to the keyword that may be in the same cell as the word "unit" if there is a table, or to the right of the word, or below the word. Once the unit label is determined, the determined label may be displayed to the user for verification. In some embodiments, this process may be performed for a relatively small number of diagrams (e.g., about 20). In this case, the determined label position may be used to determine subsequent labels in other diagrams without requiring or suggesting any validation of the determination.

[0042] At block 610, all devices and instruments (e.g., tags, valves, sensors, actuators, etc.) in the diagram are discovered for each diagram, and at block 612, all discovered devices and instruments, as well as text, are removed or otherwise suppressed from the diagram. This process may include creating a bounding box around the diagram features to be removed and setting all interior pixels to the diagram background color (e.g., white or black). Removing the text and devices facilitates discovering all lines in the diagram at block 614.

[0043] In some embodiments, lines in a diagram can be found by shrinking the diagram and converting it to grayscale. Long horizontal and vertical lines may then be removed by finding and deleting black pixels (in the case of a white background) that extend longer than a predetermined length along the X and Y axes (e.g., 7 times the width and height of the diagram, respectively). Once this is done, the remaining horizontal and vertical lines may be found by searching for clusters of black pixels (in the case of a white background). Clusters are groups of pixels separated by less than a predetermined number of pixels (e.g., less than 4 pixels). Once clusters are found, they can be connected to create horizontal and vertical lines by combining all collinear points of the clusters. Lines shorter than a predetermined length (e.g., 16 pixels) can be removed. In embodiments where the diagram is scaled before lines are removed, the removed lines may be scaled back to the original size. This allows the original line coordinates to be stored and referenced when determining connections and relationships between plant components. A variety of line types can be identified, including piping and connection symbols such as piping, process connections, electrical signals, pneumatic signals, data links, capillary tubes for filling systems, hydraulic signal lines, and guided electromagnetic or sonic signals, etc. In some embodiments, the above line extraction can be performed using a geometric line fitting algorithm.

[0044] Once the lines are extracted, a portion of the line is subjected to one or more machine learning models to obtain a type of the line, as described above. In an embodiment, the line extractor includes processor-executable instructions embodied on a storage memory device to provide the line extractor via a software environment. For example, the line extractor may be provided as processor-executable instructions including procedures, functions, routines, methods, and / or subprograms.

[0045] Following block 616, all arrows in the diagram may be found for each diagram. Such arrows indicate the flow direction in the diagram. An exemplary technique for identifying arrows in a diagram is described with respect to Figures 7A and 7B.

[0046] With reference to Figures 7A and 7B, an exemplary arrow 702 is shown. To analyze the arrow 702, in some embodiments, a detected line 704 as shown in Figure 7A is assigned a bounding box 706 having a width perpendicular to the direction of the line that is, for example, a predetermined number of pixels wide (e.g., 40 pixels). To determine whether the line 704 is an arrow, two squares 708, 710 on each end of the line can be drawn with side lengths equal to the height of the generated bounding box 706. In some embodiments, the sides of the squares 708, 710 may have different lengths. The contents of the squares 708, 710 can be analyzed using a machine learning model or algorithm trained to determine the presence of an arrowhead 712 and the direction of the arrowhead. The ML algorithm may be, for example, a convolutional neural network. Training data for the algorithm may be created by truncating the heads and tails of identified line segments by a particular size (e.g., 40 x 40 pixels). These cropped images may be stored and later classified as left arrows, right arrows, up arrows, down arrows, or no arrows. This classification may be done, for example, by a human. The identified line segments may be selected from diagrams received from different sources (e.g., different clients). In some embodiments, about 200 training images may be used for training. If both squares 708, 710 have arrowheads, the arrow 702 may be considered bidirectional. If only one of the two squares 708, 710 has an arrowhead, the arrow 702 may be considered unidirectional.

[0047] Referring back to FIG. 6, after the arrows are identified, all the equipment in the diagram (e.g., tanks, pumps, boilers, heat exchangers, etc.) are discovered for each diagram in block 618. The equipment is then assigned a name that helps to uniquely identify the equipment. Depending on the particular industry, one of several naming conventions may be used. For example, the naming convention may be <equipment type>-<equipment name / id>, where a hyphen or space acts as a delimiter. In the example of FIG. 2 above, the name assigned to the vessel may be "C-1119", where C is the type of equipment and 1119 is the identifier of the equipment in the diagram. This name may then be retained as the Regex (Regular Expression) of the tank for search purposes. Similar naming conventions may be applied to components of other diagrams.

[0048] In some embodiments, device discovery combines a geometric, or geometry-based, identity approach with machine learning, or ML-based classification.

[0049] Geometry-based identification of equipment is performed by first removing all instruments from the diagram to simplify the process (e.g., by backgrounding the instruments), and then finding all parallel lines that satisfy one or more criteria. Then, paths that connect the parallel lines are found and used to connect pairs of such parallel lines. Tanks can be found, for example, by searching for parallel lines and connecting those pairs of lines that satisfy one or more criteria. A bounding box may then be defined around each pair that is thus connected. One criterion may require that the two lines are within a predetermined distance of each other. In some embodiments, the predetermined distance may be between about 10% and 120% of the longer length of the two lines. The minimum length of such a line may be set to about 4% of the width of the diagram. The maximum length of such a line may be set to about 50% of the width of the diagram. The difference in length of two such lines may be less than about 15% of the longer length of the two lines. The start and end points of the two lines (i.e., four points) may be used to create the box. The width and / or height of the box may be expanded by a predetermined percentage. For example, if the width to height ratio is less than about 0.3, the width may be expanded by about 80%. If the width to height ratio is between about 0.3 and about 0.4, the width may be expanded by about 60%. If the width to height ratio is greater than about 0.4, the width may be expanded by about 40%. If the box expands beyond the shape boundary, the expansion may be stopped at the shape boundary. The resulting expanded box may be a bounding box. The image of the tank may be cropped at the bounding box. In some embodiments, the cropped image boundary may be erased, for example, by making about 4 pixels on one or more sides white (if the background is white).

[0050] ML-based classification can then be used to identify the equipment type. For example, a machine learning model or algorithm can be used to analyze the resulting images for the presence of tanks and identify the type of tank. The ML model or algorithm can also be trained to detect and classify tanks.

[0051] In some embodiments, the resulting image may be analyzed to determine whether it should be input to the model. For example, an image may be input to the model if it has an extended bounding box, the largest dimension of which is greater than the largest dimension of the non-extended bounding box of the cluster, and has a cluster with connected pixels (e.g., pixels with a maximum gap of about 2 pixels) that includes the start and end points of two parallel lines previously identified in the tank identification process. In some embodiments, the extended bounding box may be generated by extending one or more lengths of the non-extended bounding box based on a width-to-height ratio previously identified in the process. The image input to the model may be, for example, a 170×170 pixel black and white image. The model may be trained with a training dataset of, for example, 1950 images. Alternatively or additionally, extended training data may be used. For example, 46312 images may be used for training and 2518 images may be used for validation. Table 1 below provides details of an exemplary training data augmentation, as will be appreciated by one of ordinary skill in the art. [Table 1] Table 1: Exemplary training data augmentation

[0052] Those skilled in the art will appreciate that the above table, or a similar table, can also be used to provide training data augmentation for any of the other ML-based processes described herein.

[0053] Referring now to FIG. 8, other equipment such as pumps may also be identified. Pumps may be identified by creating a bounding box that includes the end points of one or more horizontal lines in the diagram. FIG. 8 shows an exemplary pump 800, around which a bounding box may be created. To create the bounding box, pixels may be traversed from the end point (e.g., P1) of the horizontal line 802 in each direction perpendicular to the line 802 until a black pixel (e.g., P2) is found. The distance (e.g., D) between P1 and P2 may be calculated. If the distance is between a predetermined value (e.g., 100 and 200 pixels), the bounding box may be created such that P1 and P2 are centered on opposite sides of the bounding box. The width of the bounding box may be equal to the distance D. The bounding box may be expanded by 20% of its length toward point P2. The diagram may be cropped with the bounding box. Clusters of connected pixels may be generated from the cropped image by creating groups of pixels, each group having all pixels connected together horizontally, vertically or diagonally. For clusters with a length or width exceeding a predefined threshold (e.g., 150 pixels), the area of ​​the cluster may be determined. The cluster with the largest area may have its bounding box determined, for example, by finding the minimum and maximum of x and y coordinates within the cluster. The minimum of the x and y coordinates may determine the top left corner of the bounding box. The maximum of the x and y coordinates may determine the bottom right corner of the bounding box. The image may be cropped at the bounding box of this cluster. The resulting image may be analyzed for the presence and type of pump, for example, using a machine learning algorithm trained to detect and classify pumps. For example, a similar CNN architecture may be used.

[0054] Returning again to Figure 6, connectors in the diagrams and their associated information may also be identified in block 620. Recall from above that connectors indicate connections between components located on different diagrams.

[0055] FIG. 9 illustrates an exemplary connector 902. As can be seen, the rear end 903 (end without an arrowhead) of the connector 902 has a line 904 connected to it indicating that a flow occurs from a component identified by a source identifier 906 on the line 904 to a component identified by a destination identifier 908 on the connector 902. A connector having a line connected to its front end 909 (end with an arrow) may indicate that a flow occurs from a component identified by a source identifier on the connector to a component identified by a destination identifier on the line. A connector identifier 910 in the connector 902 may indicate a diagram identifier (unit label) of a diagram that includes the component identified in the destination identifier 908. Similarly, the connector identifier 910 in the connector 902 may indicate a diagram identifier of a diagram that includes the component identified in the associated source identifier. Connectors may be identified by filtering identified lines having a length between about 20 pixels and about 40 pixels. For example, pairs of lines with starting points less than 4 pixels apart, e.g., ending points less than 4 pixels apart, may be analyzed to determine whether they are part of a connector. If the length of one line of the pair is longer than the length of the other line of the pair, and the distance between the pair is greater than about 0.1 times and less than about 0.5 times the length of the longer line, the pair may be analyzed to determine whether they are part of a connector. A tight-fitting bounding box may be created that contains both lines, and each side may be extended by half the distance between the two lines. Three exemplary resulting connector images are shown in Figures 9A, 9B, and 9C at 912, 914, and 916, respectively. The resulting images may be input to a machine learning model trained to identify the presence and orientation of a connector.

[0056] Note that the source identifier 906 may be prominently displayed or otherwise highlighted for ease of viewing by the user. This technique may be applied throughout the HMI to direct the user's attention to particular features.

[0057] To identify information related to the connector 902, a bounding box (not explicitly shown) may be generated around text 910 in the connector 902 to determine whether it is a connection identifier. If the bounding box is contained by the bounding box of the connector 902, the text may be identified as the connection identifier 910. Text located above the bounding box of the connector 902, for example within four times the distance between the parallel lines of the connector 902 (e.g., two parallel lines connected to an arrow), may be identified as a destination identifier included in 908. In some embodiments, as an alternative or addition, the text before and after the word "To" may be identified as a material name and a destination identifier 908, respectively. In this context, the material name may identify a material that flows through a pipe or other material carrier. In some embodiments, as an alternative or addition, the text before and after the word "From" may be identified as a material name and a source identifier, respectively. To identify the pipe identifier included in 906, a stride from the bounding box of the connector 902 may be taken in the direction of the line 904. In some embodiments, the stride may be twice the length of the bounding box of the connector 902. When the bounding box of the shifted connector intersects with the bounding box of a word, the word may be identified as the pipe identifier included in 906. If the bounding boxes of multiple words intersect, the word with the greatest number of characters may be identified as the pipe identifier included in 906. If the connector 902 has a line connected to its front end, one or more of the methods described above may be used to identify the source identifier and / or destination identifier.

[0058] Returning again to FIG. 6, after the ML-based asset extraction process 506 is finished, an ML-based asset relationship building process 508 can begin for the extracted assets. This process of building relationships between the extracted assets can include creating line connectivity graphs that capture the connections shown in the diagrams (e.g., PI&D, VFD, etc.) and then merging the graphs to create a composite flow graph. The merged graphs may include graphs representing diagram lines connected to equipment, graphs representing diagram lines connected to connectors, and / or graphs representing diagram lines connected to instrument components (e.g., tags). Additionally, line connectivity such as line-to-line graphs can also be generated to easily determine how various components are connected.

[0059] In block 622, a line-to-line graph may also be generated by detecting lines connected to the bounding box of a component and checking such lines for collinearity or perpendicularity. If collinearity or perpendicularity is detected and the distance between the start or end of one line and the start or end of another line is less than a predetermined value (e.g., 32 pixels), the lines may be extended so that their ends meet. Lines that form part of a component (e.g., equipment) may be removed by removing lines that have both endpoints that are within the bounding box of the component. Lines may be split to assign indices to line segments and keep track of line connections. Splitting a line into two segments is particularly useful when a device (e.g., tag, valve, etc.) is on the line. To do this, the distance to the start and end of the vertical line may be measured from the horizontal line. In some embodiments, the start and end of the horizontal line may be excluded from this measurement. If the measured distance is less than a predetermined distance (e.g., 32 pixels), a split may be made on the horizontal line at the closest point between the horizontal line and the start or end of the vertical line. If the distance from the vertical line to the start or end point of the horizontal line is less than a predetermined number (e.g., 32 pixels), a split can be made on the vertical line at the closest point between the vertical line and the start or end point of the horizontal line. In some implementations, the distance from the vertical line to the start or end point of the horizontal line can exclude the start and end points of the vertical line. The line can be assigned an index number. When a line is split at a point, the two newly created segments can be assigned index numbers. An adjacency matrix can be generated to represent the line connections.

[0060] FIG. 10A illustrates an example adjacency matrix 1002. The adjacency matrix 1002 may have a size of n×n, where n is the total number of lines. The value at position [first line index, second line index] of the adjacency matrix 1002 may be “0” to indicate that the lines at the first line index and the second line index are not connected to each other. The value at the position may be “1” to indicate that these lines are connected. It should be understood that other values ​​may also be used. Thus, the adjacency matrix 1002 may indicate whether two lines are connected to each other. For example, from left to right, node L2 has a “1” at node L1, and node L3 indicates that L2 is connected to L1 and L3. To construct a line-line graph, a node-edge (node-to-edge) graph may be generated, where nodes are assigned to line indices and edges between the nodes are assigned to connected line indices.

[0061] 10B illustrates an example line-line graph 1004 corresponding to the adjacency matrix 1002. In the figure, nodes E1 and E2 represent two devices, and nodes L1, L2, L3, L4, and L5 represent line segments. As can be seen, line segments L1, L2, L3, L4, and L5 connect E1 and E2 together, as reflected by the adjacency matrix 1002.

[0062] Returning to Figure 6, at block 624, a graph representing equipment-line (equipment-to-line) connections can be generated from the diagram as part of the ML-based asset relationship building process 508. Examples of building an equipment-line graph according to some embodiments are shown in Figures 11 and 12A-B.

[0063] FIG. 11 illustrates an example device 1102. As can be seen, a bounding box 1104 is placed around the device 1102 for computational purposes. To construct the device-line graph, the bounding box is expanded on each side by increasing the length of each side, for example, by 2.5%. Next, lines 1106 with endpoints that fall within the expanded bounding box are found, and associated nodes (e.g., L1, L2, L3, etc.) are generated for each line. In some embodiments, if two such lines are connected to each other, the line with the endpoint closer to the base of the bounding box may have associated nodes generated. The device 1102 may also have associated nodes generated (e.g., E1, E2, E3, etc.). The device-line pairs may then have associated edges generated (e.g., (E1, L1), (E2, L2), (E3, L3)).

[0064] 12A shows an example table illustrating equipment-line connections. The figure shows an equipment-line table 1202 that lists the various nodes (e.g., L1, L2, L3, etc.) and (e.g., E1, E2, E3, etc.) associated with the equipment 1102 and line segments 1106, and an edge table 1204 that lists the edges (e.g., (E1, L1), (E2, L2), (E3, L3, etc.) associated with these nodes.

[0065] 12B illustrates an example equipment line graph corresponding to equipment line table 1202 and edge table 1204. As can be seen, there are five edges, edges 1, 2, 3, 4, and 5 reflecting the five points where lines L1, L5, L6, L7, and L9 connect to equipment E1, E2, E3, and E4.

[0066] To discover a path between two pieces of equipment, the lines connected to those pieces of equipment may be identified by merging the adjacency matrix 1002 with the graph 1206 representing the diagram lines connected to the equipment. If multiple paths are discovered between two pieces of equipment, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) may be used on the graph to identify a path between the two pieces of equipment. Arrow directions associated with the path lines may indicate the direction of material flow between the equipment and indicate which equipment is the destination and which equipment is the source.

[0067] 6, in block 626, a graph representing connector line (connector-to-line) connections can be generated from the diagram as part of the ML-based asset relationship building process 508. Below is an example of building a connector line graph according to an embodiment of the present disclosure.

[0068] To construct the connector line graph, a bounding box is placed around the connectors as described with respect to FIG. 9. The bounding box may then be expanded by its long side, for example, by 0.1 times its length. The expansion may be in the direction of the arrow if the connector is an incoming connector, and in the opposite direction if the connector is an outgoing connector. Lines with endpoints that are within the expanded bounding box may then be identified. A node may then be generated for a connector, such as C1, and a node may be generated for a line connected to a connector, such as Lc1. Then, for a pair of connector lines, such as (C1, Lc1), a directed edge may be generated, with the edge direction being determined by whether the line is connected to the head or tail of the connector. To determine if a device is connected to a connector, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) may be used on the graph to determine a path between the line connected to the device and the line connected to the connector. If a connection is determined, the direction of the edge connected to the connector node may be used to determine the direction of flow between the device and the connector.

[0069] 13A shows an example table illustrating connector line connections. The figure shows a connector line table 1302 that lists the various nodes (e.g., C1, C2, Lc1, Lc2, etc.) associated with the connectors and line segments, and an edge table 1304 that lists the edges (e.g., (C1, Lc1), (C2, Lc2, etc.) associated with those nodes.

[0070] 13B illustrates an example connector line graph corresponding to connector line table 1302 and edge table 1304. As can be seen, there are five connectors C1, C2, C3, C4, and C5 connected to two devices E1 and E2 by line segments L1-L16. Table 1308 illustrates the paths generated using graph 1306.

[0071] Returning once again to Figure 6, at block 628, a graph representing device or meter line (device / meter-to-line) connections can be generated from the diagram as part of the ML-based asset relationship building process 508. Below is an example of building a device / meter-line graph according to an embodiment of the present disclosure.

[0072] A graph representing lines connected to devices or instruments (e.g., tags, sensors, etc.) may be generated by providing a bounding box around the devices. Lines having endpoints that are within the bounding box may then be identified. If fewer than two lines per bounding box are found, the length of one or more sides of the bounding box may be increased, for example, by 50%. A node may then be generated for the device, such as D1, and a node may also be generated for the line connected to the device, such as L1. A pair of an instrument and a line connected to it may have an associated edge generated. To determine if an instrument and an instrument are connected, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) may be used on the graph to determine a path between the line connected to the instrument and the line connected to the instrument.

[0073] 14A shows an example table illustrating device / instrument line connections. The figure shows a device / instrument line table 1402 that lists various nodes (e.g., D1, D2, L1, L2, L3, L4, L5, etc.) associated with devices and line segments, and an edge table 1404 that lists edges (e.g., (D1, L1), (D1, L2), (D2, L3), D2, L4), (D2, L5, etc.) associated with those nodes.

[0074] 14B illustrates example device / instrument line graphs 1406 and 1408 corresponding to connector line table 1402 and edge table 1404. As can be seen in graph 1406, device D1 has two line segments L1 and L2 connected to it. In graph 1408, device D2 has three line segments L3, L4, and L5 connected to it.

[0075] Returning once again to FIG. 6, thus far only directed graphs, i.e., graphs with a particular flow direction, have been discussed. In addition to directed graphs, there may be undirected graphs in the diagram. Any such undirected graphs can be changed to directed graphs in block 630 as part of the ML-based asset relationship building process 508. Below is an example of changing an undirected graph to a directed graph according to an embodiment of the present disclosure.

[0076] To assign direction to a graph, such as a line-line graph, equipment-line graph, or device-line graph, all line nodes in the equipment-line graph are identified. For each of these lines, all possible paths in the line-line graph are traversed up to a line that is connected to a connector or another device. The paths can then be split into segments and directions can be assigned to the lines in these segments. This can be done based on the direction of the connected connector, the presence of arrows, or the direction of the lines in adjacent paths (e.g., lines connected to connectors). For example, if one path is A->B->C->D and another is A->B->E, split these paths into three paths A->B, B->C->D, B->E. Path directions can then be assigned as follows: If one of the lines in a path is connected to a connector, assign all the lines in the path the direction of the connector (either inbound or outbound). If at least one of the lines in a path has an arrow, assign all the lines in the path the direction of the arrow. If none of the lines in a path have an arrowhead and the path is not connected to a connector, then we check the adjacent paths that are connected to the connector and assign the direction of the adjacent paths to all the lines in the path.

[0077] Returning again to Figure 6, recall that from block 612, all devices and instruments (as well as text) were removed from the diagram, resulting in a line-line graph showing only equipment-to-equipment and equipment-to-connector connections. The line-line graph can be updated to include these devices and instruments in block 632 as part of the ML-based asset relationship building process 508. This can be done, for example, by restoring the devices to the point at which they were removed.

[0078] 15A illustrates an example line-line graph 1502 and edge data table 1504 with a device removed according to block 612 of FIG. 6. As can be seen, a device has been removed from the point between the connector (C) and the equipment (E), where two dashed circles have been inserted and replaced with two corresponding edges, namely edge 5 (L3, L6) and edge 6 (L6, L7).

[0079] 15B shows the same example line-line graph 1502 and edge data table 1504, but after updating to restore the devices. As can be seen, two devices (D) have been placed between the connector (C) and the equipment (E), replacing the two edges (L3, L6) and (L6, L7). This allows the line connectivity graph to reflect not only equipment-to-equipment and connector-to-equipment connections, but also device-to-equipment, device-to-connector, and device-to-device connections.

[0080] Returning again to Figure 6, at block 634, for a given diagram (e.g., PI&D, PFD, etc.), the various line connectivity graphs generated for that diagram may be merged as part of the ML-based asset relationship building process 508. For example, if the diagram is a PI&D, all the line connectivity graphs may be merged to create a PI&D flow graph.

[0081] 16A shows an example PI&D flow graph 1602 resulting from merging the various line connectivity graphs generated for a given PI&D by the ML-based asset relationship building process 508. This PI&D flow graph 1602, unlike its constituent graphs, includes all connectors, devices or instruments, and equipment in the PI&D along with the connections between them. Thus, the PI&D flow graph 1602 captures the relationships between all assets extracted from the given PI&D.

[0082] 16B shows an example table 1604 containing data for the merged PI&D flow graph 1602. The table 1604 contains the nodes and edges of each of the line connectivity graphs (e.g., line lines, equipment lines, connector lines, device lines, etc.) generated by the ML-based asset relationship building process 508 for a given PI&D. The table 1604 also contains the nodes and edges of the merged PI&D flow graph 1602. Such a table may then be used as part of generating an HMI.

[0083] Returning again to FIG. 6, in block 636, the control loops of a given diagram (e.g., PI&D, PFD, etc.) can be added to the merged PI&D flow graph 1602 as part of the ML-based asset relationship building process 508. As mentioned above, a control loop executes some sub-process in a plant process and can be a closed or open control loop. Most control loops include several devices working together, including sensors, actuators, controllers, etc. There are generally two types of control loops: simple and complex. A simple control loop has an input, an output, and a controller, while a complex control loop is made up of several simple control loops. An example of identifying a control loop according to an embodiment of the present disclosure is shown in FIG. 17.

[0084] FIG. 17 illustrates a portion of an example diagram 1700 that includes a simple control loop. Four symbols are identified for illustrative purposes and labeled 1702, 1704, 1706, and 1708, with each symbol representing either a device (e.g., a sensor, an actuator, a controller, etc.) or a piece of equipment. Identifying the control loop in this example begins by recognizing all tags inside the symbol and all words outside the symbol. Then, for each word outside the symbol, find the closest tag inside the symbol. For example, for symbol 1702, the closest tag inside the symbol is "TIC 1803". For a given symbol, for example, all words that are more than 1.2 times the width of the symbol (i.e., words that are related to other symbols) can be ignored. The word is then checked (e.g., using a rules engine) against a list of words from an industry standard such as ISA to determine whether it represents, for example, an alarm, a signal, a tag, etc. If the word is a tag, the tag number of the closest tag is assigned to the word. For example, for the symbol 1702, the tag number "1803" is assigned to the words "TE" and "TT" to produce "TE 1803" and "TT 1803". These tagged words are then added to the symbol's list of tags. The tags for a given symbol include all tags inside and outside the symbol.

[0085] Once all tags inside and outside the symbol have been recognized, all tags with the same starting identification character and the same tag number are grouped together. In the example of Figure 17, the tags "TIC 1803", "TV 1803", "TE 1803" and "TT 1803" are grouped together. These tags are used to define a simple loop 1710 that can be included as an extracted asset in the diagram 1700.

[0086] Returning once again to Figure 6, at block 638, the plant assets are organized into an asset registry for ease of subsequent access and use. The assets are organized using a namespace to uniquely identify each asset. As one skilled in the art will appreciate, a namespace is a set of names used to identify objects so that there is no ambiguity when objects of different origins but the same name are mixed together.

[0087] At block 640, the merged flow graphs of each diagram may be merged as part of the ML-based asset relationship building process 508 to create a single plant flow graph for the entire plant. Recall that a diagram such as a PI&D or PFD represents instruments or devices and equipment that make up one unit, and the asset extraction process described above results in equipment names for a given unit that have the same unit prefix and are therefore uniquely identifiable. Device names are similarly created by prepending the unit name. Connector names are also uniquely identifiable, as each connector name includes the source equipment, destination equipment name, and pipe name.

[0088] The line index or index number (e.g., 1, 2, 3, etc.) is local to each diagram and therefore needs to be made uniquely identifiable across multiple diagrams. This can be done, for example, by prefixing the line index with the unit name or the corresponding diagram followed by the drawing number. For example, if the unit name is "UNIT1" and the diagram drawing number is "1116", the line index number "15" can be changed to "UNIT1_1116_15".

[0089] To create a single plant flow graph for the entire plant, a flow graph for each diagram in the plant can be generated as described above. If necessary, line indices may be made uniquely identifiable across different diagrams as described above. The flow graphs of the various diagrams can then be merged by inserting all nodes and edges from each diagram's flow graph (or its tabular representation) into the plant flow graph (or its tabular representation). Diagrams that are directly connected to each other have connector nodes with the same name. When this occurs, the duplicate connector nodes are removed and new edges are created in the plant flow graph by combining the lines that would have been connected by the removed connectors. All other nodes and edges must remain the same.

[0090] The resulting plant flow graph can be used to generate a visual representation of an asset hierarchy similar to the asset hierarchy 100 shown in Figure 1. The asset hierarchy and the relationships underlying the asset hierarchy can be stored as structured data, for example, in one or more JSON (JavaScript Object Notation) files, for later access and use by the industrial plant control system 300.

[0091] In some embodiments, the industrial plant control system 300 can be used to incorporate structured data about asset hierarchies and asset relationships into an ontology knowledge base, as described above. The system then dynamically builds a plant engineering domain ontology from plant engineering data sources (evolving over the plant lifecycle phases). This generally involves: (1) extracting domain entities (i.e., types and data) from structured data sources using metadata from unstructured data sources and machine learning techniques; (2) building a semantic model using entity types and adding association relationships between entities; (3) dynamically extracting entity data and building a knowledge graph based on the semantic model; and (4) providing users the ability to navigate the asset namespace and relationships to extract relevant information. Such an arrangement has many advantages.

[0092] One advantage is that automatic control configuration can be performed using process narratives. Currently, control configuration is a manual process that is time consuming and requires a strong control engineering background. In the industrial plant control system 300 described herein, even those who are not familiar with control engineering concepts should be able to easily configure the system. This results from using "process narratives" that are configured in the normal language used by plant engineers. The system automatically translates the process narratives to control strategies and configurations. Process narratives can be translated into multiple control narratives by leveraging noun-verb pairs such as "Asset.Operation" from AutomationML (see IEC62714) and State Based Control (see ISA106, ISA88, ISA95). The system uses information from asset templates, their relationships, namespaces, control strategy templates, and rules to translate the process narratives into control logic.

[0093] As another advantage, the test cases can be automatically generated based on the process narrative and documented validation. The industrial plant control system 300 uses the process narrative, the control configuration, and the asset ontology to automatically generate the test cases. The system then runs the automated test scenarios to simulate the process to validate the control application.

[0094] 18, a flow chart is shown of an exemplary method 1800 that may be used with the industrial plant control system 300. The method is for dynamically building a plant engineering domain ontology from plant engineering data sources. The method generally begins at block 1802 with inputting plant engineering data sources into the system. The data sources may include PI&D and PFD used to generate the plant asset hierarchy and asset relationships described above, as well as AutoCAD, SmartPlant, and Aveva Engineering diagrams. Other plant engineering data sources may include simulation models, instrument indexes, cable schedules, I / O assignment lists, configuration data, cross-wiring tables, cabinet load reports, system audit reports, historical data, alarms and events, and the like.

[0095] 19-29 illustrate exemplary plant engineering data sources that may be used to generate an asset hierarchy and / or knowledge graph. These plant engineering data sources are likely to be familiar to those skilled in the art. For example, FIG. 19 illustrates exemplary alarm data at 1900. FIG. 20 illustrates an exemplary cabinet load report 2000. FIG. 21 illustrates an exemplary control database at 2100. FIG. 22 illustrates an exemplary cross-wiring report at 2200. FIG. 23 illustrates an exemplary field wiring index at 2300. FIG. 24 illustrates an exemplary historical data (tag data) at 2400. FIG. 25 illustrates an exemplary instrument index at 2500. FIG. 26 illustrates an exemplary asset hierarchy at 2600. FIG. 27 illustrates an exemplary nested load report at 2700. FIG. 28 illustrates an exemplary plant asset index at 2800. FIG. 29 illustrates an exemplary process control narrative at 2900. Such data sources and other plant engineering data sources, including HMI specifications (see FIG. 34), can then be used to generate an asset hierarchy and / or knowledge graph.

[0096] Returning to FIG. 18, at block 1804, the system extracts domain entities and instances of the domain entities from the plant engineering data sources. The data sources may include structured and unstructured data sources. For example, entities (also called types or classes), properties (also called attributes), and relationships can be extracted from the data sources. The entities may be used to specify the domain and scope of the ontology, and the values ​​of the properties may be used to specify the scope of the ontology model. The semantic model can include a set of triples constructed from the domain and scope of the ontology. The triples can specify a subject entity, a predicate relationship or attribute, and an object entity or value. The subject entity can be an entity from the domain of the ontology. The object entity or value can be an entity or value from the scope of the ontology. Instances of the entities and values ​​can be extracted from the data sources. For example, the semantic model can specify a triple with a subject "Unit", a predicate "has", and an object "Device". A triple containing corresponding subject and object instances may include "Unit01" (an instance of the entity "Unit"), "has" (a predicate), "ControlValve01" (an instance of the entity "Device").

[0097] From the plant engineering data sources, multiple data extraction techniques can be used for each of both structured and unstructured data sources. The system can distinguish between structured and unstructured data based on file format or database type. Unstructured data files (e.g., email messages, word processing documents, videos, images, web pages, etc.) often contain text and multimedia content without a schema, whereas structured data files (e.g., spreadsheets, CSV files, XML files, RDBMS, time series, graph databases, etc.) contain a schema or metadata. The system can use the schema or metadata to extract domain entities from these structured data sources. For example, in a CSV file format, plain text data is separated by commas, each new line in the CSV file represents a new database row, and each database row has one or more fields separated by commas. The system can use this schema to extract domain entities from the data sources. An example can be seen in FIG. 25, where domain entities were extracted from an exemplary instrument index data source in CSV format. Among the domain entities and associated relationships extracted from this data source are plants, areas, units, loops, tags, process functions, etc., as shown in the first row of the table. Instances of these domain entities are displayed in the subsequent rows.

[0098] For unstructured data such as P&IDs, PFDs, Process Control Narratives (PCNs), and other image or unstructured text formats, the system uses machine learning to extract domain entities and relationships relevant to structured data from unstructured data sources. For reference, Process Control Narratives (PCNs) are functional instructions that describe how device mounted controls, panel mounted controls, PLCs, HMIs, and other processor-based process control system components should be configured and programmed to control and monitor a particular process, process area, or facility. PCNs are an essential link between process design and control system design. For PCNs, the system uses Named Entity Recognition (NER) techniques to extract domain entities such as pumps, valves, positions, and alarm conditions from process narratives. NER is a technique used in information processing with unstructured text. This technique labels sequences of words in the text that are names of things such as things, people's names, organizations, locations, time expressions, quantities, amounts, and percentages.

[0099] 30 is a table 3000 illustrating an example of domain entity types and instance data extracted into a tabular file. The domain entity types in the table include controller, compound, block, tag, etc., and the corresponding instance data includes (in the first row) CP2801, CF101_011900, PT038, 011900PV038, etc.

[0100] Returning to Figure 18, once the domain entities and relationships have been extracted, a user (e.g., a domain expert) builds a semantic model using the extracted entity types and associated relationships in block 1806. Figure 31 shows an example of a user interface that may be used by a domain expert to build a semantic model.

[0101] FIG. 31 shows a user interface 2100 that allows a user to specify semantic relationships between entities extracted from data sources. From this interface, a user can manually browse entity types from various data sources and filter the entities required for the semantic model. For example, a user can select two entities, such as source and target, and create a named relationship between them. A schema can be defined at a conceptual level, which is a logical model. A user can then define domain-specific relationships between entities extracted from data sources. Typical relationships are "is-a" for parent-child relationships, "has-a (part-of)" for containment relationships, etc. In this example, a user specifies that a source "Plant" is related to a target "Area" by the relationship "has Area". A user can also add additional context information as entity attributes. For example, a user can add type, severity, and priority as attributes of an alarm entity.

[0102] In some embodiments, as an alternative or addition to user definition, semantic relations may be determined automatically from, for example, primary key-foreign key relationships specified in relational database management system (RDBMS) tuples or parent-child nodes in the XML document. Properties may be added to provide additional context information (e.g., for the entity "Alarm", properties "Type", "Severity", or "Priority" may be added). The resulting semantic model may be validated against a predefined set of rules. The rules may be defined, for example, by the standard ISA-106 ("Procedure Automation for Continuous Process Operations"). With ISA-106, the triple "Device has Unit" may be flagged for correction during validation, since the standard specifies that "Unit has Device" and not vice versa. Vocabulary in the public ontologies and dictionary files may also be used for validation, as an alternative or addition to the standard.

[0103] Returning to FIG. 18, once the semantic model is built, in block 1808, the model is validated. Semantic validation refers to the process of validating that data elements are logically valid. For example, a data element such as Equipment-Has-Area would not be valid under the ISA106 schema because equipment never has an area (semantic models typically standardize entity relationship types according to industry standards). In some embodiments, the system automatically validates the domain-specific relationships between entities created in block 1806. To this end, the system may maintain standard terms, their definitions, and references to vocabularies in public ontologies and dictionary files that can be used as references for validation. The system also provides an option for domain experts to manually validate and modify system-generated relationships.

[0104] At block 1810, the validated semantic model is saved to create an ontology. In some embodiments, the semantic model is saved according to one of the following well-known semantic model formats: OWL / XML, JOSO-LD, and RDF / JSON. Alternatively, the semantic model can be persisted by converting the model into a labeled property graph model in GaphSON format. If the semantic model is converted to a W3C ontology, the semantic model knowledge graph can be persisted in an RDF triple store such as AllegroGraph, Stardog, or Amazon Neptune. If the semantic model is converted to a Labeled Property Graph (LPG), the knowledge graph can be persisted in a graph database format such as GraphDB, or Neo4j, or Azure Cosmos DB.

[0105] At block 1812, the system can use the semantic model to build and deploy a knowledge graph. As alluded to above, a knowledge graph is essentially a graph that can be used to navigate and visualize relationships between components (i.e., a physical model of data). The semantic model defines the relationships between source and target entities in the knowledge graph at a conceptual level (e.g., "Area-has-Unit"). The system extracts instances related to ontology classes and uses associated relationships and context information to build the knowledge graph. In this way, the knowledge graph represents a collection of interconnected entities that enhances a user's ability to explore desired information. The knowledge graph can be represented in a database system using LPG, RDF, or similar graph models. An exemplary visualization of a knowledge graph is shown in FIG. 32.

[0106] Referring to FIG. 32, a visualization of a knowledge graph 3200 can be seen, in which entities and relationships are shown as multiple labeled property nodes, one of which is shown at 3202. Both nodes and their relationships are named and can store properties represented as key / value pairs. Nodes can be labeled to allow easy grouping of similar nodes. Edges representing relationships have two main qualities: they always have a start node and an end node, and they are directed, making the graph a directed graph. Relationships also have properties. This is useful for providing additional metadata and semantics to the node relationships. A legend 3204 provides an indication of what each node 3202 represents on a color-coding basis. For example, blue represents a tag index list, green represents a nested load report, orange represents a cabinet load report, etc. The knowledge graph 3200 can be selected by selecting a graph view option 3206 and is one of several display screens available from an HMI according to an embodiment of the present disclosure. The user can select other display screens from the HMI by selecting the options therefor, such as upload data option 3206 for uploading data and query engine option 3208 for searching data.

[0107] FIG. 33 illustrates an example of a query display screen 3300 available from an HMI, according to an embodiment of the present disclosure. In this screen, a user can search for data by entering a natural language query into a search box 3302. For example, a user can ask the HMI to display or otherwise indicate alarms that are typically triggered during a normal plant startup process by entering the query "What alarms are expected during a startup sequence" into the search box 3302. In response, the HMI can display the requested alarms graphically (nodes with dashed circles) or in textual form by displaying the alarm's identifier. The user can also select a node that represents the type of data the user wants and drag it into the search box 3302. In the illustrated example, the user can drag and drop the six returned alarms into the search box 3302. Thanks to the knowledge graph, the HMI can quickly highlight all common assets of the six alarms, indicated by nodes without dashed circles, and gray out or suppress all other nodes into the background. The common assets returned may include the control processor in which the alarm was processed, the assets shared by these alarms, the I / O module that processes the alarm, etc., and indicate the possible root cause of the alarm.

[0108] In some embodiments, users and plant operators can also enter commands to initiate actions. The HMI can interpret commands written in natural language. For example, the command "fill Tank 01" can be entered into the command bar (not explicitly stated) to instruct the HMI to begin filling a component labeled "Tank 01."

[0109] Returning to Figure 18, once the ontology is built, in block 1814 the system can provide an asset namespace registry API for different applications to consume the data. Recall from above (Figure 6) that the plant assets are organized in an asset registry using namespaces. In some embodiments, a RESTful Application Program Interface (e.g., Azure Active Directory Graph API) may be created to allow downstream applications to extract information from the knowledge graph.

[0110] At block 1816, the system provides a way for a user to search the data based on the process narrative. In some embodiments, the user may search the data using natural language questions to search the data. The system can translate the natural language queries into, for example, SPARQL queries or Gremlin queries. Downstream applications can include, for example, HMIs that can issue commands from the translated process narratives and / or other natural language commands.

[0111] In block 1818, plant control processes and HMIs for the processes may be designed and developed (i.e., control and HMI engineering) to enable users to use the system 300 to control the plant and various plant assets. This involves processing the plant assets and asset relationships from the asset model to create an HMI asset model. The HMI asset model arranges the assets in a hierarchical structure (see FIG. 37), unlike traditional solutions that are discrete in nature (i.e., created based on individual P&ID diagrams). The HMI can then use the HMI asset model to automatically create a unique continuous HMI display that resembles a two-dimensional floor plan at the plant level, including symbols for each asset and lines extending between the symbols representing processes, electrical connections, data links, etc. A plant operator can then navigate horizontally and vertically, and zoom in and out on the two-dimensional view (i.e., along the Z-axis) to move across different assets and examine their behavior.

[0112] In some embodiments, the HMI can dynamically change the assets displayed to the user based on the user's changing position on the HMI screen and the alarms generated at run-time. The HMI assets shown at run-time can be determined primarily based on two factors: (i) static weights assigned to assets based on asset hierarchy details available in the engineering diagram (e.g., PFD, P&ID, etc.), and (ii) dynamic weights assigned to each piece of equipment that emits an alarm. Thus, at run-time, the HMI can display a view that includes assets with higher weights, or, if there are alarms, assets with alarms. The HMI can also dynamically change the view to include assets whose alarms have occurred recently. Critical equipment alarms are given more focus with a "high" or "high-high" indicator (e.g., exclamation point, red color, etc.), and the entire alarm chain gets the highest weight at run-time. In order to design the HMI, the design specifications for the HMI need to be established.

[0113] FIG. 34 illustrates an example design specification 3400 for an HMI according to an embodiment of the present disclosure. The specification 3400 specifies a hierarchical structure for the HMI that allows the HMI to display certain process control details, including process equipment details and instrumentation control details, in a hierarchical arrangement. In this example, the HMI is arranged in several hierarchical display levels 3402, 3404, 3406, 3408. Level 1 displays an overview of the process area, including information such as operational KPIs (Key Performance Indicators), alarm counters, bypass signal counters, trends, and overall plant status. The next level, level 2, displays the process unit control, including controllers, alarms, trends, and status. The next level, level 3, displays the process unit details, including smaller equipment groups, controllers, alarms, trends, ESD displays, equipment, diagnostics, and status. The next level, level 4, displays the process unit support details, including interlocks, details, "first out", procedures, documentation, and help. Other HMI design specifications may include specifications related to graphical displays, such as alarm priority colors (e.g., red, yellow, orange, etc.) and colors indicating status (e.g., out of service, etc.) and function (e.g., feedback, trend lines, etc.). The meaning of icons, symbols, and lines (e.g., solid, dash-dotted, dotted, etc.) may also be specified, along with engineering units (e.g., metric, English, etc.). These HMI specifications can then be used along with data and information extracted from various plant engineering data sources, such as those described above, to build an HMI asset model.

[0114] FIG. 35 illustrates an exemplary HMI asset model 3500 according to an embodiment of the present disclosure. The HMI asset model, as its name suggests, comprises an asset model that is processed and used by the HMI for display to the user. To this end, the HMI asset model includes the plant assets and relationships (asset hierarchy), the ontology knowledge base, and the various plant engineering data sources mentioned above. The HMI asset model is thus composed of the PFD 3502 (e.g., high level equipment, instruments, connections, etc.), the PI&D and instrument index 3504 (e.g., equipment, instruments, associations between them, alarms, piping, maintenance override graphics (MOS) and cause and effect (C&E) details, etc.), the HMI specification 3506 (e.g., navigation requirements, alarm aggregation, history, diagnostic details, etc.), the control narrative 3508 (e.g., control and monitoring parameters, set points, alarms, ranges, engineering units, etc.), and the knowledge 3510 captured via engineering displays (e.g., from maintenance records, previous corrective actions, user experience, etc.). This model is updated automatically as the content of the various data sources changes and assets are added and / or removed, making this model a dynamic HMI asset model.

[0115] Once the HMI asset model 3500 (or rather its contents) is established, the model may be used in developing an HMI according to an embodiment of the present disclosure. Developing an HMI based on an HMI asset model may include creating templates 3514 for various assets (e.g., instruments, equipment, composites and combinations thereof, etc.). These templates may include, for example, level symbols (e.g., level 2, level 3, etc.), detailed representations of instruments and equipment, attributes and actions of instruments, animated scripts, control engineering templates, integrated details of pumps, valves, etc. Developing an HMI from an HMI asset model may also include creating other asset models 3516, in which all areas and units are represented as area objects, and each composite and / or complex combination of instruments is instantiated from their templates. This process may include assigning more weights to some equipment and devices compared to other equipment and devices, which some equipment and devices are graphically emphasized. For example, new equipment and devices introduced through the control narrative may be added, and the asset hierarchy may be adjusted accordingly. Equipment and devices may be grouped based on their associations in these asset models.

[0116] In addition to the above, the HMI asset model can also be used to build control applications that control the operation of various plant assets. As mentioned above, plant assets can be extracted from unstructured data, such as engineering diagrams, using machine learning to recognize one or more assets. This procedure can also extract one or more control loops from the engineering diagram based on one or more of equipment, instruments, connectors, and lines to build a dynamic HMI asset model. An automatic control generation procedure can then be used to read one or more control loops from the dynamic asset model and generate control logic for the control application. The control logic connects the control application to the actual plant equipment to enable the control application to read or otherwise obtain process values ​​of the equipment, devices, instruments, etc., and control the operation. In other words, the HMI processes assets and asset relationships, creates an HMI asset model, and arranges the assets in a hierarchical structure. This process also links the attributes of each HMI asset with a corresponding control I / O reference so that at run time, when an operator turns on a valve in a control application in the HMI, that control value is propagated via the control I / O reference (linked in the HMI asset) to the corresponding device in the plant, thus turning on the valve in the plant.

[0117] The HMI developed from the HMI asset model can then be used by a user to monitor various displays and, for example, easily discover the root cause of an alarm. In some embodiments, such an HMI can be developed using an HMI development platform such as System Platform 2017 InTouch OMI (Operation Management Interface) available from Wonderware West, Inc., Houston, Texas. Of course, other HMI development platforms can be used within the scope of this disclosure.

[0118] Referring now to FIG. 36, in some embodiments, an HMI according to the present disclosure can process alarms raised by various devices and automatically identify the root cause of the alarm. The HMI can determine the root cause of the alarm mainly by two factors: (i) the process / material flow defined in the engineering diagram, and (ii) the management flow defined in the engineering diagram and any management narrative. For the process flow, the HMI first discovers the device nodes that cause different alarms. For each device node, the HMI searches for the equipment node that corresponds to the device node. For that equipment node, the HMI retrieves any upstream equipment nodes connected to that equipment node that also have a device that is raising an alarm. This step is repeated for adjacent nodes until the HMI finds a connected equipment that does not have a device with an alarm connected to it. This generates a chain of alarms from which the HMI can determine the root cause of the alarm by finding the equipment at the highest level in the chain that has an alarm.

[0119] In the example of FIG. 36, a two-dimensional floor plan at the plant level is displayed by the HMI on a screen 3600 including symbols 3602, 3604, 3606, and 3608 representing assets and lines 3610, 3612, and 3614 extending between the symbols representing processes, electrical connections, data links, etc. The plant operator can then navigate laterally and zoom in and out in the two-dimensional view to move across different assets and examine their behavior. In this example, material flows from equipment E1215 (pump) to equipment E1214 (boiler), causing a pressure drop at the inlet of equipment E1214 (boiler). The pressure drop causes an alarm to be generated by a device (measurement instrument) at the inlet of equipment E1214 (boiler). The problem could be with equipment E1215 or equipment E1209. However, since there is no alarm on equipment E1209, the HMI determines that the problem is with equipment E1215 (pump). In this example, the throughput of equipment E1215 (pump) has decreased, thus causing a pressure drop in the connected downstream equipment E1214 (boiler). Thus, an alarm would be raised by a measurement instrument on the boiler E1214, but the HMI would identify the root cause of the alarm in this scenario as the pump E1215. The HMI assigns additional weight to this equipment for display purposes and places an alarm light bulb 3616 next to this equipment. Other alarm light bulbs representing other alarms discovered by the HMI are aggregated at 3618.

[0120] The HMI then allows a plant operator to modify the analysis of the root cause identified by the system. For example, the operator can manually select a different piece of equipment and set it as the root cause of the alarm instead of the root cause identified by the system, and in some embodiments, can also manually identify the corresponding alarm chain. These operator modifications can be fed back into the system to dynamically update the machine learning algorithms, thereby making the system more accurate in its root cause analysis of subsequent alarms.

[0121] 37, in some embodiments, an HMI according to the present disclosure can aggregate alarms in a plant and provide a display screen 3700 showing a plant-level view of the aggregated alarms. In some embodiments, the system can aggregate alarms by getting a list of all equipment generating alarms and identifying a first equipment from the list. The system adds this equipment to a new alarm cluster and checks if nearby equipment is also generating alarms. If YES, the system adds the neighboring equipment to the current alarm cluster. This process is repeated with the next equipment on the list until all equipment and neighboring equipment are processed. The alarms aggregated at the plant level are shown in the illustrated example as a series of light bulbs 3702 in a designated area (e.g., upper left corner) of the screen 3700, although it is certain that the alarms could be in a different location or use a different icon or graphical symbol. Each of the light bulbs 3702 corresponds to a respective component (e.g., device, equipment, etc.) in the plant where an alarm occurred. As can be seen, there are five components 3704a, b, c, d, e that have generated alarms, and each component is marked with a light bulb icon 3706 to indicate the alarm condition at that component.

[0122] Upon selecting one of the components 3704a-e (e.g., by tapping, double-clicking, etc.), the HMI displays information about that component, including information about which downstream components may be the potential root cause of the alarm. In some embodiments, the system can determine the potential root cause of an alarm by identifying the alarm at the highest equipment level and then finding source equipment connected to this equipment that also has an alarm. The system then drills down on the source equipment to find subsequent connected equipment that also has an alarm. This process is repeated until it finds the lowest connected equipment that has an alarm.

[0123] FIG. 38 diagrammatically illustrates an embodiment of an HMI that can dynamically zoom in (and out) on assets according to the user / observer's location on the asset hierarchy screen. The observer's location on the hierarchy screen is determined primarily by two factors: (1) the ground position, determined by screen coordinates (e.g., x and y coordinates), and (2) the observer's height above ground. Based on the observer's location on the screen, relevant information around the asset model is provided for display. For example, consider a case where a user is located in Area 1 relative to the HMI screen and equipment 5, equipment 3, device 2, and device 4 have alarm conditions. In this example, the HMI screen shows equipment 7, 1, 2, 4, 5, and 6 and any relevant information about these equipment, but does not show equipment 8, 9, or 3, or any devices connected to equipment 3. Now assume that the user zooms in and the user's location is at unit 3. In this example, the HMI screen shows equipment 7, 1, 2, 4, 5, and 6 as above, but also shows equipment 8, 9, or 3. As the user navigates further to Equipment 5, the HMI screen shows devices 1, 2, 3, 4, and 5 and any associated information about these devices. The HMI screen automatically removes equipment above the user's level.

[0124] FIG. 39 illustrates an example HMI display screen 3900 that can dynamically zoom in (and out) on assets according to the user / observer's location in accordance with an embodiment of the disclosure. Based on the zoom level determined by the user's current location and current active alarms in the plant, the dynamic HMI display screen 3900 shows relevant information around the asset model. In the example on the left of FIG. 39, the HMI display screen 3900 shows connectivity between assets 3902 and 3904. In the example in the center, as the user zooms in, the HMI display screen 3900 remaps the connectivity to show an additional asset 3906 on the screen. In the example on the right, as the user zooms in even further, the HMI display screen 3900 remaps the connectivity again to show an additional asset 3908 on the screen.

[0125] 40A and 40B, an HMI according to an embodiment of the present disclosure may also perform alarm aggregation at an area level in addition to the plant-level alarm aggregation described above. FIG. 40A graphically illustrates an exemplary embodiment of such an HMI, where the HMI uses a two-dimensional view of the plant assets to illustrate area-level alarm aggregation. Starting with the example on the left, solid lines, such as line 4002, indicate horizontal connectivity within a plane, while dashed lines, such as line 4004, represent vertical connectivity across multiple planes (i.e., a two-dimensional view). As can be seen, area-level assets A1 and A2 have alarm conditions indicated by high and critical alarm icons, respectively. When the user selects A2 (e.g., by tapping, clicking, etc.), the HMI leads the user to the example on the right, where the user discovers a critical alarm on asset U3 and a high alarm on asset U2.

[0126] If the user selects U2, the HMI takes the user to the example on the left of FIG. 40B where the user discovers a critical alarm on device U3E2. From here, the user can visually see the source of the critical alarm on U2E1, cascaded through U3E1 and U3E2. Assuming the user selects U2E1, the HMI takes the user to the example on the right of FIG. 40B where the user is shown device level assets including D1, which is the source of the problem causing the alarm. Selecting D1 provides the user with information about that device, any trends seen on the device, any support and guidance available for the device, and any knowledge captured about the device.

[0127] Thus, as described herein, embodiments of the present disclosure provide systems and methods for controlling an industrial process automation and control system that automatically extract plant assets from engineering diagrams and other plant engineering data sources by using machine learning (ML) models and algorithms, create a plant asset registry, establish asset relationships to build an asset hierarchy from the plant assets, generate an ontology knowledge base from the plant asset hierarchy, and provide an HMI for controlling the industrial process based on the plant asset hierarchy and the ontology knowledge base.

[0128] Such embodiments of the present disclosure may include a special purpose computer including various computer hardware, as described in more detail below. Embodiments within the scope of the present disclosure also include computer-readable media that carry or have computer-executable instructions or data structures stored thereon. Such computer-readable media may be any available media accessible by a special purpose computer and comprise computer storage media and communication media. By way of example, computer storage media includes, but is not limited to, volatile and non-volatile media, and removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media are non-transitory and include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), digital versatile disk (DVD) or other optical disk storage, solid state drive (SSD), magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to transmit or store desired non-transitory information in the form of computer-executable instructions or data structures, and any other medium that can be accessed by a computer. When information is transmitted to or provided to a computer over a network or other communications connection (either wired, wireless, or a combination of wired or wireless), the computer properly regards the connection as a computer-readable medium. Thus, any such combination is properly referred to as a computer-readable medium. Combinations of the above are also included within the scope of computer-readable media. Computer-executable instructions include, for example, instructions and data that cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.

[0129] The following discussion is intended to provide a brief, general description of a suitable computing environment in which aspects of the disclosure may be implemented. Although not required, aspects of the disclosure are described in the general context of computer-executable instructions, such as program modules, executed by a computer in a network environment. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Computer-executable instructions and associated data structures, as well as program modules, represent examples of program code means for executing steps of the methods disclosed herein. A particular sequence of such executable instructions, or associated data structures, represents examples of corresponding acts for implementing the functions described in such steps.

[0130] Those skilled in the art will appreciate that aspects of the present disclosure may be practiced in networked computing environments having many types of computer system configurations, including personal computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, etc. Aspects of the present disclosure may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired, wireless, or a combination of hardwired or wireless links) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0131] An exemplary system for implementing aspects of the present disclosure includes a dedicated computing device in the form of a conventional computer including a processing unit, a system memory, and a system bus coupling various system components including the system memory to the processing unit. The system bus may include any of several types of bus structures, including a memory bus, a memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes computer storage media including non-volatile and volatile memory types. A basic input / output system (BIOS), including basic routines that help transmit information between elements within a computer, such as during start-up, may be stored in ROM. Additionally, the computer may include any device (e.g., computer, laptop, tablet, PDA, cell phone, mobile phone, multifunction television, etc.) that can wirelessly receive or transmit IP addresses from the Internet.

[0132] The computer may also include a magnetic hard disk drive for reading from and writing to a magnetic hard disk, a magnetic disk drive for reading from or writing to a removable magnetic disk, and an optical disk drive for reading from or writing to a removable optical disk, such as a CD-ROM or other optical media. The magnetic hard disk drive, the magnetic disk drive, and the optical disk drive are connected to the system bus by a hard disk drive interface, a magnetic disk drive interface, and an optical drive interface, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer-executable instructions, data structures, program modules, and other data for the computer. The exemplary environment described herein employs a magnetic hard disk, a removable magnetic disk, and a removable optical disk, although other types of computer-readable media for storing data may be used, including magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, RAM, ROM, SSDs, and the like.

[0133] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0134] The program code means, comprising one or more program modules, can be stored on a hard disk, a magnetic disk, an optical disk, a ROM and / or a RAM, and includes an operating system, one or more application programs, other program modules and program data. A user can enter commands and information into the computer through a keyboard, a pointing device, or through other input devices, such as a microphone, a joystick, a game pad, a satellite dish, a scanner, etc. These and other input devices are often connected to the processor through a serial port interface coupled to the system bus. Alternatively, the input devices may be connected through other interfaces, such as a parallel port, a game port, a Universal Serial Bus (USB), etc. A monitor or other display device is also connected to the system bus through an interface, such as a video adapter. In addition to the monitor, a personal computer typically also includes other peripheral output devices (not shown), such as speakers and a printer.

[0135] One or more aspects of the present disclosure may be embodied in computer-executable instructions (i.e., software), routines, or functions stored in system memory or non-volatile memory as application programs, program modules, and / or program data. Alternatively, the software may be stored remotely, such as a remote computer having a remote application program. Generally, program modules include routines, programs, objects, components, data structures, etc., which, when executed by a computer processor or other device, perform particular tasks or execute particular abstract data types. The computer-executable instructions may be stored in one or more tangible, non-transitory computer-readable media (e.g., hard disks, optical disks, removable storage media, solid-state memory, RAM, etc.) and executed by one or more processors or other devices. As will be appreciated by those skilled in the art, the functionality of the program modules may be combined or distributed as desired in various embodiments. Furthermore, the functionality may be embodied in whole or in part in firmware or hardware equivalents, such as integrated circuits, application-specific integrated circuits, and field programmable gate arrays (FPGAs), etc.

[0136] A computer can operate in a networked environment using logical connections to one or more remote computers. The remote computers may be another personal computer, tablet, PDA, server, router, network PC, peer device, or other common network node, and may generally include many or all of the elements described above in connection with a computer. Logical connections include, but are not limited to, local area networks (LANs) and wide area networks (WANs), presented herein by way of example only. Such network environments are commonplace in office- or enterprise-wide computer networks, intranets, and the Internet.

[0137] When used in a LAN networking environment, the computer is connected to the local network through a network interface or adapter. When used in a WAN networking environment, the computer may include a modem, a wireless link, or other means of establishing communications over a wide area network, such as the Internet. The modem, which may be internal or external, is connected to the system bus through a serial port interface. In a networked environment, program modules depicted for the computer, or portions thereof, may be stored in a remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing communications over a wide area network may be used.

[0138] Preferably, the computer executable instructions are stored in a memory, such as a hard disk drive, and executed by the computer. Advantageously, the computer processor is capable of performing all operations (e.g., executing the computer executable instructions) in real time.

[0139] The order of performing or performing operations in the embodiments of the present disclosure illustrated and described herein is not important unless otherwise specified. That is, operations may be performed in any order unless otherwise specified, and embodiments of the present disclosure may include additional operations or fewer operations than those disclosed herein. For example, performing or performing a particular operation before, contemporaneously with, or after another operation is anticipated to be within the scope of aspects of the present disclosure.

[0140] Embodiments of the present disclosure can be implemented using computer-executable instructions. The computer-executable instructions can be structured into one or more computer-executable components or modules. Aspects of the present disclosure can be implemented using any number and structure of such components or modules. For example, aspects of the present disclosure are not limited to the specific computer-executable instructions shown in the drawings and described herein, or to the specific components or modules. Other embodiments of the present disclosure may include different computer-executable instructions or components having more or less functionality than shown and described herein.

[0141] When incorporating an element of an aspect of the present disclosure, or an embodiment thereof, the articles "a," "an," "the," and "said" are intended to mean one or more of the element. The terms "comprise," "include," and "have" are intended to be inclusive and mean that there may be additional elements other than the listed elements. Having described the aspects of the present disclosure in detail, it is apparent that improvements and modifications can be made without departing from the scope of the aspects of the present disclosure as defined in the appended claims. Since various changes can be made in the structures, products, and methods described above without departing from the scope of the aspects of the present disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense.

Claims

1. 1. A control system for an industrial plant, comprising: one or more processors; a storage unit communicatively coupled to the one or more processors and configured to store processor-executable instructions thereon; the processor-executable instructions, when executed by the one or more processors, cause the control system to: a process for inputting an engineering diagram of a unit of the industrial plant, the engineering diagram including symbols representing assets of the industrial plant; a process of extracting one or more assets from the engineering diagram using machine learning to recognize the one or more assets, the one or more assets being equipment, meters, connectors, and lines, the lines relating the equipment, meters, and connectors to one another; determining one or more relationships between said equipment, meters, connectors, and lines using machine learning to recognize said one or more relationships; a process for creating a flow graph from said devices, meters, connectors, and lines and said relationships between said devices, meters, connectors, and lines; a process for creating a data table of the flow graph, the data table including constituent graphs that make up the flow graph, the data table including the nodes and edges that make up each constituent graph, the data table and the contents of the data table being used to generate an HMI for controlling the industrial plant; A control system that executes the above.

2. The control system of claim 1 , wherein the process of extracting one or more assets from the engineering diagram includes a geometry-based extraction process and a machine learning-based classification process.

3. 2. The control system of claim 1, wherein the processor executable instructions further cause the control system to execute a process to extract one or more control loops from the engineering diagram based on one or more of the equipment, instruments, connectors, and lines.

4. 2. The control system of claim 1, wherein the processor executable instructions further cause the control system to execute a process for extracting a unit identifier from the engineering diagram, the unit identifier uniquely identifying a particular unit among a plurality of units in the industrial plant.

5. 2. The control system of claim 1, wherein the processor executable instructions further cause the control system to execute a process to extract a drawing number and a revision number from the engineering diagram, the drawing number uniquely identifying the engineering diagram among a plurality of engineering diagrams of the industrial plant, and the revision number indicating a revision of the engineering diagram that is incremented whenever a change is made to the engineering diagram.

6. 5. The control system of claim 4, wherein the processor executable instructions further cause the control system to execute a process that uses the unit identifiers from the engineering diagram to assign a unique identifier to each piece of equipment, instrument, connector, and line of the engineering diagram.

7. 2. The control system of claim 1, wherein the processes for determining one or more relationships between the equipment, instruments, connectors, and lines include a process for generating a line-line graph, a process for generating an equipment-line graph, a process for generating a connector-line graph, and a process for generating an instrument-line graph.

8. 8. The control system of claim 7, wherein the processor executable instructions cause the control system to generate the flow graph for the engineering diagram by executing a process that merges the line line graph, the equipment line graph, the connector line graph, and the instrument line graph with each other.

9. 8. The control system of claim 7, wherein the processor executable instructions further cause the control system to perform a process for merging flow graphs from multiple engineering diagrams for the industrial plant to create a flow graph for the industrial plant.

10. 2. The control system of claim 1, wherein the processor executable instructions further cause the control system to execute a process for displaying the equipment, gauges, connectors, and lines and relationships between the equipment, gauges, connectors, and lines in an asset hierarchy.

11. 1. A method for controlling an industrial plant, comprising: a process for inputting an engineering diagram of a unit of the industrial plant, the engineering diagram including symbols representing assets of the industrial plant; a process of extracting one or more assets from the engineering diagram using machine learning to recognize the one or more assets, the one or more assets being equipment, meters, connectors, and lines, the lines relating the equipment, meters, and connectors to one another; determining one or more relationships between said equipment, meters, connectors, and lines using machine learning to recognize said one or more relationships; a process for creating a flow graph from said devices, meters, connectors, and lines and the relationships between said devices, meters, connectors, and lines; a process for creating a data table of the flow graph, the data table including constituent graphs that make up the flow graph, the data table including the nodes and edges that make up each constituent graph, the data table and the contents of the data table being used to generate an HMI for controlling the industrial plant; A control method comprising:

12. The method of claim 11 , wherein the process of extracting one or more assets from the engineering diagram includes a geometry-based extraction process and a machine learning-based classification process.

13. The method of claim 11 further comprising the process of extracting one or more control loops from the engineering diagram based on one or more of the equipment, instruments, connectors, and lines.

14. The method of claim 11 further comprising the process of extracting a unit identifier from the engineering diagram, the unit identifier uniquely identifying a particular unit among a plurality of units in the industrial plant.

15. 12. The method of claim 11, further comprising the process of extracting a drawing number and a revision number from the engineering diagram, the drawing number uniquely identifying the engineering diagram among a plurality of engineering diagrams of the industrial plant, and the revision number indicating a revision of the engineering diagram that is incremented whenever there is a change to the engineering diagram.

16. The method of claim 14 further comprising the process of using the unit identifiers from the engineering diagram to assign a unique identifier to each piece of equipment, gauge, connector, and line on the engineering diagram.

17. 12. The method of claim 11, wherein the process of determining one or more relationships between devices, meters, connectors, and lines includes generating a line-line graph, a device-line graph, a connector-line graph, and a meter-line graph.

18. 20. The method of claim 17, wherein the process of generating the flow graph for the engineering diagram includes a process of merging the line-line graph, the equipment-line graph, the connector-line graph, and the instrument-line graph with each other.

19. 20. The method of claim 17, further comprising the process of merging flow graphs from a plurality of engineering diagrams for the industrial plant to create a flow graph for the industrial plant.

20. The method of claim 11 further comprising the process of displaying the equipment, gauges, connectors, and lines and the relationships between the equipment, gauges, connectors, and lines in an asset hierarchy.

21. A computer readable medium storing computer readable instructions for causing one or more processors to carry out the control method of any one of claims 11 to 20.

Images