Automatically extract asset data from engineering data sources for use in generating HMIs

Abstract

Systems and methods for controlling industrial process automation and control systems can automatically extract plant assets from engineering drawings and other plant engineering data sources by using machine learning (ML) models and algorithms. The systems and methods can establish asset relationships to create a plant asset registry and build an asset hierarchy from plant assets. The systems and methods can generate an ontology knowledge base from the plant asset hierarchy and provide an HMI for controlling industrial processes based on the plant asset hierarchy and the ontology knowledge base.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Provisional Application No. 62 / 823,469, filed on March 25, 2019, entitled “Systems and Methods for Performing Industrial Plant Diagnostics and Operations”; U.S. Provisional Application No. 62 / 842,929, filed on May 3, 2019, entitled “Systems and Methods for Performing Industrial Plant Diagnostics”; and U.S. Provisional Application No. 62 / 823,377, filed on March 25, 2019, entitled “Systems and Methods for Detecting and Predicting Faults in an Industrial Process Automation System”. The subject matter of this application is also related to U.S. Non-Provisional Application No. 16 / 021,867, filed on June 28, 2018, entitled “Machine Learning Analysis of Piping and Instrumentation Diagrams”, which is incorporated herein by reference. Technical Field

[0003] Aspects of the present disclosure generally relate to industrial process automation and control systems. More specifically, aspects of the present disclosure relate to systems and methods for extracting plant assets from engineering drawings and other engineering data sources, creating a plant asset hierarchy from the plant assets, building an ontology knowledge base from the plant asset hierarchy and asset association relationships with other assets, and providing a human-machine interface ("HMI") for controlling an industrial process automation and control system based on the plant asset hierarchy and the ontology knowledge base. Background Art

[0004] A plant operator is tasked with ensuring that an industrial plant is operating properly. This requires, for example, monitoring and processing alarms triggered by plant components (e.g., pumps, valves, tanks, sensors), performing actions on plant components (e.g., shutting down components and starting up components), and generally overseeing the proper operation of the plant. To perform these tasks, a plant operator may use an HMI that provides the plant operator with a visual representation of the plant components and the data collected from the plant components. The HMI may allow the plant operator to interact with the plant components (e.g., perform actions such as shutting down components).

[0005] An HMI can be designed for a specific plant to capture components within the plant, display data used to operate the plant, and provide an interface for initiating actions that plant operators may need. Designing an HMI can be expensive and time consuming, and may require expertise in both plant and process engineering. Today, plant HMIs are created using a process-centric approach that focuses on process definitions. On the other hand, plant operators have a more asset-centric view of the plant, focusing on automation control definitions (e.g., plant, area, unit, equipment, equipment, set points for specific equipment, etc.). As a result, plant operators may find current plant HMIs unintuitive and difficult to use.

[0006] Therefore, there is a need for improvements in the area of ​​industrial plant commissioning and operation. Summary of the invention

[0007] Embodiments of the present disclosure provide systems and methods for controlling industrial process automation and control systems. The methods and systems automatically and by using machine learning (ML) models and algorithms, extract plant assets from engineering drawings and other plant engineering data sources, establish asset relationships to create a plant asset registry and build an asset hierarchy from the plant assets, generate an ontology knowledge base from the plant asset hierarchy, and provide an HMI for controlling industrial processes based on the plant asset hierarchy and the ontology knowledge base.

[0008] In general, in one aspect, embodiments of the present disclosure relate to control systems for industrial plants and corresponding methods thereof. The control system (and method thereof) particularly includes one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor executable instructions, which, when executed by the one or more processors, cause the control system to input an engineering drawing of a unit of the industrial plant, the engineering drawing including symbols representing assets of the industrial plant, and to extract one or more assets from the engineering drawing using machine learning to identify the one or more assets. The one or more assets include equipment, instruments, connectors, and lines that relate the equipment, instruments, and connectors to each other. The processor executable instructions also cause the control system to use machine learning to determine one or more relationships between equipment, instruments, connectors, and lines to identify the one or more relationships. The processor executable instructions also cause the control system to create a flow chart based on the equipment, instruments, connectors, and lines and the relationships between the equipment, instruments, connectors, and lines.

[0009] According to any one or more of the foregoing embodiments, the process of extracting one or more assets from the engineering drawing includes a geometry-based extraction process and a machine learning-based classification process. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to extract one or more control loops from the engineering drawing based on one or more of equipment, instruments, connectors, and lines. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to extract a unit identifier from the engineering drawing, wherein the unit identifier uniquely identifies a specific unit from a plurality of units in an industrial plant, and / or extract a drawing number and a revision number from the engineering drawing, wherein the drawing number uniquely identifies the engineering drawing from a plurality of engineering drawings in an industrial plant, and the revision number indicates the revision of the engineering drawing, and the revision number is incremented whenever there is a change in the engineering drawing. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to use the unit identifier from the engineering drawing to assign a unique identifier to each equipment, instrument, connector, and line of the engineering drawing. According to any one or more of the foregoing embodiments, the process of determining one or more relationships between equipment, instruments, connectors, and lines includes a process of generating a line-line graph, a process of generating an equipment-line graph, a process of generating a connector-line graph, and a process of generating an instrument-line graph. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to create a flow diagram of an engineering drawing by executing a process of merging line-line diagrams, equipment-line diagrams, connector-line diagrams, and instrument-line diagrams with each other, and / or merging flow diagrams from multiple engineering drawings of an industrial plant to create a flow diagram of an industrial plant. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to display equipment, instruments, connectors, and lines in an asset hierarchy and the relationships between equipment, instruments, connectors, and lines.

[0010] In general, on the other hand, embodiments of the present disclosure relate to control systems for industrial plants and corresponding methods thereof. The control system (and method thereof) particularly includes one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions, which, when executed by one or more processors, cause the control system to input data from multiple engineering data sources of the industrial plant, the data including structured and unstructured data, and to extract one or more domain entities from the structured and unstructured data. The processor-executable instructions also cause the control system to extract instances of one or more domain entities from the structured and unstructured data, and receive a semantic model constructed based on the one or more domain entities and instances of the one or more domain entities. The processor-executable instructions also cause the control system to create and store a knowledge graph based on the semantic model, and to extract information from the knowledge graph to construct a machine interface (HMI) display and control application for the industrial plant.

[0011] According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to extract one or more assets of the industrial plant from unstructured data using machine learning to identify the one or more assets, and / or store the one or more assets of the industrial plant in an asset hierarchy using a namespace that uniquely identifies each asset. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to provide an asset registry application programming interface (API) for allowing users and external systems to access and use the asset registry. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to input data from multiple engineering data sources by inputting one or more of the following: alarm data, cabinet loading reports, control databases, cross wiring reports, field wiring indexes, historical data, instrument indexes, asset hierarchies, nested loading reports, plant asset indexes, process control narratives, and HMI specifications. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to display the knowledge base as a plurality of interconnected labeled attribute nodes, each node representing a domain entity, and / or display the attributes and relationships of each node when a user selects a node. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to allow a user to search the knowledge base using natural language queries, display nodes responsive to the search and assets common to those nodes, and / or display a legend for the nodes based on color coding.

[0012] In general, in another aspect, embodiments of the present disclosure relate to control systems for industrial plants and corresponding methods thereof. The control system (and method thereof) includes, in particular, one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions that, when executed by the one or more processors, cause the control system to input data from multiple engineering data sources of the industrial plant and extract one or more assets of the industrial plant from the data using machine learning to identify the one or more assets. The processor-executable instructions also cause the control system to create a human-machine interface (HMI) asset model from the assets, the HMI asset model arranging the assets in a hierarchical structure, and generating an HMI display from the HMI asset model, the HMI display showing symbols representing one or more assets of the industrial plant and lines representing processes, connections, and data links between the assets. The processor-executable instructions also cause the control system to display the HMI display to the user and allow the user to navigate vertically and horizontally along the display, wherein the HMI display dynamically changes which assets are displayed based on the user's position change on the HMI display.

[0013] According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to identify all alarms currently issued for the industrial plant and the assets corresponding to the alarms and display the alarms on the HMI display, and / or display the HMI display at the plant level, which displays all currently issued alarms and the assets corresponding to the alarms identified for the industrial plant. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to determine and display on the HMI display the assets that are potential root causes of the alarms when the user selects the asset corresponding to one of the alarms, wherein the control system determines the assets that are potential root causes by finding all assets that are connected to each other and also have alarms, and / or allows the user to manually correct the root cause of the alarm on the HMI display and provide the root cause from the user as feedback to the control system. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to automatically zoom in or out of the assets displayed on the HMI display based on the change of the user's position on the HMI screen. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to display the assets in the HMI display as a two-dimensional view, wherein the HMI display allows the user to navigate vertically and horizontally within the two-dimensional view. In accordance with any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to dynamically change the assets displayed on the HMI display based on an alarm occurring on the asset, and / or display the asset on the HMI display at runtime based on one of: a static weight assigned to the asset based on the asset's position in the asset hierarchy, and a dynamic weight assigned to the asset based on an alarm issued at the asset.

[0014] In general, in yet another aspect, embodiments of the present disclosure relate to a computer-readable medium storing computer-readable instructions for causing one or more processors to perform a method according to any one or more of the preceding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A more detailed description of the disclosure of the above summary can be obtained by reference to various embodiments, some of which are shown in the accompanying drawings. Although the drawings illustrate selected embodiments of the disclosure, these drawings are not to be considered limiting of its scope, as the disclosure may admit to other equally effective embodiments.

[0016] Figure 1 An exemplary plant asset hierarchy according to an embodiment of the present disclosure is shown;

[0017] Figure 2 An exemplary engineering diagram that can be used to construct a plant asset hierarchy according to an embodiment of the present disclosure is shown;

[0018] Figure 3An exemplary industrial plant control system that may be used for a process according to an embodiment of the present disclosure is shown;

[0019] Figure 4 An exemplary intelligent processing module that may be used with an industrial plant control system according to an embodiment of the present disclosure is shown;

[0020] Figure 5 An exemplary HMI application that may be used with an industrial plant control system according to an embodiment of the present disclosure is shown;

[0021] Figure 6 An exemplary ML-based asset extraction and relationship building according to an embodiment of the present disclosure is shown;

[0022] Figure 7A-7B shows an exemplary arrow extraction according to an embodiment of the present disclosure;

[0023] Figure 8 shows an exemplary pump extraction according to an embodiment of the present disclosure;

[0024] Fig. 9 and Figure 9A-9C An exemplary connector extraction according to an embodiment of the present disclosure is shown;

[0025] Figure 10A-10B shows an exemplary line-line graph according to an embodiment of the present disclosure;

[0026] Fig.11 shows an exemplary equipment extraction according to an embodiment of the present disclosure;

[0027] Figure 12A-12B shows an exemplary equipment-circuit diagram according to an embodiment of the present disclosure;

[0028] Figure 13A-13B shows an exemplary connector-line pattern according to an embodiment of the present disclosure;

[0029] Figure 14A-14B shows an exemplary instrument-circuit diagram according to an embodiment of the present disclosure;

[0030] Figure 15A-15B A PI&D process diagram according to an embodiment of the present disclosure is shown;

[0031] Figure 16A-16B shows an exemplary line-line graph with a device removed therefrom according to an embodiment of the present disclosure;

[0032] Fig.17 An exemplary control loop according to an embodiment of the present disclosure is shown;

[0033] Fig.18An exemplary method for constructing a plant engineering domain ontology according to an embodiment of the present disclosure is shown;

[0034] Fig.19 shows exemplary alarm data according to an embodiment of the present disclosure;

[0035] Fig. 20 An exemplary cabinet loading report according to an embodiment of the present disclosure is shown;

[0036] Fig.21 An exemplary control database according to an embodiment of the present disclosure is shown;

[0037] Fig. 22 An exemplary cross-wiring report according to an embodiment of the present disclosure is shown;

[0038] Fig.23 An exemplary field wiring index according to an embodiment of the present disclosure is shown;

[0039] Fig.24 shows exemplary historical data according to an embodiment of the present disclosure;

[0040] Fig.25 shows an exemplary instrument index according to an embodiment of the present disclosure;

[0041] Fig.26 An exemplary asset hierarchy according to an embodiment of the present disclosure is shown;

[0042] Fig. 27 An exemplary nested loading report according to an embodiment of the present disclosure is shown;

[0043] Fig.28 An exemplary plant asset index according to an embodiment of the present disclosure is shown;

[0044] Fig.29 An exemplary process control narrative according to an embodiment of the present disclosure is shown;

[0045] Fig.30 An exemplary table format for data extraction according to an embodiment of the present disclosure is shown;

[0046] Fig.31 An exemplary user interface for building a semantic model using entity types according to an embodiment of the present disclosure is shown;

[0047] Fig.32 An exemplary HMI display screen displaying a knowledge graphic according to an embodiment of the present disclosure is shown;

[0048] Fig.33 An exemplary HMI display screen showing user query results according to an embodiment of the present disclosure is shown;

[0049] Fig.34 An exemplary HMI design specification according to an embodiment of the present disclosure is shown;

[0050] Fig.35 An exemplary input for constructing an HMI model according to an embodiment of the present disclosure is shown;

[0051] Fig.36 shows an exemplary HMI screen displaying alarm root cause determination according to an embodiment of the present disclosure;

[0052] Fig.37 shows an exemplary HMI screen displaying plant-level alarm aggregation according to an embodiment of the present disclosure;

[0053] Fig.38 shows an exemplary dynamic scaling according to the user's position on the HMI screen according to an embodiment of the present disclosure;

[0054] Fig.39 shows an exemplary HMI screen showing dynamic scaling according to an embodiment of the present disclosure; and

[0055] Figure 40A-40B An exemplary plant-level alarm aggregation for an HMI according to an embodiment of the present disclosure is shown.

[0056] Wherever possible, identical reference numerals are used to refer to identical elements in the drawings. However, elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation. DETAILED DESCRIPTION

[0057] The description and drawings illustrate exemplary embodiments of the present disclosure and should not be considered restrictive, and the claims define the scope of the present disclosure, including equivalents. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the scope (including equivalents) of this specification and claims. In some cases, well-known structures and techniques are not shown or described in detail to avoid confusing the present disclosure. In addition, as long as feasible, the elements and related aspects thereof described in detail with reference to one embodiment may be included in other embodiments that are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment, and the element is not described with reference to a second embodiment, the element can still be declared to be included in the second embodiment.

[0058] Reference now Figure 1, shows an example of an industrial plant asset hierarchy 100 of the type that may be created according to an embodiment of the present disclosure. The term "plant asset" as used herein generally refers to any plant equipment, instrumentation, groupings thereof, equipment associated therewith, and the like that is commonly used in an industrial plant. At the bottom of the asset hierarchy 100 are instruments (O) 102, such as sensors, monitors, actuators, and the like. Immediately above the instruments 102 are equipment (D) 104, such as vessels, heaters, exchangers, pumps, motors, mixers, and the like. One or more instruments 102 may be connected to one or more equipment 104 to form a closed or open control loop 106 that performs some sub-process in a plant process, such as mixing, heating, boiling, and the like. Several types of equipment 104 and instruments 102 may be combined to form a unit (U) 108 that serves some function in a plant process, such as a gas synthesizer unit. A plurality of such units 108 make up an area (A) 110 , while a plurality of areas 110 make up a plant site (S) 112 , and a plurality of sites 112 form an enterprise (N) 114 .

[0059] Additionally, in general, while each node in the hierarchy 100 can have child nodes and parent nodes, each child node can typically only have one parent node. However, assets such as pipes and wires can be considered equipment, so that a pipe carrying material from / to two assets can be logically divided into two pipes. Equipment or instruments can only be connected to one piece of equipment. This hierarchy 100 arranges and organizes plant assets in a logical manner and is easier for the plant control system and plant operators to understand and use.

[0060] Figure 2 An exemplary engineering drawing or diagram 200 that can be used to construct an asset hierarchy 100 (or a portion thereof) according to an embodiment of the present disclosure is shown. The engineering drawing 200 can be any diagram, such as a process flow diagram (PFD) or a piping and instrumentation diagram (P&ID), which graphically depicts how various instruments 102 (e.g., sensors) and equipment 104 (e.g., containers) in a plant are connected to each other. The example diagram 200 in this example is a P&ID, each of which displays a plurality of symbols representing the equipment or instruments, equipment, and other components that are typically required to implement a unit 108. Connectors 202 throughout the diagram 200 specify how the various PI&Ds (i.e., units) are connected together to form areas, sites, etc. Each diagram 200 also typically has an information block 204, usually in the lower right corner, which displays a drawing identifier 206, a unit identifier, and other information that helps to uniquely identify the PI&D (e.g., revision number). Other well-known parts of a PI&D that are not explicitly mentioned here include tag numbers, lines (e.g., process, electrical, data links, etc.), loops, etc.

[0061] Figure 3 An exemplary industrial plant control system 300 that can be used to process engineering drawings 200 and other plant engineering data sources according to an embodiment of the present disclosure is shown. As can be seen, the exemplary system 300 includes one or more processors 302, internal input and / or output ("I / O") interfaces 304, and memory 306, all of which are communicatively coupled and / or electrically connected to each other. These components allow the system 300 to, among other things, extract various instruments 102, equipment 104, control loops 106, units 108, areas 110, and sites 112 from the engineering drawings 200, lines (e.g., process, electrical, data links, etc.), establish relationships between assets to construct a plant asset hierarchy 100, generate an ontology knowledge base from the plant asset hierarchy 100, and provide an HMI for controlling the plant ontology knowledge base.

[0062] The system 300 can be communicatively coupled and / or electrically connected to an external I / O component 308. The external I / O component 308 allows the system 300 to interact and communicate with users and external systems. Communication can be accomplished through one or more communication networks (not explicitly shown) connected to the I / O component 308. The communication network can include a wide area network (WAN) and / or a local area network (LAN) that can be connected to other telecommunication networks, including other WANs, LANs, and / or a portion of the Internet or an intranet. The communication network can be any telecommunication network that facilitates the exchange of data, such as those networks that operate according to IEEE 802.3 (e.g., Ethernet) and / or IEEE 802.11 (e.g., wireless network) protocols. In another embodiment, the communication network is any medium that allows data to be physically transmitted through a serial or parallel communication channel (e.g., copper wire, optical fiber, computer bus, wireless communication channel, etc.).

[0063] The operation of the various components of the system 300 is well known in the art and is therefore only briefly mentioned here. The processor 302 may be adapted to execute processor executable instructions stored in the memory 306. The memory 306 may be adapted to provide the processor executable instructions to the processor 302 upon request. The instructions stored on the memory 306 include an industrial plant control application 310. The industrial plant control application 310 may include a plurality of functional modules that work together to extract assets, establish asset relationships, build asset hierarchies, generate knowledge bases, and provide HMIs, etc. These functional modules may include, for example, an image converter 312, a filtering algorithm 314, a symbol extractor module 316, a tag preprocessing algorithm 318, a tag extractor module 320, a rule engine 322, a line extractor module 324, an intelligent processing module 326, and an HMI application 328.

[0064] The image converter 312 is configured to convert the diagram 200 into an image format. In some embodiments, the image converter 312 obtains the diagram 200 in a portable document file (PDF) or other electronic data format and converts the diagram into another image format, such as a portable network graphic (PNG), a joint photographic experts group (JPEG), a graphics interchange format (GIF), etc. In some embodiments, the image converter 312 creates two image files, one for display and one for calculation (e.g., by the filtering algorithm 314, the symbol extraction 316, the label pre-processing 318, the label extractor 320, the rule engine 322, the line extractor 324, and the intelligent processing 326).

[0065] The filtering algorithm 314 is configured to process the computational image to obtain the approximate size of the symbols therein. Exemplary symbols include, but are not limited to, those symbols for instruments, control / display elements, programmable logic controllers (PLCs), valves, pumps, etc. that conform to the International Society of Automation (ISA) standards. In some embodiments, the symbol includes an identification letter (e.g., "FIC") and a tag number (e.g., "123"). Obtaining the approximate size of the symbol helps normalize the symbol for machine learning purposes (via intelligent processing 326), as discussed later herein (e.g., to avoid creating training data for symbols of different sizes).

[0066] The symbol extractor 316 is configured to detect symbols extracted from an image. In some embodiments, the symbol extractor 316 applies an image processing algorithm to identify possible areas of symbols in the image and then detects the symbol type and location in the image via a coarse symbol identification technique. The symbol extractor 316 maintains a running count of newly detected symbols in order to track the number of detected symbols and determine whether any new symbols are detected during a given execution cycle.

[0067] Label preprocessing 318 is configured to remove the symbol lines from the symbols detected in the computational image, leaving only the label component. In some embodiments, this includes centering the symbol and then removing the symbol lines from the symbol, leaving only the label component. Connected pixels are clustered, and any pixels smaller than the standard text size and larger than the standard text size are removed. Each pixel cluster is assigned a bounding box that defines the boundaries around the cluster for processing purposes. Label preprocessing 210 then finds bounding boxes of the same level vertically and in a left-to-right order. This allows label preprocessing 210 to remove non-label pixels and noise.

[0068] The label extractor 320 is configured to extract the label components of the symbols in the computational image, such as the label name and the label number. In some cases, adjacent characters in the label name and / or the label number are connected to each other and should be separated. In these cases, the label extractor 320 checks the vertical gaps in the characters of the label and segments the characters, and then performs character recognition using machine learning techniques (via intelligent processing 326). When there is no vertical gap, the label extractor 320 determines whether the aspect ratio of a given character set is greater than a predetermined threshold (e.g., 0.6, etc.). If the aspect ratio is greater than a predetermined threshold, the label extractor 320 applies segmentation using the pixel density in the vertical direction. The area showing the peak value of white pixels is a potential segmentation area in the connected characters. Thereafter, the label extractor 320 performs character recognition using machine learning techniques.

[0069] The rules engine 322 is configured to validate tags extracted from the computational image based on one or more rules. In some embodiments, the rules are based on the ISA symbol standard and are divided into two categories: a primary compliance check (e.g., a red category) and a secondary compliance check (e.g., an orange category). Exemplary primary compliance checks include, but are not limited to, verifying that the symbol is one of the valid types (e.g., a field device, a control room display, etc.), and verifying that the tag name has one or more identification letters. Exemplary secondary compliance checks include, but are not limited to, verifying that the identification letters in the tag name do not contain any numbers, and that the tag number in the tag name does not contain any alphabetic characters except the end.

[0070] The line extractor 324 is configured to extract the lines between the symbols in the computational image. In some embodiments, the extracted lines include pipes and connection symbols, such as pipes, process connections, electrical signals, pneumatic signals, data links, capillaries for filling systems, hydraulic signal lines, and guided electromagnetic or acoustic signals. As will be understood by those of ordinary skill in the art, a geometric line fitting algorithm is used to extract straight lines from the image. Once the line is extracted, a portion of the line is subjected to one or more machine learning models (via intelligent processing 326) to obtain the type of line as described above. Additional details on the operation of modules 312-324 can be found in the above-mentioned U.S. non-provisional application No. 16 / 021,867, and are incorporated herein by reference. Intelligent processing 326 and HMI application 328 will be discussed further below.

[0071] Figure 4 An exemplary implementation of the intelligent processing 326 according to an embodiment of the present disclosure is shown. As discussed, the module 326 implements machine learning models and algorithms to identify symbols in a computational image. In some embodiments, the intelligent processing 326 applies a deep neural network to a specific area of ​​the image obtained by advancing a sliding window across the entire image. Figure 4In the example of , the intelligent processing 326 includes a data input component 402, one or more machine learning models 404, an automatic feedback / correction component 406, a user application 408, a manual feedback / correction component 410, and an analyzer 412.

[0072] In general operation, the data input component 402 receives data representing drawings (e.g., P&IDs, PFDs, etc.) and, after appropriate preprocessing, feeds the data to one or more machine learning models 404. The machine learning model 404 uses machine learning and image processing techniques to extract relevant information, such as names, numbers, symbols, lines, loops, etc., from the input data. The machine learning model 404 may also use image processing and / or geometric algorithms to reduce noise and improve accuracy. The automatic feedback / correction component 406 applies rules and algorithms configured to detect errors in the output received from the machine learning model 404. These errors are used to automatically correct the model output and are fed back to the machine learning model 404 via the analyzer 412, thereby updating the learning of the machine learning model 404. The processed output from the automatic feedback / correction component 406 is then displayed to the user for verification via the user application 408 (e.g., the HMI application 328). The corrections made by the user are captured by the manual feedback / correction component 410 and fed back to the machine learning model 404 via the analyzer 412 to update the learning of the machine learning model 404. In this manner, intelligent processing 326 can continually evolve and improve its evaluation of input data and extraction of relevant information therefrom.

[0073] As described above, in some embodiments, the intelligent processing 326 applies a deep neural network to a specific area of ​​the obtained image. The deep neural network processing results in a multi-class classification of symbol candidates. In some embodiments, the symbols are classified according to the ISA symbol system. Exemplary symbols include, but are not limited to, instruments, control / display elements, programmable logic controllers (PLCs), valves, pumps, etc. In some embodiments, the intelligent processing 326 utilizes at least three types of convolutional neural networks to identify various symbol candidates. These three types include a decision network that determines whether the input is a single character or multiple characters, a single character network that recognizes single letters and numeric characters, and a multi-character network that recognizes multiple characters or words. In some embodiments, the intelligent processing 326 also utilizes context-based predictions to distinguish similar characters, such as distinguishing between the number "1" and the uppercase letter "I" and distinguishing between the letter "O" and the number "0".

[0074] Figure 5It is an exemplary implementation of the HMI application 328 according to an embodiment of the present disclosure. The HMI application 328 generally starts from box 504 and inputs data from various plant engineering data sources into the industrial plant control system 300, these data sources include plant engineering drawings (e.g., PI&D, PFD, etc.). In box 506, the data is processed to extract relevant information and assets from the data, such as names, numbers, symbols, lines, loops, etc. Via the intelligent process 316 and the ML model therein, machine learning is used to complete asset extraction. Using the ML-based asset extraction processing data described herein constitutes a practical application (e.g., designing HMI for industrial plants using a specialized automated design system). The processing system and method described herein can improve the process of HMI design by allowing image recognition models of plant components in the identification diagram to run more efficiently and accurately. This can be achieved by inputting images predetermined by a preprocessing module, which may result in positive identification of the image recognition ML model. This can also allow better training of the model, thereby improving the accuracy of the model.

[0075] At block 508, the extracted assets and other relevant information are used as input to an asset relationship building process to construct an asset hierarchy, as described in more detail later herein. The building of asset relationships is also accomplished with the aid of the intelligent process 316 and the ML model therein. In the field of HMI design, building an asset hierarchy using the ML-based asset relationship building process described herein is not well understood, conventional, or traditional. Building an asset hierarchy using the methods and systems described herein constitutes a practical application (e.g., designing HMIs for industrial plants using specialized automated design systems).

[0076] At block 510, the asset hierarchy may be used to create a knowledge graph based on the semantic model, as described in greater detail later herein. Creating a knowledge graph using the methods and systems described herein is not well understood, conventional, or traditional in the art of HMI design. Creating a knowledge graph using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0077] At block 512, the knowledge graph may be used to generate an HMI asset model (e.g., automatically using a computer), as described in more detail later herein. Generating a knowledge graph using the methods and systems described herein is not well understood, conventional, or traditional in the art of HMI design. Generating a knowledge graph using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0078] At block 514, the HMI asset model may be used to provide or build an HMI (e.g., automatically using a computer), as described in more detail later herein. Generating an HMI using the methods and systems described herein is not well understood, conventional, or traditional in the art of HMI design. Providing an HMI using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0079] Figure 6 More details are shown from Figure 5 An exemplary ML-based asset extraction process 506 and an ML-based asset relationship establishment process 508 are shown.

[0080] ML-based asset extraction 506 generally begins at block 602, where engineering drawings (e.g., PI&D, PFD, etc.) are converted from PDF to image format. At block 604, the user enters information identifying the plant site and area where each drawing is being used. At block 606, all text in the drawing is found, and at block 608, the unit identifier of the drawing is found.

[0081] In some embodiments, finding the unit identifier includes searching for certain keywords, such as "drawing" or "unit" or "part" or variations thereof, in the information block of the drawing or other predefined parts of the drawing. The predefined part can be automatically identified based on analysis of previous drawings (for example, the width of the rightmost area is 20% of the width of the P&ID and PFD, and the height of the lowest area is 20% of the height of the P&ID and PFD), or the part can be specified by the user. Once the keyword (for example, "unit") is found, the system checks the text associated with the keyword, which can be in the same cell as the word "unit" if there is a table, or to the right of the word, or below the word. Once the unit label is determined, the determined label can be displayed to the user for verification. In some embodiments, the process can be performed for a relatively small number of drawings (for example, approximately 20). In this case, the location of the determined label can be used to determine subsequent labels in other drawings without requiring or suggesting verification of the determination.

[0082] At block 610, all devices and instruments (e.g., labels, valves, sensors, actuators, etc.) in the diagram are found for each diagram, and at block 612, all found devices and instruments and text are removed or otherwise suppressed from the diagram. This process may include generating a bounding box around the diagram features to be removed and setting all interior pixels to the background color of the diagram (e.g., white or black). At block 614, the removal of text and devices makes it easier to find all lines in the diagram.

[0083] In some embodiments, the lines in the figure can be found by shrinking the figure and converting them to grayscale. Then, long horizontal and vertical lines can be removed by detecting and deleting black pixels (in the case of a white background) that extend over a predetermined length along the X-axis and Y-axis (for example, 0.7 times the width and height of the figure, respectively). Once this is done, the remaining horizontal and vertical lines can be found by searching for clusters of black pixels (in the case of a white background). A cluster is a group of pixels separated by less than a predetermined number of pixels (for example, less than four pixels). Once found, the clusters can be connected to create horizontal and vertical lines by combining all the collinear points of the clusters. Lines less than a predetermined length (for example, 16 pixels) can be removed. In an embodiment where the figure is scaled before the line is removed, the removed lines can be scaled back to their original size. This may allow the original line coordinates to be stored and referenced when determining the connections and relationships between factory components. Various line types can be identified, including pipes and connection symbols, such as pipes, process connections, electrical signals, pneumatic signals, data links, capillaries for filling systems, hydraulic signal lines, and guided electromagnetic or acoustic signals. In some embodiments, the above-mentioned line extraction can be completed using a geometric line fitting algorithm.

[0084] Once the line is extracted, a portion of the line is subjected to one or more machine learning models to obtain the type of the line as described above. In some embodiments, the line extractor includes processor-executable instructions embodied on a storage memory device to provide the line extractor via a software environment. For example, the line extractor can be provided as processor-executable instructions including procedures, functions, routines, methods, and / or subroutines.

[0085] Continuing to block 616, all arrows in the graph may be found for each graph. These arrows represent the flow direction in the graph. Fig. 7A and Figure 7B Exemplary techniques for identifying arrows in a diagram are described.

[0086] refer to Fig. 7A and 7B , an exemplary arrow 702 is shown. To analyze arrow 702, in some embodiments, as Fig. 7AThe detected line 704 shown is assigned a bounding box 706 having a width, for example, perpendicular to the direction of the line, which is a predetermined number of pixels wide (e.g., 40 pixels). In order to determine whether the line 704 is an arrow, two squares 708, 710 on each end of the line can be drawn with a side length equal to the height of the generated bounding box 706. In some embodiments, the sides of the squares 708, 710 can have different lengths. The contents of the boxes 708, 710 can be analyzed using a machine learning model or algorithm that is trained to determine the presence of an arrow 712 and the direction of the arrow. The ML algorithm can be, for example, a convolutional neural network. The training data for the algorithm can be created by cropping the head and tail of the identified line segment to a specific size (e.g., 40×40 pixels). These cropped images can be stored and later classified as a left arrow, a right arrow, an up arrow, a down arrow, or no arrow. This classification can be done by, for example, a human. The identified line segment can be selected from a graph received from different sources (e.g., different clients). In some embodiments, approximately 200 training images can be used for training. If both squares 708, 710 have arrows, then arrow 702 can be considered bidirectional. If only one of the two squares 708, 710 has an arrow, arrow 702 can be considered unidirectional.

[0087] Return to reference Figure 6 , after the arrows are identified, at block 618, all equipment (e.g., tanks, pumps, boilers, heat exchangers, etc.) in the diagram are found for each diagram. The equipment is then given a name that helps to uniquely identify the equipment. Depending on the particular industry, one of several naming conventions may be used. For example, the following naming convention may be used: <Equipment Type>-<Equipment Name / ID>, where a hyphen or space is used as a separator. Figure 2 In the example of , the name assigned to the container may be "C-1119", where C is the type of equipment and 1119 is the identifier of the equipment in the diagram. This name can then be retained as a Regex (regular expression) for the tank for search purposes. Similar naming conventions can be applied to other diagram components.

[0088] In some embodiments, the search equipment combines geometric or geometry-based identification methods with machine learning or ML-based classification.

[0089] The identification of equipment based on geometry is performed by first removing all instruments from the diagram (e.g., by backgrounding the instruments) to simplify the process, and then finding all parallel lines that meet one or more criteria. Then find the path connecting the parallel lines and use it to connect such pairs of parallel lines. For example, a storage tank can be found by searching for parallel lines and connecting those line pairs that meet one or more criteria. Then a bounding box can be defined around each pair connected in this way. A standard can require that two lines are within a predetermined distance of each other. In some embodiments, the predetermined distance can be between about 10% and 120% of the larger length of the two lines. The minimum length of such a line can be set to about 4% of the width of the diagram. The maximum length of such a line can be set to about 50% of the width of the diagram. The difference between the lengths of the two lines can be less than about 15% of the larger length of the two lines. The starting and ending points of the two lines (i.e., four points) can be used to make a box. The width and / or height of the box can be extended by a predetermined percentage. For example, if the aspect ratio is less than about 0.3, the width can be expanded by about 80%. If the aspect ratio is between about 0.3 and about 0.4, the width can be expanded by about 60%. If the aspect ratio is greater than about 0.4, the width may be expanded by about 40%. If the box is to expand beyond the graphic boundaries, the expansion may stop at the graphic boundaries. The resulting expanded box may be a bounding box. The image of the tank may be cropped at the bounding box. In some embodiments, the cropped image boundaries may be erased by making, for example, about 4 pixels of one or more sides white (in the case of a white background).

[0090] ML-based classification can be used to identify equipment types. For example, a machine learning model or algorithm can be used to analyze the presence of tanks in the resulting image and identify the type of tank. ML models or algorithms can be trained to detect and classify tanks.

[0091] In some embodiments, the resulting image can be analyzed to determine whether it is to be input into the model. For example, if the image has a cluster including connected pixels (e.g., pixels with a maximum gap of approximately 2 pixels), the cluster has an extended bounding box with a maximum dimension greater than the maximum dimension of the non-extended bounding box of the cluster, and includes the start and end points of two parallel lines identified earlier in the tank identification process, then the image can be input into the model. In some embodiments, the extended bounding box can be generated by extending one or more lengths of the non-extended bounding box based on the aspect ratio identified earlier in the process. The image input into the model can be, for example, a black and white image of 170×170 pixels. The model can be trained with a training data set of, for example, 1950 images. Alternatively or additionally, enhanced training data can be used. For example, 46,312 images can be used for training and 2518 for validation. Table 1 below shows exemplary training data augmentation details that can be understood by a person of ordinary skill in the art.

[0092]

[0093]

[0094] Table 1: Example training-data augmentation

[0095] Those skilled in the art will also appreciate that the above table or similar tables can be used to provide training data augmentation for any other ML-based process described herein.

[0096] Reference now Figure 8 Other equipment, such as pumps, may also be identified. A pump may be identified by creating a bounding box that includes the endpoints of one or more horizontal lines in the diagram. Figure 8 An exemplary pump 800 is shown around which a bounding box can be created. To create the bounding box, pixels can be traversed from the endpoint of horizontal line 802 (e.g., P1) in each direction perpendicular to line 802 until a black pixel (e.g., P2) is found. The distance (e.g., D) between P1 and P2 can be calculated. If the distance is between a predetermined value (e.g., 100 pixels and 200 pixels), a bounding box can be created so that P1 and P2 are located at the center of opposite sides of the bounding box. The width of the bounding box can be equal to the distance D. The bounding box can be extended by 20% of the length toward point P2. The image may be cropped at the bounding box. By creating groups of pixels, clusters of connected pixels can be generated from the cropped image, where each group has all pixels connected together horizontally, vertically, or diagonally. For clusters whose length or width is at a predetermined threshold (e.g., 150 pixels), the area of ​​the cluster can be determined. The cluster with the largest area can have its bounding box determined by, for example, finding the minimum and maximum values ​​of the x and y coordinates in the cluster. The minimum of the x and y coordinates may determine the upper left corner of the bounding box. The maximum of the x and y coordinates may determine the lower right corner of the bounding box. The image may be cropped at the bounding box of this cluster. The resulting image may be analyzed for the presence and type of pumps using, for example, a machine learning algorithm trained to detect and classify pumps. For example, a similar CNN architecture may be used.

[0097] Return again Figure 6 , connectors in the graph and their associated information may also be identified at block 620. Recall that connectors indicate connections between components located on different graphs.

[0098] Fig. 9An exemplary connector 902 is shown. As can be seen, the back end 903 (the end without an arrow) of the connector 902 has a line 904 connected thereto, indicating a flow from a component identified by a source identifier 906 through the line 904 through the connector 902 to a component identified by a destination identifier 908. A connector having a line connected to its front end 909 (the end with an arrow) can indicate that a flow on the line occurs from a component identified by a source identifier through the connector to a component identified by a destination identifier. A connector identifier 910 within the connector 902 can indicate a diagram identifier (unit label) of a diagram containing a component identified with the destination identifier 908. Similarly, a connector identifier 910 within the connector 902 can indicate a diagram identifier of a diagram containing a component identified with an associated source identifier. Connectors can be identified by filtering identified lines with a length between about 20 pixels and about 40 pixels. Line pairs with a start point less than, for example, four pixels and an end point less than, for example, four pixels can be analyzed to determine whether they are part of a connector. If the length of one line in a pair is greater than the length of the other line in the pair, and the distance between the pair is greater than about 0.1 times the length of the longer line and less than about 0.5 times the length of the longer line, the pair can be analyzed to determine if they are part of a connector. A tight fitting bounding box can be created that encompasses the two lines, with each side extending half the distance between the two lines. Fig. 9A , 9B 9C show three resulting exemplary connector images at 912, 914, and 916. The resulting images can be input into a machine learning model that is trained to identify the presence and orientation of connectors.

[0099] Note that source identifier 906 may be highlighted or otherwise emphasized for easy viewing by a user. This technique may be applied throughout an HMI to draw a user's attention to a particular feature.

[0100] To identify information associated with connector 902, a bounding box (not explicitly shown) may be generated around text 910 within connector 902 to determine if it is a connection identifier. If the bounding box is contained by the bounding box of connector 902, the text may be identified as connection identifier 910. Text located above the bounding box of connector 902, within, for example, four times the distance between parallel lines of connector 902 (e.g., two parallel lines connected to an arrow), may be identified as a destination identifier contained in 908. In some embodiments, alternatively or additionally, the text before and after the word "to" may be identified as a material name and destination identifier 908, respectively. In this case, the material name may identify a material flowing through a pipe or other material carrier. In some embodiments, alternatively or additionally, the text before and after the word "from" may be identified as a material name and a source identifier, respectively. To identify the pipe identifier contained in 906, a stride may be spanned from the bounding box of connector 902 in the direction of line 904. In some embodiments, the stride may be twice the length of the bounding box of connector 902. When the shifted connector bounding box intersects the bounding box of a word, the word may be identified as the pipe identifier contained in 906. If the bounding boxes of multiple words intersect, the word with the largest number of characters may be identified as the pipe identifier contained in 906. If connector 902 has a line connected to its front end, one or more of the aforementioned methods may be used to identify the source identifier and / or the destination identifier.

[0101] Return again Figure 6 , after the ML-based asset extraction process 506 is completed, an ML-based asset relationship building process 508 can be started for the extracted assets. This process of building relationships between the extracted assets can include creating a line connection graph that captures the connections shown in the diagram (e.g., PI&D, VFD, etc.), and then merging these graphs to create a composite process graph. The merged graphs can include a graph representing a diagram line connected to equipment, a graph representing a diagram line connected to a connector, and / or a graph representing a diagram line connected to an instrument component (e.g., a label). In addition, line connections, such as line-line (line-to-line) graphs, can also be generated to facilitate determining how various components are connected.

[0102] In block 622, a line-line (line-to-line) graph can be generated by detecting lines connected to the bounding box of the component and checking the collinearity or perpendicularity of these lines. If collinearity or perpendicularity is detected, and the distance between the start or end point of one line and the start or end point of another line is less than a predetermined value (e.g., 32 pixels), the lines can be extended so that their ends meet. Lines that form part of a component (e.g., equipment) can be removed by removing those lines whose endpoints are all located within the bounding box of the component. In order to assign indexes to line segments and track line connections, the line can be split. If a device (e.g., a label, a valve, etc.) is located at the top of the line, it is particularly useful to divide the line into two segments. To this end, the distance to the start and end point of the vertical line can be measured from the horizontal line. In some embodiments, the start and end point of the horizontal line can be excluded from the measurement. If the measured distance is less than a predetermined distance (e.g., 32 pixels), the horizontal line can be split at the closest point between the start or end point of the horizontal line and the vertical line. The process may be repeated for vertical lines; if the distance from the vertical line to the start or end point of the horizontal line is less than a predetermined number (e.g., 32 pixels), the vertical line may be segmented at the closest point between the start or end point of the vertical line and the horizontal line. In some embodiments, the distance from the vertical line to the start or end point of the horizontal line may exclude the start and end points of the vertical line. Lines may be assigned index numbers. When a line is segmented at a point, the two newly created segments may be assigned index numbers. An adjacency matrix may be generated to represent the connections of the lines.

[0103] Fig. 10A An exemplary proximity matrix 1002 is shown. The proximity matrix 1002 may have a size of n×n, where n is the total number of lines. The value at the position [first line index, second line index] in the proximity matrix 1002 may be "0" to indicate that the lines having the first index and the second line index, respectively, are not connected to each other. The value of this position may be "1" to indicate that the lines are connected. It should be understood that other values ​​may be used. Thus, the proximity matrix 1002 may indicate whether two lines are connected to each other. For example, from left to right, node L2 has "1" on node L1 and node L3 to indicate that L2 is connected to L1 and L3. In order to construct a line-line graph, a node-edge (node-to-edge) graph may be generated in which line indices are assigned to nodes and connected line indices are assigned to edges between nodes.

[0104] Fig. 10BAn exemplary line-line graph 1004 corresponding to the proximity matrix 1002 is shown. In the figure, nodes E1 and E2 represent two pieces of equipment, and nodes L1, L2, L3, L4, and L5 represent line segments. It can be seen that line segments L1, L2, L3, L4, and L5 connect E1 and E2 together in the manner reflected by the proximity matrix 1002.

[0105] Back to Figure 6 At block 624 , as part of the ML-based asset relationship building process 508 , a graph representing equipment-line (equipment-to-line) connections may be generated from the graph. Fig.11 and 12A - Fig. 12B An example of constructing a device-line graph according to some embodiments is shown.

[0106] Fig.11 An exemplary equipment 1102 is shown. As can be seen, a bounding box 1104 is placed around the equipment 1102 for computational purposes. To construct the equipment-line graph, the bounding box is expanded on each side by increasing the length of each side, for example by 2.5%. Lines 1106 whose endpoints are within the expanded bounding box are then found, and associated nodes (e.g., L1, L2, L3, etc.) are generated for each line. In some embodiments, if two such lines are connected to each other, the line whose endpoints are closer to the center of the bounding box can generate an associated node. Equipment 1102 can also have associated nodes generated (e.g., E1, E2, E3, etc.). Thereafter, equipment-line pairs can have associated edges generated (e.g., (E1, L1), (E2, L2), (E3, L3), etc.).

[0107] Fig. 12A An exemplary table showing equipment-line connections is shown. The figure shows an equipment-line table 1202 listing various nodes (e.g., L1, L2, L3, etc.) and (e.g., E1, E2, E3, etc.) associated with equipment 1102 and line segment 1106, and an edge table 1204 listing edges associated with these nodes (e.g., (E1, L1), (E2, L2), (E3, L3, etc.).

[0108] Fig. 12B An exemplary equipment-line graph 1206 is shown corresponding to the equipment-line table 1202 and the edge table 1204. As can be seen, there are five edges, edges 1, 2, 3, 4, and 5, reflecting the five points where lines L1, L5, L6, L7, and L9 connect to equipment E1, E2, E3, and E4.

[0109] To find a path between two pieces of equipment, the lines connected to the equipment may be identified by merging a graph 1206 representing the graph lines connected to the equipment with the proximity matrix 1002. If multiple paths are found between the two pieces of equipment, a shortest path algorithm (e.g., Dijkstra's shortest-path algorithm) may be used on the graph to identify the path between the two pieces of equipment. The direction of the arrows associated with the lines of the path may indicate the direction of the material flow between the pieces of equipment and designate which piece of equipment is the destination and which piece of equipment is the source.

[0110] Return again Figure 6 At block 626, as part of the ML-based asset relationship construction process 508, a graph representing connector-line (connector-to-line) connections may be generated from the graph. The following is an example of constructing a connector-line graph according to an embodiment of the present disclosure.

[0111] To construct the connector-circuit diagram, refer to Fig. 9 As discussed, a bounding box is placed around the connector. The bounding box can then have its long side extended, for example, by 0.1 times its length. If the connector is an incoming connector, the extension can be in the direction of an arrow, and if the connector is an outgoing connector, the extension can be in the opposite direction. Lines whose endpoints are within the extended bounding box can then be identified. Nodes such as C1 can then be generated for the connector, and nodes such as Lc1 can also be generated for the lines connected to the connector. Directed edges can then be generated for connector-line pairs (such as (C1, Lc1)), where the direction of the edge is determined by whether the line is connected to the head or tail of the connector. In order to determine whether the equipment is connected to the connector, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) can be used on the graph to determine the path between the line connected to the equipment and the line connected to the connector. If the connection is determined, the direction of the edge connected to the connector node can be used to determine the direction of the flow between the equipment and the connector.

[0112] Fig.13A An exemplary table showing connector-line connections is shown. The figure shows a connector-line table 1302 listing various nodes associated with connectors and line segments (e.g., c1, c2, Lc1, Lc2, etc.), and an edge table 1304 listing edges associated with these nodes (e.g., (C1, Lc1), (C2, Lc2, etc.).

[0113] Fig. 13BAn exemplary connector-line graph 1306 is shown corresponding to the connector-line table 1302 and the edge table 1304. It can be seen that there are five connectors, C1, C2, C3, C4, and C5, connected to two pieces of equipment, E1 and E2, via the line segments where the lines L1-L16 are located. Table 1308 shows the path generated using the graph 1306.

[0114] Return again Figure 6 At block 628, as part of the ML-based asset relationship building process 508, a graph representing the equipment or instrument-line (equipment / instrument to line) connection can be generated from the diagram. The following is an example of building an equipment / instrument-line graph according to an embodiment of the present disclosure.

[0115] A graph representing lines connected to a device or instrument (e.g., a tag, sensor, etc.) can be generated by providing a bounding box surrounding the device. Lines whose endpoints are within the bounding box can then be identified. If each bounding box has fewer than two lines, the length of one or more sides of the bounding box can be increased, for example by 50%. Nodes such as D1 can then be generated for the device, and nodes can also be generated for lines connected to the device (such as L1). Pairs of instruments and lines connected thereto can have associated edges generated. In order to determine whether an instrument and equipment are connected, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) can be used on the graph to determine the path between lines connected to the instrument and lines connected to the equipment.

[0116] Fig.14A An exemplary table showing device / instrument-line connections is shown. The figure shows a device / instrument-line table 1402 listing various nodes associated with devices and line segments (e.g., D1, D2, L1, L2, L3, L4, L5, etc.), and an edge table 1404 listing edges associated with these nodes (e.g., (D1, L1), (D1, L2), (D2, L3), D2, L4), (D2, L5), etc.).

[0117] Fig. 14B Exemplary device / instrument-line graphs 1406 and 1408 are shown corresponding to connector-line table 1402 and edge table 1404. As can be seen in graph 1406, device D1 has two line segments connected to it, L1 and L2. In graph 1408, device D2 has three line segments connected to it, L3, L4, and L5.

[0118] Back again Figure 6, so far, only directed graphs or graphs with a specific flow direction have been discussed. In addition to directed graphs, there may also be undirected graphs in the graph. As part of the ML-based asset relationship construction process 508, at block 630, any such undirected graph can be changed to a directed graph. The following is an example of changing an undirected graph to a directed graph according to an embodiment of the present disclosure.

[0119] To specify a direction for a graph (such as a line-line graph, an equipment-line graph, and a device-line graph), all line nodes in the equipment-line graph are identified. For each of these lines, all possible paths in the line-line graph are traversed to the line connected to the connector or another equipment. The path can then be divided into segments, and the lines in these segments can be assigned directions. This can be done based on the direction of the connected connector, the presence of arrows, or the direction of the lines in the adjacent paths (e.g., the lines connected to the connector). For example, if one path is A->B->C->D, and the other path is A->B->E, then these paths are divided into three paths: A->B, B->C->D, and B->E. The path direction can then be specified as follows. If one of the lines in the path is connected to a connector, the direction of the connector (incoming or outgoing) is assigned to all the lines in the path. If at least one of the lines in the path has an arrow, the direction of the arrow is assigned to all the lines in the path. If all the lines in the path do not have arrows, and the path is not connected to a connector, the adjacent paths connected to the connector are checked, and the direction of the adjacent paths is assigned to all the lines in the path.

[0120] Return to Figure 6 , recall from block 612 that all equipment and instruments (and text) were removed from the diagram, resulting in a line-line drawing showing only connections from equipment to equipment and equipment to connectors. As part of the ML-based asset relationship building process 508, the line-line drawing can now be updated at block 632 to include those equipment and instruments. This can be accomplished, for example, by restoring the equipment to the point at which they were removed.

[0121] Fig.15A Shown according to Figure 6 Block 612 of FIG. 1 removes the exemplary line-line graph 1502 and edge data table 1504 of the device. It can be seen that the device is removed from the point between the connector (C) and the equipment (E), where the two dashed circles are located and replaced by the two corresponding edges, which are edge 5 (L3, L6) and edge 6 (L6, L7).

[0122] Fig. 15BThe same exemplary line-line graph 1502 and edge data table 1504 are shown after being updated to restore the device. It can be seen that two devices (D) are now located between the connector (C) and the equipment (E), replacing the two edges (L3, L6) and (L6, L7). This allows the line connection graph to reflect not only equipment-to-equipment and connector-to-equipment connections, but also device-to-equipment, device-to-connector, and device-to-device connections.

[0123] Return again Figure 6 As part of the ML-based asset relationship building process 508, at block 634, for a given diagram (e.g., PI&D, PFD, etc.), the various wire connection graphics generated for the diagram may be merged. For example, if the diagram is a PI&D, then all wire connection graphics may be merged to create a PI&D flow diagram.

[0124] Fig.16A An exemplary PI&D flow graph 1602 is shown, which is generated by merging various line connection graphs generated for a given PI&D by the ML-based asset relationship construction process 508. The PI&D flow graph 1602, unlike its component graphs, contains all connectors, devices or instruments, and equipment in the PI&D, as well as the connections between them. Therefore, the PI&D flow graph 1602 captures the relationships between all assets extracted from a given PI&D.

[0125] Fig. 16B An exemplary table 1604 containing data for the merged PI&D flow graph 1602 is shown. The table 1604 includes nodes and edges for each of the line connection graphs (e.g., line-line, equipment-line, connector-line, device-line, etc.) generated by the ML-based asset relationship construction process 508 for a given PI&D. The table 1604 also contains the nodes and edges of the merged PI&D flow graph 1602. Such a table can be used as part of generating an HMI.

[0126] Return again Figure 6 As part of the ML-based asset relationship building process 508, at block 636, a control loop of a given diagram (e.g., PI&D, PFD, etc.) can be added to the merged PI&D process diagram 1602. As previously described, a control loop performs some sub-process in a factory process and can be a closed-loop or open-loop control loop. Most control loops involve several devices working together, including sensors, actuators, controllers, etc. There are generally two types of control loops: simple control loops and complex control loops. A simple control loop has an input, an output, and a controller, while a complex control loop includes several simple control loops. Fig.17An example of an identification control loop according to an embodiment of the present disclosure is depicted in FIG.

[0127] Fig.17 A portion of an example diagram 1700 containing a simple control loop is shown. For illustrative purposes, four symbols have been identified and labeled 1702, 1704, 1706, and 1708, each of which represents a device (e.g., a sensor, an actuator, a controller, etc.) or equipment. In this example, identifying the control loop begins by identifying all tags inside the symbol and all words outside the symbol. Next, for each word outside the symbol, the nearest tag inside the symbol is found. For example, for the symbol at 1702, the nearest tag inside the symbol is "TIC 1803". For a given symbol, all words (i.e., words associated with other symbols) whose distance is greater than, for example, 1.2 times the width of the symbol can be ignored. The word is then checked (e.g., using a rules engine) against a word list from an industry standard such as ISA to determine whether the word represents, for example, an alarm, a signal, a tag, etc. If the word is a tag, the tag number of the nearest tag is assigned to the word. For example, for the symbol at 1702, the tag number "1803" is assigned to the words "TE" and "TT" to produce "TE 1803" and "TT 1803". These tagged words are then added to the list of tags for the symbol. The tags for a given symbol include all tags both inside and outside the symbol.

[0128] Once all tags inside and outside the symbol are identified, all tags with the same starting identification letter and the same tag number are grouped together. Fig.17 In the example of , the tags “TIC 1803 ”, “TV 1803 ”, “TE 1803 ”, and “TT1803 ” are grouped together. These tags are used to define a simple loop 1710 , which can be included as an extracted asset of the graph 1700 .

[0129] Back again Figure 6 At block 638, the plant assets are organized in an asset registry for subsequent access and use. Assets are organized using namespaces to uniquely identify each asset. As understood by one of ordinary skill in the art, a namespace is a set of names used to identify objects so that there is no ambiguity when objects with the same name but different origins are mixed together.

[0130] At block 640, the merged flow diagrams of each diagram can be merged as part of the ML-based asset relationship construction process 508 to create a single plant flow diagram for the entire plant. Recall that a diagram like a PI&D or PFD represents the instruments or equipment and equipment that make up a unit, and the asset extraction process discussed above causes the equipment names for a given unit to have the same unit prefix and therefore be uniquely identifiable. Equipment names are similarly created by adding a prefix to the unit name. Connector names are also uniquely identifiable because each connector name contains the name of the source equipment, the name of the target equipment, and the name of the pipe.

[0131] The line index or index number (e.g., 1, 2, 3, etc.) is local to each drawing and therefore needs to be uniquely identified across multiple drawings. This can be done, for example, by prefixing the line index with the unit name or corresponding drawing, followed by the drawing number. For example, if the unit name is "UNIT1" and the drawing number is "1116", a line index number of "15" could be changed to "UNIT1_1116_15"

[0132] In order to create a single plant flow graph for the entire plant, a flow graph diagram for each diagram in the plant may be generated as described above. Where required, line indices may be made uniquely identifiable in different diagrams as described above. The flow graph diagrams of the various diagrams may then be merged by inserting all nodes and edges from each diagram flow graph (or its tabular representation) into the plant flow graph (or its tabular representation). Graphs that are directly connected to each other will have connector nodes with the same name. When this occurs, the duplicate connector nodes are removed and new edges are created in the plant flow graph by connecting the lines to be connected by the removed connectors. All other nodes and edges should remain unchanged.

[0133] The resulting plant flow diagram can be used to generate a similar Figure 1 A visual representation of an asset hierarchy is shown for asset hierarchy 100 . The asset hierarchy and relationships beneath the asset hierarchy may then be stored as structured data, for example, in one or more JSON (JavaScript Object Notation) files, for subsequent access and use by the industrial plant control system 300 .

[0134] In some embodiments, as described above, the industrial plant control system 300 can be used to merge structured data of asset hierarchies and asset relationships into an ontology knowledge base. The system then dynamically builds a plant engineering domain ontology from plant engineering data sources (which evolve during the plant life cycle phases). This typically includes: (1) extracting domain entities (i.e., types and data) from structured data sources (by the system) using metadata and machine learning techniques from unstructured data sources; (2) building a semantic model using entity types (by domain experts) and adding association relationships between entities; (3) the system dynamically extracts entity data (by the system) and builds a knowledge graph based on the semantic model; and (4) providing users with the ability to navigate asset namespaces and relationships to extract relevant information (by the system). This arrangement has many benefits.

[0135] One benefit is that automated control configuration can be performed using process narratives. Currently, control configuration is a time-consuming manual process and requires a strong control engineering background. Using the industrial plant control system 300 described in this article, people who are not familiar with control engineering concepts should be able to easily configure the system. This stems from the system's use of "process narratives" that include common language used by plant engineers. The system automatically converts process narratives into control strategies and configurations. By utilizing noun-verb pairs, such as "Asset.Operation" in AutomationML (see IEC 62714) and state-based control (see ISA106, ISA 88, and ISA95), process narratives can be converted into multiple control narratives. The system uses information from asset templates, their relationships, namespaces, control policy templates, and rules to translate process narratives into control logic.

[0136] Another benefit is that test cases can be automatically generated based on process narratives and documented verification. The industrial plant control system 300 automatically generates test cases using process narratives, control configurations, and asset ontologies. The system then runs automated test scenarios to simulate the process in order to verify the control application.

[0137] Reference now Fig.18, a flow chart of an exemplary method 1800 is shown that can be used with an industrial plant control system 300 to dynamically build a plant engineering domain ontology from a plant engineering data source. The method generally begins at block 1802, where a plant engineering data source is input into the system. The data sources may include PI&Ds and PFDs used to generate the above-mentioned plant asset hierarchy and asset relationships, as well as AutoCAD, SmartPlant, and Aveva engineering drawings. Other plant engineering data sources may include simulation models, instrument indexes, cable plans, I / O assignment lists, configuration data, cross-wiring tables, cabinet loading reports, system audit reports, historical data, alarms and events, etc.

[0138] Figure 19-Figure 29 Exemplary plant engineering data sources that can be used to generate asset hierarchies and / or knowledge graphs are shown. These plant engineering data sources are familiar to those skilled in the art. For example, Fig.19 Exemplary alarm data at 1900 is shown. Fig. 20 An exemplary rack loading report 2000 is shown. Fig.21 An exemplary control database is shown at 2100 . Fig. 22 An exemplary cross-wiring report at 2200 is shown. Fig.23 An exemplary field wiring index at 2300 is shown. Fig.24 Exemplary historical data (label data) at 2400 is shown. Fig.25 An exemplary instrument index at 2500 is shown. Fig.26 An exemplary asset hierarchy at 2600 is shown. Fig. 27 An exemplary nested load report at 2700 is shown. Fig.28 An exemplary plant asset index at 2800 is shown. Fig.29 An exemplary process control narrative is shown at 2900. Such data sources and other plant engineering data sources, including HMI specifications (see Fig.34 ), which can then be used to generate asset hierarchies and / or knowledge graphs.

[0139] Back to Fig.18, at block 1804, the system extracts domain entities and instances of domain entities from the plant engineering data source. The data source may include structured and unstructured data sources. For example, entities (also called types or classes), attributes (also called traits), and relationships may be extracted from the data source. Entities may be used to specify the domain and scope of the ontology, and the values ​​of the attributes may be used to specify the scope of the ontology model. The semantic model may include a set of triples constructed from the domain and scope of the ontology. The triples may specify subject entities, predicate relationships or attributives, and object entities or values. The subject entity may be an entity from the domain of the ontology. The object entity or value may be an entity or value from the scope of the ontology. Instances of entities and values ​​may be extracted from the data source. For example, the semantic model may specify triples with subject "unit", "predicate "has", and object "equipment". Triples including instances of corresponding subjects and objects may include "unit 01" (an instance of the entity "unit") "has" (predicate) "control valve 01" (an instance of the entity "equipment").

[0140] A variety of data extraction techniques can be used for each structured and unstructured data source from the plant engineering data source. The system can distinguish between structured and unstructured data based on the file format or database type. Unstructured data files (e.g., emails, word processing documents, videos, images, web pages, etc.) typically include text and multimedia content without any schema, while structured data files (e.g., spreadsheets, CSV files, XML files, RDBMS, time series, graph databases, etc.) contain schemas or metadata. The system can use the schema or metadata to extract domain entities from these structured data sources. In the CSV file format, for example, plain text data is delimited by commas, where each new row in the CSV file represents a new database row, and each database row has one or more fields separated by commas. The system can use this schema to extract domain entities from the data source. In Fig.25 An example can be seen in Figure 1, where domain entities are extracted from an exemplary instrument index data source in CSV format. As indicated in the first row of the table, the domain entities and associated relationships extracted from this data source include plants, areas, units, loops, tags, process functions, etc. Instances of these domain entities are shown in subsequent rows.

[0141] For unstructured data, such as P&ID, PFD, process control description (PCN) and other images or unstructured text formats, the system uses machine learning to extract domain entities and association relationships from unstructured data sources into structured data. For reference, a process control narrative (PCN) is a functional statement that describes how equipment mounted controls, panel mounted controls, PLCs, HMIs and other processor-based process control system components are configured and programmed to control and monitor a specific process, process area or facility. PCNs are an essential link between process design and control system design. For PCNs, the system uses named entity recognition (NER) technology to extract domain entities such as pumps, valves, locations, alarm conditions, etc. from process narratives. NER is a technology for processing unstructured text information. This technology tags sequences of words in text that are names of things, such as things, names of people, organizations, locations, time expressions, quantities, monetary values, percentages.

[0142] Fig.30 3000, which shows an example of domain entity types and instance data extracted into a file having a table format. The domain entity types in the table include controller, compound, block, label, etc., and the corresponding instance data include (for the first row) CP2801, CF101_011900, PT038, 011900PV038, etc.

[0143] Back to Fig.18 Once the domain entities and relationships have been extracted, at block 1806 , a user (eg, a domain expert) builds a semantic model using the extracted entity types and associated relationships. Fig.31 An example of a user interface is shown, which a domain expert can use to build a semantic model.

[0144] exist Fig.31 , a user interface 2100 is shown that allows a user to specify semantic relationships between entities extracted from a data source. From this interface, a user can manually browse entity types from various data sources and filter the entities required for the semantic model. For example, a user can select two entities, such as a source and a target, and create a named relationship between them. The pattern can be defined at a conceptual level, which is a logical model. Then, the user can extract domain-specific relationships between entities from the data source. Typical relationships are "is-a" for parent-child relationships, "has-a (part-of)" for inclusion relationships, and so on. In this example, the user specifies that the source "factory" is related to the target "region" through the relationship "has region". Additional contextual information can also be added as entity attributes. For example, the user can add type, severity, priority as attributes of the alarm entity.

[0145] In some embodiments, instead of or in addition to being defined by the user, semantic relationships can be automatically determined from primary and foreign key relationships specified in, for example, a relational database management system (RDBMS) tuple or a parent-child node in an XML document. Attributes can be added to provide additional contextual information (for example, for the entity "alarm", the attributes "type", "severity", or "priority" can be added). The resulting semantic model can be validated against a set of predetermined rules. These rules can be defined, for example, by the standard ISA-106 ("Procedural Automation of Continuous Process Operations"). Using ISA-106, the triple "device has unit" can be marked as correct during validation because the standard specifies that the unit has equipment, not the other way around. Public ontologies and vocabularies in dictionary files can be used for validation instead of or in addition to the standards.

[0146] Back to Fig.18 Once the semantic model has been established, the model is validated in block 1808. Semantic validation refers to the process of verifying whether a data element is logically valid. For example, a data element such as equipment has regions is invalid in ISA106 mode because equipment does not have regions (semantic models typically standardize entity relationship types based on industry standards). In some embodiments, the system automatically validates the domain-specific relationships between entities created in block 1806. To this end, the system can maintain standard terms, their definitions, and references to public ontologies and vocabularies in a dictionary file, which can all serve as references for validation. The system also provides the option of manually validating and modifying system-generated relationships by domain experts.

[0147] At block 1810, the validated semantic model is saved to create an ontology. In some embodiments, the semantic model is saved according to one of the following well-known semantic model formats: OWL / XML, JOSO-LD, and RDF / JSON. Alternatively, the semantic model can be persisted by converting the model into a tagged property graph model in GaphSON format. If the semantic model is converted into a W3C ontology, the semantic model knowledge graph can be persisted in an RDF triple store such as AllegroGraph, Stardog, or Amazon Neptune. If the semantic model is converted into an LPG (tagged property graph), the knowledge graph can be persisted in a graph database format such as GraphDB or Neo4j or Azure Cosmos DB.

[0148] At block 1812, the system may use the semantic model to build and deploy a knowledge graph. As described above, a knowledge graph is essentially a graph (i.e., a physical model of the data) that can be used to navigate and visualize relationships between components. The semantic model defines the relationships between source and target entities in the knowledge graph at a conceptual level (e.g., "a region has a unit"). The system extracts instances related to the ontology classes and builds the knowledge graph using the associated relationships and contextual information. Thus, the knowledge graph represents a collection of interconnected entities that enhances the user's ability to search for desired information. The knowledge graph may be represented in a database system using LPG, RDF, or a similar graphical model. An exemplary visualization of a knowledge graph is shown below. Fig.32 shown.

[0149] refer to Fig.32 , a visualization of a knowledge graph 3200 can be seen, where entities and relationships are displayed as multiple labeled attribute nodes, one of which is indicated as 3202. Both the nodes and their relationships are named and can store attributes represented by key / value pairs. Nodes can be labeled so that they are easily grouped together with similar nodes. Edges representing relationships have two main properties: they always have a start node and an end node, and they are directed, making the graph a directed graph. Relationships can also have attributes, which helps to provide additional metadata and semantics for the relationships of the nodes. Legend 3204 provides an indication of what each node 3202 represents based on color encoding, for example, blue represents a label index list, green represents a nested load report, orange represents a cabinet load report, and so on. According to an embodiment of the present disclosure, the knowledge graph 3200 can be selected by selecting the graph view option 3206, and is one of several display screens available from the HMI. The user can select other display screens from the HMI by selecting options, such as an upload data option 3206 for uploading data and a query engine option 3208 for searching data.

[0150] Fig.33An example of a query display screen 3300 available from an HMI according to an embodiment of the present disclosure is shown. In this screen, a user can search for data by typing a natural language query into a search box 3302. For example, a user can ask the HMI to display or otherwise indicate alarms that are typically triggered during a normal plant startup process by entering a query into the search box 3302, "What are the expected alarms during the startup sequence?" In response, the HMI can display the requested alarms in a graphical (nodes with dashed circles) or text form by displaying the identifier of the alarm. The user can also select a node representing the data type desired by the user and drag it into the search box 3302. In the example shown, the user can drag and drop the six returned alarms into the search box 3302. With the help of the knowledge graph, the HMI can quickly highlight all common assets of the six alarms, indicated by nodes without dashed circles, and gray out or suppress all other nodes to the background. The common assets returned may include the control processor where the alarm is processed, the assets shared by these alarms, the I / O modules that process the alarms, etc., indicating the possible root cause of the alarm.

[0151] In some embodiments, users and plant operators can also enter commands to initiate operations. The HMI can interpret commands written in natural language. For example, to instruct the HMI to initiate filling of a component labeled "Tank 01", the command "Fill Tank 01" can be entered into the command bar (not explicitly shown).

[0152] Back to Fig.18 Once the ontology is constructed, at block 1814, the system can provide an asset namespace registration API so that different applications can use the data. Recall the previous content ( Figure 6 ), factory assets are organized into asset registries using namespaces. In some embodiments, a RESTful application program interface (e.g., Azure Active Directory Graph API) can be generated to allow downstream applications to extract information from the knowledge graph.

[0153] At block 1816, the system provides a way for the user to retrieve data based on the process narrative. In some embodiments, the user can retrieve data using natural language questions to search for data. The system can translate the natural language query into, for example, a SPARQL query or a Gremlin query. Downstream applications can include, for example, an HMI through which commands can be issued from the converted process narrative and / or other natural language commands.

[0154] At block 1818, the plant control process and the HMI for the process (i.e., control and HMI engineering) may be designed and developed to allow users to use the system 300 to control the plant and various plant assets. This includes processing plant assets and asset relationships from the asset model to create an HMI asset model. The HMI asset model arranges assets in a hierarchical structure (see Fig.37 ), unlike traditional solutions (i.e., solutions created based on a single P&ID diagram) that are discrete in nature. The HMI can then use the HMI asset model to automatically create a unique, continuous HMI display that resembles a 2D floor plan of the plant level, with symbols for each asset and lines representing processes, electrical connections, data links, etc., running between the symbols. Plant operators can then navigate horizontally and vertically and can zoom in and out (i.e., along the Z axis) of the 2D view to move between different assets and verify their behavior.

[0155] In some embodiments, the HMI can dynamically change the assets displayed to the user based on the user's position change on the HMI screen and the alarms generated at runtime. The HMI assets displayed at runtime can be determined mainly based on two factors: (i) the static weight assigned to the asset based on the asset hierarchy details available in the engineering drawing (e.g., PFD, P&ID, etc.); and (ii) the dynamic weight assigned to each equipment that issues an alarm. Therefore, at runtime, the HMI can display a view including assets with greater weights, or if there are alarms, display assets with alarms. The HMI can also dynamically change the view to include assets that have recently issued an alarm. Alarms on critical equipment will be given a "high" or "high-high" indicator (e.g., exclamation point, red, etc.) and will receive more attention, and the entire alarm chain will receive the highest weight at runtime. In order to design the HMI, it is necessary to establish the design specifications of the HMI.

[0156] Fig.34An exemplary design specification 3400 for an HMI according to an embodiment of the present disclosure is shown. The specification 3400 specifies a hierarchical structure for the HMI that allows the HMI to display certain process control details, including process equipment details and instrument control details, in a hierarchical arrangement. In this example, the HMI is arranged in several hierarchical display levels 3402, 3404, 3406, 3408. Level 1 displays a process area overview, including information such as operational KPIs (key performance indicators), alarm counters, bypass signal counters, trends, overall plant status, etc. The next level, level 2, displays process unit controls, including controllers, alarms, trends, and status. The next level, level 3, displays process unit details, including smaller equipment groups, controllers, alarms, trends, ESD displays, equipment, diagnostics, and status. The next level, level 4, displays process unit support details, including interlocks, details, "first in", procedures, documentation, and help. Other HMI design specifications may include specifications related to graphical displays, such as alarm priority colors (e.g., red, yellow, orange, etc.), and colors indicating status (e.g., out of service, etc.) and functionality (e.g., feedback, trend lines, etc.). The meaning of icons, symbols, and lines (e.g., solid, dashed, dotted, etc.) and engineering units (e.g., metric, imperial, etc.) may also be specified. These HMI specifications, along with the data and information extracted from the various plant engineering data sources as described above, can then be used to build an HMI asset model.

[0157] Fig.35 An exemplary HMI asset model 3500 according to an embodiment of the present disclosure is shown. The HMI asset model, as the name implies, includes an asset model that is processed and used by the HMI to be displayed to the user. For the current purpose, the HMI asset model includes plant assets and relationships (asset hierarchy), an ontology knowledge base, and various plant engineering data sources discussed above. The HMI asset model thus includes PFD 3502 (e.g., high-level equipment, instruments, connections, etc.), PI&D and instrument index 3504 (e.g., equipment, instruments, associations between them, alarms, piping, maintenance override graphics (MOS) and cause and effect (C&E) details, etc.), HMI specifications 3506 (e.g., navigation requirements, alarm aggregation, historicalization, diagnostic details, etc.), control narratives 3508 (e.g., control and monitoring parameters, set points, alarms, ranges, engineering units, etc.), and knowledge captured via engineering displays 3510 (e.g., from maintenance records, previous corrective actions, user experience, etc.). As the content of the various data sources changes and assets are added and / or deleted, the model is automatically updated, making the model a dynamic HMI asset model.

[0158] Once the HMI asset model 3500 (or more precisely, its content) is established, the model can be used to develop an HMI according to an embodiment of the present disclosure. Developing an HMI based on the HMI asset model can include creating templates 3514 for various assets (e.g., instruments, equipment, composites and combinations thereof, etc.). These templates can include, for example, level symbols (e.g., level 2, level 3, etc.), detailed displays of instruments and equipment, instrument properties and actions, animation scripts, control engineering templates, comprehensive details of pumps and valves, etc. Developing an HMI based on the HMI asset model can also include creating other asset models 3516, in which all areas and units are represented as area objects, and each composite and / or complex instrument combination is instantiated according to their templates. This process may involve assigning more weights to some equipment and equipment relative to other equipment and equipment, wherein the former is graphically emphasized. New equipment and equipment introduced can be added, for example, by control narration, and the asset hierarchy can be adjusted accordingly. Equipment and equipment can be grouped based on their association in these asset models.

[0159] In addition to the above, the HMI asset model can also be used to build control applications that control the operation of various factory assets. As mentioned above, machine learning can be used to extract factory assets from unstructured data such as engineering drawings to identify one or more assets. The program can also extract one or more control loops from the engineering drawing based on one or more equipment, instruments, connectors, and lines to build a dynamic HMI asset model. Then, the automatic control generation process can be used to read one or more control loops from the dynamic asset model and generate control logic for the control application. The control logic connects the control application with the actual factory equipment to allow the control application to read or otherwise obtain process values ​​and control the operation of equipment, equipment, instruments, etc. In other words, the HMI handles assets and asset relationships and creates an HMI asset model to arrange assets in a hierarchical structure. The process also links the attributes of each HMI asset to the corresponding control I / O reference so that at runtime, if the operator opens the valve in the control application in the HMI, its control value will be propagated to the corresponding equipment in the factory via the control I / O reference (linked in the HMI asset), thereby opening the valve in the factory.

[0160] For example, an HMI developed based on the HMI asset model can be used by users to monitor various displays and easily find the root cause of an alarm. In some embodiments, such an HMI can be developed using an HMI development platform, such as the system platform 2017 InTouch OMI (Operation Management Interface) available from Wonderware West in Houston, Texas. Other HMI development platforms may certainly be used within the scope of the present disclosure.

[0161] Next reference Fig.36In some embodiments, an HMI according to the present disclosure can process alarms issued by various devices and automatically identify the root cause of the alarm. The HMI determines the root cause of the alarm mainly through two factors: (i) the process / material flow defined in the engineering drawing; and (ii) the control flow defined in the engineering drawing and any control narrative. In the case of process flow, the HMI first finds the device nodes that cause different alarms. For each device node, the HMI retrieves the equipment node corresponding to the device node. For the equipment node, the HMI retrieves any upstream equipment nodes connected to the equipment node, which also has a device that issued the alarm. Repeat this step for adjacent nodes until the HMI finds a connected device that has no device with an alarm connected to it. This creates an alarm chain, enabling the HMI to determine the root cause of the alarm by finding the highest-level equipment in the alarm chain.

[0162] exist Fig.36 In the example of , the HMI displays a two-dimensional plan view of the plant level on screen 3600, which includes symbols 3602, 3604, 3606, and 3608 representing assets and lines 3610, 3612, and 3614 extending between symbols representing processes, electrical connections, data links, etc. The plant operator can then navigate horizontally in the two-dimensional view and zoom in and out to move between different assets and verify their behavior. In this example, material flows from equipment E1215 (pump) to equipment E1214 (boiler), and there is a pressure drop at the inlet of equipment E1214 (boiler). The pressure drop causes the equipment (gauge) at the inlet of equipment E1214 (boiler) to alarm. The problem may be at equipment E1215 or at equipment E1209. However, there is no alarm at equipment E1209, so the HMI determines that the problem is at equipment E1215 (pump). In this example, the production of equipment E1215 (pump) has decreased, so there is a pressure drop at the connected downstream equipment E1214 (boiler). Therefore, although the alarm was issued by the measuring instrument at boiler E1214, in this case, the HMI identifies the root cause of the alarm as pump E1215. The HMI assigns additional weight to this equipment for display purposes and places an alarm light bulb 3616 next to the equipment. At 3618, other alarm bulbs representing other alarms found by the HMI are aggregated.

[0163] Thereafter, the HMI may allow the plant operator to correct the root cause analysis identified by the system. For example, the operator may manually select a different piece of equipment and set that piece of equipment as the root cause of the alarm instead of the root cause identified by the system, and in some embodiments may also manually identify the corresponding alarm chain. These corrections made by the operator may be fed back to the system to dynamically update the machine learning algorithm, thereby making the system more accurate for subsequent root cause analyses of alarms.

[0164] Reference now Fig.37 In some embodiments, an HMI according to the present disclosure can aggregate alarms in a plant and provide a display screen 3700 showing a plant-level view of the aggregated alarms. In some embodiments, the system can aggregate alarms by obtaining a list of all equipment that generates alarms and identifying the first equipment from the list. The system adds the equipment to a new alarm cluster and checks whether the adjacent equipment is also generating alarms. If so, the system adds the adjacent equipment to the current alarm cluster. Repeat the process for the next equipment in the list until all equipment and adjacent equipment have been processed. In the example shown, the alarms aggregated at the plant level are depicted as a series of light bulbs 3702 in a specified area (e.g., upper left corner) of the screen 3700, but different locations or different icons or graphical symbols can certainly be used for alarms. Each of the light bulbs 3702 corresponds to a corresponding component (e.g., equipment, equipment, etc.) that issues an alarm in the plant. It can be seen that there are five components 3704a, b, c, d, e that issue an alarm, each with a light bulb icon 3706 to indicate the alarm status of the component.

[0165] Selecting (e.g., by tapping, double-clicking, etc.) one of the components 3704a-e causes the HMI to display information about that component, including which downstream components may be potential root causes of the alarm. In some embodiments, the system can determine the potential root cause of the alarm by identifying the alarm at the highest equipment level, and then finding the source equipment connected to that equipment that also has the alarm. The system then drills down to the source equipment and finds the connected subsequent equipment that also has the alarm. This process is repeated until the lowest connected equipment for the alarm is found.

[0166] Fig.38An embodiment of an HMI that can dynamically zoom in (and out) on an asset based on the user / viewer's position on an asset hierarchy screen is illustrated. The viewer's position on the hierarchy screen is primarily determined by two factors: (1) the ground position as determined by screen coordinates (e.g., x and y coordinates), and (2) the viewer's height above the ground. Based on the viewer's position on the screen, relevant information surrounding the asset model is provided for viewing. For example, consider when the user is in area 1 relative to the HMI screen, and an alarm condition exists at equipment 5, equipment 3, device 2, and device 4. In this example, the HMI screen will display equipment 7, 1, 2, 4, 5, and 6 and any relevant information about the equipment, but will not display equipment 8, 9, or 3, nor any equipment connected to equipment 3. Assume that the user now zooms in so that the user's position is at unit 3. In this example, the HMI screen will display equipment 7, 1, 2, 4, 5, and 6 as before, but will also display equipment 8, 9, or 3. Once the user moves further down to equipment 5, the HMI screen will display equipment 1, 2, 3, 4, and 5 and any relevant information about these equipment. The HMI screen automatically removes any equipment above the user's level.

[0167] Fig.39 An exemplary HMI display screen 3900 is shown that can dynamically zoom in (and out) on an asset based on the location of the user / viewer, in accordance with an embodiment of the present disclosure. Based on the zoom level determined by the user's current location and currently active alarms in the plant, the dynamic HMI display screen 3900 displays relevant information surrounding the asset model. Fig.39 In the example on the left, HMI display 3900 displays connections between assets 3902 and 3904. In the middle example, the user has zoomed in, so HMI display 3900 remaps the connections to display an additional asset 3906 on the screen. In the example on the right, the user has zoomed in further, so HMI display 3900 again remaps the connections to display yet another additional asset 3908 on the screen.

[0168] Reference now Fig.40A and Fig.40B In addition to the plant-level alarm aggregation discussed above, the HMI according to an embodiment of the present disclosure can also perform alarm aggregation at the area level. Fig.40AAn exemplary embodiment of such an HMI is illustrated, in which the HMI displays zone-level alarm aggregation using a two-dimensional view of plant assets. Starting with the example on the left, solid lines such as line 4002 represent horizontal connectivity within a plane, and dashed lines such as line 4004 represent vertical connectivity across multiple planes (i.e., a two-dimensional view). It can be seen that zone-level assets A1 and A2 have alarm conditions indicated by high alarm icons and severe alarm icons, respectively. When the user selects A2 (e.g., by tapping, clicking, etc.), the HMI takes the user to the example on the right, in which the user finds a severe alarm at asset U3 and a high alarm at asset U2.

[0169] If the user selects U2, the HMI will take the user to Fig.40B Example on the left, where the user discovers a critical alarm on equipment U3E2. From here, the user will be able to visually view the source of the critical alarm at U2E1, which cascades through U3E1 and then U3E2. Assuming the user selects U2E1, the HMI takes the user to Fig.40B Example on the right, where the user will see device-level assets, including the problem source D1 that caused the alert. Selecting D1 will provide the user with information about that device, any trends seen on that device, any support and guidance available for that device, and any knowledge acquired about that device.

[0170] Therefore, as described herein, embodiments of the present disclosure provide systems and methods for controlling industrial process automation and control systems. The methods and systems automatically and by using machine learning (ML) models and algorithms, extract plant assets from engineering drawings and other plant engineering data sources, establish asset relationships to create a plant asset registry and build an asset hierarchy from the plant assets, generate an ontology knowledge base from the plant asset hierarchy, and provide an HMI for controlling industrial processes based on the plant asset hierarchy and the ontology knowledge base.

[0171] These embodiments of the present disclosure may include a special-purpose computer including various computer hardware, as described in more detail below. Embodiments within the scope of the present disclosure also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media may be any available media accessible by a special-purpose computer, and include computer storage media and communication media. As an example and not limitation, computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media are non-temporary, including but not limited to random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, solid-state drive (SSD), cassette, tape, disk storage or other magnetic storage device, or any other medium that can be used to carry or store desired non-temporary information in the form of computer-executable instructions or data structures and can be accessed by a computer. When information is transmitted or provided to a computer over a network or another communications connection (hardwired, wireless, or a combination of hardwired or wireless), the computer properly views the connection as a computer-readable medium. Therefore, any such connection is properly referred to as a computer-readable medium. Combinations of the above should also be included within the scope of computer-readable media. Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions.

[0172] The following discussion is intended to provide a brief summary description of a suitable computing environment in which aspects of the present disclosure may be implemented. Although not required, aspects of the present disclosure will be described in the general context of computer-executable instructions such as program modules executed by computers in a network environment. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code devices for performing the steps of the methods disclosed herein. Specific sequences of such executable instructions or associated data structures represent examples of corresponding actions for implementing the functions described in these steps.

[0173] Those skilled in the art will appreciate that aspects of the present disclosure can be practiced in a network computing environment with various types of computer system configurations, including personal computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, etc. Aspects of the present disclosure can also be practiced in a distributed computing environment, where tasks are performed by local and remote processing devices linked (by hardwired links, wireless links, or a combination of hardwired or wireless links) through a communications network. In a distributed computing environment, program modules can be located in local and remote memory storage devices.

[0174] An exemplary system for implementing various aspects of the present disclosure includes a special-purpose computing device in the form of a conventional computer, including a processing unit, a system memory, and a system bus that couples various system components including the system memory to the processing unit. The system bus can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes computer storage media, including non-volatile and volatile memory types. A basic input / output system (BIOS) can be stored in a read-only memory, which contains basic routines such as helping to transfer information between elements within the computer during startup. In addition, the computer can include any device that can wirelessly receive or send an IP address from the Internet (e.g., a computer, a laptop, a tablet, a PDA, a cell phone, a mobile phone, a smart TV, etc.).

[0175] The computer can also include a magnetic hard disk drive for reading and writing a magnetic hard disk, a disk drive for reading and writing a removable disk, and an optical drive for reading and writing a removable optical disk such as a CD-ROM or other optical media. The magnetic hard disk drive, the disk drive, and the optical drive are connected to the system bus via a hard disk drive interface, a disk drive interface, and an optical drive interface, respectively. The drive and its associated computer-readable medium provide the computer with non-volatile storage of computer-executable instructions, data structures, program modules, and other data. Although the exemplary environment described here employs a magnetic hard disk, a removable disk, and a removable optical disk, other types of computer-readable media for storing data can also be used, including cassettes, flash memory cards, digital video disks, Bernoulli cassettes, RAMs, ROMs, SSDs, etc.

[0176] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0177] The program code device comprising one or more program modules can be stored on a hard disk, a disk, an optical disk, a read-only memory and / or a random access memory, including an operating system, one or more application programs, other program modules and program data. A user can input commands and information to a computer by means of a keyboard, a fixed-point device or other input devices, such as a microphone, a joystick, a game controller, a dish, a scanner, etc. These and other input devices are usually connected to a processing unit by a serial port interface coupled to a system bus. Alternatively, the input device can be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). A monitor or other display device is also connected to the system bus by an interface (such as a video adapter). In addition to a monitor, a personal computer usually also includes other peripheral output devices (not shown), such as a speaker and a printer.

[0178] One or more aspects of the present disclosure may be embodied in computer executable instructions (i.e., software), routines, or functions stored in system memory or non-volatile memory as applications, program modules, and / or program data. Alternatively, the software may be stored remotely, such as on a remote computer with a remote application. Typically, program modules include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types when executed by a processor in a computer or other device. Computer executable instructions may be stored on one or more tangible, non-temporary computer-readable media (e.g., hard disks, optical disks, removable storage media, solid-state memory, RAM, etc.) and executed by one or more processors or other devices. As will be appreciated by those skilled in the art, the functions of the program modules may be combined or distributed as needed in various embodiments. In addition, the functions may be embodied in whole or in part in firmware or hardware equivalents, such as integrated circuits, application specific integrated circuits, field programmable gate arrays, etc.

[0179] Computers can be operated in a networked environment using logical connections to one or more remote computers. Each remote computer can be another personal computer, tablet computer, PDA, server, router, network PC, peer device or other public network node, and generally includes many or all elements described above relative to the computer. Logical connections include local area networks (LANs) and wide area networks (WANs), which are given as examples and not limitations herein. Such network environments are common in office-wide or enterprise-wide computer networks, intranets and the Internet.

[0180] When used in a LAN networking environment, the computer is connected to a local network through a network interface or adapter. When used in a WAN networking environment, the computer may include a modem, a wireless link, or other means for establishing communications over a wide area network (such as the Internet). The modem may be built-in or external, connected to the system bus via a serial port interface. In a networked environment, program modules or portions thereof described relative to the computer may be stored in a remote memory storage device. It should be understood that the network connections shown are exemplary, and other means of establishing communications over a wide area network may be used.

[0181] Preferably, the computer executable instructions are stored in a memory such as a hard drive and executed by the computer. Advantageously, the computer processor has the capability to perform all operations (eg, execute the computer executable instructions) in real time.

[0182] Unless otherwise noted, the execution or execution order of the operations in the embodiments of the present disclosure shown and described herein is not essential. That is, unless otherwise noted, the operations may be performed in any order, and the embodiments of the present disclosure may include more or less operations than the operations disclosed herein. For example, it is contemplated that it is within the scope of the various aspects of the present disclosure to perform or implement a particular operation before, simultaneously with, or after another operation.

[0183] Embodiments of the present disclosure may be implemented with computer executable instructions. Computer executable instructions may be organized into one or more computer executable components or modules. Aspects of the present disclosure may be implemented with such components or modules of any number and organization. For example, aspects of the present disclosure are not limited to specific computer executable instructions or specific components or modules shown in the accompanying drawings and described herein. Other embodiments of the present disclosure may include different computer executable instructions or components having more or less functionality than shown and described herein.

[0184] When introducing elements of aspects of the disclosure or embodiments thereof, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements besides the listed elements.

[0185] Having described aspects of the disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of aspects of the disclosure as defined in the appended claims. As various changes may be made to the above-described structures, products, and methods without departing from the scope of aspects of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

Claims

1. A control system for an industrial plant, comprising: one or more processors; and a storage unit, communicatively coupled to the one or more processors, and storing thereon processor-executable instructions, which, when executed by the one or more processors, cause the control system to: executing a process of inputting data from a plurality of engineering data sources of said industrial plant; performing a process of extracting assets of the industrial plant from the data using machine learning to identify the assets; Performs the process of establishing asset relationships to create a plant asset registry and build an asset hierarchy from said assets; executing a process of generating an ontology knowledge base from the asset hierarchy; performing a process of providing a human-machine interface (HMI) asset model for controlling an industrial process based on the asset hierarchy and the ontology knowledge base, the HMI asset model arranging the assets in the asset hierarchy; executing a process of generating an HMI display from the HMI asset model, the HMI display showing symbols representing assets of the industrial plant and lines representing processes, electrical connections, piping connections, pilot signal connections, and / or data links between the assets; and A process is performed that displays the HMI display to a user and allows the user to navigate vertically and horizontally along the display, wherein the HMI display dynamically changes which assets are displayed based on changes in the user's position on the HMI display.

2. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to perform a process of identifying all alarms currently issued for the industrial plant and assets corresponding to the alarms and displaying the alarms on the HMI display.

3. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to execute a process for displaying the HMI display at a plant level, the plant level display identifying all currently issued alarms for the industrial plant and the assets corresponding to the alarms.

4. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to perform a process for determining and displaying on the HMI display an asset that is a potential root cause of an alarm when the user selects an asset corresponding to one of the alarms.

5. The control system according to claim 4, wherein: The control system determines the asset that is the potential root cause by finding all assets that are connected to each other and that also have an alarm.

6. The control system according to claim 5, wherein: The processor-executable instructions further cause the control system to perform a process that allows a user to manually correct a root cause of an alarm on the HMI display and provides the root cause from the user as feedback to the control system.

7. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to perform a process for automatically zooming in or out of assets displayed on the HMI display based on a change in the user's position on the HMI display.

8. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to perform a process of displaying the asset in the HMI display as a two-dimensional view, wherein the HMI display allows a user to navigate vertically and horizontally within the two-dimensional view.

9. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to execute a process for dynamically changing an asset displayed on an HMI display based on an alarm occurring on the asset.

10. The control system according to claim 1, wherein: The processor executable instructions also cause the control system to perform a process for displaying an asset on the HMI display at runtime based on one of: a static weight assigned to the asset based on its position in an asset hierarchy, and a dynamic weight assigned to the asset based on an alarm issued at the asset.

11. A method for controlling an industrial plant, comprising: inputting data from a plurality of engineering data sources of the industrial plant; extracting assets of the industrial plant from the data to identify the assets using machine learning; Establishing asset relationships to create a plant asset registry and constructing an asset hierarchy from said assets; generating an ontology knowledge base from the asset hierarchy; Providing a human-machine interface HMI asset model for controlling an industrial process based on the asset hierarchy and the ontology knowledge base, wherein the HMI asset model arranges the assets in the asset hierarchy; generating an HMI display from the HMI asset model, the HMI display showing symbols representing assets of the industrial plant and lines representing processes, electrical connections, piping connections, pilot signal connections, and / or data links between the assets; and The HMI display is displayed to a user and allows the user to navigate vertically and horizontally along the display, wherein the HMI display dynamically changes which assets are displayed based on changes in the user's position on the HMI display.

12. The method of claim 11, further comprising identifying all alarms currently issued for the industrial plant and assets corresponding to the alarms and displaying the alarms on the HMI display.

13. The method of claim 11, further comprising displaying the HMI display at a plant level, the plant level display identifying all currently issued alarms for the industrial plant and assets corresponding to the alarms.

14. The method of claim 11, further comprising, when a user selects an asset corresponding to one of the alarms, determining and displaying on the HMI display an asset that is a potential root cause of the alarm.

15. The method according to claim 14, wherein: Identifying assets that are potential root causes includes finding all assets that are connected to each other and that also have alerts. 16 . The method of claim 11 , further comprising allowing a user to manually correct a root cause of an alarm on the HMI display and providing the root cause from the user as feedback to a control system. 17 . The method of claim 11 , further comprising automatically zooming in or out on assets displayed on the HMI display based on a change in the user's position on the HMI display.

18. The method of claim 11, further comprising displaying the assets in the HMI display as a two-dimensional view and allowing a user to navigate vertically and horizontally within the two-dimensional view.

19. The method of claim 11, further comprising dynamically changing an asset displayed on the HMI display based on an alarm occurring on the asset.

20. The method of claim 11 further comprising displaying an asset on the HMI display at runtime based on one of: a static weight assigned to the asset based on a position of the asset in an asset hierarchy, and a dynamic weight assigned to the asset based on an alarm issued at the asset.

21. A computer readable medium storing computer readable instructions for causing one or more processors to execute the method according to any one of claims 11 to 20.

Images