Automatically extract asset data from engineering data sources

Abstract

A system and method for controlling an industrial plant includes: inputting an engineering drawing of a unit of the industrial plant, the engineering drawing including symbols representing assets of the industrial plant; using machine learning to extract one or more assets from the engineering drawing to identify the one or more assets, the one or more assets including devices, instruments, connectors, and lines that relate the devices, instruments, and connectors to each other; using machine learning to determine one or more relationships between the devices, instruments, connectors, and lines to identify the one or more relationships; and creating a flow chart based on the devices, instruments, connectors, and lines and the relationships between the devices, instruments, connectors, and lines.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Provisional Application No. 62 / 823,469, filed on March 25, 2019, entitled “Systems and Methods for Performing Industrial Plant Diagnostics and Operations”; U.S. Provisional Application No. 62 / 842,929, filed on May 3, 2019, entitled “Systems and Methods for Performing Industrial Plant Diagnostics”; and U.S. Provisional Application No. 62 / 823,377, filed on March 25, 2019, entitled “Systems and Methods for Detecting and Predicting Faults in an Industrial Process Automation System”, and the above-mentioned provisional applications are incorporated herein by reference. The subject matter of this application is also related to U.S. Non-Provisional Application No. 16 / 021,867, filed on June 28, 2018, entitled “Machine Learning Analysis of Piping and Instrumentation Diagrams”, and is incorporated herein by reference. Technical Field

[0003] Aspects of the present disclosure generally relate to industrial process automation and control systems. More specifically, aspects of the present disclosure relate to systems and methods for: extracting plant assets from engineering drawings and other engineering data sources; creating a plant asset hierarchy based on the plant assets; building an ontological knowledge base based on the plant asset hierarchy and asset association relationships with other assets; and providing a human-machine interface ("HMI") for controlling industrial process automation and control systems based on the plant asset hierarchy and the ontological knowledge base. Background Art

[0004] A plant operator is tasked with ensuring that the plant is operating properly. This requires, for example, monitoring and resolving alarms triggered by plant components (e.g., pumps, valves, tanks, sensors), performing actions on plant components (e.g., shutting down components and starting up components), and generally overseeing the correct operation of the plant. To perform these tasks, a plant operator may use an HMI that provides the plant operator with a visual representation of the plant components and the data collected from the plant components. The HMI may allow the plant operator to interact with the plant components (e.g., perform actions such as shutting down components).

[0005] An HMI can be designed for a specific plant to capture components within the plant, display data used to run the plant, and provide an interface for initiating actions that plant operators may need. Designing an HMI can be expensive and time-consuming, and may require expertise in plant and process engineering. Today, plant HMIs are created using a process-centric approach that focuses on process definitions. On the other hand, plant operators have an asset-centric view of the plant that focuses on automation control definitions (e.g., plant, area, unit, equipment, device, set points for specific equipment, etc.). As a result, plant operators may find current plant HMIs unintuitive and difficult to use.

[0006] Therefore, there is a need for improvements in the area of ​​industrial plant commissioning and operation. Summary of the invention

[0007] Embodiments of the present disclosure provide systems and methods for controlling industrial process automation and control systems. The method and system automatically and by using machine learning (ML) models and algorithms, extract plant assets from engineering drawings and other plant engineering data sources; establish asset relationships to create a plant asset registry and build an asset hierarchy based on plant assets; generate an ontology knowledge base based on the plant asset hierarchy; and provide an HMI for controlling industrial processes based on the plant asset hierarchy and the ontology knowledge base.

[0008] In general, in one aspect, embodiments of the present disclosure relate to control systems for industrial plants and corresponding methods thereof. The control system (and method thereof) particularly includes one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions, which, when executed by the one or more processors, cause the control system to: input an engineering drawing of a unit of an industrial plant, the engineering drawing including symbols representing assets of the industrial plant; and extract one or more assets from the engineering drawing using machine learning to identify the one or more assets. The one or more assets include equipment, instruments, connectors, and lines that relate the equipment, instruments, and connectors to each other. The processor-executable instructions also cause the control system to use machine learning to determine one or more relationships between the equipment, instruments, connectors, and lines to identify the one or more relationships. The processor-executable instructions also cause the control system to create a flow graph based on the equipment, instruments, connectors, and lines and the relationships between the equipment, instruments, connectors, and lines.

[0009] According to any one or more of the foregoing embodiments, the process of extracting one or more assets from the engineering drawing includes a geometry-based extraction process and a machine learning-based classification process. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to extract one or more control loops from the engineering drawing based on one or more of the equipment, instruments, connectors, and lines. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to: extract a unit identifier from the engineering drawing, wherein the unit identifier uniquely identifies a specific unit from a plurality of units in an industrial plant; and / or extract a drawing number and a revision number from the engineering drawing, wherein the drawing number uniquely identifies the engineering drawing from a plurality of engineering drawings in an industrial plant, and the revision number indicates the revision of the engineering drawing, and the revision number is incremented whenever there is a change in the engineering drawing. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to use the unit identifier from the engineering drawing to assign a unique identifier to each equipment, instrument, connector, and line of the engineering drawing. According to any one or more of the foregoing embodiments, the process of determining one or more relationships between equipment, instruments, connectors, and lines includes: a process of generating a line-line diagram, a process of generating an equipment-line diagram, a process of generating a connector-line diagram, and a process of generating an instrument-line diagram. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to: create a flowchart of an engineering drawing by executing a process of merging a line-line diagram, a device-line diagram, a connector-line diagram, and an instrument-line diagram with each other; and / or merge flowcharts from multiple engineering drawings of an industrial plant to create a flowchart of an industrial plant. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to display the equipment, instruments, connectors, and lines and the relationships between the equipment, instruments, connectors, and lines in an asset hierarchy.

[0010] In general, on the other hand, embodiments of the present disclosure relate to control systems for industrial plants and corresponding methods thereof. The control system (and method thereof) particularly includes one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions, which, when executed by one or more processors, cause the control system to: input data from multiple engineering data sources of an industrial plant, the data including structured and unstructured data, and extract one or more domain entities from the structured and unstructured data. The processor-executable instructions also cause the control system to extract instances of one or more domain entities from the structured and unstructured data, and receive a semantic model constructed based on one or more domain entities and instances of one or more domain entities. The processor-executable instructions further cause the control system to create and store a knowledge graph based on the semantic model, and extract information from the knowledge graph to construct a machine interface (HMI) display and control application for an industrial plant.

[0011] According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to: extract one or more assets of the industrial plant from unstructured data using machine learning to identify the one or more assets; and / or store the one or more assets of the industrial plant in an asset hierarchy using a namespace that uniquely identifies each asset. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to provide an asset registry application programming interface (API) to allow users and external systems to access and use the asset registry. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to input data from multiple engineering data sources by inputting one or more of the following: alarm data, cabinet loading reports, control databases, cross wiring reports, field wiring indexes, historical data, instrument indexes, asset hierarchies, nested loading reports, plant asset indexes, process control narratives, and HMI specifications. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to: display the knowledge base as a plurality of interconnected labeled attribute nodes, each node representing a domain entity; and / or display the properties and relationships of each node when the node is selected by the user. According to any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to allow a user to search the knowledge base using natural language queries; display nodes responsive to the search and assets shared by those nodes; and / or display a legend for the nodes on a color-coded basis.

[0012] In general, in another aspect, embodiments of the present disclosure relate to control systems for industrial plants and corresponding methods thereof. The control system (and method thereof) includes, in particular, one or more processors and a storage unit communicatively coupled to the one or more processors. The storage unit stores processor-executable instructions, which, when executed by the one or more processors, cause the control system to input data from multiple engineering data sources of the industrial plant and extract one or more assets of the industrial plant from the data using machine learning to identify the one or more assets. The processor-executable instructions also cause the control system to create an HMI (human-machine interface, HMI) asset model based on the assets, the HMI asset model arranging the assets in a hierarchical structure; and to generate an HMI display based on the HMI asset model, the HMI display showing symbols representing one or more assets of the industrial plant and lines representing processes, connections, and data links between the assets. The processor-executable instructions also cause the control system to display the HMI display to a user and allow the user to navigate vertically and horizontally along the display, wherein the HMI display dynamically changes which assets are displayed based on changes in the user's position on the HMI display.

[0013] According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to identify all currently issued alarms for the industrial plant and the assets corresponding to the alarms, and display the alarms on the HMI display; and / or display a plant-level HMI display that displays all currently issued alarms and the assets corresponding to the alarms identified for the industrial plant. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to determine and display on the HMI display an asset that is a potential root cause of the alarm when a user selects an asset corresponding to one of the alarms, wherein the control system determines the asset as a potential root cause by finding all assets connected to each other that also have alarms; and / or allows the user to manually correct the root cause of the alarm on the HMI display, and provides the root cause from the user as feedback to the control system. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to automatically zoom in or out of the assets displayed on the HMI display based on changes in the user's position on the HMI screen. According to any one or more of the foregoing embodiments, the processor executable instructions further cause the control system to display the assets displayed by the HMI as a two-dimensional view, wherein the HMI display allows the user to navigate vertically and horizontally within the two-dimensional view. In accordance with any one or more of the foregoing embodiments, the processor executable instructions also cause the control system to dynamically change the assets displayed on the HMI display based on an alarm occurring on the asset; and / or display the asset on the HMI display at runtime based on one of: a static weight assigned to the asset based on its position in the asset hierarchy, and a dynamic weight assigned to the asset based on an alarm being issued at the asset.

[0014] In general, in yet another aspect, embodiments of the present disclosure relate to a computer-readable medium storing computer-readable instructions for causing one or more processors to perform a method according to any one or more of the foregoing embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A more detailed description of the disclosure, briefly summarized above, can be obtained by reference to various embodiments, some of which are shown in the accompanying drawings. Although the drawings illustrate selected embodiments of the disclosure, these drawings are not to be considered limiting of its scope, as the disclosure may admit to other equally effective embodiments.

[0016] Figure 1 An exemplary plant asset hierarchy structure according to an embodiment of the present disclosure is shown;

[0017] Figure 2 An exemplary engineering diagram that can be used to construct a plant asset hierarchy according to an embodiment of the present disclosure is shown;

[0018] Figure 3An exemplary industrial plant control system that may be used to process according to an embodiment of the present disclosure is shown;

[0019] Figure 4 An exemplary intelligent processing module that may be used with an industrial plant control system according to an embodiment of the present disclosure is shown;

[0020] Figure 5 An exemplary HMI application that may be used with an industrial plant control system according to an embodiment of the present disclosure is shown;

[0021] Figure 6 An exemplary ML-based asset extraction and relationship building according to an embodiment of the present disclosure is shown;

[0022] Figure 7A-7B shows an exemplary arrow extraction according to an embodiment of the present disclosure;

[0023] Figure 8 shows an exemplary pump extraction according to an embodiment of the present disclosure;

[0024] Fig. 9 and Figure 9A-9C An exemplary connector extraction according to an embodiment of the present disclosure is shown;

[0025] Figure 10A-10B shows an exemplary line-line diagram according to an embodiment of the present disclosure;

[0026] Fig.11 shows an exemplary device extraction according to an embodiment of the present disclosure;

[0027] Figure 12A-12B An exemplary device-circuit diagram according to an embodiment of the present disclosure is shown;

[0028] Figure 13A-13B shows an exemplary connector-circuit diagram according to an embodiment of the present disclosure;

[0029] Figure 14A-14B shows an exemplary instrument-circuit diagram according to an embodiment of the present disclosure;

[0030] Figures 15A-16B A PI&D flow chart according to an embodiment of the present disclosure is shown;

[0031] Figure 16A-Figure 15B shows an exemplary circuit-circuit diagram with a device removed according to an embodiment of the present disclosure;

[0032] Fig.17 An exemplary control loop according to an embodiment of the present disclosure is shown;

[0033] Fig.18 An exemplary method for constructing a plant engineering domain ontology according to an embodiment of the present disclosure is shown;

[0034] Fig.19 shows exemplary alarm data according to an embodiment of the present disclosure;

[0035] Fig. 20 An exemplary cabinet loading report according to an embodiment of the present disclosure is shown;

[0036] Fig.21 An exemplary control database according to an embodiment of the present disclosure is shown;

[0037] Fig. 22 An exemplary cross-wiring report according to an embodiment of the present disclosure is shown;

[0038] Fig.23 An exemplary field wiring index according to an embodiment of the present disclosure is shown;

[0039] Fig.24 shows exemplary historical data according to an embodiment of the present disclosure;

[0040] Fig.25 shows an exemplary instrument index according to an embodiment of the present disclosure;

[0041] Fig.26 shows an exemplary asset hierarchy structure according to an embodiment of the present disclosure;

[0042] Fig. 27 An exemplary nested loading report according to an embodiment of the present disclosure is shown;

[0043] Fig.28 An exemplary plant asset index according to an embodiment of the present disclosure is shown;

[0044] Fig.29 An exemplary process control narrative according to an embodiment of the present disclosure is shown;

[0045] Fig.30 An exemplary table format for data extraction according to an embodiment of the present disclosure is shown;

[0046] Fig.31 An exemplary user interface for building a semantic model using entity types according to an embodiment of the present disclosure is shown;

[0047] Fig.32 An exemplary HMI display screen showing a knowledge graph according to an embodiment of the present disclosure is shown;

[0048] Fig.33 An exemplary HMI display screen showing user query results according to an embodiment of the present disclosure is shown;

[0049] Fig.34An exemplary HMI design specification according to an embodiment of the present disclosure is shown;

[0050] Fig.35 An exemplary input for constructing an HMI model according to an embodiment of the present disclosure is shown;

[0051] Fig.36 shows an exemplary HMI screen illustrating alarm root cause determination according to an embodiment of the present disclosure;

[0052] Fig.37 An exemplary HMI screen is shown showing plant level alarm aggregation according to an embodiment of the present disclosure;

[0053] Fig.38 shows an exemplary dynamic scaling according to the user's position on the HMI screen according to an embodiment of the present disclosure;

[0054] Fig.39 shows an exemplary HMI screen showing dynamic scaling according to an embodiment of the present disclosure; and

[0055] Figure 40A-40B Exemplary plant-level alarm aggregation by an HMI according to an embodiment of the present disclosure is shown.

[0056] Wherever possible, identical reference numerals have been used to designate elements that are common to the figures. However, elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation. DETAILED DESCRIPTION

[0057] This description and the accompanying drawings illustrate exemplary embodiments of the present disclosure and should not be considered as restrictive, and the claims define the scope of the present disclosure, including equivalents. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the scope (including equivalents) of this specification and claims. In some cases, known structures and techniques are not shown or described in detail to avoid confusing the present disclosure. In addition, as long as feasible, the elements and related aspects thereof described in detail with reference to an embodiment may be included in other embodiments that are not specifically shown or described. For example, if an element is described in detail with reference to an embodiment, and the element is not described with reference to a second embodiment, the element can still be declared to be included in the second embodiment.

[0058] Reference now Figure 1, shows an example of an industrial plant asset hierarchy 100 that can be created according to an embodiment of the present disclosure. The term "plant asset" used herein generally refers to any plant equipment, instrumentation, groups thereof, devices associated therewith, etc. that are commonly used in industrial plants. At the bottom of the asset hierarchy 100 are instruments (O) 102, such as sensors, monitors, actuators, etc. Directly above the instruments 102 are devices (D) 104, such as vessels, heaters, exchangers, pumps, motors, mixers, etc. One or more instruments 102 can be connected to one or more devices 104 to form a closed-loop or open-loop control loop 106, which performs some sub-processes in the plant process, such as mixing, heating, boiling, etc. Several types of equipment 104 and instruments 102 can be combined to form a unit (U) 108, which serves some functions in the plant process, such as a gas synthesizer unit. Multiple such units 108 form an area (A) 110, and multiple areas 110 form a plant site (S) 112, and multiple sites 112 form an enterprise (N) 114.

[0059] Additionally, in general, while each node in the hierarchy 100 can have child nodes and parent nodes, each child node can typically only have one parent node. However, assets such as pipes and wires can be considered equipment, so that a pipe carrying material from / to two assets can be logically divided into two pipes. A device or instrument can only be connected to one equipment. This hierarchy 100 arranges and organizes plant assets in a logical manner and is easier for the plant control system and plant operators to understand and use.

[0060] Figure 2An exemplary engineering drawing or engineering diagram 200 that can be used to build an asset hierarchy 100 (or a portion thereof) according to an embodiment of the present disclosure is shown. The engineering drawing 200 can be any diagram, such as a process flow diagram (PFD) or a piping and instrument diagram (P&ID), which graphically depicts how various instruments 102 (e.g., sensors) and equipment 104 (e.g., containers) in a plant are connected to each other. The example diagram 200 in this example is a P&ID, each of which displays a plurality of symbols representing devices or instruments, equipment, and other components that are typically required to implement a unit 108. Connectors 202 located throughout the diagram 200 specify how the various PI&Ds (i.e., units) are connected together to form areas, sites, etc. Each diagram 200 also typically has an information box 204, typically in the lower right corner, which displays a drawing identifier 206, a unit identifier, and other information that helps uniquely identify the PI&D (e.g., revision number). Other well-known parts of PI&D not explicitly mentioned here include tag numbers, circuits (eg, process, electrical, data link, etc.), loops, and the like.

[0061] Figure 3 An exemplary industrial plant control system 300 that can be used to process engineering drawings 200 and other plant engineering data sources according to an embodiment of the present disclosure is shown. As can be seen, the exemplary system 300 includes one or more processors 302, internal input and / or output ("I / O") interfaces 304, and memory 306, all of which are communicatively coupled and / or electrically connected to each other. These components allow the system 300 to: extract various instruments 102, devices 104, control loops 106, units 108, areas 110, and sites 112 from engineering drawings 200, lines (e.g., process, electrical, data links, etc.); establish relationships between assets to build a plant asset hierarchy 100; generate an ontology knowledge base based on the plant asset hierarchy 100; and provide an HMI for controlling the plant ontology knowledge base; etc.

[0062] The system 300 can be communicatively coupled and / or electrically connected to an external I / O component 308. The external I / O component 308 allows the system 300 to interact and communicate with users and external systems. Communication can be accomplished by one or more communication networks (not explicitly shown) connected to the I / O component 308. The communication network may include a wide area network (WAN) and / or a local area network (LAN), wherein the local area network may be connected to other telecommunication networks, including other WANs, LANs and / or a part of the Internet or an intranet. The communication network may be any telecommunication network that facilitates data exchange, such as those networks that operate according to IEEE802.3 (e.g., Ethernet) and / or IEEE 802.11 (e.g., Wi-Fi) protocols. In another embodiment, the communication network is any medium that allows data to be physically transmitted through a serial or parallel communication channel (e.g., copper lines, optical fibers, computer buses, wireless communication channels, etc.).

[0063] The operation of the various components of the system 300 is well known in the art and is therefore only briefly mentioned here. The processor 302 may be adapted to execute processor executable instructions stored in the memory 306. The memory 306 may be adapted to provide the processor executable instructions to the processor 302 upon request. The instructions stored on the memory 306 include an industrial plant control application 310. The industrial plant control application 310 may include a plurality of functional modules that work together to extract assets, establish asset relationships, build asset hierarchies, generate knowledge bases, and provide HMIs, etc. These functional modules may include, for example, an image converter 312, a filtering algorithm 314, a symbol extractor module 316, a tag preprocessing algorithm 318, a tag extractor module 320, a rule engine 322, a line extractor module 324, an intelligent processing module 326, and an HMI application 328.

[0064] The image converter 312 is configured to convert the diagram 200 into an image format. In some embodiments, the image converter 312 obtains the diagram 200 in a portable document file (PDF) or other electronic data format and converts the diagram into another image format, such as a portable network graphics (PNG), a joint photographic experts group (JPEG), a graphics interchange format (GIF), etc. In some embodiments, the image converter 312 creates two image files: one for display and one for calculation (e.g., by the filtering algorithm 314, the symbol extraction 316, the label pre-processing 318, the label extractor 320, the rule engine 322, the line extractor 324, and the intelligent processing 326).

[0065] The filtering algorithm 314 is configured to process the computational image to obtain the approximate size of the symbols therein. Exemplary symbols include, but are not limited to, those that conform to the International Society of Automation (ISA) standards for devices, control / display elements, programmable logic controllers (PLCs), valves, pumps, and the like. In some embodiments, the symbol includes an identification letter (e.g., "FIC") and a tag number (e.g., "123"). Obtaining the approximate size of the symbol helps normalize the symbol for machine learning purposes (via intelligent processing 326), as discussed later herein (e.g., to avoid creating training data for symbols of different sizes).

[0066] The symbol extractor 316 is configured to detect symbols extracted from an image. In some embodiments, the symbol extractor 316 applies an image processing algorithm to identify possible areas of symbols in the image and then detects the symbol type and location in the image through coarse symbol recognition techniques. The symbol extractor 316 maintains a running count of newly detected symbols in order to track the number of detected symbols and determine whether any new symbols are detected during a given execution cycle.

[0067] Label preprocessing 318 is configured to remove the symbol line from the symbol detected in the computational image, leaving only the label component. In some embodiments, this involves centering the symbol and then removing the symbol line from the symbol, leaving only the label component. Connected pixels are clustered, thereby removing any pixels that are smaller than the standard text size and larger than the standard text size. Each pixel cluster is assigned a bounding box that defines the boundaries around the cluster for processing purposes. Label preprocessing 210 then finds bounding boxes of the same level vertically and in a left-to-right order. This allows label preprocessing 210 to remove non-label pixels and noise.

[0068] The label extractor 320 is configured to extract label components of symbols in the computational image, such as label names and label numbers. In some cases, adjacent characters in the label name and / or label number are joined to each other and should be separated. In these cases, the label extractor 320 checks the vertical gaps in the characters of the label and segments the characters, and then performs character recognition using machine learning techniques (via intelligent processing 326). When there is no vertical gap, the label extractor 320 determines whether the aspect ratio of a given character set is greater than a predetermined threshold (e.g., 0.6, etc.). If the aspect ratio is greater than a predetermined threshold, the label extractor 320 applies segmentation using pixel density in the vertical direction. The area showing the white pixel peak is a potential segmentation area in the joined characters. Thereafter, the label extractor 320 performs character recognition using machine learning techniques.

[0069] The rules engine 322 is configured to validate tags extracted from the computational image based on one or more rules. In some embodiments, the rules are based on the ISA symbol standard and are divided into two categories: primary compliance checks (e.g., red category) and secondary compliance checks (e.g., orange category). Exemplary primary compliance checks include, but are not limited to, verifying that the symbol is one of the valid types (e.g., field device, control room display, etc.) and verifying that the tag name has one or more identification letters. Exemplary secondary compliance checks include, but are not limited to, verifying that the identification letters in the tag name do not contain any numbers and that the tag number in the tag name does not contain any alphabetic characters except at the end.

[0070] The line extractor 324 is configured to extract the lines between the symbols in the computational image. In some embodiments, the extracted lines include pipes and connection symbols, such as pipes, process connections, electrical signals, pneumatic signals, data links, capillaries for filling systems, hydraulic signal lines, and guided electromagnetic or acoustic signals. As will be understood by those of ordinary skill in the art, a geometric line fitting algorithm is used to extract lines from the image. Once the line is extracted, a portion of the line is subjected to one or more machine learning models (via intelligent processing 326) to obtain the type of line as mentioned above. Additional details about the operation of modules 312-324 can be found in the above-mentioned U.S. non-provisional application No. 16 / 021,867, and are incorporated herein by reference. Intelligent processing 326 and HMI application 328 will be discussed further below.

[0071] Figure 4 326 according to an embodiment of the present disclosure. As discussed, this module 326 implements machine learning models and algorithms to recognize symbols in a computational image. In some embodiments, the intelligent processing 326 applies a deep neural network to a specific area of ​​the image obtained by advancing a sliding window across the entire image. Figure 4 In the example of , the intelligent processing 326 includes a data input component 402, one or more machine learning models 404, an automatic feedback / correction component 406, a user application 408, a manual feedback / correction component 410, and an analyzer 412.

[0072] In general operation, the data input component 402 receives data representing diagrams (e.g., P&IDs, PFDs, etc.) and, after appropriate preprocessing, feeds the data to one or more machine learning models 404. The machine learning model 404 uses machine learning and image processing techniques to extract relevant information, such as names, numbers, symbols, lines, loops, etc., from the input data. The machine learning model 404 may also use image processing and / or geometric algorithms to reduce noise and improve accuracy. The automatic feedback / correction component 406 applies rules and algorithms that are configured to detect errors in the output received from the machine learning model 404. These errors are used to automatically correct the model output and are fed back to the machine learning model 404 via the analyzer 412 to update the learning of the machine learning model 404. The processed output from the automatic feedback / correction component 406 is then displayed to the user for verification via the user application 408 (e.g., the HMI application 328). The corrections made by the user are captured by the manual feedback / correction component 410 and fed back to the machine learning model 404 via the analyzer 412 to update the learning of the machine learning model 404. In this manner, intelligent processing 326 may continually evolve and improve the evaluation of input data and the extraction of relevant information therefrom.

[0073] As described above, in some embodiments, the intelligent processing 326 applies a deep neural network to a specific area of ​​the obtained image. The deep neural network processing results in a multi-class classification of symbol candidates. In some embodiments, the symbols are classified according to the ISA symbol system. Exemplary symbols include, but are not limited to, instruments, control / display elements, programmable logic controllers (PLCs), valves, pumps, etc. In some embodiments, the intelligent processing 326 uses at least three types of convolutional neural networks to identify various symbol candidates. These three types include: a decision network that determines whether the input is a single character or multiple characters; a single-character network that recognizes single letters and numeric characters; and a multi-character network that recognizes multiple characters or words. In some embodiments, the intelligent processing 326 also uses context-based predictions to distinguish similar characters, such as distinguishing the uppercase letter "I" from the number "1", and distinguishing the number "0" from the letter "O", etc.

[0074] Figure 5 It is an exemplary implementation of the HMI application 328 according to an embodiment of the present disclosure. The HMI application 328 generally starts from box 504, which inputs data from various plant engineering data sources into the industrial plant control system 300, and these plant engineering data sources include plant engineering drawings (e.g., PI&D, PFD, etc.). In box 506, the data is processed to extract relevant information and assets from the data, such as names, numbers, symbols, lines, loops, etc. Via intelligent processing 316 and the ML model therein, machine learning is used to complete asset extraction. Using the ML-based asset extraction described here to process data constitutes a practical application (e.g., designing HMI for industrial plants using a specialized automated design system). The processing system and method described here can improve the process of HMI design by allowing the image recognition model (which recognizes the plant components in the diagram) to run more efficiently and accurately. This can be achieved by inputting an image that is predetermined by the pre-processing module as likely to result in a positive recognition of the image recognition model. This can also allow the model to be better trained, thereby improving the accuracy of the model.

[0075] At block 508, the extracted assets and other relevant information are used as input to an asset relationship building process to construct an asset hierarchy, as described in more detail later herein. The building of asset relationships is also accomplished with the aid of the intelligent process 316 and the ML model therein. In the art of HMI design, it is not well understood, customary, or conventional to use the ML-based asset relationship building process described herein to build an asset hierarchy. Building an asset hierarchy using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0076] At block 510, the asset hierarchy can be used to create a knowledge graph based on the semantic model, as described in more detail later herein. Using the methods and systems described herein to create a knowledge graph is not well understood, customary, or conventional in the HMI design field. Using the methods and systems described herein to create a knowledge graph constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0077] At block 512, the knowledge graph is used to generate an HMI asset model (e.g., automatically generated using a computer), as described in more detail later herein. Generating a knowledge graph using the methods and systems described herein is not well understood, customary, or conventional in the HMI design field. Generating a knowledge graph using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0078] At block 514, the HMI asset model may be used to provide or construct an HMI (e.g., automatically generated using a computer), as described in greater detail later herein. Generating an HMI using the methods and systems described herein is not well understood, customary, or conventional in the art of HMI design. Providing an HMI using the methods and systems described herein constitutes a practical application (e.g., designing an HMI for an industrial plant using a specialized automated design system).

[0079] Figure 6 In more detail, the Figure 5 An exemplary ML-based asset extraction process 506 and an ML-based asset relationship building process 508 are shown.

[0080] ML-based asset extraction 506 generally begins at block 602, where engineering drawings (e.g., PI&D, PFD, etc.) are converted from PDF to image format. At block 604, a user enters information identifying the plant site and area where each diagram is being used. At block 606, all text in the diagram is found, and at block 608, the unit identifier of the diagram is found.

[0081] In some embodiments, finding the unit identifier involves searching for certain keywords, such as "cartography" or "unit" or "section" or variations thereof, in the diagram's information box or other predetermined portions of the diagram. The predetermined portion may be automatically identified based on an analysis of previous diagrams (e.g., the rightmost area whose width is 20% of the width of the P&ID and PFD; and the bottommost area whose height is 20% of the height of the P&ID and PFD), or the portion may be specified by the user. When a keyword (e.g., "unit") is found, the system checks the text associated with the keyword, which may be in the same cell as the word "unit", to the right of the word, or below the word if there is a table. Once the unit label has been determined, the determined label may be displayed to the user for verification. In some embodiments, this process may be performed for a relatively small number of diagrams (e.g., approximately 20). In this case, the location of the determined label may be used to determine subsequent labels in other diagrams without requiring or suggesting verification of the determination.

[0082] At block 610, all devices and instruments (e.g., labels, valves, sensors, actuators, etc.) in the diagram are found, and at block 612, all found devices and instruments and text are removed from the diagram or otherwise suppressed. This process may include generating a bounding box around the diagram features to be removed and setting all interior pixels to the background color of the diagram (e.g., white or black). At block 614, the removal of text and devices makes it easier to find all lines in the diagram.

[0083] In some embodiments, the lines in the chart can be found by shrinking the chart and converting them to grayscale. Long horizontal and vertical lines can then be removed by detecting and deleting black pixels (in the case of a white background) that extend over a predetermined length along the X-axis and Y-axis (e.g., 0.7 times the width and height of the chart, respectively). Once this is done, the remaining horizontal and vertical lines can be found by searching for black pixel clusters (in the case of a white background). A cluster is a group of pixels that are spaced less than a predetermined number of pixels (e.g., less than four pixels). Once found, the clusters can be connected by combining all co-linear points of the clusters to create horizontal and vertical lines. Lines that are less than a predetermined length (e.g., 16 pixels) can be removed. In embodiments where the chart is scaled before removing the lines, the removed lines can be scaled back to their original size. This may allow the original line coordinates to be stored and referenced when determining the connections and relationships between plant components. Various line types can be identified, including pipe and connection symbols, such as pipes, process connections, electrical signals, pneumatic signals, data links, capillaries for filling systems, hydraulic signal lines, and guided electromagnetic or acoustic signals. In some embodiments, the above line extraction can be performed using a geometric line fitting algorithm.

[0084] Once the line is extracted, a portion of the line is subjected to one or more machine learning models to obtain the type of line as mentioned above. In some embodiments, the line extractor includes processor-executable instructions contained on a storage memory device to provide the line extractor through a software environment. For example, the line extractor can be provided as processor-executable instructions including procedures, functions, routines, methods and / or subroutines.

[0085] Continuing to block 616, all arrows in the diagram may be found for each diagram. These arrows represent the flow direction in the diagram. Fig. 7A and Figure 7B Exemplary techniques for identifying arrows in a diagram are described.

[0086] refer to Fig. 7A and Figure 7B , an exemplary arrow 702 is shown. To analyze arrow 702, in some embodiments, as Fig. 7AThe detected line 704 shown is assigned a bounding box 706 having a width, for example, perpendicular to the direction of the line, which is a predetermined number of pixels wide (e.g., 40 pixels). In order to determine whether the line 704 is an arrow, two squares 708, 710 on each end of the line can be drawn with a side length equal to the height of the generated bounding box 706. In some embodiments, the sides of the squares 708, 710 can have different lengths. The content of the squares 708, 710 can be analyzed using a machine learning model or algorithm that is trained to determine the presence of the arrow 712 and the direction of the arrow. The ML algorithm can be, for example, a convolutional neural network. Training data for the algorithm can be created by cropping the head and tail of the identified segment at a specific size (e.g., 40×40 pixels). These cropped images can be stored and later classified as a left arrow, a right arrow, an up arrow, a down arrow, or no arrow. This classification can be done by, for example, a human. The identified segment can be selected from charts received from different sources (e.g., different clients). In some embodiments, approximately 200 training images may be used for training. If both squares 708, 710 have arrows, then arrow 702 may be considered bidirectional. If only one of the two squares 708, 710 has an arrow, arrow 702 may be considered unidirectional.

[0087] Return to reference Figure 6 , after the arrows are identified, at block 618, all equipment in the diagram (e.g., tanks, pumps, boilers, heat exchangers, etc.) are found. The equipment is then assigned a name that helps to uniquely identify the equipment. Depending on the particular industry, one of several naming conventions may be used. For example, the following naming convention may be used: <Equipment Type>-<Equipment Name / ID>, where a hyphen or space is used as a separator. Figure 2 In the example of , the name assigned to the container may be "C-1119", where C is the type of device and 1119 is the identifier of the device in the diagram. This name can then be saved as a Regex for the tank for search purposes. Similar naming conventions can be applied to other diagram components.

[0088] In some embodiments, finding the device is a combination of geometry-based or geometric-based recognition methods and machine learning or ML-based classification.

[0089] The device identification based on geometry is performed in the following manner: first, all instruments are removed from the diagram (for example, by backgrounding the instruments) to simplify the process, and then all parallel lines that meet one or more criteria are found. Then a path that joins with the parallel lines is found and used to connect pairs of such parallel lines. For example, a tank can be found by searching for parallel lines and connecting those pairs of lines that meet one or more criteria. Then a bounding box can be defined around each pair connected in this way. A standard can require that two lines are within a predetermined distance of each other. In some embodiments, the predetermined distance can be between about 10% and 120% of the larger length of the two lines. The minimum length of such a line can be set to about 4% of the width of the diagram. The maximum length of such a line can be set to about 50% of the width of the diagram. The difference between the lengths of the two lines can be less than about 15% of the larger length of the two lines. The starting and ending points of the two lines (i.e., four points) can be used to make the box. The width and / or height of the box can be extended by a predetermined percentage. For example, if the aspect ratio is less than about 0.3, the width can be extended by about 80%. If the aspect ratio is between about 0.3 and about 0.4, the width can be extended by about 60%. If the aspect ratio is greater than about 0.4, the width may be expanded by about 40%. If the box is to be expanded beyond the graphic boundary, the expansion may stop at the graphic boundary. The resulting expanded box may be a bounding box. The image of the tank may be cropped at the bounding box. In some embodiments, the cropped image boundary may be erased by making, for example, about 4 pixels of one or more sides white (in the case of a white background).

[0090] ML-based classification can then be used to identify the type of device. For example, a machine learning model or algorithm can be used to analyze the presence of cans in the resulting image and identify the type of can. An ML model or algorithm can be trained to detect and classify cans.

[0091] In some embodiments, the resulting image can be analyzed to determine whether to input it into the model. For example, if the image has a cluster of connected pixels (e.g., pixels with a maximum gap of approximately 2 pixels), the maximum size of the extended bounding box of the cluster is greater than the maximum size of the non-extended bounding box of the cluster, and includes the start and end points of two parallel lines identified earlier in the tank recognition process, then the image can be input into the model. In some embodiments, the extended bounding box can be generated by extending one or more lengths of the non-extended bounding box based on the aspect ratio identified earlier in the process. The image input into the model can be, for example, a black and white image of 170×170 pixels. The model can be trained with a training data set of, for example, 1950 images. Alternatively or additionally, enhanced training data can be used. For example, 46,312 images can be used for training and 2518 for validation. Table 1 below shows the details of exemplary training data enhancement that can be understood by a person of ordinary skill in the art.

[0092]

[0093] Table 1: Example training data augmentation

[0094] Those skilled in the art will also appreciate that the above table or similar tables can be used to provide training data augmentation for any other ML-based process described herein.

[0095] Reference now Figure 8 , you can also identify other equipment, such as pumps. You can identify a pump by creating a bounding box that includes the endpoints of one or more horizontal lines in the diagram. Figure 8An exemplary pump 800 is shown around which a bounding box can be created. To create the bounding box, pixels can be traversed from the endpoint of the horizontal line 802 (e.g., P1) in each direction perpendicular to the line 802 until a black pixel (e.g., P2) is found. The distance (e.g., D) between P1 and P2 can be calculated. If the distance is between a predetermined value (e.g., 100 pixels and 200 pixels), a bounding box can be created so that P1 and P2 are located at the center of opposite sides of the bounding box. The width of the bounding box can be equal to the distance D. The bounding box can be extended 20% of the length toward the P2 point. The chart can be cropped at the bounding box. By creating pixel groups, connected pixel clusters can be generated from the cropped image, where each group has all pixels connected together horizontally, vertically, or diagonally. For clusters whose length or width is above a predetermined threshold (e.g., 150 pixels), the area of ​​the cluster can be determined. The bounding box of the cluster with the largest area can be determined by, for example, finding the minimum and maximum values ​​of the x and y coordinates in the cluster. The minimum of the x and y coordinates may determine the upper left corner of the bounding box. The maximum of the x and y coordinates may determine the lower right corner of the bounding box. The image may be cropped at the bounding box of this cluster. The resulting image may be analyzed for the presence and type of pumps using, for example, a machine learning algorithm trained to detect and classify pumps. For example, a similar CNN architecture may be used.

[0096] Return again Figure 6 , in block 620, connectors in the graph and their related information may also be identified. Recalling above, a connector represents a connection between components located on different graphs.

[0097] Fig. 9An exemplary connector 902 is shown. As can be seen, the back end 903 (the end without arrows) of the connector 902 has a line 904 connected thereto, which indicates that the following flow occurs: from the component identified by the source identifier 906, through the line 904; and through the connector 902, to the component identified by the destination identifier 908. The front end 909 (the end with arrows) of the connector has a line connected thereto, which can indicate that the following flow occurs: from the component identified by the source identifier, through the connector; through the line to the component identified by the destination identifier. The connector identifier 910 within the connector 902 can indicate the sheet identifier (unit label) of the sheet containing the component identified with the destination identifier 908. Similarly, the connector identifier 910 within the connector 902 can indicate the sheet identifier of the sheet containing the component identified with the associated source identifier. The connector can be identified by filtering the lines with a length between about 20 pixels and about 40 pixels in the identified lines. Pairs of lines with a start point spacing of less than, for example, four pixels and an end point spacing of less than, for example, four pixels can be analyzed to determine whether they are part of a connector. If the length of one line in a pair is greater than the length of the other line in the pair, and the distance between the pair is greater than about 0.1 times the length of the longer line and less than about 0.5 times the length of the longer line, then the pair can be analyzed to determine if they are part of a connector. A tight fitting bounding box can be created that encompasses the two lines, with each side extending by half the distance between the two lines. Fig.9A , Fig. 9B and Fig. 9C Three resulting exemplary connector images are shown, respectively, at 912, 914, and 916. The resulting images may be input into a machine learning model that is trained to recognize the presence and orientation of a connector.

[0098] Note that source identifier 906 may be highlighted or otherwise emphasized for easier viewing by the user. This technique may be applied throughout the HMI to draw the user's attention to specific features.

[0099] To identify information associated with connector 902, a bounding box (not explicitly shown) may be generated around text 910 within connector 902 to determine if it is a connection identifier. If the bounding box of connector 902 contains the bounding box, the text may be identified as connection identifier 910. Text located above the bounding box of connector 902, for example within four times the distance between parallel lines of connector 902 (e.g., two parallel lines connected to an arrow), may be identified as a destination identifier contained in 908. In some embodiments, alternatively or additionally, the text before and after the word "To" may be identified as a material name and destination identifier 908, respectively. In this case, the material name may identify a material flowing through a pipe or other material carrier. In some embodiments, alternatively or additionally, the text before and after the word "From" may be identified as a material name and a source identifier, respectively. To identify the pipe identifier contained in 906, a step may be taken from the bounding box of connector 902 in the direction of line 904. In some embodiments, the step may be twice the length of the bounding box of connector 902. When the shifted connector bounding box intersects the bounding box of a word, the word may be identified as a pipe identifier contained in 906. If the bounding boxes of multiple words intersect, the word with the largest number of characters may be identified as the pipe identifier contained in 906. If connector 902 has a line connected to its front end, one or more of the aforementioned methods may be used to identify a source identifier and / or a destination identifier.

[0100] Return again Figure 6 , after the ML-based asset extraction process 506 has concluded, the ML-based asset relationship construction process 508 can begin for the extracted assets. This process of building relationships between the extracted assets can involve: creating a wiring connection diagram that captures the connections shown in the diagrams (e.g., PI&D, VFD, etc.); and then merging these wiring connection diagrams to create a composite flow chart. The merged diagrams may include: diagrams representing diagram lines connected to instruments, diagrams representing diagram lines connected to connectors, and / or diagrams representing diagram lines connected to instrument components (e.g., labels). In addition, wiring connection diagrams, such as line-to-line (line-to-line) diagrams, can also be generated to facilitate determining how various components are connected.

[0101] In block 622, a line-line (line-to-line) graph may be generated by detecting lines connected to the bounding box of the component and checking the collinearity or perpendicularity of these lines. If collinearity or perpendicularity is detected and the distance between the start or end point of one line and the start or end point of another line is less than a predetermined value (e.g., 32 pixels), the lines may be extended so that their ends meet. Lines that form part of a component (e.g., a device) may be removed by removing those lines whose endpoints on both sides are within the bounding box of the component. In order to assign indexes to line segments and keep track of line connections, the line may be split. If a device (e.g., a label, a valve, etc.) is located directly above the line, it is particularly useful to split the line into two segments. To this end, the distance to the start and end points of the vertical line may be measured from the horizontal line. In some embodiments, the start and end points of the horizontal line may be excluded from the measurement. If the measured distance is less than a predetermined distance (e.g., 32 pixels), the split may be performed in the horizontal line at the closest point between the start or end point of the horizontal line and the vertical line. The process may be repeated for vertical lines; if the distance from the vertical line to the start or end point of the horizontal line is less than a predetermined number (e.g., 32 pixels), a split may be made in the vertical line at the closest point between the vertical line and the start or end point of the horizontal line. In some embodiments, the distance from the vertical line to the start or end point of the horizontal line may exclude the start and end points of the vertical line. Lines may be assigned index numbers. When a line is split at a point, the two newly created segments may be assigned index numbers. An adjacency matrix representing the connections of the lines may be generated.

[0102] Fig. 10A An exemplary adjacency matrix 1002 is shown. The adjacency matrix 1002 may have a size of n×n, where n is the total number of lines. The value at position [first line index, second line index] in the adjacency matrix 1002 may be "0" to indicate that the lines having the first and second line indexes, respectively, are not connected to each other. The value of this position may be "1" to indicate that the lines are connected. It should be understood that other values ​​may be used. Thus, the adjacency matrix 1002 may indicate whether two lines are connected to each other. For example, from left to right, node L2 has "1" at node L1 and node L3 to indicate that L2 is connected to L1 and L3. In order to construct a line-line graph, a node-edge (node-to-edge) graph may be generated in which line indices are assigned to nodes and connecting line indices are assigned to edges between nodes.

[0103] Fig. 10BAn exemplary line graph 1004 corresponding to the adjacency matrix 1002 is shown. In the figure, nodes E1 and E2 represent two devices, and nodes L1, L2, L3, L4, and L5 represent line segments. It can be seen that line segments L1, L2, L3, L4, and L5 connect E1 and E2 together in the manner reflected by the adjacency matrix 1002.

[0104] Back to Figure 6 At block 624 , as part of the ML-based asset relationship construction process 508 , a graph representing device-to-wire connections may be generated from the graph. Fig.11 and Figure 12A-12B An example of constructing a device-circuit diagram according to some embodiments is shown.

[0105] Fig.11 An example of a device 1102 is shown. As can be seen, a bounding box 1104 is placed around the device 1102 for computational purposes. In order to construct a device-line graph, the bounding box is expanded on each side by increasing the length of each side, for example, by 2.5%. Then, lines 1106 whose endpoints are located in the expanded bounding box are found, and associated nodes (e.g., L1, L2, L3, etc.) are generated for each line. In some embodiments, if two such lines are connected to each other, associated nodes can be generated for the line whose endpoint is closer to the center of the bounding box. Associated nodes (e.g., E1, E2, E3, etc.) can also be generated for device 1102. Thereafter, associated edges (e.g., (E1, L1), (E2, L2), (E3, L3), etc.) can be generated for device-line pairs.

[0106] Fig. 12A An exemplary table showing device-line connections is shown. The figure shows: a device-line table 1202 that lists various nodes (e.g., L1, L2, L3, etc.) and (e.g., E1, E2, E3, etc.) associated with a device 1102 and a line segment 1106; and an edge table 1204 that lists the edges associated with these nodes (e.g., (E1, L1), (E2, L2), (E3, L3), etc.).

[0107] Fig. 12B An exemplary device-line graph 1206 is shown corresponding to the device-line table 1202 and the edge table 1204. As can be seen, there are five edges: edges 1, 2, 3, 4, and 5, which reflect the five points where lines L1, L5, L6, L7, and L9 connect to devices E1, E2, E3, and E4.

[0108] To find a path between two devices, the lines connected to each device may be identified by merging the graph 1206 representing the graph lines connected to the devices with the adjacency matrix 1002. If multiple paths are found between the two devices, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) may be used on the graph to identify the path between the two devices. The direction of the arrows associated with the lines of the path may indicate the direction of material flow between the devices and specify which device is the destination and which device is the source.

[0109] Return again Figure 6 At block 626, as part of the ML-based asset relationship construction process 508, a diagram representing connector-wire (connector-to-wire) connections may be generated from the graph. The following is an example of constructing a connector-wire diagram according to an embodiment of the present disclosure.

[0110] To construct a connector-circuit diagram, refer to Fig. 9 As discussed, a bounding box is placed around the connector. The bounding box can then have its long side extended, for example, by 0.1 times its length. If the connector is an incoming connector, the extension can be in the direction of the arrow; if the connector is an outgoing connector, the extension can be in the opposite direction. Then, the line whose endpoint is located within the extended bounding box can be identified. A node, such as C1, can then be generated for the connector; and a node, such as Lc1, can also be generated for the line connected to the connector. Then, a directed edge can be generated for the connector-line pair (e.g., (C1, Lc1)), where the edge direction is determined by whether the line is connected to the head or tail of the connector. In order to determine whether a device is connected to a connector, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) can be used on the graph to determine the path between the line connected to the device and the line connected to the connector. If the connection is determined, the direction of the edge connected to the connector node can be used to determine the flow direction between the device and the connector.

[0111] Fig.13A An exemplary table showing connector-line connections is shown. The figure shows: a connector-line table 1302 that lists various nodes associated with connectors and line segments (e.g., C1, C2, Lc1, Lc2, etc.); and an edge table 1304 that lists the edges associated with these nodes (e.g., (C1, Lc1), (C2, Lc2), etc.).

[0112] Fig. 13BAn exemplary connector-wire graph 1306 is shown corresponding to the connector-wire table 1302 and the edge table 1304. As can be seen, there are five connectors: C1, C2, C3, C4, and C5, which are connected to two devices: E1 and E2 via the line segments where the lines L1-L16 are located. Table 1308 shows the paths generated using the graph 1306.

[0113] Return again Figure 6 At block 628, as part of the ML-based asset relationship building process 508, a diagram representing device or equipment-line (device / instrument to line) connections may be generated from the diagram. The following is an example of building a device / instrument-line diagram according to an embodiment of the present disclosure.

[0114] A graph representing lines connected to a device or equipment (e.g., a tag, sensor, etc.) can be generated by providing a bounding box around the device. Lines whose endpoints are within the bounding box can then be identified. If fewer than two lines are found per bounding box, the length of one or more edges of the bounding box can be increased, for example by 50%. A node, such as D1, can then be generated for the device; and a node, such as L1, can also be generated for the line connected to the device. Associated edges can be generated for pairs of instruments and the lines connected to them. In order to determine whether an instrument and a piece of equipment are connected, a shortest path algorithm (e.g., Dijkstra's shortest path algorithm) can be used on the graph to determine the path between the line connected to the instrument and the line connected to the device.

[0115] Fig.14A An exemplary table showing device / equipment-line connections is shown. The figure shows: a device / equipment-line table 1402, which lists various nodes associated with devices and line segments (e.g., D1, D2, L1, L2, L3, L4, L5, etc.); and an edge table 1404, which lists the edges associated with these nodes (e.g., (D1, L1), (D1, L2), (D2, L3), (D2, L4), (D2, L5), etc.).

[0116] Fig. 14B Exemplary device / equipment wiring diagrams 1406 and 1408 are shown corresponding to the connector-wire table 1402 and the edge table 1404. As can be seen in the diagram 1406, the device D1 has two line segments connected to it: L1 and L2. In the curved wiring diagram 1408, the device D2 has three line segments connected to it: L3, L4, and L5.

[0117] Back again Figure 6, so far, only directed graphs or graphs with a specific flow direction have been discussed. In addition to directed graphs, there may also be undirected graphs in the graph. As part of the ML-based asset relationship construction process 508, at block 630, any such undirected graph can be changed to a directed graph. The following is an example of changing an undirected graph to a directed graph according to an embodiment of the present disclosure.

[0118] In order to assign directions to diagrams such as line-line diagrams, equipment-line diagrams, and device-line diagrams, it is necessary to identify all line nodes in the equipment-line diagram. For each of these lines, all possible paths in the line-line diagram are traversed until the line connected to a connector or another device. Then, the path can be divided into segments, and the lines in these segments can be assigned directions. This can be done based on the direction of the connected connector, the presence of arrows, or the direction of the lines in the adjacent paths (e.g., the lines connected to the connector). For example, if one path is A->B->C->D and the other path is A->B->E, then these paths are divided into three paths: A->B, B->C->D, and B->E. Then, two directions can be assigned as follows. If a line in the path is connected to a connector, the direction of the connector (incoming or outgoing) is assigned to all lines in the path. If at least one line in the path has an arrow, the direction of the arrow is assigned to all lines in the path. If all lines in the path do not have arrows and the path is not connected to a connector, the adjacent paths connected to the connector are checked and the directions of the adjacent paths are assigned to all lines in the path.

[0119] Return again Figure 6 , recalling from block 612, all devices and instruments (and text) were removed from the diagram, resulting in a line-line diagram that only shows connections from device to device and device to connector. As part of the ML-based asset relationship building process 508, at block 632, the line-line diagram can now be updated to include those devices and instruments. This can be accomplished, for example, by restoring the devices to the point at which they were removed.

[0120] Fig.15A An exemplary line-line graph 1502 and edge data table 1504 are shown, wherein the edge data table 1504 has been Figure 6 612. It can be seen that the device is removed from the points between the connector (C) and the device (E), where the two dashed circles are located and the points are replaced by two corresponding edges, namely: edge 5 (L3, L6) and edge 6 (L6, L7).

[0121] Fig. 15BThe same exemplary line-line diagram 1502 and edge data table 1504 are shown after an update has been made to restore the devices. It can be seen that two devices (D) are now located between the connector (C) and the equipment (E), replacing the two edges (L3, L6) and (L6, L7). This allows the line connection diagram to reflect not only device-to-device and connector-to-device connections, but also device-to-device, device-to-connector, and device-to-device connections.

[0122] Back again Figure 6 At block 634, for a given diagram (e.g., PI&D, PFD, etc.), various wiring diagrams generated for the diagram may be merged as part of the ML-based asset relationship construction process 508. For example, if the diagram is a PI&D, then all wiring diagrams may be merged to create a PI&D flow diagram.

[0123] Fig.16A An exemplary PI&D flow chart 1602 is shown, which is generated by the ML-based asset relationship construction process 508 for the merger of various wiring connection diagrams generated for a given PI&D. The PI&D flow chart 1602 is different from its component diagrams, which contains all connectors, devices or instruments, and equipment in the PI&D and the connections between them. Therefore, the PI&D flow chart 1602 captures the relationships between all assets extracted from the given PI&D.

[0124] Fig. 16B An exemplary table 1604 containing data for the merged PI&D flowchart 1602 is shown. The table 1604 includes nodes and edges for each wiring diagram (e.g., line-line, device-line, connector-line, apparatus-line, etc.). The table 1604 also contains the nodes and edges of the merged PI&D flowchart 1602. Such a table can then be used as part of generating an HMI.

[0125] Return again Figure 6 , at block 636, as part of the ML-based asset relationship building process 508, a control loop for a given diagram (e.g., PI&D, PFD, etc.) can be added to the merged PI&D flowchart 1602. As previously described, a control loop performs some sub-process in a plant process and can be a closed-loop or open-loop control loop. Most control loops involve several devices working together, including sensors, actuators, controllers, etc. There are generally two types of control loops: simple control loops and complex control loops. A simple control loop has an input, an output, and a controller, while a complex control loop consists of several simple control loops. Fig.17 An example of identifying a control loop according to an embodiment of the present disclosure is depicted in FIG.

[0126] Fig.17 A portion of an example diagram 1700 containing a simple control loop is shown. For illustrative purposes, four symbols have been identified and labeled 1702, 1704, 1706, and 1708, each symbol representing a device (e.g., a sensor, an actuator, a controller, etc.) or a piece of equipment. In this example, identifying the control loop begins by identifying all labels inside the symbol and all words outside the symbol. Next, for each word outside the symbol, the nearest label within the symbol is found. For example, for the symbol at 1702, the nearest label within the symbol is "TIC 1803". For a given symbol, all words that are greater than, for example, 1.2 times the width of the symbol (i.e., words associated with other symbols) can be ignored. The word is then checked (e.g., using a rules engine) against a list of words from an industry standard such as an ISA to determine whether the word represents, for example, an alarm, a signal, a label, etc. If the word is a label, the label number of the nearest label is assigned to the word. For example, for the symbol at 1702, the tag number "1803" is assigned to the words "TE" and "TT" to generate "TE 1803" and "TT 1803". These tagged words are then added to the tag list for that symbol. The tags for a given symbol include all tags inside and outside the symbol.

[0127] Once all tags inside and outside the symbol are identified, all tags with the same starting identification letter and the same tag number are grouped together. Fig.17 In the example of , the tags “TIC 1803 ”, “TV 1803 ”, “TE 1803 ”, and “TT1803 ” are grouped together. These tags are used to define a simple loop 1710 , which can be included as an extracted asset of the diagram 1700 .

[0128] Return again Figure 6 At block 638, the plant assets are organized in an asset registry for subsequent access and use. Assets are organized using namespaces so that each asset is uniquely identified. As will be appreciated by one of ordinary skill in the art, a namespace is a set of names used to identify objects so that there is no ambiguity when objects with the same name but different origins are mixed together.

[0129] At box 640, as part of the ML-based asset relationship construction process 508, the merged flow charts of each diagram can be merged to create a single plant flow chart for the entire plant. Recall that a diagram like a PI&D or PFD represents the instruments or devices and equipment that make up a unit, and recall that the asset extraction process discussed above causes the device names for a given unit to have the same unit prefix and are therefore uniquely identifiable. The device name is similarly created by prefixing the unit name. Connector names are also uniquely identifiable because each connector name contains the name of the source device, the name of the target device, and the name of the pipe.

[0130] The line index or index number (e.g., 1, 2, 3, etc.) is local to each sheet and therefore needs to be made uniquely identifiable across multiple sheets. This can be achieved, for example, by prefixing the line index with the unit name or corresponding sheet followed by the drawing number. For example, if the unit name is "UNIT1" and the sheet drawing number is "1116", then the line index number "15" can be changed to "UNIT1_1116_15".

[0131] In order to create a single plant flow chart for the entire plant, a flow chart for each chart in the plant may be generated as described above. Where required, line indices may be made uniquely identifiable across different charts as described above. The flow charts of the various charts may then be merged by inserting all nodes and edges from each chart flow chart (or its tabular representation) into the plant flow chart (or its tabular representation). Charts that are directly connected to each other will have connector nodes with the same name. When this occurs, the duplicate connector nodes are removed and new edges are created in the plant flow chart by connecting the lines to be connected by the removed connectors. All other nodes and edges should remain the same.

[0132] The resulting plant flow diagram can be used to generate a flow chart similar to Figure 1 A visual representation of an asset hierarchy is shown of the asset hierarchy 100. The asset hierarchy and relationships under the asset hierarchy may then be stored as structured data, for example, in one or more JSON (JavaScript Object Notation) files, for subsequent access and use by the industrial plant control system 300.

[0133] In some embodiments, the industrial plant control system 300 can be used to merge structured data of asset hierarchies and asset relationships into an ontology knowledge base as described above. The system then dynamically builds a plant engineering domain ontology based on the plant engineering data source (which evolves during the plant life cycle phase). This typically involves: (1) extracting domain entities (i.e., types and data) from structured data sources (by the system) using metadata and machine learning techniques from unstructured data sources; (2) building a semantic model using entity types (by domain experts) and also adding association relationships between entities; (3) dynamically extracting entity data (by the system) and building a knowledge graph based on the semantic model; and (4) providing users with the ability to navigate asset namespaces and relationships to extract relevant information (by the system). This arrangement has many benefits.

[0134] As a benefit, automated control configuration can be performed using process narratives. Currently, control configuration is a time-consuming manual process and requires a strong control engineering background. Using the industrial plant control system 300 described here, people who are not familiar with control engineering concepts should be able to easily configure the system. This stems from the fact that the system uses "process narratives" composed of ordinary language used by plant engineers. The system automatically converts process narratives into control strategies and configurations. The process narrative can be converted into multiple control narratives by utilizing noun-verb pairs, such as "Asset.Operation" in Automation ML (see IEC 62714) and state-based control (see ISA 106, ISA 88, and ISA 95). The system uses information from asset templates, their relationships, namespaces, control policy templates, and rules to convert process narratives into control logic.

[0135] As another benefit, test cases can be automatically self-generated based on process descriptions and documented verification. The industrial plant control system 300 uses process descriptions, control configurations, and asset ontologies to automatically generate test cases. The system then runs automated test scenarios to simulate the process in order to verify the control application.

[0136] Reference now Fig.18, a flow chart of an exemplary method 1800 is shown, which can be used with an industrial plant control system 300 to dynamically build a plant engineering domain ontology from a plant engineering data source. The method generally starts from box 1802, where the plant engineering data source is input into the system. The data source can include PI&D and PDF for generating the above-mentioned plant asset hierarchy and asset relationships, as well as AutoCAD, SmartPlant and Aveva engineering drawings. Other plant engineering data sources can include simulation models, instrument indexes, cable schedules, I / O allocation lists, configuration data, cross wiring tables, cabinet loading reports, system audit reports, historical data, alarms and events, etc.

[0137] Figure 19-Figure 29 Exemplary plant engineering data sources that can be used to generate asset hierarchies and / or knowledge graphs are shown. These plant engineering data sources are familiar to those skilled in the art. For example, Fig.19 At 1900, exemplary alarm data is shown. Fig. 20 An exemplary rack loading report 2000 is shown. Fig.21 An exemplary control database is shown at 2100 . Fig. 22 An exemplary cross-wiring report is shown at 2200 . Fig.23 An exemplary field wiring index is shown at 2300 . Fig.24 Exemplary historical data (tag data) is shown at 2400 . Fig.25 An exemplary instrument index is shown at 2500 . Fig.26 An exemplary asset hierarchy is shown at 2600 . Fig. 27 An exemplary nest loading report is shown at 2700 . Fig.28 An exemplary plant asset index is shown at 2800 . Fig.29 An exemplary process control narrative is shown at 2900. Such data sources and other plant engineering data sources, including HMI specifications (see Fig.34 ) to generate asset hierarchies and / or knowledge graphs.

[0138] Back to Fig.18In block 1804, the system extracts domain entities and instances of domain entities from the plant engineering data source. The data source may include structured and unstructured data sources. For example, entities (also called types or classes), attributes (also called properties), and relationships may be extracted from the data source. Entities may be used to specify the domain and range of the ontology, and attribute values ​​may be used to specify the scope of the ontology model. The semantic model may include a set of triples constructed according to the domain and range of the ontology. The triple may specify a subject entity, a predicate relationship or attribute, and an object entity or value. The subject entity may be an entity from the domain of the ontology. The object entity or value may be an entity or value from the scope of the ontology. Instances of entities and values ​​may be extracted from the data source. For example, the semantic model may specify a triple with the subject "unit", the predicate "has", and the object "device". A triple consisting of instances of the corresponding subject and object may include "unit 01" (an instance of the entity "unit") "has" (predicate) "control valve 01" (an instance of the entity "device").

[0139] A variety of data extraction techniques can be used for each data source (both structured and unstructured) from the plant engineering data source. The system can distinguish between structured and unstructured data based on the file format or database type. Unstructured data files (e.g., emails, word processing documents, videos, images, web pages, etc.) often include text and multimedia content without any schema, while structured data files (such as spreadsheets, CSV files, XML files, RDBMS, time series, graph databases, etc.) contain schemas or metadata. The system can use the schema or metadata to extract domain entities from these structured data sources. For example, in the CSV file format, plain text data is delimited by commas, where each new line in the CSV file represents a new database row, and each database row has one or more fields separated by commas. The system can use this pattern to extract domain entities from the data source. In Fig.25 An example can be seen in Figure 1, where domain entities are extracted from an exemplary instrument index data source in CSV format. As shown in the first row of the table, the domain entities and associated relationships extracted from this data source include plant, area, unit, loop, tag, process function, etc. Instances of these domain entities are shown in subsequent rows.

[0140] For unstructured data, such as P&ID, PFD, Process Control Narrative (PCN) and other images or unstructured text formats, the system uses machine learning to extract domain entities and association relationships from unstructured data sources into structured data. For reference, a process control narrative (PCN) is a functional statement that describes how device mounted controls, panel mounted controls, PLCs, HMIs and other processor-based process control system components should be configured and programmed to control and monitor a specific process, process area or facility. PCN is an essential link between process design and control system design. For PCN, the system uses Named Entity Recognition (NER) technology to extract domain entities such as pumps, valves, locations, alarm conditions, etc. from process narratives. NER is a technology used in information processing with unstructured text. This technology tags sequences of words in text, which are names of things, such as things, names of people, organizations, locations, time expressions, quantities, monetary values, percentages.

[0141] Fig.30 3000 is a table showing an example of domain entity types and instance data extracted into a file having a tabular format. The domain entity types in the table include controller, compound, block, tag, etc., and the corresponding instance data include (for the first row) CP2801, CF101_011900, PT038, 011900PV038, etc.

[0142] Back to Fig.18 Once the domain entities and relationships have been extracted, at block 1806 , a user (eg, a domain expert) builds a semantic model using the extracted entity types and associated relationships. Fig.31 An example of a user interface that can be used by a domain expert to build a semantic model is shown.

[0143] exist Fig.31, a user interface 2100 is shown that allows a user to specify semantic relationships between entities extracted from a data source. From this interface, a user can manually browse entity types from various data sources and filter entities required for the semantic model. For example, a user can select two entities, such as a source and a target, and create a named relationship between them. A schema can be defined at a conceptual level, which is a logical model. Then, the user can specify domain-specific relationships between entities extracted from the data source. Typical relationships are "is-a (is ...)" representing a parent-child relationship, "has-a (has ...) (part-of (is a part of ...))" representing an inclusive relationship, and so on. In this example, the user specifies that the source "factory" is related to the target "region" through the relationship "has region". The user can also add additional contextual information as entity attributes. For example, the user can add type, severity, priority, as attributes of the alarm entity.

[0144] In some embodiments, instead of or in addition to being defined by the user, semantic relationships can be automatically determined based on primary key and foreign key relationships specified in parent-child nodes in, for example, a Relational Database Management System (RDBMS) tuple or an XML document. Attributes can be added to provide additional contextual information (for example, for the entity "alarm", the attributes "type", "severity" or "priority" can be added). The resulting semantic model can be validated against a set of predetermined rules. These rules can be defined, for example, by the standard ISA-106 ("Procedural Automation of Continuous Processing Operations"). Using ISA-106, the triple "device has unit" may be marked for correction during validation because the standard specifies that the unit has the device, not the other way around. Instead of or in addition to the standard, public ontologies and vocabularies in dictionary files can also be used for validation.

[0145] Back to Fig.18 Once the semantic model has been established, the model is validated in box 1808. Semantic validation refers to the process of verifying whether a data element is logically valid. For example, a data element like "a device has a region" would be invalid in ISA 106 mode because a device would not have a region (semantic models typically standardize entity relationship types according to industry standards). In some embodiments, the system automatically validates the domain-specific relationships between the entities created in box 1806. To this end, the system can maintain standard terms, their definitions, and references to common ontologies and vocabularies in dictionary files, which can all serve as references for validation. The system also provides the option of manually validating and modifying system-generated relationships by domain experts.

[0146] In box 1810, the validated semantic model is saved to create an ontology. In some embodiments, the semantic model is saved according to one of the following well-known semantic model formats: OWL / XML, JOSO-LD, and RDF / JSON. As an alternative, the semantic model can be persisted by converting the model into a Labeled Property Graph Model model in GaphSON format. If the semantic model is converted into a W3C ontology, the semantic model knowledge graph can be persisted in an RDF triple store, such as AllegroGraph, Stardog, or Amazon Neptune. If the semantic model is converted into an LPG (labeled property graph), the knowledge graph for this can be persisted in a graph database (GraphDatabase) format, such as GraphDB or Neo4j or Azure Cosmos DB.

[0147] In box 1812, the system can use the semantic model to build and deploy the knowledge graph. As described above, the knowledge graph is basically a graph that can be used to navigate and visualize the relationships between components (i.e., a physical model of the data). The semantic model defines the relationship between source entities and target entities in the knowledge graph at the conceptual level (e.g., "region-has-unit"). The system extracts instances related to the ontology classes and uses the association relationships and context information to build the knowledge graph. Therefore, the knowledge graph represents a collection of interlinked entities that enhances the user's ability to search for required information. LPG, RDF, or similar graph models can be used in database systems to represent knowledge graphs. Fig.32 An exemplary visualization of a knowledge graph is depicted in .

[0148] refer to Fig.32, a visualization of the knowledge graph 3200 can be seen, where entities and relationships are shown as multiple labeled-property nodes, one of which is indicated as 3202. Both the nodes and their relationships are named and can store properties represented by key / value pairs. Nodes can be labeled to easily group similar nodes. The edges representing relationships have two main properties: they always have a start node and an end node, and they are directed, making the graph a directed graph. Relationships can also have attributes, which helps to provide additional metadata and semantics for the relationships of the nodes. The legend 3204 provides an indication of what each node 3202 represents on a color-coded basis, for example, blue represents a label index list, green represents a nested load report, orange represents a cabinet load report, and so on. According to an embodiment of the present disclosure, the knowledge graph 3200 can be selected by selecting the graph view option 3206, and the knowledge graph 3200 is one of several display screens that can be obtained from the HMI. The user may select other display screens from the HMI by selecting options for other display screens, such as an upload data option 3206 for uploading data and a query engine option 3208 for searching data.

[0149] Fig.33 An example of a query display screen 3300 that can be obtained from an HMI according to an embodiment of the present disclosure is shown. In this screen, a user can search for data by typing a natural language query into a search box 3302. For example, a user can ask the HMI to display or otherwise indicate alarms that are typically triggered during a normal plant startup process by entering the query "What are the expected alarms during the startup sequence?" into the search box 3302. In response, the HMI can display the requested alarms graphically (nodes with dashed circles), or the HMI can display the requested alarms in text form by displaying the identifier of the alarm. The user can also select a node representing the data type desired by the user and drag it into the search box 3302. In the example shown, the user can drag and drop the six returned alarms into the search box 3302. With the help of the knowledge graph, the HMI can quickly highlight all common assets for the six alarms, which are indicated by nodes without dashed circles; and the HMI grays out all other nodes or suppresses them to the background. The common assets returned may include: the control process where the alarm is processed, the assets shared by these alarms, the I / O modules that processed the alarm, etc., and the common assets returned indicate the possible root cause of the alarm.

[0150] In some embodiments, users and plant operators can also enter commands to initiate operations. The HMI can interpret commands written in natural language. For example, to instruct the HMI to start filling the component labeled "Tank 01", the command "Fill Tank 01" can be entered into the command bar (not explicitly shown).

[0151] Back to Fig.18 Once the ontology is constructed, in block 1814, the system can provide an asset namespace registry API to allow different applications to consume the data. Figure 6 ), factory assets are organized into asset registries using namespaces. In some embodiments, a RESTful application program interface (e.g., Azure Active Directory Graph API) can be generated to allow downstream applications to extract information from the knowledge graph.

[0152] At block 1816, the system provides a way for a user to retrieve data based on the process narrative. In some embodiments, the user can retrieve data using natural language questions to search for data. The system can translate the natural language query into, for example, a SPARQL query or a Gremlin query. Downstream applications can include, for example, an HMI through which commands from the converted process narrative and / or other natural language commands can be issued.

[0153] At block 1818, a plant control process and an HMI for the process (i.e., control and HMI engineering) may be designed and developed to allow users to use the system 300 to control the plant and various plant assets. This involves processing plant assets and asset relationships from the asset model to create an HMI asset model. The HMI asset model arranges assets in a hierarchical structure (see Fig.37 ), unlike traditional solutions that are discrete in nature (i.e., created based on a single P&ID diagram). The HMI can then use the HMI asset model to automatically create a unique, continuous HMI display that resembles a 2D floor plan of the plant level, including symbols for each asset and lines representing processes, electrical connections, data links, etc., running between the symbols. Plant operators can then navigate horizontally and vertically, and can zoom in and out (i.e., along the Z axis) of this 2D view to move across different assets and verify their behavior.

[0154] In some embodiments, the HMI can dynamically change the assets displayed to the user based on changes in the user's position on the HMI screen and alarms generated at runtime. The HMI assets displayed at runtime can be determined based primarily on two factors: (i) static weights, which are assigned to assets based on the asset hierarchy details available in engineering drawings (e.g., PFD, P&ID, etc.); and (ii) dynamic weights, which are assigned to each device that issues an alarm. Thus, at runtime, the HMI can display a view that includes assets with greater weights, or if an alarm exists, display assets with alarms. The HMI can also dynamically change the view to include assets that have recently issued an alarm. Alarms on critical equipment will be given a "high" or "high-high" indication (e.g., exclamation point, red, etc.) and will receive more attention, and the complete alarm chain will receive the highest weight at runtime. In order to design an HMI, it is necessary to establish a design specification for the HMI.

[0155] Fig.34 An exemplary design specification 3400 of an HMI according to an embodiment of the present disclosure is shown. The specification 3400 specifies a hierarchical structure of the HMI, which allows the HMI to display certain process control details, including process equipment details and instrument control details, in a hierarchical arrangement. In this example, the HMI is arranged in several hierarchical display levels 3402, 3404, 3406, 3408. Level 1 displays a process area overview, including information such as operational KPIs (key performance indicators), alarm counters, bypass signal counters, trends, overall plant status, etc. The next level, level 2, displays process unit controls, including controllers, alarms, trends, and status. The next level, level 3, displays process unit details, including smaller equipment groups, controllers, alarms, trends, ESD displays, equipment, diagnostics, and status. The next level, level 4, displays process unit support details, including interlocks, details, "first-out" processes, documentation, and help. Other HMI design specifications may include specifications related to graphical displays, such as alarm priority colors (e.g., red, yellow, orange, etc.), and colors indicating status (e.g., out of service, etc.) and functionality (e.g., feedback, trend lines, etc.). The meaning of icons, symbols, and lines (e.g., solid, dashed, dot-dash, etc.) and engineering units (e.g., metric, imperial, etc.) may also be specified. These HMI specifications, along with data and information extracted from various plant engineering data sources as described above, may then be used to build an HMI asset model.

[0156] Fig.35An exemplary HMI asset model 3500 according to an embodiment of the present disclosure is shown. The HMI asset model, as the name implies, includes an asset model that is processed and used by the HMI to be displayed to the user. For the current purpose, the HMI asset model includes the plant assets and relationships (asset hierarchy) discussed above, the ontology knowledge base, and various plant engineering data sources. The HMI asset model is therefore composed of the following: PFD 3502 (e.g., high-level equipment, instruments, connections, etc.), PI&D and instrument index 3504 (e.g., equipment, instruments, associations between them, alarms, piping, maintenance override graphics (MOS) and cause and effect (C&E) details, etc.), HMI specifications 3506 (such as navigation requirements, alarm summaries, historicalization, diagnostic details, etc.), control narratives 3508 (e.g., control and monitoring parameters, set points, alarms, ranges, engineering units, etc.), and knowledge captured via engineering displays 3510 (e.g., captured from maintenance records, previous corrective actions, user experience, etc.). As the content of the various data sources changes and assets are added and / or removed, the model automatically updates, making this a dynamic HMI asset model.

[0157] Once the HMI asset model 3500 (or more precisely, its content) is established, the model can be used to develop an HMI according to an embodiment of the present disclosure. Developing an HMI based on an HMI asset model can involve: creating templates 3514 for various assets (e.g., instruments, equipment, composites and combinations thereof, etc.). These templates can include: for example, level symbols (e.g., level 2, level 3, etc.), detailed displays of instruments and equipment, instrument properties and actions, animation scripts, control engineering templates, comprehensive details of pumps, valves, etc. Developing an HMI from an HMI asset model can also involve: creating other asset models 3516, in which all areas and units are represented as area objects, and each composite and / or complex instrument combination is instantiated from their templates. The process can involve assigning more weight to some equipment and devices relative to other equipment and devices, wherein the latter are graphically emphasized. New equipment and devices (e.g., introduced by control narration) can be added, and the asset hierarchy structure can be adjusted accordingly. Equipment and devices can be grouped according to their association in these asset models.

[0158] In addition to the above, the HMI asset model can also be used to build a control application that controls the operation of various factory assets. As mentioned above, machine learning can be used to extract factory assets from unstructured data such as engineering drawings to identify one or more assets. The process can also extract one or more control loops from the engineering drawing based on one or more devices, instruments, connectors, and lines to build a dynamic HMI asset model. Then, the automatic control generation process can be used to read one or more control loops from the dynamic asset model and generate control logic for the control application. The control logic connects the control application with the actual factory equipment to allow the control application to read or otherwise obtain process values ​​and control the operation of equipment, devices, equipment, etc. In other words, the HMI processes assets and asset relationships and creates an HMI asset model to arrange assets in a hierarchical structure. The process also links the properties of each HMI asset with the corresponding control I / O reference so that at runtime, if the operator opens a valve in the control application in the HMI, its control value will be propagated to the corresponding device in the factory via the control I / O reference (linked in the HMI asset), thereby opening the valve in the factory.

[0159] For example, an HMI developed from an HMI asset model can be used by a user to monitor various displays and easily find the root cause of an alarm. In some embodiments, such an HMI can be developed using an HMI development platform, such as the System Platform 2017 InTouch OMI (Operation Management Interface) available from Wonderware West of Houston, Texas. Other HMI development platforms may certainly be used within the scope of the present disclosure.

[0160] Next reference Fig.36 In some embodiments, an HMI according to the present disclosure can process alarms issued by various devices and automatically identify the root cause of the alarm. The HMI determines the root cause of the alarm mainly through two factors: (i) the process / material flow defined in the engineering drawing; and (ii) the control flow defined in the engineering drawing and any control narrative. In the case of process flow, the HMI first finds the device nodes that cause different alarms. For each device node, the HMI retrieves the device node corresponding to the device node. For the device node, the HMI retrieves any upstream device nodes connected to the device node, which also has the device that issued the alarm. Repeat this step for adjacent nodes until the HMI finds the connected device of the unconnected device with the alarm. This forms an alarm chain, which enables the HMI to determine the root cause of the alarm by finding the device with the alarm at the highest level in the chain.

[0161] exist Fig.36In the example of , a two-dimensional plan view of the plant level is displayed by the HMI on screen 3600, which includes symbols 3602, 3604, 3606, and 3608 representing assets, and lines 3610, 3612, and 3614 extending between the symbols representing processes, electrical connections, data links, etc. The plant operator can then navigate horizontally in the two-dimensional view and zoom in and out to move across different assets and verify their actions. In this example, material flows from equipment E1215 (pump) to equipment E1214 (boiler), and there is a pressure drop at the inlet of equipment E1214 (boiler). The pressure drop causes the device (gauge) at the inlet of equipment E1214 (boiler) to alarm. The problem may be with equipment E1215 or with equipment E1209. However, there is no alarm at equipment E1209, so the HMI determines that the problem is at equipment E1215 (pump). In this example, the throughput of device E1215 (the pump) has been reduced, and therefore there is a pressure drop at the connected downstream device E1214 (the boiler). Therefore, even though the alarm was raised by the measuring instrument at boiler E1214, in this case the HMI identifies the root cause of the alarm as pump E1215. The HMI assigns additional weight to this device for display purposes and places an alarm light bulb 3616 next to this device. At 3618, other alarm light bulbs representing other alarms discovered by the HMI are aggregated.

[0162] Thereafter, the HMI may allow the plant operator to correct the root cause analysis identified by the system. For example, the operator may manually select a different device and set that device as the root cause of the alarm instead of the root cause identified by the system, and in some embodiments may also manually identify the corresponding alarm chain. These corrections made by the operator may be fed back to the system to dynamically update the machine learning algorithm, thereby making the system more accurate for subsequent root cause analyses of alarms.

[0163] Reference now Fig.37In some embodiments, an HMI according to the present disclosure can aggregate alarms in a plant and provide a display screen 3700 showing a plant-level view of the aggregated alarms. In some embodiments, the system can aggregate alarms by obtaining a list of all devices that generate alarms and identifying a first device from the list. The system adds the device to a new alarm cluster and checks whether an adjacent device is also generating an alarm. If so, the system adds the adjacent device to the current alarm cluster. The process is repeated for the next device in the list until all devices and adjacent devices have been processed. In the example shown, the aggregated alarms at the plant level are depicted as a series of light bulbs 3702 in a designated area (e.g., upper left corner) of the screen 3700, although different locations or different icons or graphical symbols can certainly be used for these alarms. Each light bulb 3702 corresponds to a corresponding component (e.g., device, equipment, etc.) for which an alarm has been issued in the plant. As can be seen, there are five components 3704a, b, c, d, e for which alarms have been issued, each with a light bulb icon 3706 to indicate the alarm condition at the component.

[0164] Selecting (e.g., by clicking, double-clicking, etc.) one of the components 3704a-e causes the HMI to display information about the component, including which downstream components may be potential root causes of the alarm. In some embodiments, the system can determine the potential root cause of the alarm by identifying the alarm at the highest device level, then finding the source device connected to the device that also has the alarm. The system then drills down to the source device to find the connected subsequent devices that also have the alarm. This process is repeated until the bottom-most connected device with the alarm is found.

[0165] Fig.38An embodiment of an HMI that can dynamically zoom in (and out) of an asset based on the user / viewer's position on the asset hierarchy screen is illustrated. The viewer's position on the hierarchy screen is primarily determined by two factors: (1) the position on the ground as determined by screen coordinates (e.g., x and y coordinates), and (2) the viewer's height above the ground. Based on the viewer's position on the screen, relevant information surrounding the asset model is provided for viewing. For example, consider when the user is located at area 1 relative to the HMI screen, and an alarm condition exists at equipment 5, equipment 3, device 2, and device 4. In this example, the HMI screen will display equipment 7, 1, 2, 4, 5, and 6 and any relevant information about the equipment, but will not display equipment 8, 9, or 3, nor any equipment connected to equipment 3. Assume that the user now zooms in so that the user's position is at unit 3. In this example, the HMI screen will display equipment 7, 1, 2, 4, 5, and 6 as before, but will also display equipment 8, 9, or 3. Once the user moves further down to equipment 5, the HMI screen will display devices 1, 2, 3, 4, and 5 and any relevant information about these devices. The HMI screen automatically removes any equipment above the user's level.

[0166] Fig.39 An exemplary HMI display screen 3900 is shown that can dynamically zoom in (and out) on an asset based on the location of the user / viewer according to an embodiment of the present disclosure. The dynamic HMI display screen 3900 displays relevant information surrounding the asset model based on the zoom level determined by the user's current location and currently active alarms in the plant. Fig.39 In the example on the left, HMI display screen 3900 displays a connection between assets 3902 and 3904. In the middle example, the user has zoomed in, so HMI display screen 3900 remaps the connection to display an additional asset 3906 on the screen. In the example on the right, the user has zoomed in further, so HMI display screen 3900 again remaps the connection to display yet another additional asset 3908 on the screen.

[0167] Reference now Fig.40A and Fig.40B In addition to the alarm aggregation at the factory level discussed above, the HMI according to the embodiment of the present disclosure can also perform alarm aggregation at the area level. Fig.40AAn exemplary embodiment of such an HMI is illustrated, where the HMI uses a two-dimensional view of plant assets to display area-level alarm aggregation. Starting with the example on the left, solid lines (such as line 4002) represent horizontal connections within a plane, and dashed lines (such as line 4004) represent vertical connections across multiple planes (i.e., two-dimensional views). It can be seen that area-level assets A1 and A2 have alarm conditions indicated by high alarm icons and critical alarm icons, respectively. When the user selects A2 (e.g., by tapping, clicking, etc.), the HMI takes the user to the example on the right, where the user finds a critical alarm at asset U3 and a high alarm at asset U2.

[0168] If the user selects U2, the HMI takes the user to Fig.40B Example on the left, where the user discovers a critical alarm at device U3E2. From here, the user will be able to visually see the source of the critical alarm at U2E1, which cascades through U3E1 and then U3E2. Assuming the user selects U2E1, the HMI takes the user to Fig.40B Example on the right, where the user will be shown device-level assets, including the source D1 of the problem that caused the alarm. Selecting D1 will provide the user with information about that device, any trends seen on that device, any support and guidance available for that device, and any knowledge captured about that device.

[0169] Therefore, as described herein, embodiments of the present disclosure provide systems and methods for controlling industrial process automation and control systems. The method and system automatically and by using machine learning (ML) models and algorithms, extract plant assets from engineering drawings and other plant engineering data sources; establish asset relationships to create a plant asset registry and build an asset hierarchy based on plant assets; generate an ontology knowledge base based on the plant asset hierarchy; and provide an HMI for controlling industrial processes based on the plant asset hierarchy and the ontology knowledge base.

[0170] These embodiments of the present disclosure may include a special-purpose computer, which includes various computer hardware, as described in more detail below. Embodiments within the scope of the present disclosure also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a special-purpose computer, and include computer storage media and communication media. By way of example and not limitation, computer storage media include both volatile and non-volatile, removable and non-removable media for storing information (such as computer-readable instructions, data structures, program modules or other data) implemented in any method or technology. Computer storage media is non-temporary, and includes but is not limited to random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk ROM (CD-ROM), digital versatile disk (DVD) or other optical disk storage devices, solid-state drives (SSD), cassettes, tapes, magnetic disk storage devices or other magnetic storage devices, or can be used to carry or store desired non-temporary information in the form of computer-executable instructions or data structures and any other media that can be accessed by a computer. When information is transferred or provided to a computer over a network or another communications connection (hardwired, wireless, or a combination of hardwired or wireless), the computer properly views the connection as a computer-readable medium. Therefore, any such connection is properly referred to as a computer-readable medium. The above combinations should also be included within the scope of computer-readable media. Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a certain function or group of functions.

[0171] The following discussion is intended to provide a brief, general description of a suitable computing environment in which various aspects of the present disclosure may be implemented. Although not required, various aspects of the present disclosure will be described in the general context of computer-executable instructions (such as program modules) executed by computers in a network environment. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code components for performing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps.

[0172] Those skilled in the art will appreciate that aspects of the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, etc. Aspects of the present disclosure can also be practiced in a distributed computing environment in which tasks are performed by local and remote processing devices linked (by hardwired links, wireless links, or by a combination of hardwired or wireless links) through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0173] Exemplary systems for implementing various aspects of the present disclosure include a dedicated computing device in the form of a conventional computer, including a processing unit, a system memory, and a system bus that couples various system components including the system memory to the processing unit. The system bus can be any of a variety of types of bus structures, including a memory bus or a memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes computer storage media, including non-volatile and volatile memory types. A basic input / output system (BIOS) can be stored in a ROM, which contains basic routines that help transfer information between elements in a computer (such as during startup). In addition, a computer can include any device (e.g., a computer, a laptop computer, a tablet computer, a PDA, a mobile phone, a mobile phone, a smart TV, etc.) that can wirelessly receive an IP address from the Internet or wirelessly send an IP address to the Internet.

[0174] The computer may also include a magnetic hard drive for reading from and writing to a magnetic hard drive, a disk drive for reading from or writing to a removable disk, and an optical drive for reading from or writing to a removable optical disk (such as a CD-ROM or other optical media). The magnetic hard drive, the disk drive, and the optical drive are connected to the system bus via a hard drive interface, a disk drive interface, and an optical drive interface, respectively. The drive and its associated computer-readable medium provide the computer with non-volatile storage of computer-executable instructions, data structures, program modules, and other data. Although the exemplary environment described herein employs a magnetic hard drive, a removable disk, and a removable optical disk, other types of computer-readable media for storing data may also be used, including magnetic tapes, flash memory cards, digital video disks, Bernoulli tape cassettes, RAM, ROM, SSDs, etc.

[0175] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0176] The program code device comprising one or more program modules can be stored on a hard disk, a disk, an optical disk, a ROM and / or a RAM, including an operating system, one or more application programs, other program modules and program data. The user can input commands and information into the computer through a keyboard, a pointing device or other input devices (such as a microphone, a joystick, a game controller, a satellite antenna, a scanner, etc.). These and other input devices are usually connected to the processing unit through a serial port interface coupled to a system bus. Alternatively, the input device can be connected through other interfaces such as a parallel port, a game port or a universal serial bus (USB). A monitor or another display device is also connected to the system bus via an interface such as a video adapter. In addition to the monitor, a personal computer also generally includes other peripheral output devices (not shown), such as a speaker and a printer.

[0177] One or more aspects of the present disclosure may be embodied in computer executable instructions (i.e., software), routines or functions stored in system memory or non-volatile memory as applications, program modules and / or program data. Alternatively, the software may be stored remotely, such as on a remote computer with a remote application. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types when executed by a processor in a computer or other device. Computer executable instructions may be stored on one or more tangible, non-temporary computer-readable media (e.g., hard disks, optical disks, removable storage media, solid-state memory, RAM, etc.) and executed by one or more processors or other devices. As will be appreciated by those skilled in the art, in various embodiments, the functionality of program modules may be combined or distributed as needed. In addition, the functionality may be embodied in whole or in part in firmware or hardware equivalents (such as integrated circuits, application specific integrated circuits, field programmable gate arrays (FPGAs), etc.).

[0178] A computer can operate in a networked environment using a logical connection with one or more remote computers. The remote computer can each be another personal computer, a tablet computer, a PDA, a server, a router, a network PC, a peer device or other public network node, and typically includes many or all of the elements described above relative to the computer. Logical connections include local area networks (LANs) and wide area networks (WANs) presented here by way of example and not limitation. Such networked environments are common in office-wide or enterprise-wide computer networks, intranets and the Internet.

[0179] When used in a LAN networking environment, the computer is connected to a local network through a network interface or adapter. When used in a WAN networking environment, the computer may include a modem, a wireless link, or other components for establishing communications on a wide area network (such as the Internet). The modem (which may be internal or external) is connected to the system bus via a serial port interface. In a networking environment, program modules or portions thereof depicted relative to the computer may be stored in a remote memory storage device. It should be understood that the network connections shown are exemplary, and other components for establishing communications on a wide area network may be used.

[0180] Preferably, the computer executable instructions are stored in a memory such as a hard drive and executed by the computer. Advantageously, the computer processor has the capability to perform all operations (eg, execute the computer executable instructions) in real time.

[0181] Unless otherwise noted, the execution / performance order of the operations in the embodiments of the present disclosure shown and described herein is not necessary. That is, unless otherwise noted, operations may be performed in any order, and embodiments of the present disclosure may include more or less operations than those disclosed herein. For example, it is contemplated that performing a particular operation before, at the same time as, or after another operation is within the scope of the various aspects of the present disclosure.

[0182] Embodiments of the present disclosure may be implemented with computer executable instructions. Computer executable instructions may be organized into one or more computer executable components or modules. Aspects of the present disclosure may be implemented with such components or modules of any number and organization. For example, aspects of the present disclosure are not limited to specific computer executable instructions or specific components or modules shown in the figures and described herein. Other embodiments of the present disclosure may include different computer executable instructions or components having more or less functionality than shown and described herein.

[0183] When introducing elements of aspects of the present disclosure or embodiments thereof, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements.

[0184] Having described in detail various aspects of the present disclosure, it will be apparent that many modifications and changes may be made without departing from the scope of various aspects of the present disclosure as defined in the appended claims. As various changes may be made in the above-described structures, products and methods without departing from the scope of various aspects of the present disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative rather than limiting.

Claims

1. A control system for an industrial plant, comprising: one or more processors; as well as a memory unit communicatively coupled to the one or more processors and storing thereon processor-executable instructions that, when executed by the one or more processors, cause the control system to: executing a process of importing an engineering drawing of a unit of an industrial plant, the engineering drawing including symbols representing assets of the industrial plant; performing a process of extracting one or more assets from the engineering drawing using machine learning to identify the one or more assets, the one or more assets including equipment, instruments, connectors, and wiring that relates the equipment, instruments, and connectors to one another; performing a process of determining one or more relationships between devices, instruments, connectors, and lines to each other using machine learning to identify the one or more relationships; Performs the process of creating flow charts from equipment, instruments, connectors and wiring and the relationships between equipment, instruments, connectors and wiring; and A process of creating a data table for the flowchart is performed, the data table including component graphs constituting the flowchart and also including nodes and edges constituting each component graph, wherein the data table and its contents are used to generate an HMI for controlling the industrial plant.

2. The control system of claim 1, wherein the process of extracting one or more assets from the engineering drawing comprises a geometry-based extraction process and a machine learning-based classification process.

3. The control system of claim 1 , wherein the processor executable instructions further cause the control system to execute a process of extracting one or more control loops from the engineering drawing based on one or more of devices, instruments, connectors, and wiring.

4. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to perform a process of extracting a unit identifier from the engineering drawing, wherein the unit identifier uniquely identifies a particular unit from among a plurality of units in the industrial plant.

5. The control system according to claim 1, wherein: The processor executable instructions also cause the control system to execute a process of extracting a drawing number and a revision number from the engineering drawing, wherein the drawing number uniquely identifies the engineering drawing from a plurality of engineering drawings of the industrial plant, and the revision number indicates a revision of the engineering drawing, and the revision number is incremented each time there is a change in the engineering drawing.

6. The control system according to claim 4, wherein: The processor-executable instructions further cause the control system to perform a process of assigning a unique identifier to each device, instrument, connector, and line of the engineering drawing using a unit identifier from the engineering drawing.

7. The control system according to claim 1, wherein: The process of determining one or more relationships between the devices, instruments, connectors and lines includes a process of generating a line-line diagram, a process of generating a device-line diagram, a process of generating a connector-line diagram and a process of generating an instrument-line diagram.

8. The control system according to claim 7, wherein: The processor-executable instructions further cause the control system to create a flow chart of the engineering diagram by executing a process that merges the wire-wire diagram, the device-wire diagram, the connector-wire diagram, and the instrument-wire diagram with one another.

9. The control system according to claim 7, wherein: The processor-executable instructions further cause the control system to perform a process of merging flow diagrams from a plurality of engineering drawings of the industrial plant to create a flow diagram of the industrial plant.

10. The control system according to claim 1, wherein: The processor-executable instructions further cause the control system to perform a process of displaying devices, instruments, connectors, and wiring and relationships between the devices, instruments, connectors, and wiring in an asset hierarchy.

11. A method of controlling an industrial plant, comprising: inputting an engineering drawing of a unit of an industrial plant, the engineering drawing including symbols representing assets of the industrial plant; extracting one or more assets from the engineering drawing using machine learning to identify the one or more assets, the one or more assets comprising equipment, instruments, connectors, and wiring that relates the equipment, instruments, and connectors to one another; using machine learning to determine one or more relationships between devices, instruments, connectors, and lines to identify the one or more relationships; Create flow charts based on equipment, instruments, connectors and lines and the relationships between them; and A data table is created for the flowchart, the data table including component graphs constituting the flowchart and also including nodes and edges constituting each component graph, wherein the data table is used to generate an HMI for controlling the industrial plant.

12. The method of claim 11, wherein extracting one or more assets from the engineering drawing comprises performing a geometry-based extraction process and a machine learning-based classification process.

13. The method of claim 11, further comprising extracting one or more control loops from the engineering drawing based on one or more of devices, instruments, connectors, and wiring.

14. The method of claim 11, further comprising extracting a unit identifier from the engineering drawing, wherein the unit identifier uniquely identifies a particular unit from among a plurality of units in the industrial plant.

15. The method according to claim 11 also includes extracting a drawing number and a revision number from the engineering drawing, wherein the drawing number uniquely identifies the engineering drawing from multiple engineering drawings of the industrial plant, and the revision number indicates a revision of the engineering drawing, and the revision number is incremented each time there is a change in the engineering drawing.

16. The method of claim 14, further comprising assigning a unique identifier to each device, instrument, connector, and line of the engineering drawing using a unit identifier from the engineering drawing.

17. The method of claim 11, wherein determining one or more relationships between the devices, instruments, connectors, and lines comprises generating a line-line diagram, a device-line diagram, a connector-line diagram, and an instrument-line diagram.

18. The method of claim 17, wherein creating the flowchart of the engineering drawing includes merging the line-line diagram, the device-line diagram, the connector-line diagram, and the instrument-line diagram with one another.

19. The method of claim 17, further comprising merging flowcharts from a plurality of engineering drawings of the industrial plant to create a flowchart of the industrial plant.

20. The method of claim 11, further comprising displaying devices, instruments, connectors, and wiring and relationships between the devices, instruments, connectors, and wiring in an asset hierarchy.

21. A computer readable medium storing computer readable instructions for causing one or more processors to perform the method according to any one of claims 11 to 20.

Images