Method for generating learning data for identification model of piping and instrument diagram, and a computer program recorded on a recording medium for executing the same

Abstract

The present invention proposes a method for generating training data for an artificial intelligence (AI) model for recognizing images related to a Piping and Instrument Diagram (P&ID). The method may include the steps of: acquiring a single image containing a Piping and Instrument Diagram (P&ID) from a database (DB); preparing a list containing one or more symbols and texts that may be included in the Piping and Instrument Diagram (P&ID); and generating training data for an artificial intelligence (AI) model for recognizing symbols and texts related to the Piping and Instrument Diagram (P&ID) from an image by applying modifications to the symbols and texts included in the acquired image, or by replacing the symbols or texts included in the acquired image with the symbols or texts included in the list. According to the present invention, by augmenting the training data for recognizing images related to the Piping and Instrument Diagram (P&ID), it becomes possible to construct an artificial intelligence (AI) model with an improved recognition rate.

Description

Technology Field

[0001] The present invention relates to industrial processes technology. More specifically, it relates to a method for generating learning data for an Artificial Intelligence (AI) model for recognizing images of Piping and Instrument Diagrams (P&ID), and a computer program recorded on a recording medium for executing the same. Background Technology

[0002] A Piping and Instrumentation Diagram (P&ID) is a technical drawing that visually represents the structure and interactions of pipelines, devices, instruments, control systems, and other components in an industrial process system.

[0003] Process Piping and Instrumentation Diagrams (P&ID) visualize the flow of industrial processes to verify how fluids, such as liquids or gases, move through pipes and equipment. They provide reference information necessary for engineering, construction, maintenance, and operation, visually represent the location, role, and control methods of various equipment and instruments, and can help in understanding key devices and emergency procedures related to facility safety.

[0004] A Piping and Instrumentation Diagram (P&ID) may include equipment, pipe lines, instruments, tags, flow directions, lines, and control loops.

[0005] In addition, process piping and instrumentation diagrams (P&ID) can be prepared and interpreted in accordance with standards specified by ISO (International Organization for Standardization) 14617, ISA (International Society of Automation) S5.1 or ASME (American Society of Mechanical Engineers).

[0006] Meanwhile, Artificial Intelligence (AI) refers to a technology that artificially implements some or all of human learning, reasoning, and perceptual abilities using computer programs.

[0007] Large Language Models (LLMs) are artificial intelligence technologies capable of understanding and generating language based on large-scale data. LLMs can be implemented based on Transformers with Artificial Neural Network (ANN) structures. Furthermore, LLMs are characterized by their ability to understand the relationships between context and words by utilizing self-attention. Representative LLMs include OpenAI's GPT (Generative Pre-trained Transformer), Google's PaLM (Pathways Language Model), and Meta's LLaMA (Large Language Model Meta AI).

[0008] Vision recognition is an artificial intelligence technology capable of understanding and analyzing images or videos. Such vision recognition can be utilized in image classification, object detection, segmentation, face recognition, action recognition, image captioning, and optical character recognition.

[0009] The YOLO (You Only Look Once) model, one of the technologies related to vision recognition, is a deep learning model designed for real-time object detection. Based on a single neural network, the YOLO model can detect objects by passing an image through it only once. Additionally, the YOLO model is characterized by its ability to perform multi-object detection by dividing the target image into small grids and simultaneously predicting the probability of an object existing in each grid cell.

[0010] Meanwhile, a graph is a data structure composed of a finite set of non-empty nodes and a finite set of edges consisting of pairs of nodes. Here, the nodes of a graph are sometimes referred to as vertices or points, and the edges are sometimes referred to as edges, lines, or relationships.

[0011] Graph databases (GDBs) represent data and the relationships between data as nodes and edges of a graph, and are non-relational databases that do not use a structured query language (No SQL). Depending on their purpose of use, these graph databases can be classified into properties graphs and Resource Description Framework (RDF) graphs.

[0012] Meanwhile, traditional paper or CAD (Computer Aided Design)-based Piping and Instrumentation Diagrams (P&ID) had various limitations in terms of information management or utilization.

[0013] Recently, various means have been proposed to digitize Piping and Instrumentation Diagrams (P&ID) to improve the efficiency, safety, and data usability of industrial processes; however, so far, this has been limited to converting P&IDs related to specific industrial fields into digital data according to predefined rules.

[0014] Furthermore, the means to support the utilization of digitized data related to Piping and Instrumentation Diagrams (P&ID) for various purposes by employing technologies such as artificial intelligence (AI) and graphs during the digitization process are still insufficient. Prior art literature

[0015] Korean Registered Patent Publication No. 10-2757455, 'Device and method for automatically generating process piping instrumentation diagrams for a water spray smart plant, and a computer-readable medium recording the method', (Published Jan. 21, 2025) The problem to be solved

[0016] One objective of the present invention is to provide a method for generating training data for an artificial intelligence (AI) model for recognizing images of process piping and instrumentation diagrams (P&ID).

[0017] Another objective of the present invention is to provide a computer program recorded on a recording medium for implementing a method of generating training data for an artificial intelligence (AI) model for recognizing images of process piping and instrumentation diagrams (P&ID).

[0018] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0019] To achieve the technical objectives described above, the present invention proposes a method for generating training data for an artificial intelligence (AI) model for recognizing images of process piping and instrumentation diagrams (P&ID).

[0020] The above method may include the steps of: acquiring an image containing a Piping and Instrument Diagram (P&ID) from a database (DB); preparing a list containing one or more symbols and texts that may be included in the Piping and Instrument Diagram (P&ID); and generating training data for an Artificial Intelligence (AI) model to recognize symbols and texts related to the Piping and Instrument Diagram (P&ID) from an image by applying modifications to the symbols and texts included in the acquired image, or by replacing the symbols or texts included in the acquired image with the symbols or texts included in the list.

[0021] Specifically, the step of acquiring the image may involve the artificial intelligence (AI) model identifying a project related to the process piping and instrumentation diagram (P&ID) to be used, generating a query including keywords related to the identified project, and then querying the database (DB) using the generated query to acquire the image.

[0022] The step of generating the above training data may involve identifying one or more symbols related to a process piping and instrumentation diagram (P&ID) from the acquired image, applying geometric deformation by randomly performing one or more operations among rotation, scaling, translation, and flipping on all or part of the identified one or more symbols, readjusting the lines included in the process piping and instrumentation diagram (P&ID) so that the lines connected to the symbols before the geometric deformation are applied remain connected to the symbols after the geometric deformation is applied, and applying image deformation by randomly performing one or more operations among brightness or contrast adjustment, color jittering, background noise addition, and grid overlay addition on the image with the readjusted lines.

[0023] Alternatively, the step of generating the training data may search for a symbol among one or more symbols included in the list that can replace the symbol identified from the image, and after replacing the symbol identified from the image with the searched symbol, readjust the lines included in the process piping and instrumentation diagram (P&ID) so that the line connected to the symbol before the replacement is connected as is to the symbol after the replacement.

[0024] The step of generating the above training data may involve identifying one or more texts related to a process piping and instrumentation diagram (P&ID) from the acquired image, applying text variations by randomly performing one or more operations among font variation, warping and skewing, and blur effect on all or part of the identified text, and applying image variations by randomly performing one or more operations among texture variation, annotation, and color reverse on the image to which the text variations have been applied.

[0025] After the step of generating the training data, the above method may further include a step of verifying whether the symbols and lines included in the generated training data are correctly positioned from a fluid dynamics perspective.

[0026] In this case, the verification step may verify pressure drop, stagnation zone, turbulence, cavitation, fluid separation, and multi-phase flow for the Process Piping and Instrumentation Diagram (P&ID) by setting fluid properties and boundary conditions based on attribute values ​​assigned to the project related to the Process Piping and Instrumentation Diagram (P&ID) where the artificial intelligence (AI) model is to be used, and by performing Computational Fluid Dynamics (CFD) analysis on the generated training data according to the set fluid properties and boundary conditions. Here, the fluid properties may include one or more of the fluid type, density, viscosity, compressibility, temperature, and pressure. Additionally, the boundary conditions may include one or more of the fluid flow rate, velocity, atmospheric pressure, and friction coefficient within the pipe.

[0027] The verification step described above may divide the generated training data into multiple regions based on the type of symbol to be included in each region, apply a guideword to each divided region to derive risk factors for each region, and perform a risk assessment of the generated training data in correspondence with the derived risk factors for each region. Here, the guideword may be a standardized keyword to identify abnormal conditions that may occur depending on process flow or operating conditions.

[0028] Additionally, the verification step may involve obtaining a Process Flow Diagram (PFD) stored in the database (DB) that is matched with the Process Piping and Instrumentation Diagram (P&ID), and performing verification by comparing the generated training data with the obtained Process Flow Diagram (PFD) according to a pre-set checklist. Here, the checklist may include one or more pieces of information regarding equipment configuration, piping configuration, instrument configuration, process conditions, and safety devices.

[0029] To achieve the technical objectives described above, the present invention proposes a computer program recorded on a recording medium for executing a method of generating training data for an artificial intelligence (AI) model for recognizing images of a process piping and instrumentation diagram (P&ID). The computer program may be combined with a computing device comprising a memory and a processor for processing instructions residing in the memory. Furthermore, the computer program may be recorded on a recording medium to execute the steps of: the processor acquiring an image containing a process piping and instrumentation diagram (P&ID) from a database (DB); the processor preparing a list containing one or more symbols and texts that may be included in the process piping and instrumentation diagram (P&ID); and the processor applying modifications to the symbols and texts included in the acquired image, or replacing the symbols or texts included in the acquired image with the symbols or texts included in the list, thereby generating training data for an artificial intelligence (AI) model for recognizing symbols and texts related to the process piping and instrumentation diagram (P&ID) from an image.

[0030] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0031] According to embodiments of the present invention, by augmenting training data for recognizing images of process piping and instrumentation diagrams (P&ID), it is possible to construct an artificial intelligence (AI) model with an improved recognition rate.

[0032] Specifically, since the creation methods of Piping and Instrumentation Diagrams (P&ID) vary by industrial site or engineering firm, the shapes and sizes of symbols included in the P&IDs, as well as the languages ​​and fonts of the text, vary; however, the amount of data available from external sources is limited due to reasons such as project security. By securing P&ID images from diverse environments through the augmentation of training data, the present invention enables the acquisition of an artificial intelligence (AI) model capable of providing generalized performance in actual operating environments.

[0033] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0034] FIG. 1 is an illustrative diagram showing an example of a graph generated according to a digitization system according to an embodiment of the present invention. FIGS. 2 and FIGS. 3 are configuration diagrams showing a digitization system according to some embodiments of the present invention. FIG. 4 is a logical configuration diagram of a digitization server according to one embodiment of the present invention. FIGS. 5 and 6 are exemplary diagrams for explaining the process of augmenting learning data according to an embodiment of the present invention. FIGS. 7 to 9 are illustrative diagrams for explaining the process of generating a graph according to an embodiment of the present invention. FIGS. 10 and 11 are illustrative diagrams for explaining the process of verifying data according to one embodiment of the present invention. FIG. 12 is an illustrative diagram for explaining the process of converting a graph into an ontology according to one embodiment of the present invention. FIG. 13 is an exemplary diagram illustrating the process of automatically generating a process piping and instrumentation diagram (P&ID) image according to one embodiment of the present invention. FIGS. 14 and FIGS. 15 are exemplary diagrams for explaining the process of answering a user's query according to an embodiment of the present invention. FIG. 16 is a hardware configuration diagram of a digitization server according to one embodiment of the present invention. FIG. 17 is a flowchart of a digitization method according to one embodiment of the present invention. FIG. 18 is a flowchart for specifically explaining a method for generating training data according to one embodiment of the present invention. FIG. 19 is a flowchart for specifically explaining a graph generation method according to one embodiment of the present invention. FIG. 20 is a flowchart for specifically explaining a data verification method according to one embodiment of the present invention. FIG. 21 is a flowchart for specifically explaining an ontology conversion method according to one embodiment of the present invention. FIG. 22 is a flowchart for specifically explaining a method for automatically generating process piping and instrumentation diagram (P&ID) images according to one embodiment of the present invention. FIG. 23 is a flowchart for specifically explaining a process piping and instrumentation diagram (P&ID) management method according to one embodiment of the present invention. Specific details for implementing the invention

[0035] It should be noted that technical terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Furthermore, unless specifically defined otherwise in this specification, technical terms used herein shall be interpreted in the sense generally understood by those skilled in the art to which the invention pertains, and shall not be interpreted in an overly broad or overly narrow sense. Additionally, if a technical term used in this specification is an incorrect technical term that fails to accurately express the spirit of the invention, it shall be understood as being replaced by a technical term that can be correctly understood by those skilled in the art. Moreover, general terms used in the invention shall be interpreted according to their prior definitions or the context, and shall not be interpreted in an overly narrow sense.

[0036] Additionally, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "composed of" or "have" should not be interpreted as necessarily including all of the various components or steps described in the specification, and should be interpreted as potentially including some of the components or steps, or including additional components or steps.

[0037] Additionally, terms including ordinal numbers, such as first, second, etc., used herein may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0038] When it is stated that one component is "connected" or "connected" to another component, it may be directly connected or connected to that other component, or there may be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0039] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are given the same reference number, and redundant descriptions thereof will be omitted. Furthermore, in describing the present invention, if it is determined that a detailed description of related known technology may obscure the essence of the present invention, such detailed description will be omitted. Additionally, it should be noted that the attached drawings are intended only to facilitate an easy understanding of the concept of the present invention and should not be interpreted as limiting the concept of the present invention. The concept of the present invention should be interpreted as extending to all modifications, equivalents, and substitutions other than those shown in the attached drawings.

[0041] As mentioned above, various means have been proposed to digitize Piping and Instrumentation Diagrams (P&ID) to improve process efficiency, safety, and data usability; however, so far, these efforts have been limited to converting P&IDs related to specific industrial fields into digital data according to predefined rules. Furthermore, there is still a lack of means to support the utilization of digitized P&ID data for various purposes by employing technologies such as artificial intelligence (AI) and graphs during the digitization process.

[0042] To overcome this situation, the present invention proposes various means to convert, verify, and manage data related to process piping and instrumentation diagrams (P&ID) by utilizing technologies such as artificial intelligence (AI) and graphs.

[0044] FIG. 1 is an illustrative diagram showing an example of a graph generated according to a digitization system according to an embodiment of the present invention.

[0045] FIG. 1 illustrates a graph automatically generated by vision recognition of an image containing a process piping and instrumentation diagram (P&ID) as an example of features that can be performed by a digitization system according to the present invention. As shown in FIG. 1, a digitization system according to various embodiments of the present invention can provide means for converting, verifying, and managing data related to a process piping and instrumentation diagram (P&ID) into other types of data.

[0046] For example, a digitization system according to one embodiment of the present invention can recognize an image of a process piping and instrumentation diagram (P&ID) and generate a graph. Through this, beyond simply digitizing data related to the process piping and instrumentation diagram (P&ID), it becomes possible to obtain data that can be utilized for analyzing connection relationships of piping and instrumentation, verifying network validity, simulating flow rates, analyzing the impact of devices and piping, managing change history, and performing emergency simulations.

[0047] A digitization system according to another embodiment of the present invention can augment training data for recognizing images of Process Piping and Instrumentation Diagrams (P&ID). Since the creation methods of P&IDs vary by industrial site or engineering company, the shapes and sizes of symbols included in the P&IDs, as well as the language and fonts of the text, vary; however, the number of data available from external sources is limited due to reasons such as project security. By securing P&ID images from various environments through the augmentation of training data, the present invention can secure an artificial intelligence (AI) model capable of providing generalized performance in actual operating environments.

[0048] A digitization system according to another embodiment of the present invention can convert graphs related to process piping and instrumentation diagrams (P&ID) into an ontology. Through this, it is possible to respond more flexibly to sophisticated queries related to process piping and instrumentation diagrams (P&ID) and to secure data that can be utilized for intelligent inference, linkage and integration of heterogeneous data, semantic-based event analysis, and anomaly detection.

[0049] A digitization system according to another embodiment of the present invention can verify a vision-recognized graph from an image based on XML data related to a Piping and Instrumentation Diagram (P&ID). Through this, a foundation can be established for digital transformation and automation related to the Piping and Instrumentation Diagram (P&ID).

[0050] A digitization system according to another embodiment of the present invention can manage piping and instrumentation based on graphs generated from a Process Piping and Instrumentation Diagram (P&ID). Through this, it can provide more accurate answers based on graphs and artificial intelligence (AI) in response to user queries related to identifying connection relationships in the P&ID, tracing flow paths, inspecting piping and instrumentation, strengthening emergency response, analyzing the impact of changes, and verifying regulatory compliance.

[0051] A digitization system according to another embodiment of the present invention can automatically generate Piping and Instrumentation Diagram (P&ID) images using artificial intelligence (AI). This reduces the time and cost of P&ID-related tasks that were previously performed manually. Furthermore, it ensures consistency in P&ID design, prevents errors, and allows for more flexible responses to changes in P&ID design. Moreover, it expands the possibilities for digital twins and data linkage.

[0052] Hereinafter, a digitization system according to various embodiments of the present invention having these characteristics will be described in more detail.

[0054] FIGS. 2 and FIGS. 3 are configuration diagrams showing a digitization system according to some embodiments of the present invention.

[0055] As illustrated in FIG. 2, a digitization system according to one embodiment of the present invention may be configured to include a digitization server (100) and a storage (200). The digitization server (100) and the storage (200) of the digitization system may transmit and receive data through a secure line that directly connects each device to each other.

[0056] Alternatively, as illustrated in FIG. 3, the digitization server (100) and storage (200) of the digitization system according to another embodiment of the present invention may transmit and receive data through a network that combines one or more of a public wired communication network and a mobile communication network.

[0057] In this case, the shared wired network may include one or more of Ethernet, Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH), but is not limited thereto. Additionally, the mobile network may include one or more of Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), and 5th / 6th generation mobile telecommunication, but is not limited thereto.

[0058] To describe each component, the digitization server (100) can convert, verify, and manage data related to the process piping and instrumentation diagram (P&ID) into other types of data.

[0059] The digitization server (100) is a main device for implementing features according to various embodiments of the present invention, and is not limited to the term server, but may be any one of fixed computing devices such as a desktop or a workstation, but is not limited thereto.

[0060] The specific configuration and operation of such a digitization server (100) will be explained in more detail later with reference to FIGS. 4 to 23.

[0061] With the following configuration, the storage (200) can temporarily or permanently store data to be used in the digitization server (100). Characteristically, the storage (200) according to one embodiment of the present invention may be configured to include a Graph Database (GDB) and a Resource Description Framework (RDF) storage. Additionally, the storage (200) may be configured to include a general database (DB) for storing various types of data that do not correspond to a graph or ontology.

[0062] Specifically, the graph database (GDB) of the storage (200) can store data related to graphs used or created by the digitization server (100). For example, the graph database (GDB) may be implemented based on Neo4j, but is not limited thereto. Additionally, the graph database (GDB) can perform operations such as searching, storing, and exploring graphs in response to a request from the digitization server (100), and provide the result of the performed operations to the digitization server (100).

[0063] The RDF repository of the repository (200) can store RDF-based triples regarding an ontology generated by the digitization server (100). For example, the RDF repository may be implemented based on Apache Jena, but is not limited thereto. Additionally, the RDF repository can perform operations such as storing, searching, and analyzing semantic relationships of the ontology at the request of the digitization server (100), and provide the result of the performed operations to the digitization server (100).

[0064] As the components of the digitization system described above merely represent functionally distinct elements, two or more components may be implemented together in an actual physical environment, or a single component may be implemented separately in an actual physical environment. For example, the storage (200) may be implemented as an integrated component of the digitization server (100).

[0066] Hereinafter, the logical configuration of the digitization server (100) having the characteristics described above will be explained in more detail.

[0067] FIG. 4 is a logical configuration diagram of a digitization server according to one embodiment of the present invention.

[0068] As illustrated in FIG. 4, a digitization server (100) according to one embodiment of the present invention may be configured to include a communication unit (105), an input / output unit (110), an artificial intelligence management unit (120), a database management unit (130), a data conversion unit (140), and a P&ID service unit (150).

[0069] As the components of the digitization server (100) described above are merely functionally distinct elements, two or more components may be implemented as integrated elements in an actual physical environment, or a single component may be implemented as separated elements in an actual physical environment.

[0070] To explain each component, the communication unit (105) can transmit and receive data related to the process piping and instrumentation diagram (P&ID) with the storage unit (200). Not limited thereto, the communication unit (105) may also transmit and receive data related to the process piping and instrumentation diagram (P&ID) with another external storage unit (not shown).

[0071] With the following configuration, the input / output unit (110) can receive or output various types of data required for the operation of the digitization server (100).

[0072] Specifically, the input / output unit (110) can receive reference values, threshold values, data lists, etc., from an external source that are necessary for augmenting training data, generating graphs, verifying graphs, generating ontologies, and generating process piping and instrumentation diagram (P&ID) images. Additionally, the input / output unit (110) can receive queries from an external source for managing process piping and instrumentation diagrams (P&ID).

[0073] Additionally, the input / output unit (110) can output generated or converted training data, graphs, ontology, and process piping and instrumentation diagram (P&ID) images, etc., to the outside. Additionally, the input / output unit (110) can output answers to queries analyzed based on graphs and ontology based on the process piping and instrumentation diagram (P&ID) to the outside.

[0075] With the following configuration, the artificial intelligence management unit (120) can manage artificial intelligence (AI) for processing data regarding process piping and instrumentation diagrams (P&ID).

[0076] Specifically, an artificial intelligence management unit (120) according to one embodiment of the present invention can manage an artificial intelligence (AI) model that can be used to automatically generate a graph from an image containing a process piping and instrumentation diagram (P&ID). To this end, the artificial intelligence management unit (120) may be configured to include a data management module (122) and a model learning module (124).

[0077] First, the data management module (122) of the artificial intelligence management department (120) can manage data related to artificial intelligence (AI).

[0078] Specifically, the data management module (122) can perform labeling on data that can be used for artificial intelligence (AI) learning. For example, the data management module (122) can perform labeling on symbols, lines, and text detected from process piping and instrumentation diagram (P&ID) images.

[0079] The data management module (122) can manage versions of data that can be used for artificial intelligence (AI) learning. For example, the data management module (122) can use a Data Version Control (DVC) tool to manage versions of a dataset for training an artificial intelligence (AI) model and track change history. Additionally, the data management module (122) can manage the correlation between the training data and the training results.

[0080] Characteristically, a data management module (122) according to one embodiment of the present invention can augment learning data from an image containing a process piping and instrumentation diagram (P&ID). Hereinafter, with reference to FIGS. 5 and 6, the process of the data management module (122) augmenting learning data will be described in more detail.

[0082] FIGS. 5 and 6 are exemplary diagrams for explaining the process of augmenting learning data according to an embodiment of the present invention.

[0083] Referring to FIG. 5, a data management module (122) according to one embodiment of the present invention can obtain an image containing a process piping and instrumentation diagram (P&ID) from a database (DB). In this case, the database (DB) may be a general database for storing data that does not correspond to a graph or ontology in the storage (200), but is not limited thereto and may be a graph database (GDB).

[0084] For example, the data management module (122) can identify a project related to a process piping and instrumentation diagram (P&ID) to which an artificial intelligence model (AI) will be used. The data management module (122) can generate a query by including keywords related to the identified project. Then, the data management module (122) can query a database (DB) using the generated query to obtain a single image containing the process piping and instrumentation diagram (P&ID).

[0085] The data management module (122) can prepare a list containing one or more symbols and text that may be included in a process piping and instrumentation diagram (P&ID). For this purpose, the list containing one or more symbols and text that may be included in a process piping and instrumentation diagram (P&ID) may be stored in a database (DB) by matching it with the domain of the process piping and instrumentation diagram (P&ID), or it may be managed directly within the data management module (122).

[0086] Here, symbols are graphic representation elements that express various facilities and functions, such as equipment, measuring instruments, and valves included in a Piping and Instrumentation Diagram (P&ID), using standardized shapes or symbols. Additionally, text is text information inserted to identify components or explain information regarding components, such as the names, line numbers, fluid types, process conditions, and instrument tags of equipment or piping included in the P&ID.

[0087] Additionally, the data management module (122) may apply direct modifications to all or part of the symbols and text included in the image of the process piping and instrumentation diagram (P&ID). Alternatively, the data management module (122) may replace all or part of the symbols and text included in the image of the process piping and instrumentation diagram (P&ID) with symbols or text included in a list. Through this, the data management module (122) can generate new training data (i.e., data augmentation) to train an artificial intelligence (AI) model for recognizing symbols and text related to the process piping and instrumentation diagram (P&ID) from the image.

[0088] First, we will explain the process of modifying the symbols included in the Piping and Instrumentation Diagram (P&ID) images.

[0089] The data management module (122) can identify one or more symbols related to the process piping and instrumentation diagram (P&ID) from an image obtained from a database (DB). The data management module (122) can apply geometric deformation to all or part of the identified symbols by randomly performing one or more operations such as rotation, scaling, translation, and flipping. Additionally, the data management module (122) can readjust the lines included in the process piping and instrumentation diagram (P&ID) so that the lines connected to the symbols before the geometric deformation is applied remain connected to the symbols after the geometric deformation is applied.

[0090] Here, a line is a graphic element represented by a line to indicate the flow path of a process medium, such as a fluid, gas, or vapor.

[0091] The data management module (122) can apply image transformations by randomly performing one or more operations such as brightness or contrast adjustment, color jittering, background noise addition, and grid overlay addition on an image in which lines have been readjusted.

[0092] Alternatively, the data management module (122) can search for a symbol among one or more symbols included in a prepared list that can replace a symbol identified from an image of a process piping and instrumentation diagram (P&ID). The data management module (122) can replace the symbol identified from the image with a symbol found from the list. And, the data management module (122) can readjust the lines included in the process piping and instrumentation diagram (P&ID) so that the line connected to the symbol before replacement is connected to the symbol after replacement.

[0093] Next, the process of modifying text included in the P&ID image is explained.

[0094] The data management module (122) can identify one or more texts related to process piping and instrumentation diagrams (P&ID) from an image obtained from a database (DB). The data management module (122) can apply text variations by randomly performing one or more operations among font variation, text warping and skewing, and blur effect on all or part of the identified texts.

[0095] The data management module (122) can apply image deformation by randomly performing one or more operations among texture variation, annotation, and color reverse on an image to which text deformation has been applied.

[0096] Next, referring to FIG. 6, the data management module (122) can verify whether the symbols and lines included in the newly generated training data are correctly arranged from a fluid dynamics perspective.

[0097] To this end, the data management module (122) can verify the newly generated training data by performing one or more of the following: verification using Computational Fluid Dynamics (CFD) simulation, verification through HAZOP (Hazard and Operability Study) analysis, and verification through comparison with a Process Flow Diagram (PFD).

[0098] Specifically, the data management module (122) can set fluid properties and boundary conditions based on attribute values ​​assigned to a process piping and instrumentation diagram (P&ID) related project in which an artificial intelligence (AI) model will be used. Additionally, the data management module (122) can verify pressure drop, stagnation zone, turbulence, cavitation, fluid separation, and multi-phase flow for the process piping and instrumentation diagram (P&ID) by performing computational fluid dynamics (CFD) analysis based on fluid properties and boundary conditions on the generated training data.

[0099] Here, fluid properties may include one or more of fluid type, density, viscosity, compressibility, temperature, and pressure. Additionally, boundary conditions may include one or more of fluid flow rate, velocity, atmospheric pressure, and friction coefficient within the pipe.

[0100] Meanwhile, the data management module (122) can divide the generated training data into multiple regions based on the type of symbol to be included in each region. For example, the data management module (122) can divide the generated training data (i.e., an image including a process piping and instrumentation diagram (P&ID)) into a pump region, a heat exchanger region, a reactor region, a storage tank region, etc.

[0101] The data management module (122) can derive risk factors for each area by applying guidewords to each divided area. Then, the data management module (122) can assess the risk level of the generated training data in response to the derived risk factors for each area.

[0102] Here, the guide word can be a standardized keyword to identify abnormal conditions that may occur depending on the process flow or operating conditions. For example, the guide word can be any one of "More," meaning "excessive flow," "Less," meaning "low flow," "No," meaning "no flow," "Reverse," meaning "reverse flow," and "Other," meaning "fluid contamination." Such guide words can be applied in accordance with the IEC (International Electrotechnical Commission) 61882 standard.

[0103] Meanwhile, the data management module (122) can obtain a process flow diagram (PFD) stored by matching it with a process piping and instrumentation diagram (P&ID) from a database (DB). The data management module (122) can perform verification by comparing the generated training data and the process flow diagram (PFD) obtained from the database (DB) according to a pre-set checklist.

[0104] Here, the checklist may include one or more of the following information regarding equipment configuration, piping configuration, instrument configuration, process conditions, and safety devices.

[0105] And, the data management module (122) can store the training data that is determined to be free of problems as a result of verification in the database (DB).

[0106] Referring again to FIG. 4, the following components are described. The model learning module (124) can train an artificial intelligence (AI) model that can be used to automatically generate a graph from an image containing a process piping and instrumentation diagram (P&ID).

[0107] Specifically, the model training module (124) can train a YOLO (You Only Look Once) model using a dataset consisting of images containing process piping and instrumentation diagrams (P&ID) managed by the data management module (122) and symbols contained in the process piping and instrumentation diagrams (P&ID). Here, the YOLO model is an artificial intelligence (AI) model for detecting objects from images containing process piping and instrumentation diagrams (P&ID).

[0108] The model training module (124) can train a Tesseract model using a dataset consisting of images containing process piping and instrumentation diagrams (P&ID) managed by the data management module (122) and text contained in the process piping and instrumentation diagrams (P&ID). Here, the Tesseract model is an artificial intelligence (AI) model for performing optical character recognition on images containing process piping and instrumentation diagrams (P&ID).

[0109] The model training module (124) may perform automatic machine learning to periodically train the YOLO model and the Tesseract model. The model training module (124) may manage the training history, parameters, and dataset versions for the YOLO model and the Tesseract model. Additionally, the model training module (124) may monitor the performance of the YOLO model and the Tesseract model and evaluate the performance of each model.

[0111] With the following configuration, the data conversion unit (130) can convert data related to the process piping and instrumentation diagram (P&ID) into other types of data.

[0112] Specifically, a data conversion unit (130) according to various embodiments of the present invention can convert an image containing a process piping and instrumentation diagram (P&ID) into a graph and verify the converted graph. The data conversion unit (130) can convert a graph related to the process piping and instrumentation diagram (P&ID) into an ontology. In addition, the data conversion unit (130) can generate an image of the process piping and instrumentation diagram (P&ID) based on the graph. To this end, the data conversion unit (130) may be configured to include a graph conversion module (132), a graph verification module (134), an ontology conversion module (136), and an image generation module (138).

[0114] First, with reference to FIGS. 7 to 9, the graph conversion module (132) will be described in more detail.

[0115] FIGS. 7 to 9 are illustrative diagrams for explaining the process of generating a graph according to an embodiment of the present invention.

[0116] Referring to FIG. 7, a graph conversion module (132) according to one embodiment of the present invention can automatically generate a graph by vision recognizing an image containing a process piping and instrumentation diagram (P&ID).

[0117] First, the graph conversion module (132) can preprocess an image containing a process piping and instrumentation diagram (P&ID).

[0118] Specifically, the graph conversion module (132) can adjust the size or resolution of an image containing a process piping and instrumentation diagram (P&ID) to correspond to a preset size or resolution.

[0119] The graph transformation module (132) can reduce noise contained in an image containing a process piping and instrumentation diagram (P&ID). To this end, the graph transformation module (132) can apply one or more of grayscale transformation, Gaussian blur, and bilateral filter to an image containing a process piping and instrumentation diagram (P&ID). Additionally, the graph transformation module (132) can convert the noise-removed image into a black and white image by binarizing it.

[0120] The graph conversion module (132) can separate a legend area from a black-and-white converted image based on the type of process piping and instrumentation diagram (P&ID) to be converted. Generally, the legend area has the shape of a rectangular box and is positioned on the right or bottom of the process piping and instrumentation diagram (P&ID), and has the characteristic of having symbols and text regularly arranged inside. Accordingly, the graph conversion module (132) can detect the contours of the black-and-white converted image to detect candidate areas of the rectangular box, and among the detected candidate areas, identify the area with the highest text density as the legend area.

[0121] The graph conversion module (132) can crop the separated legend area by row and then determine whether the symbols and text included in the cropped area by row are arranged according to a predefined regular expression. In this case, the regular expression may be defined based on the ISO (International Organization for Standardization) 10628 standard or the ANSI / ISA (American National Standards Institute / International Society of Automation) 5.1 standard.

[0122] If the symbols and text included in the cropped area are arranged according to a regular expression, the graph transformation module (132) can additionally map the symbols and text included in each cropped area according to the regular expression to a pre-configured legend ontology on the repository (200). Alternatively, if the arrangement of the symbols and text included in the cropped area does not follow a regular expression, the graph transformation module (132) can recognize the symbols and text included in each cropped area using a pre-trained Vision-Language Model (VLM), and then map the recognized symbols and text to a legend ontology.

[0123] Next, the graph conversion module (132) can recognize symbols, lines, and text related to the process piping and instrumentation diagram (P&ID) from the preprocessed image using an artificial intelligence (AI) model. Here, a symbol is a graphic representation element that represents various facilities and functions, such as equipment, measuring instruments, and valves included in the process piping and instrumentation diagram (P&ID), as a standardized shape or symbol. A line is a graphic element represented as a line to indicate the flow path of a process medium, such as a fluid, gas, or steam. And, text is character information inserted to identify components such as the name of equipment or piping included in the process piping and instrumentation diagram (P&ID), line number, fluid type, process condition, instrument tag, etc., or to explain information regarding the components.

[0124] First, the graph conversion module (132) can recognize symbols related to the process piping and instrumentation diagram (P&ID) using the YOLO model.

[0125] Specifically, the graph transformation module (132) can divide the preprocessed image into multiple grid cells according to a pre-set window size. The graph transformation module (132) can apply various anchor boxes having pre-defined sizes and ratios based on the types of symbols that may appear in the process piping and instrumentation diagram (P&ID) to each divided grid cell and calculate the probability (confidence score) that an object exists within each grid cell. The graph transformation module (132) can identify one or more bounding boxes corresponding to anchor boxes where the probability of an object existing is greater than or equal to a pre-set threshold value. And, the graph transformation module (132) can recognize symbols related to the process piping and instrumentation diagram (P&ID) by removing bounding boxes among one or more identified bounding boxes whose Intersection over Union (IoU) index is greater than or equal to a preset threshold, and calculating a classification score for the class of the object based only on the remaining bounding boxes.

[0126] In this case, the probability of an object existing and the probability for a class can be calculated using an artificial intelligence (AI) model trained to minimize the localization loss (loss due to the difference in coordinates between the bounding box and the actual object), the confidence loss (loss due to the difference in probability between when the actual object exists and when it does not exist within the grid cell), and the classification loss (loss due to the difference between the class of the actual object and the predicted class).

[0127] Next, the graph transformation module (132) can detect edges of an image in which symbols are recognized and perform a Hough transform on the detected edge images to detect lines and curves contained within the image. To this end, the graph transformation module (132) can detect edges of the image based on a Canny edge detection algorithm, but is not limited thereto. Additionally, if necessary, the graph transformation module (132) may perform morphological transformations of the detected lines by performing opening / closing operations.

[0128] Next, the graph conversion module (132) can skeletonize an image containing a process piping and instrumentation diagram (P&ID) by first removing symbols from the image and then repeatedly performing erosion and dilation operations on the image from which symbols have been removed until the thickness of the recognized lines (detected straight lines and curves) does not change, thereby thinning the lines.

[0129] And, the graph conversion module (132) can recognize text related to the process piping and instrumentation diagram (P&ID) using a tesseract model.

[0130] Specifically, the graph transformation module (132) can segment a skeletonized image into text blocks that may contain one or more texts. The graph transformation module (132) can extract a feature vector by analyzing connected components for pixels for each segmented text block. To this end, the graph transformation module (132) can perform blob detection for each text block to perform analysis of connected components (i.e., character region detection). Then, the graph transformation module (132) can recognize character units contained in each text block by inputting the extracted feature vector into a pre-trained LSTM (Long Short-Term Memory) model. In this case, the LSTM model may be a model trained based on a dictionary consisting only of terms related to process piping and instrumentation diagrams (P&ID).

[0131] Meanwhile, there may be symbols and text that are not recognized by the YOLO model and tesseract model of the graph transformation module (132), even though they are included in the image.

[0132] As illustrated in FIGS. 8 and 9, to process unrecognized symbols and text, the graph transformation module (132) may provide a User Interface (UI) that allows a user to manually input information about the symbol, targeting a bounding box where the probability of an object existing is greater than a preset threshold, but the probability of the class of the calculated object is lower than the threshold. Additionally, the graph transformation module (132) may provide a UI that allows a user to manually input text, targeting a text block where the character unit is not recognized by the LSTM model. In this way, information about the symbol and text input by the user may be accumulated and recorded in the storage (200) to retrain the YOLO model and the Tesseract model.

[0133] Next, the graph conversion module (132) can generate a graph corresponding to the process piping and instrumentation diagram (P&ID) based on the connectivity of the symbols and lines recognized from the process piping and instrumentation diagram (P&ID) and the position of the text within the image.

[0134] Specifically, the graph conversion module (132) can analyze connectivity by identifying the intersection points between the symbols and lines recognized from the process piping and instrumentation diagram (P&ID), the intersection points and branch points between the lines, and the end points of the lines.

[0135] To this end, the graph conversion module (132) may set a buffer area of ​​a pre-set size centered on a symbol, and then determine a line passing through the buffer area as a line intersecting the symbol. Additionally, the graph conversion module (132) may match text recognized from the process piping and instrumentation diagram (P&ID) to the nearest symbol or line. In this case, the graph conversion module (132) may perform the matching while excluding unmatchable symbols or lines based on the attributes of the recognized text.

[0136] Next, the graph conversion module (132) can generate a graph by setting a symbol recognized from the process piping and instrumentation diagram (P&ID) as a node of the graph, setting a line recognized from the process piping and instrumentation diagram (P&ID) as an edge of the graph, and setting text recognized from the process piping and instrumentation diagram (P&ID) as an attribute of the node and edge.

[0137] The graph transformation module (132) can selectively set all or part of the intersections, branches, and terminals identified through connectivity analysis as nodes on a previously generated graph and add them to the graph.

[0138] And, the graph conversion module (132) can complete the conversion process into a graph by storing the generated graph in the graph database (GDB) of the storage (200).

[0140] Next, with reference to FIGS. 10 and FIGS. 11, the graph verification module (134) will be described in more detail.

[0141] FIGS. 10 and 11 are illustrative diagrams for explaining the process of verifying data according to one embodiment of the present invention.

[0142] Referring to FIGS. 10 and FIGS. 11, a graph verification module (134) according to one embodiment of the present invention can verify a graph generated by vision recognition from an image based on XML (eXtensible Markup Language) data.

[0143] First, the graph verification module (134) can generate a first graph from an XML-formatted document in which a process piping and instrumentation diagram (P&ID) is stored. To do this, the graph verification module (134) can obtain an XML-formatted document from a database (DB) of a storage (200).

[0144] Specifically, the graph verification module (134) can parse a document in XML format and then identify equipment and pipelines related to the process piping and instrumentation diagram (P&ID) from the parsed document. The graph verification module (134) can identify attributes set on the equipment and pipelines identified from the document in XML format. Here, the attributes set on the equipment or pipelines may include one or more of a tag, type, coordinate, from-to connection relationship of the pipelines, and direction of the fluid for identifying the equipment or pipelines.

[0145] And, the graph verification module (134) can generate a first graph corresponding to a process piping and instrumentation diagram (P&ID) by setting the identified equipment as a node of the first graph, setting the identified piping as an edge of the first graph, and setting the attributes set for the identified equipment and piping as attributes of the nodes and edges of the first graph.

[0146] Next, if a graph generated from an image containing a process piping and instrumentation diagram (P&ID) does not yet exist, the graph verification module (134) can generate a second graph from an image containing a process piping and instrumentation diagram (P&ID) through the graph conversion module (132). The process of generating a second graph from an image containing a process piping and instrumentation diagram (P&ID) is the same as described above with reference to FIGS. 7 to 9, so it will not be described redundantly.

[0147] Next, the graph verification module (134) can verify the second graph generated from the image based on the consistency between the first graph generated from an XML-formatted document and the second graph generated from the image. To determine the consistency of these graphs, the graph verification module (134) can determine whether all nodes and edges included in the first graph match each of all nodes and edges included in the second graph. Additionally, the graph verification module (134) can determine whether the network structure of the first graph and the network structure of the second graph match each other. Here, the network structure of the graph is a structural characteristic of the graph represented by one or more nodes connected by edges.

[0148] Specifically, the graph verification module (134) can determine whether the number of nodes, the tag assigned to identify each node, and the type of each node perfectly match each other for the nodes included in the first graph and the second graph, respectively.

[0149] If the number of nodes included in the first graph and the second graph, the tag assigned to identify each node, and the type of each node do not perfectly match each other, the graph verification module (134) can extract lists of nodes included in the first graph and the second graph, respectively, and then calculate the similarity based on the Levenshtein distance between the nodes included in the two extracted lists.

[0150] The graph verification module (134) can determine that the nodes of the first graph and the second graph are matched with each other if the calculated similarity is greater than or equal to a preset threshold, and determine that the nodes of the first graph and the second graph are not matched with each other if the calculated similarity is less than the threshold.

[0151] Next, the graph verification module (134) can determine whether the number of edges, the connection relationship (from-to) of each edge, and the directionality of each edge perfectly match each other, for the edges included in the first graph and the second graph, respectively.

[0152] If the number of edges included in the first graph and the second graph, the connection relationship of each edge, and the directionality of each edge do not perfectly match each other, the graph verification module (134) can regenerate the second graph through the graph conversion module (132) after readjusting the size of the buffer area set around the symbol to analyze the connectivity of the symbol and line related to the process piping and instrumentation diagram (P&ID).

[0153] After regenerating the second graph, the graph verification module (134) can determine again whether the number of edges, the connection relationship (from-to) of each edge, and the directionality of each edge perfectly match each other, for the edges included in the first graph and the second graph, respectively.

[0154] When re-evaluated, if the number of edges included in the first graph and the second graph, the connection relationship of each edge, and the direction of each edge perfectly match each other, the graph verification module (134) determines that the edges of the first graph and the second graph match each other, and if the number of edges included in the first graph and the second graph, the connection relationship of each edge, and the direction of each edge still do not perfectly match each other, the graph verification module (134) can determine that the edges of the first graph and the second graph do not match each other.

[0155] Next, the graph verification module (134) can determine whether the network structure of the first graph and the network structure of the second graph perfectly match each other. To this end, the graph verification module (134) can perform a graph isomorphism test on the first graph and the second graph.

[0156] If the network structure of the first graph and the network structure of the second graph do not perfectly match each other, the graph verification module (134) can determine whether the network structure of the first graph and the network structure of the second graph match each other through the Graph Edit Distance (GED) or subgraph isomorphism.

[0157] First, we will explain the process of determining whether a graph network structure is matched using Graph Edit Distance (GED).

[0158] If the network structure of the first graph and the network structure of the second graph do not perfectly match each other, the graph verification module (134) can identify edit operations required for the second graph so that the network structure of the first graph and the network structure of the second graph become identical to each other. Here, the edit operations may include operations for inserting, deleting, or modifying nodes, and operations for inserting, deleting, or modifying edges.

[0159] The graph verification module (134) can calculate the minimum cost required to perform each identified editing operation. The graph verification module (134) can determine that the network structure of the first graph and the network structure of the second graph match each other if the calculated minimum cost is less than or equal to a preset threshold, and determine that the network structure of the first graph and the network structure of the second graph do not match each other if the calculated minimum cost exceeds the threshold.

[0160] In contrast, the process of determining whether a graph network structure matches through subgraph isomorphism checks is explained.

[0161] If the network structure of the first graph and the network structure of the second graph do not perfectly match each other, the graph verification module (134) can set one of the nodes included in the second graph as a reference node.

[0162] The graph verification module (134) selects one candidate node that can be matched to the reference node from among the nodes included in the first graph based on the reference node, and checks whether the connection relationship of the reference node in the second graph and the connection relationship of the candidate node in the first graph match (feasibility check). If the connection relationship of the reference node and the connection relationship of the candidate node match each other, the node adjacent to the reference node can be set as the new reference node. Additionally, the graph verification module (134) can recursively perform the process of selecting the aforementioned candidate node based on the new reference node and checking whether the connection relationship matches.

[0163] As a result of such recursive execution, the graph verification module (134) can determine that the network structure of the first graph and the network structure of the second graph match each other if all nodes of the first graph match all nodes of the second graph, and determine that the network structure of the first graph and the network structure of the second graph do not match each other if all nodes of the first graph do not match all nodes of the second graph.

[0164] Next, the graph verification module (134) can update the second graph based on the results of the verification performed as described above.

[0165] Specifically, the graph verification module (134) can update the second graph by performing one or more of the following based on the verification results: adding missing nodes or edges in the second graph, correcting nodes or edges of the second graph that are misidentified from the image, or supplementing attributes for nodes or edges of the second graph that are not identified from the image.

[0166] And, the graph verification module (134) can create a consistency report that can be used to tune the artificial intelligence (AI) model used to generate the second graph from the image based on the history of the update, and can store the created consistency report in a database (DB).

[0168] Next, with reference to FIG. 12, the ontology conversion module (136) will be described in more detail.

[0169] FIG. 12 is an illustrative diagram for explaining the process of converting a graph into an ontology according to one embodiment of the present invention.

[0170] Referring to FIG. 12, an ontology conversion module (136) according to one embodiment of the present invention can convert a graph generated from a process piping and instrumentation diagram (P&ID) into an ontology having semantic information.

[0171] First, the ontology conversion module (136) can obtain a graph generated from a process piping and instrumentation diagram (P&ID) from the graph database (GDB) of the repository (200).

[0172] The ontology conversion module (136) can identify a domain model of a process piping and instrumentation diagram (P&ID). Here, the domain model of the process piping and instrumentation diagram (P&ID) may be a model that defines two or more objects included in the process piping and instrumentation diagram (P&ID), the relationships between said objects, and attributes. And, the ontology conversion module (136) can define an ontology definition corresponding to the identified domain model. Here, the ontology definition may be a semantic model formalized to be usable in a computing environment.

[0173] Next, the ontology conversion module (136) can generate an ontology using the components of a graph obtained from a graph database (GDB) based on the identified domain model and the definition of the ontology.

[0174] Specifically, the ontology transformation module (136) can map nodes included in the graph to classes of the ontology. The ontology transformation module (136) can map edges included in the graph to relationships of the ontology. In addition, the ontology transformation module (136) can map attributes of nodes and edges included in the graph to data properties of the ontology.

[0175] After mapping the classes, relationships, and data attributes as described above, the ontology transformation module (136) can identify semantic information about the components included in the graph based on the domain model and the ontology definition. Then, based on the identified semantic information, the ontology transformation module (136) can set up ontology triples for the classes, relationships, and data attributes mapped from the graph. Here, the ontology triple can serve as a basic unit for expressing the semantic information of the ontology.

[0176] Next, the ontology conversion module (136) can expand and refine the previously created ontology based on the domain model and ontology definition.

[0177] Specifically, the ontology transformation module (136) can classify the components of the graph by type based on the domain model and the ontology definition. Based on the classified types, the ontology transformation module (136) can define a new superclass that encompasses the same type. Furthermore, the ontology transformation module (136) can provide a hierarchy to the ontology by reflecting the defined superclass in the previously created ontology.

[0178] The ontology transformation module (136) can identify common attributes and relationships among classes belonging to the same layer within the ontology based on the assigned layer structure. The ontology transformation module (136) can apply abstraction to the ontology by establishing a definition for the layer using the identified common attributes and relationships.

[0179] The ontology conversion module (136) can integrate duplicate classes into one if duplicate classes exist among the classes included in the ontology generated from the graph and the classes added through the assignment of the hierarchy.

[0180] Additionally, the ontology conversion module (136) may correct the definitions of relationships, data attributes, and layers established through the application of abstraction to the ontology generated from the graph, if there are any definitions of relationships, data attributes, and layers that do not conform to the domain model and ontology definitions, so as to conform to the domain model and ontology definitions.

[0181] And, the ontology conversion module (136) can store the ontology, which has been expanded and refined, in an RDF repository within the repository (200).

[0182] Based on the ontology converted in this way, when a question regarding a process piping and instrumentation diagram (P&ID) is input from a user, the ontology conversion module (136) may use the ontology to derive an answer to the input question.

[0183] For example, the ontology conversion module (136) can identify the type of query entered by the user by using a pre-trained Natural Language Processing (NLP) artificial intelligence (AI) model. The ontology conversion module (136) can generate a dynamic query based on the identified type of query and query the ontology using the generated dynamic query. Additionally, the ontology conversion module (136) can derive an answer to the query using a value obtained from the ontology. The process of deriving an answer to the user query will be described in more detail later with reference to FIGS. 14 and FIGS. 15.

[0185] Next, with reference to FIG. 13, the image generation module (138) will be described in more detail.

[0186] FIG. 13 is an exemplary diagram illustrating the process of automatically generating a process piping and instrumentation diagram (P&ID) image according to one embodiment of the present invention.

[0187] Referring to FIG. 13, an image generation module (138) according to one embodiment of the present invention can automatically generate a process piping and instrumentation diagram (P&ID) image based on a graph related to the process piping and instrumentation diagram (P&ID).

[0188] First, the image generation module (138) can obtain a graph related to a process piping and instrumentation diagram (P&ID) from the graph database (GDB) of the repository (200).

[0189] The image generation module (138) can determine the layout of an image for generating a process piping and instrumentation diagram (P&ID) image by analyzing the network structure of the graph based on the attributes of the nodes, edges, and nodes and edges included in the graph obtained from the graph database (GDB).

[0190] Specifically, the image generation module (138) can determine whether the node's image coordinates (X, Y) are included in the attributes of the node included in the graph. If the node's attributes include the node's image coordinates (X, Y), the image generation module (138) can determine the layout of the node based on the coordinates (X, Y) included in the attributes.

[0191] If the node's attributes do not include the node's image coordinates (X, Y), the image generation module (138) can determine the image coordinates for each node included in the graph by applying a pre-set layout algorithm. In this case, the image generation module (138) can determine the layout of the image by applying a hierarchical layout and an orthogonal layout in combination.

[0192] More specifically, the image generation module (138) can identify nodes belonging to the same group according to a functional grouping rule that defines two or more nodes included in a graph as a single group. Here, the functional grouping rule may be a rule for grouping two or more devices to achieve a common purpose according to fluid, signal, or power flow. The image generation module (138) can determine the order of nodes belonging to the same group according to the flow direction included in the functional grouping rule. Based on the determined order, the image generation module (138) can determine the coordinates on the image for each node.

[0193] To this end, the image generation module (138) can convert the graph into a Directed Acyclic Graph (DAG). If the graph cannot be converted into a Directed Acyclic Graph (DAG) because it contains cycles, the image generation module (138) can retry the conversion into a Directed Acyclic Graph (DAG) after adding a virtual node or edge in the middle of the cycle.

[0194] The image generation module (138) can separate and assign nodes included in the transformed directed acyclic graph to multiple layers. To do this, the image generation module (138) can calculate the in-degree of each node included in the directed acyclic graph. The image generation module (138) can enqueue a node with a calculated in-degree of 0 as the starting point of level 1. The image generation module (138) can repeatedly perform the process of dequeuing a node inserted into the queue, visiting another node connected from the dequeued node, assigning a level to the visited other node that is 1 greater than that of the dequeued node, and then inserting it into the queue, until the queue is empty. Through this process, the image generation module (138) assigns a level to each node included in the directed acyclic graph and separates and assigns them to multiple layers according to the level assigned to each node.

[0195] The image generation module (138) can adjust the order within the layers so that the crossing of nodes belonging to the same layer is minimized for each separated layer. Additionally, the image generation module (138) can determine the vertical (Y) coordinate of each node on the image based on the level of the layer and determine the horizontal (X) coordinate of each node on the image based on the order of the nodes within the layer.

[0196] The image generation module (138) can generate an image for visually representing a process piping and instrumentation diagram (P&ID) based on the layout of the determined image.

[0197] Specifically, the image generation module (138) can visualize each node included in the graph on an image based on horizontal and vertical coordinates (X, Y) determined for each node. After the nodes are visualized, the image generation module (138) can visualize the edges included in the graph on an image according to orthogonal layout rules defined so that the edges are composed only of horizontal or vertical lines. Additionally, the image generation module (138) can visualize the attributes of the nodes and edges on an image based on the coordinates of the nodes and edges on the image.

[0198] For visualization of nodes, edges, and attributes, the image generation module (138) can query symbols, lines, and text corresponding to the nodes, edges, and attributes of the graph, respectively, from a legend ontology configured within the repository (200). Then, the image generation module (138) can perform visualization of the nodes, edges, and attributes of the graph by placing a vector image corresponding to the symbols, lines, and text queried from the legend ontology onto the image.

[0199] Furthermore, the image generation module (138) may add an arrow image to match the line included in the visualized image based on the direction of the edge. Additionally, the image generation module (138) may assign a color or pattern to the line in the image based on the type of flow included in the attribute of the edge. Here, the type of flow may correspond to any one of water, steam, fuel, signal, and power.

[0200] And, the image generation module (138) can store the image of the generated process piping and instrumentation diagram (P&ID) in the database (DB) of the storage (200).

[0202] With the following configuration, the P&ID service unit (140) can provide a service that manages piping and instrumentation related to the process piping and instrumentation diagram (P&ID) based on various data converted by the data conversion unit (130).

[0203] Specifically, the P&ID service unit (140) can provide information that allows for real-time structural analysis of piping and instrumentation and easy identification of connection relationships. The P&ID service unit (140) can provide information that allows for tracking and analyzing fluid flow paths. The P&ID service unit (140) can provide information that allows for maintaining and periodically inspecting equipment and facilities. The P&ID service unit (140) can provide information that allows for responding to emergency situations according to emergency scenarios. In addition, the P&ID service unit (140) can provide information that analyzes the impact that may occur when designing equipment and piping. Furthermore, the provision of information by the P&ID service unit (140) can be performed in the form of providing answers in response to user inquiries.

[0205] With reference to FIGS. 14 and FIGS. 15, the P&ID service section (140) will be described in more detail.

[0206] FIGS. 14 and FIGS. 15 are exemplary diagrams for explaining the process of answering a user's query according to an embodiment of the present invention.

[0207] Referring to FIG. 14, a P&ID service unit (140) according to one embodiment of the present invention can provide an answer to a user's query by utilizing a combination of graphs and ontologies.

[0208] First, when a user inputs a question about a process piping and instrumentation diagram (P&ID), the P&ID service department (140) can analyze the user's question using a pre-trained natural language processing (NLP) artificial intelligence (AI) model.

[0209] Specifically, the P&ID service unit (140) can tokenize a user's query into semantic units and then classify the intent of the query included in the tokenized semantic units using a classifier trained based on the domain model of the process piping and instrumentation diagram (P&ID). For example, the intent of the query can be classified as "inquiry into equipment attributes," "inquiry into process paths," "safety analysis," "inquiry into maintenance information," "analysis of design change impacts," etc., but is not limited thereto.

[0210] The P&ID service unit (140) can identify the entity of the query included in the tokenized semantic unit using a Named Entity Recognition (NER) algorithm. For example, the entity of the query can be identified by the names of equipment, piping, instruments, etc., such as "P-201", "P504", "R-102", "M-08", etc.

[0211] The P&ID service unit (140) can extract attributes of a query that match a tokenized semantic unit from among the attributes included in a predefined list of attributes based on the identified target of the query. For example, attributes of the query can be extracted as "inspection cycle", "installation date", "fluid type", "pressure", "temperature", "connected equipment", etc.

[0212] Once the analysis of the user query is complete, the P&ID service unit (140) can determine the database to process the user query. To this end, the P&ID service unit (140) can classify the type of query corresponding to the intent of the query, the target of the query, and the attributes of the query by using a pre-trained artificial intelligence (AI) or rule-based algorithm.

[0213] If the type of classified query corresponds to structural exploration of the graph or real-time data query of the graph, the P&ID service unit (140) can set only the graph database (GDB) of the storage (200) as the query target.

[0214] Alternatively, if the type of classified query corresponds to a query requiring semantic information, the P&ID service unit (140) may set an ontology generated from a graph and stored in an RDF repository as an additional query target, in addition to the graph database (GDB) of the repository (200).

[0215] In response to the configured query target, the P&ID service unit (140) can query a graph database (GDB) that stores a graph corresponding to the process piping and instrumentation diagram (P&ID) based on the analysis results of the query. Additionally, the P&ID service unit (140) can query an RDF repository that stores an ontology corresponding to the process piping and instrumentation diagram (P&ID) based on the analysis results of the query.

[0216] For example, if the type of classified query corresponds to any one of the following: management of inspection criteria by type of equipment included in the process piping and instrumentation diagram (P&ID), inquiry into safety or risk assessment of the process piping and instrumentation diagram (P&ID), inquiry into compliance with safety regulations of the process piping and instrumentation diagram (P&ID), and analysis of the impact of design changes of the process piping and instrumentation diagram (P&ID), the P&ID service unit (140) can query both graphs and ontologies.

[0217] For querying graphs and ontologies, the P&ID service unit (140) can generate a query based on the intent of the query, the target of the query, and the attributes of the query.

[0218] More specifically, the P&ID service unit (140) can extract two or more sample queries from a pre-set sample query list in which one or more of the query intent, query target, and query attribute match. The P&ID service unit (140) can generate a few-shot prompt by including the query intent, query target, and query attribute and the two or more sample queries extracted from the sample query list. Then, the P&ID service unit (140) can generate a query by inputting the few-shot prompt into a pre-trained artificial intelligence (AI) model.

[0219] Alternatively, the P&ID service unit (140) can search for a template that matches the intent of the query from a pre-configured query template store. The P&ID service unit (140) may also generate a query by mapping parameters using a pre-trained artificial intelligence (AI) model to include the intent of the query, the target of the query, and the attributes of the query from the searched template.

[0220] Next, the P&ID service unit (140) can generate an answer to a user's query based on results retrieved from the graph database (GDB), or results retrieved from the graph database (GDB) and the RDF repository, respectively.

[0221] For example, the P&ID service unit (140) can combine semantic information of an ontology with information regarding components of a process piping and instrumentation diagram (P&ID) retrieved from a graph database (GDB). Then, the P&ID service unit (140) can generate an answer related to the information regarding components of the process piping and instrumentation diagram (P&ID) combined with semantic information using a natural language processing (NLP) artificial intelligence (AI) model.

[0222] Meanwhile, as illustrated in FIG. 15, the P&ID service unit (140) can provide answers not only through text but also through visual graphics of the process piping and instrumentation diagram (P&ID).

[0223] To this end, the P&ID service unit (140) can identify the image coordinates (X, Y) and size (width, height) of components (i.e., symbols, lines, and text) included in the process piping and instrumentation diagram (P&ID) from the graph database (GDB). Based on the identified image coordinates and size, the P&ID service unit (140) can perform image processing to highlight the components related to the answer on the image containing the process piping and instrumentation diagram (P&ID).

[0224] And, the P&ID service unit (140) can output an answer including visual graphics.

[0226] Below, the hardware of the digitization server (100) for realizing the logical configuration described above will be explained in more detail.

[0227] FIG. 16 is a hardware configuration diagram of a digitization server according to one embodiment of the present invention.

[0228] As illustrated in FIG. 16, the digitization server (100) may be configured to include a processor (150), memory (155), transceiver (160), input / output device (165), data bus (170) and storage (175).

[0229] Specifically, the processor (150) can implement the operation and function of the digitization server (100) based on instructions according to software (180a) that implements a digitization method residing in memory (155).

[0230] Software (180b) that implements a digitization method stored in storage (175) can be loaded into memory (155).

[0231] The input / output device (165) can receive signals necessary for the operation of the digitization server (100) or output calculation results to the outside according to the command of the processor (150).

[0232] The data bus (170) is connected to the processor (150), memory (155), transceiver (160), input / output device (165), and storage (175), respectively, and can serve as a passage for transmitting signals between each component.

[0233] Storage (175) may store an Application Programming Interface (API), library files, resource files, etc., necessary for the execution of software (180a) in which a digitization method according to embodiments of the present invention is implemented. Storage (175) may store software (180b) in which a digitization method according to embodiments of the present invention is implemented. Additionally, storage (175) may store data necessary for the operation of an artificial intelligence model (AI).

[0234] According to one embodiment of the present invention, software (180a, 180b) for implementing a method for generating training data that resides in memory (155) or is stored in storage (175) may be a computer program recorded on a recording medium to execute the steps of: a processor (150) acquiring an image containing a process piping and instrumentation diagram (P&ID) from a database (DB); a processor (150) preparing a list containing one or more symbols and texts that may be included in the process piping and instrumentation diagram (P&ID); and a processor (150) applying a modification to the symbols and texts included in the acquired image, or replacing the symbols or texts included in the acquired image with the symbols or texts included in the list, thereby generating training data for an artificial intelligence (AI) model for recognizing symbols and texts related to the process piping and instrumentation diagram (P&ID) from an image.

[0235] According to another embodiment of the present invention, software (180a, 180b) for implementing a graph generation method residing in memory (155) or stored in storage (175) may be a computer program recorded on a recording medium to execute the steps of: a processor (150) preprocessing an image containing a process piping and instrumentation diagram (P&ID); a processor (150) recognizing symbols, lines, and text related to the process piping and instrumentation diagram (P&ID) from the preprocessed image using an artificial intelligence (AI) model; and a processor (150) generating a graph corresponding to the process piping and instrumentation diagram (P&ID) based on the connectivity of the recognized symbols and lines and the location of the text within the image.

[0236] According to another embodiment of the present invention, software (180a, 180b) for implementing a data verification method residing in memory (155) or stored in storage (175) may be a computer program recorded on a recording medium to execute the steps of: the processor (150) generating a first graph from an XML-format document in which a process piping and instrumentation diagram (P&ID) is stored; the processor (150) generating a second graph from an image containing the process piping and instrumentation diagram (P&ID); and the processor (150) verifying the second graph based on the consistency between the first graph and the second graph.

[0237] According to another embodiment of the present invention, software (180a, 180b) for implementing an ontology conversion method residing in memory (155) or stored in storage (175) may be a computer program recorded on a recording medium to execute the steps of: a processor (150) obtaining a graph generated from a process piping and instrumentation diagram (P&ID) from a graph database (GDB); a processor (150) identifying a domain model of the process piping and instrumentation diagram (P&ID) and defining an ontology definition corresponding to the identified domain model; and a processor (150) generating an ontology using components of a graph based on the identified domain model and ontology definition.

[0238] According to another embodiment of the present invention, software (180a, 180b) for implementing a method for automatically generating a process piping and instrumentation diagram (P&ID) image that resides in memory (155) or is stored in storage (175) may be a computer program recorded on a recording medium to execute the steps of: a processor (150) obtaining a graph related to a process piping and instrumentation diagram (P&ID) from a graph database (GDB); a processor (150) analyzing the network structure of the graph based on the nodes, edges, and attributes of the nodes and edges included in the graph to determine the layout of the image; and a processor (150) generating an image to visually represent the process piping and instrumentation diagram (P&ID) based on the determined layout.

[0239] According to another embodiment of the present invention, software (180a, 180b) for implementing a process piping and instrumentation diagram (P&ID) management method residing in memory (155) or stored in storage (175) may be a computer program recorded on a recording medium to execute the steps of: a processor (150) analyzing the input query using a pre-trained natural language processing (NLP) artificial intelligence (AI) model when a query regarding the process piping and instrumentation diagram (P&ID) is input; a processor (150) querying a graph database (GDB) that stores a graph corresponding to the process piping and instrumentation diagram (P&ID) based on the analysis result of the query; and a processor (150) generating an answer corresponding to the query based on the result queried from the graph database (GDB).

[0241] More specifically, the processor (150) may be configured to include one or more of a Central Processing Unit (CPU), an Application-Specific Integrated Circuit (ASIC), a chipset, and a logic circuit, but is not limited thereto.

[0242] The memory (155) may be configured to include one or more of ROM (Read-Only Memory), RAM (Random Access Memory), flash memory, and memory card, but is not limited thereto.

[0243] The input / output device (160) may be configured to include one or more input devices such as a button, switch, keyboard, mouse, and joystick, and output devices such as an LCD (Liquid Crystal Display), LED (Light Emitting Diode), Organic LED (OLED), Active Matrix OLED (AMOLED), printer, and plotter, but is not limited thereto.

[0244] When the embodiments included in this specification are implemented in software, the above-described method may be implemented as modules (processes, functions, etc.) that each perform the above-described functions. Each module may reside in memory (155) and be executed by a processor (150). Memory (155) may exist inside or outside the processor (150) and may be connected to the processor (150) by various well-known means.

[0245] Each component illustrated in FIG. 16 may be implemented by various means (e.g., hardware, firmware, software, or a combination thereof). When implemented by hardware, one embodiment of the present invention may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0246] In addition, when implemented by firmware or software, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc., that performs the functions or operations described above, and may be recorded on a recording medium readable through various computer means. Here, the recording medium may include program instructions, data files, data structures, etc., either alone or in combination.

[0247] The program instructions recorded on the recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks); magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

[0248] Examples of program instructions may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Such hardware devices may be configured to operate as one or more software to perform the operation of the present invention, and vice versa.

[0250] Hereinafter, the operation of the digitization server (100) as described above will be explained in more detail.

[0251] FIG. 17 is a flowchart of a digitization method according to one embodiment of the present invention.

[0252] Referring to FIG. 17, the digitization server (100) can train an artificial intelligence (AI) model that can be used to automatically generate a graph from an image containing a process piping and instrumentation diagram (P&ID) (S100). In this regard, it will be described in more detail later with reference to FIG. 18.

[0253] The digitization server (100) can automatically generate a graph from an image containing a process piping and instrumentation diagram (P&ID) (S200). In this regard, it will be described in more detail later with reference to FIG. 19.

[0254] The digitization server (100) can verify a graph generated from an image based on XML data (S300). In this regard, it will be described in more detail later with reference to FIG. 20.

[0255] The digitization server (100) can convert a graph generated from a process piping and instrumentation diagram (P&ID) into an ontology having semantic information (S400). In this regard, it will be described in more detail later with reference to FIG. 21.

[0256] In addition, the digitization server (100) can provide services to the user based on data such as graphs and ontology generated or converted in relation to the process piping and instrumentation diagram (P&ID) (S500). Here, the services that can be provided to the user may include a service for automatically generating images of the process piping and instrumentation diagram (P&ID) and a service for providing answers to the user's queries. In this regard, it will be described in more detail later with reference to FIGS. 22 and FIGS. 23.

[0258] FIG. 18 is a flowchart for specifically explaining a method for generating training data according to one embodiment of the present invention.

[0259] Referring to FIG. 18, the digitization server (100) can obtain one image containing a process piping and instrumentation diagram (P&ID) from a database (DB) (S110).

[0260] Specifically, the digitization server (100) can identify a project related to a process piping and instrumentation diagram (P&ID) to be used with an artificial intelligence model (AI), generate a query including keywords related to the identified project, and then query a database (DB) using the generated query to obtain a single image containing the process piping and instrumentation diagram (P&ID).

[0261] The digitization server (100) can modify the symbols included in the image of the process piping and instrumentation diagram (P&ID) (S120).

[0262] Specifically, the digitization server (100) can identify one or more symbols related to the process piping and instrumentation diagram (P&ID) from an image, and then randomly perform one or more operations among rotation, enlargement or reduction, translation, and inversion on all or part of the identified symbols to apply geometric deformation, and can readjust the lines included in the process piping and instrumentation diagram (P&ID) so that the lines connected to the symbols before the geometric deformation is applied remain connected to the symbols after the geometric deformation is applied.

[0263] And, the digitization server (100) can apply image transformation by randomly performing one or more operations such as brightness or contrast adjustment, color change, adding background noise and adding a grid pattern on an image in which the lines have been readjusted.

[0264] Alternatively, the digitization server (100) may search for a symbol that can replace a symbol identified from an image of a process piping and instrumentation diagram (P&ID) among one or more symbols included in a prepared list, and after replacing the symbol identified from the image with a symbol found in the list, readjust the line included in the process piping and instrumentation diagram (P&ID) so that the line connected to the symbol before replacement is connected to the symbol after replacement.

[0265] The digitization server (100) can modify the text included in the image of the process piping and instrumentation diagram (P&ID) (S130).

[0266] Specifically, one or more texts related to a process piping and instrumentation diagram (P&ID) can be identified from an image, and text deformation can be applied by randomly performing one or more operations among font deformation, text distortion, and blur addition on all or part of the identified texts.

[0267] And, the digitization server (100) can apply image transformation by randomly performing one or more operations such as texture change, annotation addition, and color inversion on an image to which text transformation has been applied.

[0268] The digitization server (100) can verify whether the symbols and lines included in the newly generated training data are correctly arranged from a fluid dynamics perspective (S140).

[0269] Specifically, the digitization server (100) and the data management module (122) can verify newly generated training data by performing one or more of verification using computational fluid dynamics (CFD) simulation, verification through HAZOP analysis, and verification through comparison with process flow diagrams (PFD).

[0270] The digitization server (100) can store training data that is determined to be free of problems as a result of verification in a database (DB) (S150). Then, the digitization server (100) can use the training data stored in the database (DB) to train an artificial intelligence (AI) model that can be used to automatically generate a graph from an image (S160).

[0272] FIG. 19 is a flowchart for specifically explaining a graph generation method according to one embodiment of the present invention.

[0273] Referring to FIG. 19, the digitization server (100) can preprocess an image containing a process piping and instrumentation diagram (P&ID) (S210).

[0274] Specifically, the digitization server (100) can adjust the size or resolution of an image containing a process piping and instrumentation diagram (P&ID) to correspond to a preset size or resolution. The digitization server (100) can remove noise contained in the image containing the process piping and instrumentation diagram (P&ID), and can convert the noise-removed image into a black and white image by binarizing it.

[0275] The digitization server (100) can separate a legend area from a preprocessed image based on the type of process piping and instrumentation diagram (P&ID) (S220).

[0276] Specifically, the digitization server (100) can crop the legend area by column and then determine whether the symbols and text included in the cropped area by column are arranged according to a predefined regular expression. If the symbols and text included in the cropped area are arranged according to the regular expression, the digitization server (100) can additionally map the symbols and text included in each cropped area according to the regular expression to a pre-configured legend ontology on the repository (200). Alternatively, if the arrangement of the symbols and text included in the cropped area does not follow the regular expression, the digitization server (100) can recognize the symbols and text included in each cropped area using a pre-learned VLM and then map the recognized symbols and text to a legend ontology.

[0277] The digitization server (100) can recognize symbols related to the process piping and instrumentation diagram (P&ID) using the YOLO model (S230).

[0278] Specifically, the digitization server (100) can divide a preprocessed image into multiple grid cells according to a pre-set window size. The digitization server (100) can apply various anchor boxes having pre-defined sizes and ratios based on the types of symbols that may appear in the Process Piping and Instrumentation Diagram (P&ID) to each divided grid cell and calculate the probability that an object exists within each grid cell. The digitization server (100) can identify one or more bounding boxes corresponding to anchor boxes where the probability of an object existing is greater than or equal to a pre-set threshold. Then, the digitization server (100) can recognize symbols related to the Process Piping and Instrumentation Diagram (P&ID) by removing bounding boxes among the identified one or more bounding boxes where the IoU index is greater than or equal to a pre-set threshold, and calculating the probability of the object class based only on the remaining bounding boxes.

[0279] The digitization server (100) can skeletonize an image containing a process piping and instrumentation diagram (P&ID) (S240).

[0280] Specifically, the digitization server (100) can detect the edges of an image in which a symbol is recognized and perform a Hough transform on the detected edge images to detect straight lines and curves included in the image. The digitization server (100) can skeletonize an image containing a process piping and instrumentation diagram (P&ID) by first removing the symbol from the image and then repeatedly performing erosion and dilation operations on the image from which the symbol has been removed until the thickness of the recognized lines (detected straight lines and curves) does not change, thereby thinning the lines.

[0281] The digitization server (100) can recognize text related to the process piping and instrumentation diagram (P&ID) using a tesseract model (S250).

[0282] Specifically, the digitization server (100) can divide a skeletonized image into text blocks that may contain one or more texts. The digitization server (100) can extract feature vectors by analyzing the connectivity components of pixels for each divided text block. To this end, the digitization server (100) can recognize character units included in each text block by inputting the extracted feature vectors into a pre-trained LSTM model.

[0283] The digitization server (100) can process unrecognized items by providing a user with a UI that allows input of information and text regarding symbols, targeting symbols and text that are not recognized by the YOLO model and the Tesseract model, and then receiving information and text regarding symbols from the user (S260).

[0284] The digitization server (100) can analyze connectivity by identifying the intersection between the symbols and lines recognized from the process piping and instrumentation diagram (P&ID), the intersection between the lines and the branching point and the end point of the line (S270).

[0285] Specifically, the digitization server (100) can set a buffer area of ​​a preset size centered on a symbol, and then determine a line passing through the buffer area as a line intersecting the symbol. The digitization server (100) can match text recognized from the process piping and instrumentation diagram (P&ID) to the nearest symbol or line.

[0286] The digitization server (100) can generate a graph corresponding to a process piping and instrumentation diagram (P&ID) based on the connectivity of symbols and lines and the position of text within the image (S280).

[0287] Specifically, the digitization server (100) can generate a graph by setting symbols recognized from the process piping and instrumentation diagram (P&ID) as nodes of the graph, lines recognized from the process piping and instrumentation diagram (P&ID) as edges of the graph, and text recognized from the process piping and instrumentation diagram (P&ID) as attributes of the nodes and edges. Additionally, the digitization server (100) can selectively set all or part of the intersections, branches, and terminals identified through connectivity analysis of the generated graph as nodes and add them to the graph.

[0288] And, the digitization server (100) can store the generated graph in the graph database (GDB) of the storage (200) (S290).

[0290] FIG. 20 is a flowchart for specifically explaining a data verification method according to one embodiment of the present invention.

[0291] Referring to FIG. 20, the digitization server (100) can generate a first graph from an XML format document in which a process piping and instrumentation diagram (P&ID) is stored (S310).

[0292] Specifically, the digitization server (100) can parse a document in XML format and then identify equipment and piping related to the process piping and instrumentation diagram (P&ID) from the parsed document. The digitization server (100) can identify attributes set on the identified equipment and piping from the XML format document. Then, the digitization server (100) can create a first graph corresponding to the process piping and instrumentation diagram (P&ID) by setting the identified equipment as a node of the first graph, setting the identified piping as an edge of the first graph, and setting the attributes set on the identified equipment and piping as attributes of the nodes and edges of the first graph.

[0293] The digitization server (100) can generate a second graph from an image containing a process piping and instrumentation diagram (P&ID) (S320). The method for generating a graph from an image containing a process piping and instrumentation diagram (P&ID) is the same as described with reference to FIG. 19, so it is not described redundantly.

[0294] The digitization server (100) can verify the second graph generated from the image based on the consistency between the first graph generated from the XML-formatted document and the second graph generated from the image (S330).

[0295] To determine the consistency of the graphs, the digitization server (100) can determine whether all nodes and edges included in the first graph match each of all nodes and edges included in the second graph. Additionally, the digitization server (100) can determine whether the network structure of the first graph and the network structure of the second graph match each other.

[0296] Specifically, the digitization server (100) can extract lists of nodes included in the first graph and the second graph, respectively, and then calculate a similarity based on the Levenstein distance between the nodes included in the two extracted lists. The digitization server (100) can determine that the nodes of the first graph and the second graph match each other if the calculated similarity is greater than or equal to a preset threshold value, and determine that the nodes of the first graph and the second graph do not match each other if the calculated similarity is less than the threshold value.

[0297] The digitization server (100) can determine whether the edges of the first graph and the second graph match each other by re-adjusting the size of the buffer area set around the symbol to analyze the connectivity of the symbol and the line, and determining whether the number of edges included in the first graph and the second graph, the connection relationship of each edge, and the direction of each edge perfectly match each other.

[0298] The digitization server (100) can determine whether the network structure of the first graph and the network structure of the second graph perfectly match each other by performing a graph isomorphism check on the first graph and the second graph. If the network structure of the first graph and the network structure of the second graph do not perfectly match each other, the digitization server (100) can determine whether the network structure of the first graph and the network structure of the second graph match each other through a graph edit distance (GED) or a subgraph isomorphism check.

[0299] The digitization server (100) can update the second graph based on the results of the verification (S340).

[0300] Specifically, the digitization server (100) can update the second graph by performing one or more of the following based on the verification results: adding missing nodes or edges in the second graph, correcting nodes or edges in the second graph that are misidentified from the image, or supplementing attributes for nodes or edges in the second graph that are not identified from the image.

[0301] And, the digitization server (100) can create a consistency report that can be used to tune the artificial intelligence (AI) model used to generate the second graph from the image based on the history of the update, and can store the created consistency report in a database (DB) (S350).

[0303] FIG. 21 is a flowchart for specifically explaining an ontology conversion method according to one embodiment of the present invention.

[0304] Referring to FIG. 21, the digitization server (100) can obtain a graph generated from a process piping and instrumentation diagram (P&ID) from a graph database (GDB) (S410).

[0305] The digitization server (100) can identify a domain model of a process piping and instrumentation diagram (P&ID) and define an ontology definition corresponding to the identified domain model (S420). Here, the domain model of the process piping and instrumentation diagram (P&ID) may be a model that defines two or more objects included in the process piping and instrumentation diagram (P&ID), the relationships between said objects, and attributes. And, the ontology definition may be a semantic model formalized so that it can be used in a computing environment.

[0306] The digitization server (100) can generate an ontology based on the definition of the identified domain model and ontology, using the components of the graph obtained from the graph database (GDB) (S430).

[0307] Specifically, the digitization server (100) can map nodes included in a graph to classes of an ontology. The digitization server (100) can map edges included in a graph to relationships of the ontology. Additionally, the digitization server (100) can map attributes of nodes and edges included in a graph to data attributes of the ontology. Furthermore, the digitization server (100) can identify semantic information regarding components included in a graph based on a domain model and an ontology definition, and then set up ontology triples for classes, relationships, and data attributes.

[0308] The digitization server (100) can expand and refine the previously created ontology based on the domain model and ontology definition (S440).

[0309] Specifically, the digitization server (100) can classify the components of the graph by type based on the domain model and the ontology definition. Based on the classified types, the digitization server (100) can define a new superclass that can encompass the same type. Then, the digitization server (100) can give a hierarchical structure to the ontology by reflecting the defined superclass in the previously created ontology.

[0310] The digitization server (100) can identify common attributes and relationships among classes belonging to the same layer within the ontology based on the assigned layer structure. The digitization server (100) can apply abstraction to the ontology by establishing a definition for the layer using the identified common attributes and relationships.

[0311] The digitization server (100) can integrate duplicate classes into one if there are duplicate classes among the classes included in the ontology generated from the graph and the classes added through the assignment of the hierarchy.

[0312] Additionally, the digitization server (100) may correct the definitions of relationships, data attributes, and layers that do not conform to the domain model and ontology definitions when there are relationships, data attributes, and layers that do not conform to the domain model and ontology definitions, by applying the relationships and data attributes and abstractions included in the ontology generated from the graph.

[0313] And, the digitization server (100) can store the ontology, which has been expanded and refined, in the RDF repository within the repository (200) (S450).

[0315] FIG. 22 is a flowchart for specifically explaining a method for automatically generating process piping and instrumentation diagram (P&ID) images according to one embodiment of the present invention.

[0316] Referring to FIG. 22, the digitization server (100) can obtain a graph related to a process piping and instrumentation diagram (P&ID) from a graph database (GDB) (S505).

[0317] The digitization server (100) can determine the layout of an image for generating a process piping and instrumentation diagram (P&ID) image by analyzing the network structure of the graph based on the attributes of the nodes, edges, and nodes and edges included in the graph obtained from the graph database (GDB) (S510).

[0318] Specifically, if the node's attributes include the node's image coordinates (X, Y), the digitization server (100) can determine the layout of the node based on the coordinates (X, Y) included in the attributes. If the node's attributes do not include the node's image coordinates (X, Y), the digitization server (100) can determine the image coordinates for each node included in the graph by applying a pre-set layout algorithm. In this case, the digitization server (100) can determine the layout of the image by applying a hierarchical layout and an orthogonal layout in combination.

[0319] The digitization server (100) can generate an image to visually represent a process piping and instrumentation diagram (P&ID) based on the layout of the determined image (S515).

[0320] Specifically, the digitization server (100) can visualize each node included in the graph on an image based on horizontal and vertical coordinates (X, Y) determined for each node. After the nodes are visualized, the digitization server (100) can visualize the edges included in the graph on an image according to a right-angle layout rule defined so that the edges are composed only of horizontal or vertical lines. Additionally, the digitization server (100) can visualize the attributes of the nodes and edges on an image based on the coordinates of the nodes and edges on the image.

[0321] For visualization of nodes, edges, and attributes, the digitization server (100) can query symbols, lines, and text corresponding to the nodes, edges, and attributes of the graph, respectively, from the legend ontology. Then, the digitization server (100) can perform visualization of the nodes, edges, and attributes of the graph by placing vector images corresponding to the symbols, lines, and text queried from the legend ontology onto the image.

[0322] Furthermore, the digitization server (100) may add arrow images to match lines included in the visualized image based on the directionality of the edges. Additionally, the digitization server (100) may assign colors or patterns to lines in the image based on the type of flow included in the attributes of the edges.

[0323] And, the digitization server (100) can store the image of the generated process piping and instrumentation diagram (P&ID) in the database (DB) of the storage (200) (S520).

[0325] FIG. 23 is a flowchart for specifically explaining a process piping and instrumentation diagram (P&ID) management method according to one embodiment of the present invention.

[0326] Referring to FIG. 23, the digitization server (100) can receive a query from a user regarding a process piping and instrumentation diagram (P&ID) (S555).

[0327] The digitization server (100) can analyze a user's query using a pre-trained natural language processing (NLP) artificial intelligence (AI) model (S560).

[0328] Specifically, the digitization server (100) can tokenize a user's query into semantic units and then classify the intent of the query included in the tokenized semantic units using a classifier trained based on a domain model of a process piping and instrumentation diagram (P&ID). The digitization server (100) can identify the target of the query included in the tokenized semantic units using a Named Entity Recognition (NER) algorithm. Based on the identified target of the query, the digitization server (100) can extract the attributes of the query that match the tokenized semantic units from among the attributes included in a predefined attribute list.

[0329] The digitization server (100) can determine the database to process the user's query (S565).

[0330] Specifically, the digitization server (100) can classify types of queries corresponding to the intent of the query, the target of the query, and the attributes of the query by using a pre-trained artificial intelligence (AI) or rule-based algorithm. If the classified query type corresponds to structural exploration of a graph or real-time data retrieval of a graph, the digitization server (100) can set only the graph database (GDB) as the query target. Alternatively, if the classified query type corresponds to a query requiring semantic information, the digitization server (100) can set both the graph database (GDB) and the ontologies stored in the RDF repository as query targets.

[0331] The digitization server (100) can query the graph database (GDB) in response to the configured query target. Additionally, the digitization server (100) can also query the RDF repository (S570). To this end, the digitization server (100) can generate a query based on the intent of the query, the target of the query, and the attributes of the query.

[0332] Specifically, the digitization server (100) can extract two or more sample queries from a pre-set sample query list in which one or more of the query intent, query target, and query attribute match. The digitization server (100) can generate a few-shot prompt by including the query intent, query target, and query attribute and the two or more sample queries extracted from the sample query list. Then, the digitization server (100) can generate a query by inputting the few-shot prompt into a pre-trained artificial intelligence (AI) model.

[0333] Alternatively, the digitization server (100) can search for a template that matches the intent of the query from a pre-configured query template repository. The digitization server (100) may also generate a query by mapping parameters using a pre-trained artificial intelligence (AI) model to include the intent of the query, the target of the query, and the attributes of the query from the searched template.

[0334] The digitization server (100) can generate an answer corresponding to a user's query based on results retrieved from the graph database (GDB) or results retrieved from the graph database (GDB) and the RDF repository, respectively (S575).

[0335] The digitization server (100) can generate visual graphics to provide answers not only through text but also through images of process piping and instrumentation diagrams (P&ID) (S580).

[0336] Specifically, the digitization server (100) can identify the image coordinates (X, Y) and size of components (i.e., symbols, lines, and text) included in the process piping and instrumentation diagram (P&ID) from the graph database (GDB). Based on the identified image coordinates and size, the digitization server (100) can perform image processing to highlight the components related to the answer on the image containing the process piping and instrumentation diagram (P&ID).

[0337] And, the digitization server (100) can output an answer including visual graphics to the user (S585).

[0339] As described above, preferred embodiments of the present invention have been disclosed in this specification and drawings; however, it is obvious to those skilled in the art that other variations based on the technical spirit of the present invention are possible in addition to the embodiments disclosed herein. Furthermore, although specific terms have been used in this specification and drawings, they are used merely in a general sense to facilitate the explanation of the technical content of the present invention and to aid in understanding the invention, and are not intended to limit the scope of the present invention. Accordingly, the detailed description above should not be interpreted restrictively in any respect and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included within the scope of the present invention. Explanation of the symbols

[0340] 100 : Digitization Server 200 : Storage 105 : Communications Department 110 : Input / Output section 120 : Artificial Intelligence Management Department 122 : Data Management Module 124: Model Training Module 130 : Data conversion unit 132 : Graphic conversion module 134 : Graph Validation Module 136 : Ontology Transformation Module 138 : Image generation module 140 : P&ID Service Department

Claims

Claim 1 A method for generating training data, comprising: a step of acquiring an image containing a Piping and Instrument Diagram (P&ID) from a database (DB); a step of preparing a list containing one or more symbols and texts that may be included in the Piping and Instrument Diagram (P&ID); and a step of generating training data for an Artificial Intelligence (AI) model to recognize symbols and texts related to the Piping and Instrument Diagram (P&ID) from an image by applying modifications to the symbols and texts included in the acquired image or by replacing the symbols or texts included in the acquired image with the symbols or texts included in the list. Claim 2 A method for generating training data according to claim 1, wherein the step of acquiring the image is characterized by identifying a project related to a process piping and instrumentation diagram (P&ID) to be used by the artificial intelligence (AI) model, generating a query including keywords related to the identified project, and then querying the database (DB) using the generated query to acquire the image. Claim 3 A method for generating training data according to claim 1, wherein the step of generating training data is characterized by identifying one or more symbols related to a process piping and instrumentation diagram (P&ID) from the acquired image, applying geometric deformation by randomly performing one or more operations among rotation, scaling, translation, and flipping on all or part of the identified one or more symbols, readjusting the lines included in the process piping and instrumentation diagram (P&ID) so that the lines connected to the symbols before the geometric deformation is applied remain connected to the symbols after the geometric deformation is applied, and applying image deformation by randomly performing one or more operations among brightness or contrast adjustment, color jittering, background noise addition, and grid overlay addition on the image in which the lines are readjusted. Claim 4 A method for generating training data according to claim 3, wherein the step of generating training data involves searching for a symbol among one or more symbols included in the list that can replace a symbol identified from the image, replacing the symbol identified from the image with the searched symbol, and then readjusting the lines included in the process piping and instrumentation diagram (P&ID) so that the line connected to the symbol before the replacement is connected as is to the symbol after the replacement. Claim 5 A method for generating training data according to claim 1, wherein the step of generating training data involves identifying one or more texts related to a process piping and instrumentation diagram (P&ID) from the acquired image, applying text variation by randomly performing one or more operations among font variation, warping and skewing, and blur effect on all or part of the identified text, and applying image variation by randomly performing one or more operations among texture variation, annotation, and color reverse on the image to which the text variation has been applied. Claim 6 A method for generating training data according to claim 1, further comprising, after the step of generating the training data, a step of verifying whether the symbols and lines included in the generated training data are correctly arranged from a fluid dynamics perspective. Claim 7 A method for generating training data according to claim 6, wherein the verification step involves setting fluid properties and boundary conditions based on attribute values ​​assigned to a project related to a process piping and instrumentation diagram (P&ID) in which the artificial intelligence (AI) model is to be used, and performing Computational Fluid Dynamics (CFD) analysis on the generated training data according to the set fluid properties and boundary conditions to verify pressure drop, stagnation zone, turbulence, cavitation, fluid separation, and multi-phase flow for the process piping and instrumentation diagram (P&ID), wherein the fluid properties include one or more of fluid type, density, viscosity, compressibility, temperature, and pressure, and the boundary conditions include one or more of fluid flow rate, velocity, atmospheric pressure, and friction coefficient within the pipe. Claim 8 A method for generating training data according to claim 6, wherein the verification step divides the generated training data into multiple regions based on the type of symbol to be included in each region, applies a guideword to each divided region to derive a risk factor for each region, and performs a risk assessment of the generated training data in correspondence with the derived risk factor for each region, wherein the guideword is a standardized keyword for identifying abnormal conditions that may occur depending on process flow or operating conditions. Claim 9 A method for generating learning data according to claim 6, wherein the verification step involves obtaining a Process Flow Diagram (PFD) stored in the database (DB) that is matched with the Process Piping and Instrumentation Diagram (P&ID), and performing verification by mutually comparing the generated learning data and the obtained Process Flow Diagram (PFD) according to a pre-set checklist, wherein the checklist includes one or more of information regarding equipment configuration, piping configuration, instrument configuration, process conditions, and safety devices. Claim 10 A computer program recorded on a recording medium for executing the steps of: the processor acquiring an image containing a process piping and instrumentation diagram (P&ID) from a database (DB); the processor preparing a list containing one or more symbols and texts that may be included in the process piping and instrumentation diagram (P&ID); and the processor generating training data for an artificial intelligence (AI) model to recognize symbols and texts related to the process piping and instrumentation diagram (P&ID) from an image by applying modifications to the symbols and texts included in the acquired image or by replacing the symbols or texts included in the acquired image with the symbols or texts included in the list.

Images