System and method for generating and modifying engineering schematic diagrams

Abstract

A method (100) for generating and modifying an engineering schematic diagram is disclosed. The method includes receiving a natural language input via a user interface, the input including a user intent to perform an operation on an engineering schematic. The method uses a language model to generate a structured representation of the input, the structured representation identifying components, actions, and attributes. A knowledge graph query compatible with the standardized ontology is generated based on the representation. These queries are executed to retrieve component, connection, and attribute data from the knowledge graph. A systematic representation of the schematic diagram is generated using the retrieved data and the specified action. The method then renders a visual representation of the engineering schematic based on the system representation, and outputs it via a user interface. This approach allows intuitive schematic modification using natural language processing and knowledge graph techniques.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of computer-aided design and engineering. More specifically, the present invention relates to systems and methods for generating and modifying engineering schematics using natural language processing and knowledge graph technology. BACKGROUND

[0002] Engineering schematics, such as piping and instrumentation diagrams (P&IDs), process flow diagrams (PFDs), heating, ventilation, and air conditioning (HVAC) layouts, and electrical wiring diagrams, are indispensable tools in various industrial sectors. These schematics serve as visual representations of complex systems, enabling engineers and technicians to design, analyze, and maintain industrial processes and facilities. While the conceptual design and detailed design processes of such systems involve significant thought and are creative processes, the actual process of drawing these schematics is a repetitive, manual process, where much of the effort is in realizing the ideas, rather than ideation itself. Despite being a skilled trade, it can still require iterations to ensure clarity, correctness, and compliance with standards. Thus, the task of translating a system design into a schematic is manual-intensive and iterative.

[0003] Due to the designer's heavy reliance on their familiarity and proficiency with the authoring tool, the fluency / interactivity / seamlessness of translating their ideas into a detailed design is not yet at the level that it can be. They must invest a specified effort to draw it out correctly before they can iterate on it to improve it. Furthermore, similar to problems in the field of computer programming, the challenge of maintaining existing schematics and change management, if not more challenging, is comparable to that of creating from scratch. This is because a complete understanding of the schematic of the working system and its association with the physical system is a prerequisite for maintaining existing schematics.

[0004] Some computer-aided design tools have been developed to assist in creating and modifying engineering schematics. These tools typically provide a library of standard components, a drag-and-drop interface, and automatic layout algorithms. While these solutions have improved efficiency to some extent, they still require users to have in-depth knowledge of the software interface and the ability to manually manipulate individual components. Overall, such conventional approaches have the following limitations. They typically have a steep learning curve, requiring significant training of the user to become proficient. The process of translating high-level design concepts into detailed schematics is still largely manual, limiting the speed at which engineers can iterate on their designs. These tools typically lack the ability to understand and interpret natural language instructions, forcing users to interact with the system through rigid, pre-defined commands and menu structures.

[0005] Accordingly, there is a need for a solution that can reduce / eliminate the gap from conception to detailed rendering of a schematic by automating most of the drawing process; enable intelligent understanding of the underlying knowledge of a schematic to help an analysis system evaluate it; and provide a natural means for a human expert to interact with the schematic representation of the system at work.

[0006] The present invention aims to provide a system and method for generating and modifying engineering schematics that can bridge the gap between natural human expression of design intent and the technical requirements of schematic creation. The system and method of the present invention provide a solution for interpreting natural language instructions, understanding underlying engineering principles, and correspondingly generating or modifying schematics, while ensuring compliance with industry standards and best practices. SUMMARY

[0007] The object of the present invention is achieved by a method for generating and modifying engineering schematics. The method employs a processing unit to perform a series of steps to achieve this object. The method comprises receiving, via a user interface, a natural language input related to an engineering schematic. Herein, the natural language input comprises a user intent to perform one or more operations on the engineering schematic. The method further comprises generating a structured representation of the natural language input by processing the natural language input using a language model. The structured representation comprises one or more of: identified engineering components, actions to be performed on the identified engineering components, and attributes associated with the identified engineering components. The method further comprises generating one or more knowledge graph queries for a knowledge graph based on the structured representation. Herein, the knowledge graph queries are compatible with the knowledge graph based on a standardized ontology for a domain of the engineering schematic. The method further comprises executing the one or more knowledge graph queries to retrieve from the knowledge graph one or more of: component data corresponding to the identified engineering components, connection data specifying allowable connections between the identified engineering components, and attribute data specifying allowable attributes for the identified engineering components and allowable connections. The method further comprises generating a system representation corresponding to the engineering schematic using the knowledge graph based on the retrieved component data, connection data, and attribute data and the specified actions in the structured representation. The method further comprises rendering a visual representation of the engineering schematic based on the generated system representation. The method further comprises outputting, via the user interface, the visual representation of the engineering schematic.

[0008] In one or more embodiments, the language model is a large language model trained on domain-specific engineering terminology and syntax.

[0009] In one or more embodiments, processing the natural language input using the language model further includes interpreting an explicit intent stated in the natural language input and deriving an implicit intent based on at least one of: a context from previous user interactions, domain-specific rules, and industry best practices.

[0010] In one or more embodiments, generating the one or more knowledge graph queries includes analyzing the structured representation to identify constituent components. The method further includes preparing hints based on a catalog of examples for different types of intents. The method then includes generating the one or more knowledge graph queries using the prepared hints.

[0011] In one or more embodiments, the knowledge graph includes nodes representing engineering components and edges representing connections between engineering components. Herein, generating the system representation using the knowledge graph includes at least one of: adding nodes and edges within the knowledge graph, modifying existing nodes or edges, and removing nodes or edges.

[0012] In one or more embodiments, generating the system representation using the knowledge graph includes comparing specifications in the structured representation to a database of standardized components. The method further includes identifying a standardized component corresponding to the specification based on the comparison. The method then includes generating the system representation using the identified standardized component.

[0013] In one or more embodiments, rendering the visual representation of the engineering schematic includes determining a spatial arrangement of engineering components using a layout algorithm. The method then includes rendering the engineering components and their connections based on the determined spatial arrangement.

[0014] In one or more embodiments, the method further includes receiving user input via a user interface to modify the visual representation of the engineering schematic. The method further includes updating the system representation based on the user input. The method further includes re-rendering the visual representation of the engineering schematic based on the updated system representation.

[0015] The object of the present invention is also achieved by a system comprising one or more processing units and a memory unit communicatively coupled with the one or more processing units. Herein, the memory unit comprises an engineering schematic design module stored in the form of machine readable instructions executable by the one or more processing units, wherein the engineering schematic design module is configured to perform the aforementioned method steps for generating and modifying an engineering schematic.

[0016] The object of the present invention is also achieved by a computer program product having stored therein machine readable instructions which, when executed by the one or more processing units, cause the one or more processing units to perform the aforementioned method steps.

[0017] However, other aspects, features, and advantages of the present application will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the application. The application can be implemented in various ways, including as a method, system, computer program product, and / or the like. The present application can be implemented in many different ways and is not limited to the embodiments described below, which are presented as examples of the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] A more complete appreciation of the application and its many attendant advantages will be readily understood by reference to the following detailed description, taken in conjunction with the accompanying drawings, in which: Figure 1 is a flowchart of a method for generating and modifying engineering schematics according to one or more embodiments of the application; Figure 2 is a block diagram representation of a system for generating and modifying engineering schematics according to one or more embodiments of the application; and Figure 3 is a flowchart depicting a process for generating and modifying engineering schematics from natural language input using a knowledge graph according to one or more embodiments of the application. DETAILED DESCRIPTION

[0019] Various embodiments are described in the detailed description, drawings, and claims below. Similar reference characters denote corresponding features consistently throughout the attached drawings and the detailed description. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It will be evident, however, to one skilled in the art that the embodiments can be practiced without these specific details.

[0020] Examples of methods, systems, and computer program products for generating and modifying engineering schematics are disclosed herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the application. It will be evident, however, to one skilled in the art that embodiments of the application can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring embodiments of the application.

[0021] For the purposes of the present invention, the term "engineering diagram" refers to a representation used to document and communicate the design, structure, and operation of complex systems in various industries. Engineering diagrams can include, but are not limited to, piping and instrumentation diagrams (P&IDs), process flow diagrams (PFDs), electrical schematics, and mechanical drawings. Engineering diagrams are typically composed of graphical symbols representing components and equipment, lines or arrows representing connections or flow, and textual annotations providing additional information. The process of generating and modifying engineering diagrams refers to the creation and updating of these detailed technical drawings. This involves adding, removing, or altering components and connections within the diagram, ensuring that the drawing accurately reflects the intended design and functionality of the engineered system.

[0022] Referring to Figure 1 FIG. 1 illustrates a flowchart of a method (denoted by reference numeral 100) for generating and modifying engineering diagrams, in accordance with an embodiment of the present invention. The method 100 is implemented to enable users to interact with the system using natural language inputs, which are then interpreted and translated into actionable modifications to the engineering diagrams. The method 100 combines the interpretive capabilities of large language models (LLMs), the representational capabilities of knowledge graphs (KGs), and human-centered interaction methods to enable the creation, modification, and validation of engineering diagrams. The method 100 significantly reduces the manual effort involved in the design of the diagrams, improves the accuracy of the design, and facilitates seamless updates and modifications.

[0023] Referring to Figure 2FIG. 1 illustrates a block diagram of a system 100 for generating and modifying engineering schematics, in accordance with one or more embodiments of the present application. It can be appreciated that the system 100 described herein can be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. One or more embodiments of the present embodiment can take the form of a computer program product which can include program modules accessible from a computer-usable or computer-readable medium storing program codes designed and constructed for use with or in conjunction with one or more computers, processors, or instruction execution systems. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the programs for use by or in connection with the instruction execution system, apparatus, or device. The medium can be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium of which the signal bearer is excluded from the definition of physical computer-readable medium including a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid disk, and an optical disk such as a compact disk read-only memory (CD-ROM), compact disk recordable (CD-R), and digital versatile disk (DVD). Both the processor and the program codes for implementing each aspect of the technology can be centralized or distributed (or a combination thereof), as known to those skilled in the art.

[0024] In one example, the system 200 can be embodied as a computer program product programmed for generating and modifying engineering schematics. The system 200 can be incorporated into one or more physical packages (e.g., chips). In one example, a physical package includes a structural assembly (e.g., a base board) with one or more materials, components, and / or wires on it, which are configured to provide one or more characteristics such as physical strength, conservation of size, and / or limitation of electrical interaction. It is contemplated that in certain embodiments, a computing device can implement a single chip. As illustrated, the system 200 includes a communication mechanism, such as a bus 202, for communicating information between components of the system 200. The system 200 includes one or more processing units 204 and one or more memory units 206. In this regard, the memory unit 206 is communicatively coupled to the processing unit 204. In one example, the memory unit 206 can be embodied as a computer-readable medium on which a computer program can be stored and loaded into the system 200 so as to implement the steps described herein for achieving the purposes.

[0025] Generally, as used herein, the term "processing unit" refers to a computational element operable to respond to and process instructions that drive the system 200. Optionally, the processing unit includes, but is not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other type of processing circuit. Moreover, the term "processing unit" can refer to one or more individual processors, processing devices, and various elements associated with the processing devices that can be shared by other processing devices. Additionally, the one or more individual processors, processing devices, and elements are arranged in various architectures for responding to and processing instructions that drive the system 200.

[0026] Herein, the memory unit 206 can be volatile memory and / or non-volatile memory. The memory unit 206 can be coupled to communicate with the processing unit 204. The processing unit 204 can execute instructions and / or code stored in the memory unit 206. Various computer-readable storage media can be stored in and accessed from the memory unit 206. The memory unit 206 can include any suitable elements for storing data and machine-readable instructions, such as read-only memory, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, hard disk drives, removable media drives used for processing compact discs, digital video discs, magnetic disks, magnetic cassettes, memory cards, and the like.

[0027] In particular, the processing unit 204 has a connection to the bus 202 to execute instructions and process information stored in the memory unit 206. The processing unit 204 can include one or more processing cores, each core configured to execute independently. A multi-core processor enables multiprocessing within a single physical package. Examples of multi-core processors include two, four, eight or more numbers of processing cores. Alternatively or additionally, the processing unit 204 can include one or more microprocessors configured in tandem via the bus 202 to enable independent execution of instructions, pipelining, and multithreading. The processing unit 204 can also be accompanied with one or more specialized components to perform certain processing functions and tasks such as one or more digital signal processors (DSP) and / or one or more application-specific integrated circuits (ASIC). Other specialized components to aid in performing the inventive functions described herein include one or more field- programmable gate arrays (FPGA) (not shown), one or more controllers (not shown), or one or more other specialized computer chips.

[0028] The system 200 can also include an interface 208, such as a communication interface (the terms are used interchangeably), which can enable the system 200 to communicate with other systems to receive and transmit information. The communication interface 208 can include a medium (e.g., a communication channel) through which the system 200 communicates with other systems. Examples of the communication interface 208 can include, but are not limited to, a communication channel in a computer cluster, a local area network communication channel (LAN), a cellular communication channel, a wireless sensor communication channel (WSN), a cloud communication channel, a metropolitan area network communication channel (MAN), and / or the Internet. Optionally, the communication interface 208 can include one or more of a wired connection, a wireless network, a cellular network (such as 2G, 3G, 4G, 5G mobile networks), and a Zigbee connection.

[0029] The system 200 also includes a database 210. As used herein, the database 210 is an organized collection of structured data, typically stored in a computer system, and designed for ease of access, management, and updating. The database 210 can be in the form of a central repository of information that can be queried, analyzed, and processed to support various applications and business processes. In the system 200, the database 210 provides a mechanism for storing, retrieving, updating, and deleting data, and typically includes functions such as data validation, security, backup and recovery, and data modeling.

[0030] The system 200 also includes an input device 212 and an output device 214. The input device 212 can take various forms depending on the specific application of the system 200. In one example, the input device 212 can include one or more of a keyboard, a mouse, a touch screen display, a microphone, a camera, or any other hardware component that enables a user to interact with the system 200. Further, the output device 214 can be in the form of a display, a printer, a communication channel, and the like, without any limitation.

[0031] In the present system 200, the processing unit 204 and the accompanying components have a connection to the memory unit 206 via the bus 202. The memory unit 206 includes both dynamic memory (e.g., RAM, disk, writable optical disk, etc.) and static memory (e.g., ROM, CD-ROM, etc.) for storing executable instructions that, when executed, carry out the method steps described herein for generating and modifying engineering schematics. Specifically, the memory unit 206 includes an engineering schematic design module 216 to perform the steps for the purpose.

[0032] In conjunction with Figures 1-2described below) can be performed in the system 200, or particularly in the processing unit 204 of the system 200, for generating and modifying engineering schematics. It can be appreciated that although the method 100 is illustrated and described as a series of steps, it is contemplated that various embodiments of the method 100 can be performed in any order or different combination, and need not include all of the illustrated steps.

[0033] At step 102, the method 100 includes receiving, via a user interface, a natural language input related to an engineering schematic. For this purpose, the user interface, such as the interface 208, is configured to accept text input from a user, which allows the user to describe desired modifications or additions to the engineering schematic using everyday language. The natural language input is related to an engineering schematic, which can be a piping and instrumentation diagram (P&ID), a process flow diagram (PFD), a heating, ventilation, and air conditioning (HVAC) layout, or an electrical wiring diagram. The natural language input includes a user intent to perform one or more operations on the engineering schematic. The operations can include adding new components, modifying existing components, removing components, or establishing connections between components.

[0034] The natural language input serves as the primary mode of interaction between the user and the system 200, bridging the gap between human expression of design intent and the technical requirements of schematic creation. This approach significantly reduces the learning curve associated with traditional computer-aided design software, as the user can express their design intent in familiar language without having to learn complex software interfaces or command structures. For example, the user can input a statement such as "Create a flow path between the two tanks and specify the properties of the tanks, i.e., their heights." This natural language input contains the user intent to create a flow path between two tanks and specify the properties of the tanks, i.e., their heights. This input is interpreted to extract the relevant information needed to modify the engineering schematic accordingly. In the given example, more is the interpretation of the user intent and breaking it down into the following components: Moving crude oil from tank A to tank B, tank A height is 20m and tank B height is 40m Action: move; source: tank A; destination: tank B; numeric properties: (height_source: 20m, height_destination: 40m); object: fluid (crude oil) Moving crude oil from tank A to tank B, tank A height is 20m and tank B height is 40m Figure 3 This structure is merely a representative example. The actual structuring can vary and the entire process of providing the correct input (utterance, current state, context, domain, organization) is discussed in more detail later.

[0035] ​The method 100 is able to account for both explicit and implicit intent from natural language input. Explicit intent is directly stated in the input, such as the request to move crude oil between storage tanks in the previous example. Implicit intent is derived based on context from previous user interactions, domain-specific rules, and industry best practices. For example, if a user previously specified that all fluid transfers should use centrifugal pumps, it can be implicitly understood that a pump needs to be added between the storage tanks, even though this is not explicitly stated in the current input.

[0036] As used herein, the user interface can be part of the input device 212 of the system 200 for receiving natural language input. In the current implementation, the user interface is designed to be user-friendly and intuitive, allowing engineers to express their design requirements in a natural way to them. This approach eliminates the need for users to learn a complex query language or navigate through multiple menus and options to find the information they need. In some examples, the user interface can also provide functionality to help users formulate their queries. This can include autocomplete suggestions based on commonly used terms in engineering schematics, or a list of example queries to help users understand the types of problems the system can handle. These functionalities can help users, especially new users to the system, to most effectively utilize the natural language input capability.

[0037] The natural language input (query) serves as the starting point for the process of generating recommendations. By allowing users to express their needs in natural language, the method 100 can capture specific element requests as well as contextual information about the project, such as the type of facility being designed or specific operational requirements. This contextual information allows for accurate and relevant recommendations to be generated in subsequent steps of the process.

[0038] At step 104, the method 100 includes generating a structured representation of the natural language input by processing the natural language input using a language model. This is done to transform the natural language input into a format that can be effectively used to search a knowledge graph (as discussed later) and generate relevant recommendations. This structured representation is formatted in a way that can be directly used to search the knowledge graph. In the case of using SPARQL, the generated structured representation can include specific graph patterns to match against nodes and edges in the knowledge graph. The conversion from natural language input to structured representation is a process of mapping natural language concepts to the specific structures and terminology used in the knowledge graph. This can involve expanding abbreviations, resolving ambiguities, and inferring additional relevant search terms based on the context of the query.

[0039] Here, the structured representation includes one or more of the following: identified engineering components, actions to be performed on the identified engineering components, and attributes associated with the identified engineering components. The identified engineering components in the structured representation refer to specific elements of the engineering diagram that are mentioned or implied in the natural language input. These components can include, but are not limited to, tanks, pumps, valves, pipes, heat exchangers, and sensors. The actions to be performed on the identified engineering components are also captured in the structured representation. These actions correspond to the operations that the user intends to perform on the engineering diagram. These actions can include adding new components, removing existing components, modifying component properties, and establishing connections between components. The attributes associated with the identified engineering components provide specific details about the components or the actions to be performed. These attributes can include physical properties such as dimensions, capacity, or materials; operational parameters like flow rate, temperature, or pressure; or location information such as coordinates or relative positions within the diagram.

[0040] The generated structured representation serves as an intermediate step between the natural language input and the technical operations. Specifically, the structured representation acts as a bridge, translating the human intent into a format that can be used to query the knowledge graph and ultimately generate or modify the engineering diagram. This structured format ensures that all relevant information from the user's natural language input is captured and organized in a way that facilitates further processing.

[0041] In one embodiment, the language model is a large language model (LLM) trained on domain-specific engineering terminology and syntax. This specialized training enables the model to accurately interpret and understand the nuances of engineering language used in the context of creating and modifying diagrams. The LLM is exposed to a large amount of engineering documents, technical specifications, industry standards, and engineering diagram examples during its training phase. This exposure allows the LLM to develop an understanding of the vocabulary, phrasing, and conceptual relationships commonly found in the engineering domain. The domain-specific training also enables the LLM to understand the syntax commonly used when describing engineering systems, including how components are typically described in relation to each other, how specifications are typically stated, and how modifications to systems are typically expressed. This expertise allows the LLM to more accurately parse and interpret natural language input related to engineering diagrams, extracting relevant information and intent even when the input uses highly technical language or domain-specific language.

[0042] In the case of an LLM, deriving the structured output (as discussed above) is performed by prompting the LLM. The training or fine-tuning of such an LLM is outside the scope of the present invention and is therefore not discussed here. The next step is to translate this set of requirements into components of the domain and their attributes again by using specific prompts to the LLM (which can be the same or different from the ones mentioned above). The expected output is: While a simplistic example is stated for purposes of explanation, the intent is actually a complex collection that can consist of what is explicitly stated by the user, and then there can be implicit intent as well to add a layer of explanation, such as context based on user interaction history (e.g., domain specific language such as “star pump”), domain specific rules (such as not allowing mixing of two different pH fluids, or using food grade materials for piping), and best practices.

[0043] In another embodiment, the conversion from natural language utterance to structured form can be handled by Controlled Natural Language (CNL), natural language translation, first order logic, etc. For example, a CNL model uses a restricted version of natural language designed for the specific purpose of describing engineering schematics. The CNL approach provides increased accuracy in intent understanding at the expense of some flexibility in the input phrasing. In the case of CNL, the human intent is described through a controlled natural language rather than a fully free-form natural language text. This is a restricted version of natural language designed to meet specific purposes within a specific domain. CNL uses a well-defined subset of language syntax, grammar, and semantics, but adds terminology required for the technical domain. Thus, the vocabulary is familiar to the technical domain expert / user. Taking the P&ID use case as an example, a CNL data model can include concepts such as action, source, destination, subject, object, and quantity. Here, the user query “ ​ ​ ” can then be interpreted to the intended form based on the CNL mapping.

[0044] It can be appreciated that using CNL reduces flexibility compared to the LLM case using free text, but this approach provides certain benefits, such as: a) user familiarity with the domain vocabulary; b) increased accuracy in intent understanding; c) less error and greater robustness; d) as a form of guidance to the user, where the UI can even be enhanced to be more intuitive (e.g., allowing the user to specify the required concepts, providing options for selection) rather than requiring the user to start from a blank text field; e) improved efficiency and accuracy of machine processing; f) input and intermediate output structured, reducing the need for pre / post processing; and g) better adherence to industry standards by designing the CNL data model to align with standards.

[0045] In this embodiment, processing the natural language input using the language model involves interpreting explicit intents and deriving implicit intents. Specifically, the method 100 includes interpreting explicit intents stated in the natural language input. When interpreting explicit intents stated in the natural language input, the language model identifies and extracts clearly stated instructions, specifications, or requests. This includes identifying explicitly mentioned components, actions, and properties. For example, if the input states “add a pump with a flow rate of 100 liters per minute,” the model interprets this as an explicit intent to add a pump component with a specific flow rate property. In addition, the method 100 includes deriving implicit intents based on at least one of the following: context from previous user interactions, domain-specific rules, and industry best practices. This involves inferring additional information or requirements that are not explicitly stated in the current input. The model can consider the history of previous interactions with the user to understand preferences or repeated patterns in their design approach. The model can also apply domain-specific rules, such as automatically including a necessary safety valve when certain conditions are met, even if this is not explicitly requested. Furthermore, the model incorporates industry best practices, possibly suggesting modifications or additions that align with standard engineering principles. For example, if the user requests a connection between two storage tanks at different heights, the model can implicitly understand that a pump is required, even if this is not explicitly mentioned, based on the principle that a pump is needed to transport fluids to a certain height. This combination of explicit and implicit intent interpretation allows for the generation of an accurate structured representation that captures both the stated requirements of the user and the unstated but necessary considerations defined by engineering principles and best practices.

[0046] At step 106, the method 100 includes generating one or more knowledge graph queries based on the structured representation as a knowledge graph. These knowledge graph queries are designed to interact with a knowledge graph that represents domain knowledge related to engineering schematics. The process of generating these queries involves translating the information contained in the structured representation into a format that can be used to effectively query the knowledge graph. The knowledge graph queries are generated based on the structured representation, which includes the identified engineering components, actions to be performed, and associated properties. Each element of the structured representation can correspond to one or more queries, depending on the complexity of the user intent and the information required from the knowledge graph.

[0047] The knowledge graph query is based on a standardized ontology compatible with the engineering diagram domain. This standardized ontology ensures consistency across terms and relationships for different types of engineering diagrams. For example, in the case of piping and instrumentation diagrams (P&IDs), the method 100 can use the DEXPI (Data Exchange for Process Industries) ontology or a customized version thereof. Using a standardized ontology allows for the generation of queries that accurately map to the structure and content of the knowledge graph. In the current implementation, the generated queries are typically in a format such as SPARQL (SPARQL Protocol and RDF Query Language), which is designed for querying graph databases. These queries are structured to navigate the relationships between nodes in the knowledge graph, retrieve relevant data, and potentially modify the graph structure when needed by the user intent.

[0048] In the present embodiment, the process of generating one or more knowledge graph queries involves a series of steps designed to translate the structured representation of the user intent into specific queries that can interact with the knowledge graph. First, the method 100 involves analyzing the structured representation to identify the constituent components. This includes examining each element, including the identified engineering components, actions to be performed, and associated attributes. This analysis determines which specific pieces of information need to be retrieved from or manipulated in the knowledge graph. Then, the method 100 involves preparing prompts based on a catalog of examples for different types of intents. This catalog serves as a repository of pre-defined query templates that correspond to various common operations and queries in the engineering diagram domain. For the present purposes, the appropriate template is selected based on the identified constituent components and the overall intent derived from the structured representation. Finally, the method 100 involves generating the one or more knowledge graph queries using the prepared prompts. This combines the selected query template with specific information from the structured representation to create fully formed queries, typically in a language like SPARQL. The method 100 can involve populating placeholders in the query template with specific component names, attributes, and other relevant details from the structured representation. This process allows for the conversion of the user's natural language input into precise, structured queries that can effectively interact with the knowledge graph while ensuring consistency with its standardized ontology.

[0049] In general, the prompts can be prepared through one-shot learning, for example, based on a catalog of examples for different types of intents such as "create," "modify," and "delete." Once the prompts themselves are constructed, these prompts are provided to create the SPARQL queries that ultimately interact with the knowledge graph. An example query to add two tanks to the knowledge graph is as follows.

[0050] It will be appreciated that such knowledge graph queries are used to create, extend, or otherwise modify the underlying knowledge graph corresponding to the system representation.

[0051] The method 100 ensures that the information required to implement the user intent can be accurately retrieved and manipulated while maintaining consistency with industry standards and best practices represented in the knowledge graph by generating knowledge graph queries that are compatible with the standardized ontology. For example, if the structured representation indicates that a pump with a specific flow rate requirement needs to be added, the method 100 can generate a query to retrieve information about compatible pump types, their specifications, and their connection requirements with other components in the engineering system. The query can seek a pump that satisfies the specified flow rate, is suitable for the type of fluid being moved, and is compatible with existing components in the engineering schematic. This approach allows the comprehensive domain knowledge encoded in the knowledge graph to be leveraged to generate accurate and standards-compliant engineering schematics.

[0052] At step 108, the method 100 includes executing the one or more knowledge graph queries to retrieve from the knowledge graph one or more of: component data corresponding to the identified engineering components, connection data specifying allowable connections between the identified engineering components, and attribute data specifying allowable attributes for the identified engineering components and allowable connections. This retrieval process involves accessing the knowledge graph, which is structured based on the standardized ontology for the engineering schematic domain. Here, the component data includes detailed information about each component type, such as its standard representation, typical use cases, and general specifications. For example, if a pump is identified in the user input, the system retrieves data about various types of pumps, their standard symbols in schematics, and their general characteristics. The connection data specifying allowable connections between the identified engineering components defines how different components can be connected in an engineering schematic, ensuring that the system only proposes or allows connections that are physically and logically valid within the engineering schematic domain. For example, the connection data can specify that a valve can be connected to a pipe but cannot be connected directly to a tank without an intermediate pipe. The attribute data specifying allowable attributes for the identified engineering components and allowable connections defines the range of valid properties and parameters for each component and connection type. For a pump, this can include allowable flow rates, pressure ratings, or materials of manufacture. For connections between components, this can specify allowable diameters, materials, or operating conditions. By retrieving this comprehensive set of data, the method 100 ensures that all subsequent steps in generating or modifying the engineering schematic are based on accurate, standardized information that conforms to industry norms and best practices.

[0053] At step 110, the method 100 includes generating a system representation corresponding to the engineering schematic using the knowledge graph based on the retrieved component data, connection data, and property data, and the specified actions in the structured representation. This generation process involves creating an internal model of the engineering schematic that incorporates all relevant information obtained from the knowledge graph and user input. The process begins by mapping the identified engineering components from the structured representation to their corresponding representations in the retrieved component data. Each component is instantiated in the system representation with the associated properties and attributes specified in the property data. Then, the process establishes connections between these components based on the connection data and the specified actions in the structured representation. These connections are created according to the allowable connections defined in the connection data, ensuring that only valid and logical connections are made. Then, the specified actions from the structured representation, such as adding, modifying, or removing components, are applied to the evolving system representation. Throughout the process, the knowledge graph is continuously referenced to verify that all components, connections, and properties in the system representation comply with standardized ontologies and domain-specific rules. If the specified actions in the structured representation involve modifying an existing engineering schematic, the process first loads the existing engineering schematic into the system representation and then applies the necessary modifications. The resulting system representation serves as a complete and accurate model of the engineering schematic, incorporating all user-specified changes while maintaining compliance with industry standards and best practices encoded in the knowledge graph.

[0054] As used herein, the knowledge graph represents engineering components, and the edges represent connections between engineering components. This structure allows for a flexible and comprehensive representation of engineering systems, where each node defines the properties and attributes of a particular component, and each edge defines the nature of the relationship or connection between components. When generating a system representation using the knowledge graph, the system performs operations that directly manipulate this graph structure. These operations include at least one of the following: adding nodes and edges within the knowledge graph, modifying existing nodes or edges, and removing nodes or edges. The process of adding nodes involves creating new instances of engineering components in the system representation, each represented by a node with associated attributes specified in the retrieved component data. Adding edges requires establishing new connections between existing or newly created nodes, adhering to the allowable connections defined in the connection data. Modifying existing nodes or edges is performed when user input requires changes to the properties or relationships of components already present in the system representation. This can involve updating attribute values of nodes or altering the properties of edges to reflect changes in connections. Removing nodes or edges is performed when user input specifies the deletion of components or the disconnection of previously connected components. These modifications to the knowledge graph structure allow the system to accurately reflect the user's intended changes to the engineering schematic while maintaining the integrity and validity of the system representation.

[0055] In particular, the process of generating a system representation using the knowledge graph in the method 100 involves multiple steps to ensure the use of standardized components. First, the method 100 involves comparing the specification in the structured representation to a database of standardized components. This database, such as the database 210, contains information about industry standard components, their properties, and typical uses. The comparison process involves matching parameters such as component type, size, capacity, material, and other relevant attributes. Next, the method 100 involves identifying a standardized component that corresponds to the specification based on the comparison. When a perfect match is found in the database, the standardized component is selected for the system representation. In cases where a perfect match is not available, the method 100 identifies the closest matching standardized component or a combination of components that satisfies the specified requirements. Finally, the method 100 involves generating the system representation using the identified standardized components. Each identified standardized component is instantiated as a node in the knowledge graph, complete with all of its standard properties and attributes. The method 100 then establishes the necessary connections between the components based on the structured representation and the allowable connections defined in the knowledge graph, represented as edges in the graph. In this way, the method 100 ensures that the generated system representation adheres to industry specifications and standards, facilitating compatibility, maintenance, and compliance with regulatory requirements in the resulting engineering schematic.

[0056] At step 112, the method 100 includes rendering a visual representation of the engineering schematic based on the generated system representation. This rendering process translates the system representation, which exists as a set of structured nodes and edges in the knowledge graph, into a visual format that users can easily understand and manipulate. The rendering process begins by retrieving the visual symbols or icons associated with each engineering component in the system representation. These symbols are typically standardized within a particular engineering domain and are stored in the knowledge graph as part of the component data. The method 100 then involves determining the spatial arrangement of the engineering components using a layout algorithm. Here, the layout algorithm can implement a suitable networking layout, such as an orthographic layout, which appears to be the closest to the field. The spatial arrangement of the engineering components is determined on a two-dimensional canvas. This arrangement is guided by the connections between the components defined in the system representation and the standard layout practices for the type of engineering schematic being created. Finally, the method 100 involves rendering the engineering components and their connections based on the determined spatial arrangement. This step includes drawing the connections between the components, which are represented as lines or arrows depending on the nature of the connection and the standards of the particular engineering domain. Labels, annotations, and other textual information from the system representation are added to the visual representation, providing necessary details about the components, connections, and system parameters. The rendered visual representation serves as a graphical interface through which users can view, understand, and interact with the engineering schematic.

[0057] At step 114, the method 100 includes outputting, via a user interface, a visual representation of the engineering schematic. This output step involves displaying the rendered visual representation on an output device, such as the output device 214, e.g., a screen or other display device accessible to the user. The user interface presents the engineering schematic in a format that allows for clear visualization of all components, connections, and related details. In addition to the visual representation, the method 100 can output supplemental information via the user interface, such as a legend explaining the meaning of symbols, a list of components and their attributes, or summary statistics about the engineering schematic. The visual representation is not merely an image, but a vector graphic representation that is tightly linked to the underlying knowledge graph through the drawing canvas, such that selecting, editing, and deleting individual parts of the rendered engineering schematic are all possible. Overall, the output visual representation serves as the final product of the schematic generation or modification process and the starting point for further user interaction, allowing the user to visually verify the results of their natural language input and potentially make further modifications or refinements to the engineering schematic.

[0058] In some embodiments, the method 100 also includes additional steps that facilitate iterative modification of the engineering schematic. This involves receiving user input via the user interface to modify the visual representation of the engineering schematic. The user input can take various forms, such as a drag-and-drop operation to reposition a component, a mouse click to select and modify a component or connection, or a text input to change a component property or add an annotation. Upon receiving such user input, the method 100 proceeds to update the system representation based on the user input. This update process involves translating the user’s modifications back into the structured format of the system representation. For example, if the user moved a component in the visual representation, the method 100 updates the position attribute of the corresponding node in the system representation. After updating the system representation, the method 100 includes re-rendering the visual representation of the engineering schematic based on the updated system representation. This re-rendering process follows the same principles as the initial rendering, translating the modified system representation into the updated visual format. This iterative process enables the user to further refine the output engineering schematic as needed.

[0059] In some embodiments, the method 100 further includes validating the outputted engineering schematic. The validation process can involve accessing historical data stored within the knowledge graph. This historical data can include information about past engineering schematic designs, including successful implementations and designs that can have encountered problems. This data includes detailed information about combinations of elements and connections that have been used in various contexts, their performance metrics, any problems encountered, and modifications made over time. The validation process can flag potential problems or inconsistencies for further review. For example, if a certain connection in the outputted engineering schematic has been associated with performance problems in past designs, such a connection can be highlighted to draw the user's attention. The results of this validation process are used to refine the engineering schematic.

[0060] In one implementation, the method 100 further includes implementing application programming interfaces (APIs) to integrate with engineering design tools. These APIs serve as a bridge between the system and various engineering design tools commonly used in the industry. These design tools can include computer-aided design (CAD) software, specialized piping and instrumentation diagram (P&ID) tools, or other engineering-specific applications, including but not limited to COMOS and CAD-CAM. This API implementation provides the ability to export the generated / modified engineering schematic back into the user's existing design environment. This integration capability enhances the utility of the system 200, allowing engineers to leverage its advanced functionality within their familiar design environment, thereby improving the efficiency and consistency of the engineering design process.

[0061] Referring now to ​ , a schematic diagram illustrating an architecture for generating and modifying engineering schematics using natural language processing and knowledge graph technology is shown. The process 300 includes a number of interconnected steps. The process 300 implements a user interface, represented by reference numeral 301. The process 300 begins with user input and proceeds through various stages of natural language processing, intent understanding, knowledge graph manipulation, and visual rendering.

[0062] At step 302, process 300 receives natural language input related to an engineering schematic. This input includes a user intent to perform one or more operations on the engineering schematic, such as adding, extending, or modifying components or connections. At step 304, process 300 implements natural language processing (NLP) techniques to parse the intent from the natural language input. This involves analyzing the linguistic structure of the input to extract meaningful information. Process 300 then involves analyzing the parsed intent to determine both explicit and implicit intent. Specifically, step 306 involves identifying instructions or requests for additions to the engineering schematic provided in the natural language input (such as, add this component or group of components). Step 308 involves identifying instructions or requests for modifications to the engineering schematic provided in the natural language input (such as, correct this component or group of components). Step 310 involves understanding the instructions to exactly identify which components or groups of components are being referenced. At the same time, process 300 analyzes for implicit intent. At step 312, process 300 derives additional context from the user interaction history and resolves any anaphoric references (pronouns or other words that refer to previously mentioned entities) in the natural language input. Step 314 involves applying domain-specific rules to the interpreted intent, ensuring that the proposed changes align with established engineering principles and constraints. Step 316 incorporates best practices from industry experience for modifications or additions that align with standard engineering principles, even if the user did not explicitly request.

[0063] Process 300 also involves knowledge graph (KG) query generation. Specifically, step 318 involves constructing prompts based on the analyzed explicit and implicit intent. These prompts serve as an intermediate structure that bridges the gap between the natural language input and the formal queries required to interact with a knowledge graph. At step 320, process 300 uses these constructed prompts to generate specific knowledge graph queries by applying one or more selection, update, or filtering techniques. These queries are formulated in a language compatible with the knowledge graph, such as SPARQL (SPARQL Protocol and RDF Query Language).

[0064] The process 300 also involves generating a KG representation. Step 322 focuses on creating or modifying a knowledge graph representation of the engineering system represented by the engineering schematic. This step involves adding, updating, or removing nodes and edges within the knowledge graph to reflect the changes specified in the user input and interpreted by the previous steps. At step 324, the process 300 executes SPARQL queries for visualizing the engineering schematic. These queries retrieve the necessary information from the updated knowledge graph to create a visual representation of the schematic. Step 326 involves rendering the drawing on the canvas. This involves translating the abstract representation from the knowledge graph into a visual format, placing components and connections in the appropriate locations, and applying any necessary visual styles or annotations. The resulting visual representation of the engineering schematic (denoted by reference numeral 350) is then displayed to the user through the user interface 301. This visual representation 350 of the engineering schematic incorporates all the instructions / modifications specified in the original natural language input.

[0065] The process 300 incorporates several components that interact with the individual steps. Component 328 represents an interaction history that maintains a record of previous user inputs and system responses. This history provides valuable context for interpreting new instructions and deriving implicit intent in steps 312-316. Component 330 is a knowledge graph representation of the system at the current time. This represents the current state of the engineering schematic as a knowledge graph, reflecting all previous modifications. This knowledge graph representation is continuously updated throughout the process, particularly in step 322. Component 332 represents domain knowledge graphs, which are a collection of standardized knowledge graphs containing comprehensive information about engineering components, connections, and best practices for specific domains. These graphs are consulted throughout the process 300, particularly in steps 314-316 for applying domain rules and best practices, and in steps 322-324 for creating and querying the system representation.

[0066] The method 100 of the present invention significantly reduces the manual effort and domain expertise required to create and modify complex engineering schematics. By allowing users to interact using natural language input, the method 100 bridges the gap between human expression of design intent and the technical requirements of schematic creation. The use of standardized ontology-based knowledge graphs ensures that all modifications adhere to industry standards and best practices, even when these are not explicitly specified in the user input.

[0067] Conventional methods for engineering diagram creation and modification are manually repetitive and require a significant level of expertise in both the engineering domain and specialized drawing tools. These conventional methods often require a comprehensive understanding of the entire system prior to any modifications, making the process of changing existing diagrams challenging and time consuming. In contrast, the present method 100 allows users to specify changes using natural language, eliminating the need for extensive training in specialized software. The ability to interpret both explicit and implicit intent enables the method 100 to make appropriate modifications while taking into account domain-specific rules and best practices, reducing the risk of errors and inconsistencies.

[0068] Furthermore, the use of a knowledge graph to represent engineering diagrams allows for more efficient and accurate modifications. Unlike traditional systems, where each change can require manual adjustments to multiple interconnected components, the present method 100 can automatically make changes throughout the system representation, ensuring consistency and reducing the likelihood of oversights. The ability to generate knowledge graph queries based on natural language input also enables more flexible and powerful interactions with the underlying diagram data, allowing for complex modifications that are difficult or time-consuming to achieve with traditional tools.

[0069] While the application has been described in detail with reference to certain embodiments thereof, it is understood that the application is not limited to those embodiments. Many modifications and changes can come to mind of the skilled in the art in light of the application without departing from the scope of the various embodiments of the application described herein. It is therefore intended that the scope of the application be indicated by the following claims rather than by the description preceding them. All changes, modifications and variations that come within the meaning and range of equivalency of the claims are to be considered as being within the scope of the application.

Claims

1. A computer-implemented method (100) for generating and modifying engineering schematics, the method (100) comprising implementing a processing unit to: receive, via a user interface, a natural language input related to an engineering schematic, wherein the natural language input comprises a user intent to perform one or more operations on the engineering schematic; generate a structured representation of the natural language input by processing the natural language input using a language model, wherein the structured representation comprises one or more of: identified engineering components, actions to be performed on the identified engineering components, and attributes associated with the identified engineering components; generate one or more knowledge graph queries for a knowledge graph based on the structured representation, wherein the knowledge graph queries are compatible with the knowledge graph based on a standardized ontology for a domain of the engineering schematic; execute the one or more knowledge graph queries to retrieve from the knowledge graph one or more of: component data corresponding to the identified engineering components, connection data specifying allowable connections between the identified engineering components, and attribute data specifying allowable attributes for the identified engineering components and the allowable connections; generate a system representation corresponding to the engineering schematic using the knowledge graph based on the retrieved component data, connection data, and attribute data and the specified actions in the structured representation; render a visual representation of the engineering schematic based on the generated system representation; and output, via a user interface, the visual representation of the engineering schematic.

2. The method (100) of claim 1, wherein the language model is a large language model trained on domain-specific engineering terminology and syntax.

3. The method (100) of claim 1, wherein processing the natural language input using the language model further comprises: interpreting explicit intents stated in the natural language input; and deriving implicit intents based on at least one of: context from previous user interactions, domain-specific rules, and industry best practices.

4. The method (100) of claim 1, wherein generating the one or more knowledge graph queries comprises: analyzing the structured representation to identify constituent components; preparing prompts based on a catalog of examples for different types of intents; and generating the one or more knowledge graph queries using the prepared prompts.

5. The method (100) of claim 1, wherein the knowledge graph comprises nodes representing engineering components and edges representing connections between the engineering components, and wherein generating the system representation using the knowledge graph comprises at least one of: adding nodes and edges within the knowledge graph, modifying existing nodes or edges, and removing nodes or edges.

6. The method (100) of claim 1, wherein generating the system representation using the knowledge graph comprises: comparing specifications in the structured representation to a standardized component database; identifying standardized components corresponding to the specifications based on the comparison; and generating the system representation using the identified standardized components. ​ ​ ​ ​ 7. The method (100) of claim 1, wherein rendering the visual representation of the engineering schematic comprises: retrieving a visual symbol or icon associated with each engineering component in the system representation; determining a spatial arrangement of the engineering components using a layout algorithm; and rendering the engineering components and their connections based on the determined spatial arrangement.

8. The method (100) of claim 1, further comprising: receiving user input via a user interface to modify the visual representation of the engineering schematic; updating the system representation based on the user input; and re-rendering the visual representation of the engineering schematic based on the updated system representation.

9. A system (200) comprising: one or more processing units (204); and a memory unit (206) communicatively coupled with the one or more processing units (204), wherein the memory unit (206) comprises an engineering schematic design module stored in the form of machine-readable instructions executable by the one or more processing units (204), wherein the engineering schematic design module is configured to perform the steps of the method (100) for generating and modifying an engineering schematic according to any one of claims 1 to 8.

10. A computer program product having stored thereon machine-readable instructions which, when executed by one or more processing units (204), cause the one or more processing units (204) to perform the method (100) according to any one of claims 1 to 8.

Images