System and method for automated parsing of schematic

Abstract

To provide systems, methods, and computer program products for generating a digital representation of a system from engineering documents of the system comprising one or more schematics and a components table.SOLUTION: A method for automated parsing of schematics comprises: Step 510 of preprocessing non-digital schematics of a physical system; Step 520 of classifying, using a deep learning algorithm, each of a plurality of symbols in the one or more schematics as a component and each group of related symbols as an assembly; Step 530 of determining connections between the components and the assemblies; Step 540 of associating a subset of the components and the assemblies with entries in the components table; and Step 550 of generating a digital representation of the system from the components, the assemblies, the connections, and the associations.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 899,011, filed Sep. 11, 2019, which is incorporated by reference in its entirety. [Background technology]

[0002] Engineering design documents, such as circuit diagrams and piping and instrumentation diagrams, can help engineers understand the constituent components of a system, how those components are connected, and how the system operates. A complete set of engineering design documents for a system may include a schematic, a component table that provides more detail about specific components within the schematic, and a list of actions to be taken on the system (e.g., maintenance actions, calibrations, etc.). Some or all of such engineering design documents may not be digitized. Summary of the Invention [Means for solving the problem]

[0003] The present disclosure provides a system, method, and computer program product for generating a digital representation of a physical system from an engineering design document of the physical system. The engineering design document may include a non-digital schematic diagram of the physical system and a corresponding component table. The schematic diagram may span many different pages. The system described herein may generate a digital representation of a physical system by detecting symbols in the non-digital schematic diagram. The system may then use a classifier to classify each of the detected symbols as a particular component. In some cases, detection and classification may be performed simultaneously by a single algorithm or model, e.g., an object detection algorithm. The system may additionally classify related groups of components as assemblies, e.g., by detecting dashed lines that bound the groups in the schematic diagram. The system may then determine connections between components and assemblies, e.g., by detecting solid lines that connect the components and assemblies in the schematic diagram. In some cases, the system may also associate a subset of components and assemblies with entries in the component table. The component table may provide more detailed information about a component that the schematic diagram does not provide. Finally, the system can generate a digital representation of the physical system from the components, assemblies, connections, and associations. The digital representation of the physical system may include a digital model of the physical system and a machine-readable bill of materials. The digital model may be computer code that defines the components, connections, and state of the physical system. The digital model may be a static model or a time-varying model.

[0004] The system described above can facilitate the rapid and accurate conversion of engineering design documents into comprehensive digital models. In particular, the present system's use of deep learning to identify and classify objects within engineering design documents can reduce the amount of domain knowledge (e.g., knowledge of the geometry of specific symbols used within the engineering design documents) required to generate such digital models. Instead, a limited amount of label training data can be used to generate highly accurate deep learning models. The generality of deep learning also allows the methods described herein to be used across different industries with different types of engineering design documents.

[0005] In one aspect, the present disclosure provides a method for generating a digital representation of a system from an engineering design document of the system. The engineering design document may comprise one or more circuit diagrams and a bill of materials. The method may include (a) using a deep learning algorithm to classify (i) each of a plurality of symbols in the one or more circuit diagrams as a component and (ii) each group of related symbols as an assembly, (b) determining connections between the components and the assemblies, (c) associating a subset of the components and assemblies with entries in the bill of materials, and (d) generating a digital representation of the system from the components, assemblies, connections, and associations. The digital representation of the system may include at least a digital model of the system and a machine-readable bill of materials.

[0006] In some implementations, the method further includes, prior to (a), preprocessing the one or more circuit diagrams by removing white space or noise from the one or more circuit diagrams.

[0007] In some implementations, the method further includes, prior to (a), determining a library of components to find in the one or more schematics. In some such implementations, the method further includes (i) determining that individual symbols in the one or more schematics are not present in the library of components, and (ii) adding the individual symbols and their corresponding components to the library of components.

[0008] In some implementations, (a) includes detecting a plurality of symbols in the one or more schematics using a binary classifier. The detecting may include a sliding window approach. The deep learning algorithm may comprise the binary classifier. In some implementations, (a) further includes classifying each of the detected symbols as a component using a multi-class deep neural network. The deep learning algorithm may comprise a multi-class deep neural network. The multi-class deep neural network may be selected from the group consisting of a feedforward neural network and a convolutional neural network. In some implementations, a single algorithm (e.g., an object detection algorithm) performs both detection and classification of the symbols.

[0009] In some implementations, classifying each group of related symbols as an assembly includes, for each group, detecting dashed lines in one or more schematic diagrams that bound the group.

[0010] In some implementations, (b) includes detecting solid lines connecting the components and assemblies. The solid lines can be traversed and modeled using a graph search algorithm.

[0011] In some implementations, (c) includes detecting and identifying, within the one or more schematics, identification numbers associated with the components and assemblies in the subset of components and assemblies using optical character recognition, and matching the detected identification numbers with identification numbers in the component table. Detecting the identification numbers may include using a Hough transform.

[0012] In some implementations, the digital model may comprise computer code that defines the components, connections, and states of the system. The digital model may be a static model. The digital model may be a time-varying model.

[0013] Another aspect of the disclosure provides a non-transitory computer-readable medium comprising machine-executable code that, upon execution by one or more computer processors, implements any of the methods described above or elsewhere herein.

[0014] Another aspect of the present disclosure provides a system comprising one or more computer processors and a computer memory coupled thereto, the computer memory comprising machine executable code that, upon execution by the one or more computer processors, implements any of the methods described above or elsewhere herein.

[0015] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, in which only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the present disclosure. Thus, the drawings and description should be regarded as illustrative in nature, and not restrictive. The present invention provides, for example, the following items. (Item 1) 1. A method for generating a digital representation of a system from engineering design documents of the system comprising one or more circuit diagrams and a component list, the method comprising: (a) using a deep learning algorithm to classify (i) each of a plurality of symbols in the one or more circuit diagrams as a component, and (ii) each group of related symbols as an assembly; (b) determining a connection between the component and the assembly; (c) associating a subset of the components and assemblies with entries in the component table; (d) generating a digital representation of the system from the components, the assemblies, the connections, and the associations, the digital representation of the system including at least a digital model of the system and a machine-readable bill of materials; A method comprising: (Item 2) 2. The method of claim 1, further comprising, prior to (a), preprocessing the one or more circuit diagrams by removing white space or noise from the one or more circuit diagrams. (Item 3) 2. The method of claim 1, further comprising, prior to (a), determining a library of components for detection within the one or more circuit diagrams. (Item 4) 4. The method of claim 3, further comprising: (i) determining that individual symbols in the one or more schematic diagrams are not present in the library of components; and (ii) adding the individual symbols and their corresponding components to the library of components. (Item 5) 2. The method of claim 1, wherein (a) includes detecting the plurality of symbols in the one or more circuit diagrams using a binary classifier, the deep learning algorithm comprising the binary classifier, and the detecting includes a sliding window approach. (Item 6) 6. The method of claim 5, wherein (a) further comprises classifying each of the detected symbols as a component using a multi-class deep neural network, the deep learning algorithm comprising the multi-class deep neural network. (Item 7) 6. The method of claim 5, wherein (a) further comprises using an autoencoder to classify each of the detected symbols as a component, and the deep learning algorithm comprises the autoencoder. (Item 8) 7. The method of claim 6, wherein the multi-class deep neural network is selected from the group consisting of a feedforward neural network and a convolutional neural network. (Item 9) 7. The method of claim 6, wherein the multi-class deep neural network is trained using a semi-supervised learning process. (Item 10) 2. The method of claim 1, wherein (a) includes using a single object detection algorithm to detect and classify the plurality of symbols in the one or more schematic diagrams. (Item 11) 2. The method of claim 1, wherein classifying each group of related symbols as an assembly includes, for each group, detecting dashed lines in the one or more circuit diagrams that bound the group. (Item 12) 2. The method of claim 1, further comprising detecting a continuous line connecting the component and the assembly. (Item 13) Item 13. The method of item 12, wherein the detecting step comprises using a graph-searching algorithm. (Item 14) (c) is detecting and identifying, within the one or more schematic diagrams, identification numbers associated with the components and assemblies in the subset of components and assemblies using optical character recognition; matching said detected identification number with an identification number in said component table; The method according to item 1, comprising: (Item 15) Item 15. The method of item 14, wherein the step of detecting the identification number includes using a Hough transform. (Item 16) 2. The method of claim 1, wherein the digital model comprises computer code that defines the components, the connections, and the state of the system. (Item 17) Item 17. The method according to item 16, wherein the digital model is a static model. (Item 18) Item 17. The method according to item 16, wherein the digital model is a time-varying model. (Item 19) One or more non-transitory computer storage media having instructions stored thereon that, when executed by one or more computers, cause the one or more computers to: (a) using a deep learning algorithm to classify (i) each of a plurality of symbols in the one or more schematic diagrams as a component, and (ii) each group of related symbols as an assembly; (b) determining a connection between the component and the assembly; (c) associating a subset of the components and assemblies with entries in the component table; (d) generating a digital representation of the system from the components, the assemblies, the connections, and the associations, the digital representation of the system including at least a digital model of the system and a machine-readable bill of materials;

[0023] 1. One or more non-transitory computer storage media operable to cause a computer to perform operations including: (Item 20) 20. The non-transitory computer storage medium of claim 19, wherein the operations further include, prior to (a), preprocessing the one or more circuit diagrams by removing white space or noise from the one or more circuit diagrams. (Item 21) 20. The non-transitory computer storage medium of claim 19, wherein the operations further include, prior to (a), determining a library of components to find in the one or more circuit diagrams. (Item 22) 22. The non-transitory computer storage medium of claim 21, wherein the operations further include (i) determining that individual symbols in the one or more circuit diagrams are not present in the library of components; and (ii) adding the individual symbols and their corresponding components to the library of components. (Item 23) 20. The non-transitory computer storage medium of claim 19, wherein (a) includes detecting the plurality of symbols in the one or more circuit diagrams using a binary classifier, the deep learning algorithm comprising the binary classifier, and the detecting includes a sliding window approach. (Item 24) 24. The non-transitory computer storage medium of claim 23, wherein (a) further comprises classifying each of the detected symbols as a component using a multi-class deep neural network, the deep learning algorithm comprising the multi-class deep neural network. (Item 25) 24. The non-transitory computer storage medium of claim 23, wherein (a) further comprises using an autoencoder to classify each of the detected symbols as a component, and wherein the deep learning algorithm comprises the autoencoder. (Item 26) 25. The non-transitory computer storage medium of claim 24, wherein the multi-class deep neural network is selected from the group consisting of a feedforward neural network and a convolutional neural network. (Item 27) 25. The non-transitory computer storage medium of claim 24, wherein the multi-class deep neural network is trained using a semi-supervised learning process. (Item 28) 20. The non-transitory computer storage medium of claim 19, wherein (a) comprises using a single object detection algorithm to detect and classify the plurality of symbols in the one or more circuit diagrams. (Item 29) 20. The non-transitory computer storage medium of claim 19, wherein classifying each group of related symbols as an assembly includes, for each group, detecting dashed lines in the one or more circuit diagrams that bound the group. (Item 30) 20. The non-transitory computer storage medium of claim 19, wherein (b) comprises detecting a solid line connecting the component and the assembly. (Item 31) Item 31. The non-transitory computer storage medium of item 30, wherein the detecting step includes using a graph-searching algorithm. (Item 32) (c) is detecting and identifying, within the one or more schematic diagrams, identification numbers associated with the components and assemblies in the subset of components and assemblies using optical character recognition; matching said detected identification number with an identification number in said component table; 20. The non-transitory computer storage medium of item 19, comprising: (Item 33) Item 33. The non-transitory computer storage medium of item 32, wherein detecting the identification number includes using a Hough transform. (Item 34) 20. The non-transitory computer storage medium of claim 19, wherein the digital model comprises computer code that defines the components, the connections, and the state of the system. (Item 35) Item 35. The non-transitory computer storage medium of item 34, wherein the digital model is a static model. (Item 36) Item 35. The non-transitory computer storage medium of item 34, wherein the digital model is a time-varying model.

[0016] (Incorporated by reference) All publications, patents, and patent applications mentioned herein are incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated herein. To the extent that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, it is intended that the present specification supersede and / or supersede any such conflicting material. [Brief description of the drawings]

[0017] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "Figure and FIG.").

[0018] [Figure 1] FIG. 1 diagrammatically illustrates a system for generating a digital representation of a physical system from engineering design documents of the physical system.

[0019] [Diagram 2] FIG. 2 is a schematic diagram of a physical system in which detected symbols are identified by red boxes.

[0020] [Diagram 3]FIG. 3 is a schematic diagram of the physical system in which detected assemblies are identified by red boxes.

[0021] [Figure 4] FIG. 4 shows an exemplary digital model as it may be displayed within the user interface of an application.

[0022] [Diagram 5] FIG. 5 is a flowchart of an exemplary process for generating a digital representation of a physical system from an engineering design document of the physical system.

[0023] [Figure 6] FIG. 6 illustrates a computer system that is programmed or otherwise configured to implement the methods provided herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Detailed Description While various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It is understood that various alternatives to the embodiments of the invention described herein may be employed.

[0025] Each time the term "at least," "greater than," or "greater than or equal to" precedes a first number in a series of two or more numerical values, the term "at least," "greater than," or "greater than or equal to" applies to each and every numerical value in the series. For example, "greater than or equal to 1, 2, or 3" is equivalent to "greater than or equal to 1," "greater than or equal to 2," or "greater than or equal to 3."

[0026] Each time the term "less than", "less than", or "less than or equal to" precedes a first number in a series of two or more numerical values, the term "less than", "less than", or "less than or equal to" applies to each and every numerical value in the series. For example, "less than or equal to 3, 2, or 1" is equivalent to "less than or equal to 3", "less than or equal to 2", or "less than or equal to 1".

[0027] The present disclosure provides a system, method, and computer program product for generating a digital representation of a physical system from an engineering design document of the physical system. Specifically, the system described herein can generate a digital representation of a physical system by detecting symbols in a non-digital circuit diagram. The system can then use a classifier to classify each of the detected symbols as a particular component. The system can additionally classify related groups of components as assemblies, for example, by detecting dashed lines that bound the groups in the circuit diagram. The system can then determine connections between components and assemblies, for example, by detecting solid lines that connect the components and assemblies in the circuit diagram. In some cases, the system can also associate a subset of components and assemblies with entries in a component table. The component table may provide more detailed information about a component that the circuit diagram does not provide. Finally, the system can generate a digital representation of the physical system from the components, assemblies, connections, and associations. The digital representation of the physical system may include a digital model of the system and a machine-readable bill of materials.

[0028] FIG. 1 diagrammatically illustrates a system 100 for generating a digital representation of a physical system from an engineering design document of the physical system. The physical system may be, for example, a processing pipeline. The processing pipeline may be an oil and gas processing pipeline, a chemical or material synthesis pipeline (e.g., a pipeline for the synthesis of paint, fertilizer, textile products, rubber, steel, or aluminum), a drug synthesis pipeline, or a food production pipeline, to name a few examples. The physical system may alternatively be a mechanical system, a building, an infrastructure (e.g., a road, a bridge, an electric grid, or a public water system), a vehicle, an electronic device, a circuit, or the like. In general, the physical system may be any system that can be represented by a circuit diagram or schematic diagram.

[0029] Engineering design documentation may include a non-digital (eg, paper) schematic diagram of a physical system, a corresponding bill of components, and a list of actions to be taken on the physical system (eg, maintenance actions, calibrations, etc.).

[0030] System 100 may also have a pre-processing subsystem 110. In general, pre-processing subsystem 110 may simplify the schematic to make subsequent processing steps easier. For example, pre-processing subsystem 110 may scan a physical copy of the schematic into a digital image (e.g., Portable Network Graphics (PNG) or JPEG), resize or change the color of the image, remove edges, white space, and other noise from the image, and remove non-symbolic text from the image. In this disclosure, a digital image of a schematic may simply be referred to as a schematic.

[0031] System 100 may also include a classification subsystem 120. Classification subsystem 120 may receive the preprocessed schematic and detect and classify symbols, assemblies (e.g., related groups of symbols), and connections within the schematic.

[0032] The classification subsystem 120 may include a detection module 122 and a classification module 124. The detection module 122 may be configured to detect symbols and assemblies in a schematic. That is, the detection module 122 may be configured to distinguish between parts of the schematic that contain symbols and assemblies and parts that do not. To detect symbols in a schematic, the detection module 122 may implement a binary classifier configured to generate an output indicating that a particular part of the schematic is a symbol or is not. The binary classifier may be a deep neural network, such as a feed-forward neural network or a convolutional neural network (CNN), or it may be a support vector machine (SVM), an autoencoder, or any other machine learning algorithm or approach. In one embodiment, the binary classifier is a CNN. The CNN may have at least about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more convolutional layers. ReLU activation layers and max pooling layers may be interspersed between the convolutional layers. A CNN may also have a fully connected layer. The output layer of a CNN may have a sigmoid or softmax activation, which produces a certain probability.

[0033] A binary classifier may be trained using a supervised, semi-supervised, or unsupervised learning process. In one implementation, a binary classifier can be trained based on a small amount of labeled training data (e.g., a schematic in which parts of the schematic are labeled as containing or not containing a symbol) and a large amount of unlabeled training data (e.g., a schematic in which parts of the schematic are not labeled as containing or not containing a symbol). Figure 2 is a schematic of a physical system in which detected symbols are identified by red boxes.

[0034] In some cases, the detection module 122 may implement a ternary classifier instead of a binary classifier. A ternary classifier may have a structure and function similar to a binary classifier, but may classify areas in a schematic as (1) containing a tag (i.e., a digital representation of a sensor, typically represented by a circle inscribed within a square), (2) containing a locally mounted instrument (i.e., a physical instrument such as a pressure gauge or temperature reader, typically represented by a circle), or (3) not containing any symbols. Tags and LMIs are important because they often have corresponding entries in a time-series sensor database.

[0035] To detect symbols in a circuit diagram, the detection module 122 may use a sliding window approach. That is, a particular circuit diagram may be divided into multiple overlapping windows. The classifier described above may return an output for each window. The output for each window may indicate whether the window has at least a portion of a symbol within its boundaries or does not have a symbol, or whether the window contains a tag, an LMI, or does not contain any symbol. The size and stride of the sliding window may be based in part on the average or median size of the symbols, or any other statistical metric or heuristic. In one embodiment, the window may be 100 pixels by 100 pixels. In the case of the sliding window approach, the output of the binary classifier may undergo a post-processing step before classification as a particular component. For example, a non-maximum suppression algorithm may be applied to the output to filter out one or more of the multiple windows that have the same symbol (e.g., due to overlapping windows).

[0036] To detect assemblies in a schematic, the detection module 122 can implement an algorithm configured to detect dashed lines that bound groups of symbols. Such dashed lines may indicate that the symbols are constituent components of an assembly. Figure 3 is a schematic diagram of a physical system in which detected assemblies are identified by red boxes.

[0037] To find connections in a schematic, the detection module 122 can implement a similarity algorithm configured to find solid lines connecting components and assemblies. In one embodiment, the detection module 122 uses a schematic search algorithm to traverse the components and solid lines representing the connections between the components. The schematic search algorithm can traverse the components and solid lines by finding and following adjacent black pixels in the schematic. The black pixels can be pixels that satisfy a threshold pixel intensity. The schematic search algorithm can represent pixels as nodes in the schematic. A particular pixel may or may not be part of a component, and this information can be contained in the node corresponding to that pixel. A symbol is represented in the schematic as a collection of nodes corresponding to the pixels that form the symbol. A schematic edge can be a connection between neighboring pixels with up to eight edges per node. A symbol is represented in the schematic as a collection of nodes corresponding to the pixels that form the symbol.

[0038] Using this graphical representation of the diagram, connections between symbols can be identified through a depth-first search (DFS). Specifically, for each detected symbol, a DFS is initialized from one of the nodes in the symbol. The DFS traverses the graph along its black nodes, hitting (and tracing) connected symbols along that path. The search terminates once all valid paths have been exhausted.

[0039] The classification module 124, on the other hand, may be configured to classify the detected symbol as a component. To classify a symbol as a component, the classification module 124 may implement a multi-class deep neural network configured to generate an output indicating which of a plurality of components a particular symbol corresponds to. The plurality of components may be determined prior to the step of training the multi-class deep neural network. The components may be stored in a component library. If during classification, the multi-class deep neural network fails to classify a particular symbol as a component, this may indicate that the symbol is a new symbol. The new symbol may be added to the component library.

[0040] The multi-class deep neural network may be, for example, a feedforward neural network or a CNN. The multi-class deep neural network can be trained using a supervised, semi-supervised, or unsupervised learning process. In one implementation, the multi-class deep neural network can be trained based on a small amount of labeled training data (e.g., circuit diagrams in which symbols are labeled as corresponding specific components) and a large amount of unlabeled training data (e.g., circuit diagrams in which symbols are not labeled as corresponding to specific components).

[0041] In general, the neural networks described herein may employ multiple layers of operations to predict one or more outputs from one or more inputs. A neural network may include one or more hidden layers that are mounted between an input layer and an output layer. The output of each layer may serve as an input to another layer, for example, the next hidden layer or the output layer. Each layer of a neural network may define one or more transformation operations to be performed on the inputs to the layer. Such transformation operations may be referred to as neurons. The output of a particular neuron may be a weighted sum of the inputs to the neuron, adjusted with a bias and multiplied by an activation function, for example, a rectified linear unit (ReLU) or a sigmoid function. Training a neural network may involve providing inputs to an untrained or partially trained version of the neural network to generate a predicted output, comparing the predicted output to an expected output, and updating the weights and biases of the algorithm to account for the difference between the predicted and expected outputs. Specifically, a cost function can be used to calculate the difference between the predicted and expected outputs. By calculating the derivative of the cost function with respect to the weights and biases of the network, the weights and biases can be iteratively adjusted over multiple cycles to minimize the cost function. Training can be completed when the predicted outputs meet a convergence condition, such as obtaining a small calculated cost.

[0042] A CNN is a neural network in which neurons in some layers, called convolutional layers, receive pixels from only small portions of the input data set. These small portions may be referred to as the receptive fields of the neurons. Each neuron in such a convolutional layer may have the same weights. In this way, the convolutional layer can detect features in any portion of the input data set. A CNN may also have pooling layers that combine the outputs of neuron clusters in the convolutional and fully connected layers, similar to traditional layers in feedforward neural networks. CNNs may be particularly good at detecting and classifying objects (e.g., products) in images.

[0043] In some cases, classification module 124 may implement an autoencoder to classify symbols. The autoencoder may comprise an encoder and a decoder. The encoder may be configured to generate a compressed representation of an input symbol, and the decoder may be configured to reconstruct the input symbol from the compressed representation. The decoder may be trained (e.g., through a supervised learning process) based on examples of known symbols. Classification module 124 may compare the output of the decoder for a given input symbol to known signatures for various components in a component library (e.g., using a distance-based algorithm (e.g., cosine similarity)). The symbol may be classified as the component with the most similar or closest signature.

[0044] In some cases, a single object detection algorithm can both detect and classify symbols. Such an object detection algorithm may take any of the forms described above.

[0045] The classification system 120 can provide the classified components and assemblies to the subsystem 130. The subsystem 130 can also receive a components table of the physical system. The subsystem 130 can be configured to associate some or all of the components and assemblies with entries in the components table. The components table can provide additional information about components that do not appear in the circuit diagram, such as the size and material composition of a capacitor.

[0046] The subsystem 130 may include an optical character recognition (OCR) module 132. The OCR module 132 may implement an OCR algorithm configured to recognize text associated with the classified components and assemblies. More specifically, the OCR algorithm may be configured to detect and identify identification numbers associated with the components and assemblies. In some cases, the identification numbers may be located within or otherwise near a circle adjacent to the classified components and assemblies. In such cases, the subsystem 130 may use a Hough transform to detect the circle. The OCR algorithm may additionally recognize text, including the identification numbers and other data, within the component table. In some cases, the text may be detected using an Efficient and Accurate Scene Text Detector (EAST), a pipeline that uses a neural network to produce a bounding box where text is present in an image.

[0047] Identification numbers in the schematic can correspond to identification numbers in the component table. The OCR module 132 can provide identification numbers from the schematic to a matching module 134, which can match such identification numbers with identification numbers in the component table, thereby associating corresponding components and assemblies with entries in the component table.

[0048] The system 100 may additionally include a model builder subsystem 140. The model builder subsystem 140 may receive classified components, assemblies, and connections from the classification subsystem 120 and associations from the subsystem 130 and generate a digital representation of the physical system. The digital representation of the physical system may include a digital model of the physical system and a machine-readable bill of materials. The machine-readable bill of materials may be an asset hierarchy table that identifies the various components in a schematic or set of schematics, the text associated with the components, and the components connected to each component. The digital model may be computer code that defines the components in the physical system, the connections between the components, and the state of the physical system. The components may be further defined by their parameters (e.g., dimensions, functions, inputs and outputs, throughput, operating variables, etc.), which may be stored in a component library and retrieved when the digital model is generated. The digital model may be a time-varying model or a static model.

[0049] The system of FIG. 1 and its components can be implemented on one or more computing devices. The computing device can be a server, a desktop or laptop computer, an electronic tablet, a mobile device, or the like. The computing device can be located in one or more locations. The computing device can have a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like. The computing device can additionally have a memory, such as a dynamic or static random access memory, a read-only memory, a flash memory, a hard drive, or the like. The memory can be configured to store instructions that, upon execution, cause the computing device to implement the functionality of the pre-processing subsystem 110, the classification subsystem 120, the subsystem 130, or the model builder subsystem 140. The computing device can additionally have a network communication device. The network communication device can enable the computing devices to communicate with each other and any number of user devices via a network. The network can be a wired or wireless network. For example, the network can be an optical fiber network, an Ethernet network, a satellite network, a cellular network, a Wi-Fi network, a Bluetooth network, or the like. In other implementations, the computing device can be a number of distributed computing devices accessible through the Internet. Such computing devices can be considered cloud computing devices.

[0050] 4 shows an example of a digital model as displayed within a user interface of an application. The user interface can allow a user to validate the digital model. The user interface can show recognized components in blue and recognized assemblies in red. The user interface can allow a user to filter the components and assemblies by type or by status (e.g., "classified" or "unclassified"). The user interface can also allow a user to manually reclassify components and assemblies or edit other parameters of the components and assemblies.

[0051] 5 is a flow chart of an exemplary process for generating a digital representation of a physical system from an engineering design document of the physical system. The engineering design document may be a non-digital (e.g., paper) schematic diagram of the physical system and a corresponding bill of materials. The process may be performed by one or more computer systems that are appropriately programmed in one or more locations.

[0052] The system may include pre-processing the non-digital circuit diagram (510), which may involve scanning the non-digital circuit diagram into a digital image and removing white space and other noise from the digital image.

[0053] The system may use a machine learning algorithm to classify (i) each of the multiple symbols in the preprocessed schematic as a component and (ii) each group of related symbols as an assembly (510). The machine learning algorithm may comprise a binary classifier and a multi-class deep neural network or an autoencoder, and classifying the symbols as components may involve detecting the symbols using a binary classifier and classifying each of the symbols as a component using a multi-class deep neural network or an autoencoder. The multi-class deep neural network may be a feed-forward neural network or a convolutional neural network. The multi-class deep neural network may be trained using a semi-supervised learning process. The classes of the multi-class deep neural network may be determined prior to the training step. The classes may be components that are typically found in engineering design documents of that type. If an individual symbol in one of the schematics does not exist in the component library, the system may add the corresponding component to the library. In some cases, the system may implement a single object detection algorithm that simultaneously detects and classifies symbols as components. Classifying each group of related symbols as an assembly may include, for each group, detecting dashed lines in one or more schematic diagrams that bound the group.

[0054] The system can determine connections between components, assemblies, and between components and assemblies by detecting and traversing solid lines connecting various combinations of these (530). The system can traverse the solid lines using a graph search algorithm, as described above.

[0055] The system can associate a subset of the components and assemblies with entries in the component table by (i) detecting and identifying, in the schematic, identification numbers associated with the components and assemblies using optical character recognition, and (ii) matching the detected identification numbers with identification numbers in the component table (540). The steps of detecting and identifying the identification numbers can involve using a Hough transform to identify circles within which the identification numbers are located. The circles can be adjacent to or otherwise proximate to corresponding components in the schematic.

[0056] Using the classified components and assemblies, connections, and associations, the system can generate a digital representation of the physical system (550). The digital representation can include a digital model of the physical system and a machine-readable bill of materials. The digital model can be computer code that defines the components, connections, and state of the physical system. The digital model can be a static model or a time-varying model. (Example)

[0057] In one example, the system as described herein has a ternary CNN classifier to classify the area of ​​a schematic as containing tags, LMI, or not containing any symbols. The CNN has three convolutional layers interspersed between a ReLU activation layer and a max pooling layer, followed by two fully connected dense layers. The first hidden layer has 64 units with ReLU activation. The final output layer has three units with softmax activation to predict the probability that the input image belonged to each of the three classes. The CNN was trained on 18 labeled piping and instrumentation diagrams, including 308 tag crops and 687 LMI crops. The training example was augmented with rotation, shift, shear, magnification, and inversion to make the CNN invariant to these transformations. Dropout was applied to the dense layers to add some regularization and improve the generalization performance of the CNN. The symbol crops were split into a training (60%) and validation (40%) set, and the network was trained to minimize the cross-entropy loss on the validation set. Both the training and validation losses converged by 100 epochs of training. The training and validation sets were then combined, and the final model was trained on the entire dataset over 100 epochs. For 11 unseen piping and instrumentation diagrams, tags were classified with 100% precision and 98% recall using a classification probability threshold of 0.95. LMIs were classified with 85% precision and 90% recall using a classification probability threshold of 0.95.

[0058] Computer Systems The present disclosure provides a computer system that is programmed to implement the method of the present disclosure. Figure 6 shows a computer system 601 that is programmed or otherwise configured to implement the system 100 of Figure 1. The computer system 601 additionally coordinates various aspects of the process of Figure 5 of the present disclosure. The computer system 601 can be a user's electronic device or a computer system that is remotely located with respect to the electronic device. The electronic device can be a mobile electronic device.

[0059] The computer system 601 includes a central processing unit (CPU, also referred to herein as "processor" and "computer processor") 605, which may be a single-core or multi-core processor, or multiple processors for parallel processing. The computer system 601 also includes memory or memory locations 610 (e.g., random access memory, read-only memory, flash memory), an electronic storage unit 615 (e.g., hard disk), a communication interface 620 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 625, such as cache, other memory, data storage devices, and / or electronic display adapters. The memory 610, the storage unit 615, the interface 620, and the peripheral devices 625 communicate with the CPU 605 through a communication bus (solid lines), such as a motherboard. The storage unit 615 may be a data storage unit (or data repository) for storing data. The computer system 601 can be operatively coupled to a computer network ("network") 630 with the aid of the communication interface 620. Network 630 may be the Internet, an intranet and / or an extranet, or an intranet and / or an extranet in communication with the Internet. Network 630 may in some cases be a telecommunications and / or data network. Network 630 may include one or more computer servers, which may enable distributed computing such as cloud computing. Network 630 may in some cases implement a peer-to-peer network, which may enable devices coupled to computer system 1101 to behave as clients or servers with the aid of computer system 1101.

[0060] The CPU 1105 can execute sequences of machine-readable instructions, which may be embodied in a program or software. The instructions may be stored in a memory location, such as memory 610. The instructions can be directed to the CPU 605, which can subsequently program or otherwise configure the CPU 605 to implement the methods of the present disclosure. Examples of operations performed by the CPU 605 can include fetch, decode, execute, and writeback.

[0061] The CPU 605 may be part of a circuit, such as an integrated circuit. One or more other components of the system 601 may be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0062] The storage unit 615 can store files such as drivers, libraries, and saved programs. The storage unit 615 can store user data, such as user preferences and user programs. The computer system 601 can include one or more additional data storage units that are external to the computer system 601, in some cases, such as located on a remote server that communicates with the computer system 601 through an intranet or the Internet.

[0063] Computer system 601 can communicate with one or more remote computer systems through network 630. For example, computer system 601 can communicate with a user's remote computer system. Examples of remote computer systems include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., Apple® iPad®, Samsung® Galaxy Tab), a phone, a smartphone (e.g., Apple® iPhone®, Android®-enabled devices, Blackberry®), or a personal digital assistant. A user can access computer system 601 via network 630.

[0064] Methods as described herein may be implemented using machine (e.g., computer processor) executable code stored on electronic storage locations of computer system 601, such as, for example, on memory 610 or electronic storage unit 615. Machine executable or machine readable code may be provided in the form of software. During use, the code may be executed by processor 605. In some cases, the code may be read from storage unit 615 and stored on memory 610 for quick access by processor 605. In some circumstances, electronic storage unit 615 may be omitted and machine executable instructions are stored on memory 610.

[0065] The code can be pre-compiled and configured for use with a machine having a processor adapted to execute the code, or can be compiled during run-time. The code can be provided in a programming language that can be selected to allow the code to be executed in a pre-compiled or immediately compiled manner.

[0066] Aspects of the systems and methods provided herein, such as computer system 601, can be embodied in programming. Various aspects of the technology can be considered as a "product" or "article of manufacture" typically in the form of machine (or processor) executable code and / or associated data carried on or embodied in some type of machine-readable medium. The machine executable code can be stored on an electronic storage unit, such as a memory (e.g., read-only memory, random access memory, flash memory) or a hard disk. A "storage" type medium can include any or all of the tangible memory of a computer, processor, or equivalent, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, that can provide non-transitory storage at any time for software programming. All or a portion of the software may be communicated from time to time over the Internet or various other telecommunications networks. Such communication may, for example, enable loading of the software from one computer or processor to another, for example, from a management server or host computer to a computer platform of an application server. Thus, other types of media that may bear software elements include optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical fixed networks, and via various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, or the like, may also be considered media bearing the software. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0067] Thus, a machine-readable medium such as a computer executable code may take many forms, including, but not limited to, a tangible storage medium, a carrier wave medium, or a physical transmission medium. Non-volatile storage media include optical or magnetic disks, such as any of the storage devices in any computer or equivalent, such as those that may be used to implement the databases, etc., shown in the figures. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire and optical fibers, including the wires that comprise a bus in a computer system. Carrier wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media thus include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch cards, paper tape, any other physical storage media with patterns of holes, RAM, ROM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves that transmit data or instructions, cables or links that transmit such carrier waves, or any other medium from which a computer may read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0068] The computer system 601 can include or communicate with an electronic display 635 that has a user interface (UI) 640 for providing, for example, the digital model of Figure 4. Examples of a UI include, but are not limited to, a graphical user interface (GUI) and a web-based user interface.

[0069] The methods and systems of the present disclosure can be implemented using one or more algorithms. The algorithms can be implemented using software upon execution by the central processing unit 605. The algorithms can be, for example, one of the neural networks described in this disclosure.

[0070] While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the present invention be limited by the specific examples provided herein. Although the present invention has been described with reference to the above specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the present invention. Furthermore, it should be understood that all aspects of the present invention are not limited to the specific depictions, configurations, or relative proportions set forth herein, which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention. It is therefore contemplated that the present invention also covers any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the present invention, and that methods and structures within the scope of these claims and their equivalents are covered thereby.

Claims

[Claim 1] The invention described in this specification.

Images