System and method for automatic parsing of schematics

Abstract

This disclosure provides systems, methods, and computer program products for generating a digital representation of a system based on an engineering document comprising one or more schematic diagrams and a list of parts. Example methods may include: (a) using a deep learning algorithm, (i) classifying each of a plurality of symbols in one or more schematic diagrams as a part, and (ii) classifying each group of related symbols as an assembly; (b) determining connections between parts and assemblies; (c) associating subsets of parts and assemblies with entries in a list of parts; and (d) generating a digital representation of the system based on parts, assemblies, connections, and associations. The digital representation of the system may include at least a digital model of the system and a machine-readable list of materials.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 899,011, filed September 11, 2019, the entirety of which is incorporated by reference herein. BACKGROUND

[0003] Engineering documents, such as schematics, piping diagrams, and instrument diagrams, can help engineers understand the constituent parts of a system, how those parts are connected, and how the system operates. A complete set of system engineering documents can include diagrams, parts lists that provide more detailed information about particular parts in the diagrams, and lists of actions taken on the system (e.g., maintenance actions, calibrations, etc.). Some or all of this class of engineering documents can not be digitized. SUMMARY

[0004] The present disclosure provides systems, methods, and computer program products for generating a digital representation of a physical system from engineering documents of the physical system. The engineering documents can include non-digital schematics of the physical system and a corresponding parts list. The schematics can span many different pages. The systems described herein can generate the digital representation of the physical system by detecting symbols in the non-digital schematics. The system can then use a classifier to classify each detected symbol as a particular part. In some cases, the detection and classification can be performed simultaneously by a single algorithm or model (e.g., an object detection algorithm). Additionally, the system can classify groups of related parts as assemblies, for example, by detecting dashed lines in the schematics that delineate groups. Thereafter, the system can determine connections between the parts and assemblies, for example, by detecting solid lines in the schematics that connect the parts and assemblies. In some cases, the system can also associate subsets of the parts and assemblies with entries in the parts list. The parts list can provide more detailed information about certain parts that are not provided by the schematics. Finally, the system can generate the digital representation of the physical system from the parts, assemblies, connections, and associations. The digital representation of the physical system can include a digital model of the physical system and a bill of materials that is machine-readable. The digital model can be computer code that defines the parts, connections, and states of the physical system. The digital model can be a static model or a time-varying model.

[0005] The above-described systems can facilitate the rapid and accurate conversion of engineering documents into comprehensive digital models. In particular, the use of deep learning by the systems to identify and classify objects in the engineering documents can reduce the amount of domain knowledge (e.g., geometric knowledge of particular symbols used in the engineering documents) required to generate such digital models. Instead, a limited amount of labeled training data can be used to generate highly accurate deep learning models. The ubiquity of deep learning also allows the methods described herein to be used in different industries with different types of engineering documents.

[0006] In an aspect, the disclosure provides a method for generating a digital representation of a system from engineering documents of the system. The engineering documents can include one or more schematics and a parts list. The method can include: (a) using a deep learning algorithm, (i) classifying each symbol of a plurality of symbols in the one or more schematics as a component, and (ii) classifying each group of related symbols as an assembly; (b) determining connections between the components and the assemblies; (c) associating a subset of the components and the assemblies with entries in the parts list; and (d) generating the digital representation of the system from the components, the assemblies, the connections, and the associations. The digital representation of the system can include at least a digital model of the system and a bill of materials that is machine-readable.

[0007] In some implementations, the method further includes, prior to (a), pre-processing the one or more schematics by removing white space or noise from the one or more schematics.

[0008] In some implementations, the method further includes, prior to (a), determining a library of components to be detected in the one or more schematics. In some such implementations, the method further includes: (i) determining that a respective symbol in the one or more schematics is not in the library of components, and (ii) adding the respective symbol and its corresponding component to the library of components.

[0009] In some implementations, (a) includes using a binary classifier to detect the plurality of symbols in the one or more schematics. The detecting can include a sliding window approach. The deep learning algorithm can include the binary classifier. In some implementations, (a) further includes using a multiclass deep neural network to classify each of the detected symbols as a component. The deep learning algorithm can include the multiclass deep neural network. The multiclass deep neural network can be selected from a group including: a feedforward neural network and a convolutional neural network. In some implementations, a single algorithm (e.g., an object detection algorithm) performs both the detecting and the classifying of the symbols.

[0010] In some implementations, classifying each group of related symbols as an assembly includes detecting, for each group, a dashed line in the one or more schematics that depicts the group.

[0011] In some implementations, (b) includes detecting solid lines connecting the components and the assemblies. The solid lines can be traversed and modeled using a graph search algorithm.

[0012] In some implementations, (c) includes: using optical character recognition, detecting and identifying, in the one or more schematics, identification numbers associated with components and assemblies in the subset of components and assemblies; and matching the detected identification numbers to identification numbers in the parts list. Detecting the identification numbers can include using a Hough transform.

[0013] In some implementations, the digital model can include computer code that defines components, connections, and states of the system. The digital model can be a static model. The digital model can be a time-varying model.

[0014] Another aspect of the disclosure provides a non-transitory computer-readable medium comprising machine executable code that, when executed by one or more computer processors, implements any of the methods above or elsewhere herein.

[0015] Another aspect of the disclosure provides a system comprising one or more computer processors and computer memory coupled with the same. The computer memory comprises machine executable code that, when executed by the one or more computer processors, implements any of the methods above or elsewhere herein.

[0016] Other aspects and advantages of the disclosure will become readily apparent to those skilled in the art and the novel features will be pointed out hereinafter, the only description and drawings being given for descriptive purposes.

[0017] INCORPORATION BY REFERENCE

[0018] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent there is a contradiction between the disclosure herein and that of any such incorporated publication, patent or patent application, the present specification shall control. BRIEF DESCRIPTION OF DRAWINGS

[0019] The novel features of the application are set forth with particularity in the claims that follow. A better understanding of the features and advantages of the present application will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the application are utilized, and the accompanying drawings (also “Figure” and “FIG.” and are also referred to as the “illustrations” and “drawings”), of which:

[0020] FIG. 1 A system for generating a digital representation of a physical system from engineering documents of the physical system is schematically illustrated.

[0021] FIG. 2 A schematic view of a physical system in which detected symbols are identified by red boxes.

[0022] FIG. 3is a schematic diagram of a physical system with detected components identified by red boxes.

[0023] FIG. 4 An example digital model as displayed in an application's user interface is shown.

[0024] FIG. 5 is a flowchart of an example process for generating a digital representation of a physical system from engineering documents of the physical system.

[0025] FIG. 6 A computer system programmed or otherwise configured to implement the methods provided herein is shown. DETAILED DESCRIPTION

[0026] While the preferred embodiments of the application have been shown and described herein, it will be readily apparent to those of ordinary skill in the art that many changes, modifications and substitutions can be made thereto without departing from the application. It is to be understood that the application is not limited to the embodiments described herein, but can be practiced with modification and alteration within the scope and spirit of the following claims. Specifically, it should be noted that the terms "comprises", "comprising", "includes", "including" and "contains", "containing" as used herein are taken to specify the presence of stated features, integers, steps or components but do not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0027] Whenever the term "at least", "greater than" or "greater than or equal to" precedes the first numerical value in a series of two or more numerical values, the term "at least", "greater than" or "greater than or equal to" applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0028] Whenever the term "not greater than", "less than" or "less than or equal to" precedes the first numerical value in a series of two or more numerical values, the term "not greater than", "less than" or "less than or equal to" applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0029] The present disclosure provides systems, methods, and computer program products for generating a digital representation of a physical system from engineering documents of the physical system. Specifically, the systems described herein can generate a digital representation of a physical system by detecting symbols in a non-digital schematic. The systems can then use a classifier to classify each detected symbol as a particular component. Additionally, the systems can classify groups of related components as assemblies, e.g., by detecting dashed lines in the schematic that delineate groups. Thereafter, the systems can determine connections between components and assemblies, e.g., by detecting solid lines in the schematic that connect components and assemblies. In some cases, the systems can also associate subsets of components and assemblies with entries in a component table. The component table can provide more detailed information about certain components that are not provided by the schematic. Finally, the systems can generate a digital representation of the physical system from the components, assemblies, connections, and associations. The digital representation of the physical system can include a digital model of the system as well as a bill of materials that is machine-readable.

[0030] FIG. 1 A system 100 for generating a digital representation of a physical system from engineering documents of the physical system is schematically illustrated. For example, the physical system can be a process pipeline. By way of example, the process pipeline can be an oil and gas process pipeline, a chemical or materials synthesis pipeline (e.g., a pipeline for synthesizing paint, fertilizer, textiles, rubber, steel, or aluminum), a pharmaceutical synthesis pipeline, or a food production pipeline. The physical system can alternatively be a mechanical system, a building, an infrastructure (e.g., a road, a bridge, an electrical grid, or a municipal water supply system), a vehicle, an electronic device or circuit, etc. In general, the physical system can be any system that can be represented by a schematic or diagram.

[0031] The engineering documents can include a non-digital (e.g., paper) schematic of the physical system, a corresponding component table, and a list of actions taken on the physical system (e.g., maintenance actions, calibrations, etc.).

[0032] The system 100 can have a preprocessing subsystem 110. In general, the preprocessing subsystem 110 can clean up the schematic to make subsequent processing steps easier. For example, the preprocessing subsystem 110 can scan a physical copy of the schematic into a digital image (e.g., a Portable Network Graphics (PNG) or JPEG), resize or change the colors of the image, remove borders, whitespace, and other noise from the image, and remove non-symbolic text from the image. In this disclosure, the digital image of the schematic can simply be referred to as the schematic.

[0033] The system 100 can also include a classification subsystem 120. The classification subsystem 120 can receive the preprocessed schematic and detect and classify symbols, assemblies (e.g., groups of related symbols), and connections in the schematic.

[0034] The classification subsystem 120 can have a detection module 122 and a classification module 124. The detection module 122 can be configured to detect symbols and components in a schematic. That is, the detection module 122 can be configured to distinguish between portions of a schematic that contain symbols and components and portions of a schematic that do not contain symbols and components. To detect symbols in a schematic, the detection module 122 can implement a binary classifier configured to generate an output indicating whether a particular portion of a schematic is or is not a symbol. The binary classifier can be a deep neural network, such as a feedforward neural network or a convolutional neural network (CNN), or it can be a support vector machine (SVM), an autoencoder, or any other machine learning algorithm or method. In one example, the binary classifier is a CNN. The CNN can have at least about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more convolutional layers. ReLU activations and max pooling layers can be interspersed among the convolutional layers. The CNN can also have fully connected layers. The output layer of the CNN can have a sigmoid or softmax activation that produces a probability.

[0035] The binary classifier can be trained using a supervised, semi-supervised, or unsupervised learning process. In one implementation, the binary classifier can be trained on a small amount of labeled training data (e.g., schematics whose portions are labeled as containing or not containing symbols) and a large amount of unlabeled training data (e.g., schematics whose portions are not labeled as containing or not containing symbols). FIG. 2 is a schematic of a physical system in which the detected symbols are identified by red boxes.

[0036] In some cases, the detection module 122 can implement a ternary classifier instead of a binary classifier. The ternary classifier can have a similar structure and functionality as the binary classifier, but can classify regions within a schematic as (1) containing a tag (i.e., a digital representation of a sensor, which is typically represented by a circle inscribed in a square), (2) containing a locally mounted instrument (i.e., a physical instrument such as a pressure gauge or temperature reading, which is typically represented by a circle), or (3) not containing a symbol. Tags and LMIs are important because they typically have corresponding entries in a time series sensor database.

[0037] To detect symbols in a schematic, the detection module 122 can use a sliding window approach. That is, a particular schematic can be divided into a plurality of overlapping windows. The classifier described above can return an output for each window. The output for each window can indicate whether the window has at least a portion of a symbol within its bounds, or whether the window contains a label, LMI, or no symbol. The size and stride of the sliding window can be based in part on the mean or median size of a symbol, or any other statistical measure or heuristic. In one example, the window can be 100 pixels x 100 pixels. In the case of the sliding window approach, the output of the binary classifier can undergo a post-processing step before being classified as a particular component. For example, a non-maximum suppression algorithm can be applied to the output to filter out one or more of the multiple windows that have the same symbol (e.g., due to window overlap).

[0038] To detect components in a schematic, the detection module 122 can implement an algorithm configured to detect dashed lines that delineate groups of symbols. Such dashed lines can indicate that the symbols are constituent components of a component. FIG. 3 is a schematic of a physical system in which the detected components are identified by red boxes.

[0039] To detect connections in a schematic, the detection module 122 can implement a similar algorithm configured to detect solid lines that connect components and groups. In one example, the detection module 122 uses a graph search algorithm to traverse the components and solid lines that represent connections between the components. The graph search algorithm can traverse the components and solid lines by detecting and tracking adjacent black pixels in the schematic. A black pixel can be a pixel that satisfies a threshold pixel intensity. The graph search algorithm can represent pixels as nodes of a graph. A particular pixel can or can not be part of a component, and this information can be included in the node corresponding to that pixel. A symbol is represented in the graph as a set of nodes corresponding to the pixels that form the symbol.

[0040] With this graph representation of the graph, connections between symbols can be identified by a depth-first search (DFS). Specifically, for each detected symbol, a DFS is initialized from one of the nodes in the symbol. The DFS traverses the graph along black nodes, hitting (and keeping track of) connected symbols along the path. Once all valid paths are exhausted, the search terminates.

[0041] In another aspect, the classification module 124 can be configured to classify the detected symbols as components. To classify the symbols as components, the classification module 124 can implement a multi-class deep neural network configured to generate an output indicating which of a plurality of components a particular symbol corresponds to. The plurality of components can be determined prior to training the multi-class deep neural network. The components can be stored in a component library. If, during classification, the multi-class deep neural network fails to classify a particular symbol as a component, the symbol can be indicated as a new symbol. The new symbol can be added to the component library.

[0042] For example, the multi-class deep neural network can be a feed-forward neural network or a CNN. The multi-class deep neural network can be trained using a supervised, semi-supervised, or unsupervised learning process. In one implementation, the multi-class deep neural network can be trained on a small amount of labeled training data (e.g., symbols labeled as schematics corresponding to particular components) and a large amount of unlabeled training data (e.g., symbols not labeled as schematics corresponding to particular components).

[0043] Generally, the neural networks described herein can employ multiple layers of operations to predict one or more outputs from one or more inputs. The neural network can include one or more hidden layers between an input layer and an output layer. The output of each layer can be used as input to another layer (e.g., a next hidden layer or the output layer). Each layer of the neural network can specify one or more transformation operations to be performed on the input to that layer. Such a transformation operation can be referred to as a neuron. The output of a particular neuron can be a weighted sum of the inputs to the neuron, adjusted with a bias and multiplied by an activation function, such as a rectified linear unit (ReLU) or a sigmoid function. Training a neural network can involve providing an untrained or partially trained version of the neural network with inputs to generate predicted outputs, comparing the predicted outputs to expected outputs, and updating the weights and biases of the algorithm to account for the difference between the predicted outputs and the expected outputs. Specifically, a cost function can be used to compute the difference between the predicted outputs and the expected outputs. By computing the derivative of the cost function with respect to the network weights and biases, the weights and biases can be iteratively adjusted over multiple cycles to minimize the cost function. Training can be complete when the predicted outputs satisfy a convergence condition (e.g., achieve a small magnitude of computed cost).

[0044] A CNN is a neural network in which the neurons in some layers, referred to as convolutional layers, receive pixels from only a small portion of the input dataset. These small portions can be referred to as the receptive fields of the neurons. Each neuron in such a convolutional layer can have the same weights. In this way, the convolutional layer can detect features in any portion of the input dataset. A CNN can also have pooling layers that combine the outputs of clusters of neurons in the convolutional layers, as well as fully connected layers similar to traditional layers in a feed-forward neural network. CNNs can be particularly good at detecting and classifying objects (e.g., products) in images.

[0045] In some cases, the classification module 124 can instead implement an autoencoder to classify the symbols. An autoencoder can include an encoder and a decoder. The encoder can be configured to generate a compressed representation of an input symbol, and the decoder can be configured to reconstruct the input symbol from the compressed representation. The decoder can be trained on examples of known symbols (e.g., through a supervised learning process). The classification module 124 can compare the output of the decoder for a given input symbol to known signatures for various components in the component library (e.g., using a distance-based algorithm such as cosine similarity). The symbol can be classified as the component with the most similar or closest signature.

[0046] In some cases, a single object detection algorithm can both detect symbols and classify symbols. Such an object detection algorithm can take any of the forms described above.

[0047] The classification system 120 can provide the classified components and assemblies to a subsystem 130. The subsystem 130 can also receive a component table for the physical system. The subsystem 130 can be configured to associate some or all of the components and assemblies with entries in the component table. The component table can provide additional information about components that do not appear in the schematic, such as the size of a capacitor or the composition of a material.

[0048] The subsystem 130 can have an optical character recognition (OCR) module 132. The OCR module 132 can implement an OCR algorithm configured to recognize text associated with the classified components and assemblies. More specifically, the OCR algorithm can be configured to detect and identify identification numbers associated with the components and assemblies. In some cases, the identification numbers can be located in circles that are adjacent to or otherwise proximate to the classified components and assemblies. In such cases, the subsystem 130 can use a Hough transform to detect the circles. Additionally, the OCR algorithm can recognize text in the component table, including identification numbers and other data. In some cases, an efficient and accurate scene text detector (EAST) can be used to detect the text, where EAST is a pipeline that uses a neural network to produce bounding boxes for text present in an image.

[0049] The identification numbers in the schematic can correspond to identification numbers in the component table. The OCR module 132 can provide the identification numbers from the schematic to a matching module 134, which can match the identification numbers to identification numbers in the component table, thereby associating the corresponding components and assemblies with entries in the component table.

[0050] The system 100 can additionally include a model builder subsystem 140. The model builder subsystem 140 can receive the classified components, assemblies, and connections from the classification subsystem 120 and the associations from the subsystem 130 and generate a digital representation of the physical system. The digital representation of the physical system can include a digital model of the physical system and a machine-readable bill of materials. The machine-readable bill of materials can be a table of assets hierarchy identifying various components in the schematic or set of schematics, text associated with the components, and components connected to each component. The digital model can be computer code defining the components in the physical system, the connections between the components, and the state of the physical system. The components can be further defined by their parameters (e.g., dimensions, functionality, inputs and outputs, throughput, operating variables, etc.), which can be stored in a component library and retrieved when creating the digital model. The digital model can be a time-varying model or a static model.

[0051] FIG. 1 The system and its components can be implemented on one or more computing devices. The computing devices can be servers, desktop or laptop computers, electronic tablets, or mobile devices, etc. The computing devices can be located in one or more locations. The computing devices can have general purpose processors, graphics processing units (GPUs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs), etc. The computing devices can additionally have memory, such as dynamic or static random access memory, read only memory, flash memory, or hard drives, etc. The memory can be configured to store instructions that, when executed, cause the computing devices to implement the functionality of the pre-processing subsystem 110, the classification subsystem 120, the subsystem 130, or the model builder subsystem 140. The computing devices can additionally have network communication devices. The network communication devices can enable the computing devices to communicate with each other and with any number of user devices over a network. The network can be a wired or wireless network. For example, the network can be a fiber optic network, a satellite network, a cellular network, or a Bluetooth® network, etc. In other implementations, the computing devices can be several distributed computing devices that are accessible over the Internet. Such computing devices can be considered cloud computing devices.

[0052] FIG. 4 An example of a digital model as displayed in a user interface of an application is shown. The user interface can allow a user to validate the digital model. The user interface can show the identified components in blue and the identified assemblies in red. The user interface can enable the user to filter the components and assemblies by type or status (e.g., classified or unclassified). The user interface can also enable the user to manually reclassify the components and assemblies or edit other parameters of the components and assemblies.

[0053] FIG. 5is a flowchart of an example process for generating a digital representation of a physical system from engineering documents of the physical system. The engineering documents can be physical, non-digital (e.g., paper) schematics and corresponding parts lists. The process can be performed by a system of one or more suitably programmed computers in one or more locations.

[0054] The system can pre-process the non-digital schematics (510). Pre-processing the non-digital schematics can involve scanning the non-digital schematics into digital images and removing white space and other noise from the digital images.

[0055] The system can use a machine learning algorithm to (i) classify each of a plurality of symbols in the pre-processed schematics as a part and (ii) classify each set of related symbols as an assembly (510). The machine learning algorithm can include a binary classifier and a multi-class deep neural network or autoencoder, and classifying the symbols as parts can involve using the binary classifier to detect the symbols and using the multi-class deep neural network or autoencoder to classify each symbol as a part. The multi-class deep neural network can be a feed-forward neural network or a convolutional neural network. The multi-class deep neural network can be trained using a semi-supervised learning process. The classes of the multi-class deep neural network can be determined prior to training. The classes can be parts that are commonly found in engineering documents of that type. If a corresponding symbol in a schematic is not in the parts library, the system can add a corresponding part to the library. In some cases, the system can implement a single object detection algorithm that simultaneously detects and classifies symbols as parts. Classifying each set of related symbols as an assembly can include detecting a dashed line in one or more schematics that delineates the set.

[0056] The system can determine connections between parts, between assemblies, and between parts and assemblies by detecting and traversing solid lines connecting these various combinations (530). As previously described, the system can use a graph search algorithm to traverse the solid lines.

[0057] The system can associate a subset of the parts and assemblies with entries in the parts list by (i) using optical character recognition to detect and identify identification numbers in the schematics that are associated with the parts and assemblies and (ii) matching the detected identification numbers to identification numbers in the parts list (540). Detecting and identifying the identification numbers can involve using a Hough transform to identify circles in which the identification numbers reside. The circles can be adjacent to or otherwise proximate to corresponding parts in the schematics.

[0058] Using the classified parts and assemblies, the connections, and the associations, the system can generate a digital representation of the physical system (550). The digital representation can include a digital model of the physical system and a machine-readable bill of materials. The digital model can be computer code that defines the parts, connections, and states of the physical system. The digital model can be a static model or a time-varying model.

[0059] Example

[0060] In one example, the system described herein features a three-way CNN classifier for classifying regions of a schematic diagram as containing a label, an LMI (Labeled Motion) diagram, or not containing a symbol. The CNN has three convolutional layers spread between ReLU activation and max-pooling layers, followed by two fully connected dense layers. The first hidden layer has 64 units with ReLU activation. The final output layer has three units with softmax activation to predict the probability that the input image belongs to each of the three classes. The CNN is trained on 18 labeled pipeline and instrumentation graphs, including 308 labeled croppings and 687 LMI croppings. The training examples are augmented by rotating, shifting, shearing, scaling, and flipping to keep the CNN invariant to these transformations. Dropout is applied in the dense layers to add some regularization and improve the CNN's generalization performance. Symbol croppings are split into training (60%) and validation (40%) sets, and the network is trained to minimize the cross-entropy loss on the validation set. Both training and validation losses converge after 100 training epochs. The training and validation sets were then combined to train the final model on the entire dataset over 100 epochs. Labels were classified with 100% precision and 98% recall across 11 unseen pipeline and instrumentation graphs using a classification probability threshold of 0.95. LMI was classified with 85% precision and 90% recall using a classification probability threshold of 0.95.

[0061] COMPUTER SYSTEM

[0062] This disclosure provides a computer system that is programmed to implement the methods of this disclosure. FIG. 6 It shows that it is programmed or otherwise configured to implement FIG. 1 The computer system 601 of the system 100. The computer system 601 further regulates the present disclosure. FIG. 5 The computer system 601 can be a user's electronic device or a computer system located remotely relative to the electronic device. The electronic device can be a mobile electronic device.

[0063] The computer system 601 includes a central processing unit (CPU, also "processor" and "computer processor") 605 that can be a single core or multi core processor, or a plurality of processors for parallel processing. The computer system 601 also includes memory or memory location 610 (e.g., random access memory, read only memory, flash memory), electronic storage unit 615 (e.g., hard disk), communication interface 620 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 625, such as cache, other memory, data storage, and / or electronic display adapters. The memory 610, storage unit 615, interface 620, and peripheral devices 625 are in communication with the CPU 605 through a communication bus (solid lines), such as a motherboard. The storage unit 615 can be a data storage unit (or data repository) for storing data. The computer system 601 can be operatively coupled to a computer network ("network") 630 with the aid of the communication interface 620. The network 630 can be the Internet, an internet and / or an extranet, or an intranet and / or extranet that in turn can use a private communication protocol to communicate with an internet, intranet, and / or extranet. The network 630 can include one or more computer servers, which can implement a distributed computing solution. In some cases, the network 630 can implement a peer-to-peer network, which can enable devices coupled to the computer system 1101 to behave as a client or a server.

[0064] The CPU 1105 can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions can be stored in a memory location, such as the memory 610. The instructions can be directed to the CPU 605, which can subsequently program or otherwise configure the CPU 605 to implement methods of the present disclosure. Examples of operations performed by the CPU 605 can include fetch, decode, execute, and writeback.

[0065] The CPU 605 can be part of a circuit, such as an integrated circuit. One or more other components of the system 601 can be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0066] The storage unit 615 can store files, such as drivers, libraries and saved programs. The storage unit 615 can store user data, e.g., user preferences and user programs. The computer system 601 in some cases can include one or more additional data storage units that are external to the computer system 601, such as located on a remote server that is in communication with the computer system 601 through an intranet or the Internet.

[0067] Computer system 601 can communicate with one or more remote computer systems via network 630. For example, computer system 601 can communicate with a user's remote computer system. Examples of remote computer systems include personal computers (e.g., portable PCs), tablet computers, or tablet PCs (e.g., tablet PCs with a large capacity). GalaxyTab), telephone, smartphone (e.g., iPhone, Android-compatible devices (or personal digital assistant). Users can access computer system 601 via network 630.

[0068] The methods described herein can be implemented by machine-executable code (e.g., a computer processor) stored in an electronic storage location of computer system 601, such as memory 610 or electronic storage unit 615. The machine-executable or machine-readable code can be provided in software form. During use, this code can be executed by processor 605. In some cases, the code can be retrieved from storage unit 615 and stored in memory 610 for access by processor 605 at any time. In some cases, electronic storage unit 615 can be excluded, and machine-executable instructions are stored on memory 610.

[0069] Code can be pre-compiled and configured for use with machines that have processors suitable for executing the code, or it can be compiled during runtime. The code can be provided in a programming language, which can be selected to enable the code to be executed either pre-compiled or compiled.

[0070] Aspects of the systems and methods provided herein, such as computer system 601, can be embodied in programming. Various aspects of this technology can be considered "products" or "artifacts," which typically exist or are contained in a machine-readable medium in the form of machine (or processor) executable code and / or associated data. Machine-executable code can be stored on electronic storage units, such as memory (e.g., read-only memory, random access memory, flash memory) or hard disks. Media of the "storage" type can include any or all tangible memory of a computer, processor, etc., or related modules thereof, such as various semiconductor memories, tape drives, disk drives, etc., which can readily provide non-transitory storage for software programming. All or part of the software can sometimes be communicated via the Internet or various other telecommunications networks. For example, such communication enables software to be loaded from one computer or processor to another, such as from a management server or host computer to a computer platform for an application server. Therefore, another type of media that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as physical interfaces between local devices, via wired and optical terrestrial networks, and via various air links. The physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as the medium carrying software. As used herein, unless limited to non-transitory, tangible "storage" media, the term "readable medium" for a computer or machine refers to any medium that participates in providing instructions to a processor for execution.

[0071] Therefore, machine-readable media, such as computer-executable code, can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical discs or disks, any storage device such as any computer, such as those used to implement the database shown in the figure. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that form the bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched cardstock tapes, any other physical storage media with a perforated pattern, RAM, ROM, PROM and EPROM, FLASH-EPROM, any other memory chips or cassette tapes, carriers for transmitting data or instructions, cables or links for transmitting such media, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer-readable media may involve passing one or more sequences of one or more instructions to a processor for execution.

[0072] The computer system 601 can include or be in communication with an electronic display 635, which can include a user interface (UI) 640 for providing digital models, for example. FIG. 4 Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0073] The methods and systems of the present disclosure can be implemented by one or more algorithms. The algorithms can be implemented in software when executed by the central processing unit 605. The algorithm can be, for example, one of the neural networks described in the present disclosure.

[0074] While preferred embodiments of the application have been shown and described herein, it is to be understood that the application has been presented by way of example only. It is not intended to limit the application to the specific examples described. Although the present application has been described with reference to the aforementioned embodiment, the description and illustrations serve only to illustrate the present application and should not be construed in a limiting sense. Numerous alternative embodiments, which will be apparent to those skilled in the art, will be readily suggested by the description and illustrations herein. Furthermore, it should be understood that all aspects of the application are not limited to the specific depictions, configurations or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the application described herein can be employed in practicing the application. Accordingly, the present application is intended to embrace all such alternatives, modifications and variances that fall within the scope of the present application. It is intended that the scope of the application be defined by the following claims and that equivalents or alternatives within the scope and spirit of the application should be within the scope of the claims.

Claims

1. A method for generating a digital representation of a system from engineering documents of the system, the engineering documents comprising one or more schematics and a component table, the method comprising: (a) using a deep learning algorithm, (i) classifying each symbol of a plurality of symbols in the one or more schematics as a component, and (ii) classifying each set of related symbols as an assembly, wherein classifying each set of related symbols as an assembly comprises detecting, for each set, a dashed line in the one or more schematics that delineates the set; (b) determining connections between the components and the assemblies; (c) associating a subset of the components and the assemblies with existing entries in the component table, wherein the component table stores information about one or more components that do not appear in the one or more schematics; and (d) generating the digital representation of the system from the components, the assemblies, the connections, and the associations, wherein the digital representation of the system comprises at least a digital model of the system and a machine-readable bill of materials.

2. The method of claim 1, further comprising: Prior to (a), pre-processing the one or more schematics by removing white space or noise from the one or more schematics.

3. The method of claim 1, further comprising: Prior to (a), determining a library of components to be detected in the one or more schematics.

4. The method of claim 3, further comprising: (i) determining that a respective symbol in the one or more schematics is not in the library of components, and (ii) adding the respective symbol and its corresponding component to the library of components.

5. The method of claim 1, wherein (a) comprises using a binary classifier to detect the plurality of symbols in the one or more schematics, wherein the deep learning algorithm comprises the binary classifier, and wherein the detecting comprises a sliding window approach.

6. The method of claim 5, wherein (a) further comprises using a multiclass deep neural network to classify each of the detected symbols as a component, wherein the deep learning algorithm comprises the multiclass deep neural network.

7. The method of claim 5, wherein (a) further comprises using an autoencoder to classify each of the detected symbols as a component, wherein the deep learning algorithm comprises the autoencoder.

8. The method of claim 6, wherein the multiclass deep neural network is selected from the group consisting of: a feedforward neural network and a convolutional neural network.

9. The method of claim 6, wherein the multiclass deep neural network is trained using a semi-supervised learning process.

10. The method of claim 1, wherein (a) comprises using a single object detection algorithm to detect and classify the plurality of symbols in the one or more schematics.

11. The method of claim 1, wherein (b) comprises detecting solid lines that connect the components and the assemblies.

12. The method of claim 11, wherein the detecting comprises using a graph search algorithm.

13. The method of claim 1, wherein (c) comprises: detecting and identifying, using optical character recognition, in the one or more schematics, identification numbers associated with the components and the assemblies in the subset of the components and assemblies; and matching the detected identification numbers to identification numbers in the component table.

14. The method of claim 13, wherein detecting the identification numbers comprises using a Hough transform.

15. The method of claim 1, wherein the digital model comprises computer code defining the components, the connections, and states of the system.

16. The method of claim 15, wherein the digital model is a static model.

17. The method of claim 15, wherein the digital model is a time-varying model.

18. One or more non-transitory computer storage media storing instructions that are operable, when executed by one or more computers, to cause the one or more computers to operate to generate a digital representation of a system from engineering documents of the system, the engineering documents comprising one or more schematics and a component table, the operations comprising: (a) using a deep learning algorithm, (i) classifying each symbol in a plurality of symbols in the one or more schematics as a component, and (ii) classifying each set of related symbols as an assembly, wherein classifying each set of related symbols as an assembly comprises: detecting, for each set, a dashed line in the one or more schematics that delineates the set; (b) determining connections between the components and the assemblies; (c) associating a subset of the components and the assemblies with existing entries in the component table, wherein the component table stores information about one or more components that do not appear in the one or more schematics; and (d) generating a digital representation of the system from the components, the assemblies, the connections, and the associations, wherein the digital representation of the system comprises at least a digital model of the system and a machine-readable bill of materials.

19. The non-transitory computer storage medium of claim 18, wherein the operations further comprise: prior to (a), pre-processing the one or more schematics by removing white space or noise from the one or more schematics.

20. The non-transitory computer storage medium of claim 18, wherein the operations further comprise: prior to (a), determining a library of components to detect in the one or more schematics.

21. The non-transitory computer storage medium of claim 20, wherein the operations further comprise: (i) determining that a respective symbol in the one or more schematics is not in the library of components, and (ii) adding the respective symbol and its corresponding component to the library of components.

22. The non-transitory computer storage media of claim 18, wherein (a) comprises using a binary classifier to detect the plurality of symbols in the one or more schematics, wherein the deep learning algorithm comprises the binary classifier, and wherein the detecting comprises a sliding window approach.

23. The non-transitory computer storage media of claim 22, wherein (a) further comprises using a multiclass deep neural network to classify each of the detected symbols as a component, wherein the deep learning algorithm comprises the multiclass deep neural network.

24. The non-transitory computer storage medium of claim 22, wherein (a) further comprises classifying each of the detected symbols as components using an autoencoder, wherein the deep learning algorithm comprises the autoencoder.

25. The non-transitory computer storage medium of claim 23, wherein the multiclass deep neural network is selected from the group consisting of: a feedforward neural network and a convolutional neural network.

26. The non-transitory computer storage medium of claim 23, wherein the multiclass deep neural network is trained using a semi-supervised learning process.

27. The non-transitory computer storage medium of claim 18, wherein (a) comprises detecting and classifying the plurality of symbols in the one or more schematics using a single object detection algorithm.

28. The non-transitory computer storage medium of claim 18, wherein (b) comprises detecting solid lines connecting the components and the assemblies.

29. The non-transitory computer storage medium of claim 28, wherein the detecting comprises using a graph search algorithm.

30. The non-transitory computer storage medium of claim 18, wherein (c) comprises: detecting and identifying identification numbers associated with the components and the assemblies in a subset of the components and assemblies in the one or more schematics using optical character recognition; and matching the detected identification numbers to identification numbers in the component table.

31. The non-transitory computer storage medium of claim 30, wherein detecting the identification numbers comprises using a Hough transform.

32. The non-transitory computer storage medium of claim 18, wherein the digital model comprises computer code defining the components, the connections, and states of the system.

33. The non-transitory computer storage medium of claim 32, wherein the digital model is a static model.

34. The non-transitory computer storage medium of claim 32, wherein the digital model is a time-varying model.

Images