Techniques for extracting machine-readable information from a P&ID

Abstract

In example embodiments, techniques are provided for using machine learning to extract machine-readable labels for text boxes and symbols in P&IDs in image-only format. A P&ID data extraction application uses an optical character recognition (OCR) algorithm to predict labels for P&ID text boxes. The P&ID data extraction application uses a first machine learning algorithm to detect symbols in a P&ID and returns, for each symbol, a predicted bounding box and a predicted kind of equipment. One or more of the predicted bounding boxes can be pruned by non-maximum suppression to avoid overlapping detections. The P&ID data extraction application uses a second machine learning algorithm to infer attributes of each detected symbol with remaining predicted bounding boxes. The P&ID data extraction application stores the predicted bounding boxes and labels including the predicted kind of equipment and inferred attributes in a machine-readable format.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the use of piping and instrumentation diagrams (P&IDs) in plant and infrastructure projects, and more particularly to techniques for extracting machine-readable information from P&IDs in image-only format. BACKGROUND

[0002] Many engineering disciplines utilize functional diagrams. In contrast to physical models that represent the specific location, size, and shape of elements, functional diagrams are independent of location, size, and shape and instead focus on the process flow. In the field of plant and infrastructure design and maintenance, a common type of functional diagram is the P&ID. Figure 1 is a view that is part of an example P&ID 100. The P&ID includes symbols 110 representing elements (e.g., valves, pumps, vessels, instruments, etc.), text boxes 120 providing descriptions (e.g., names, codes, properties, etc.) of the elements, connections 130 representing connections (e.g., pipes, wires, etc.) between the elements, and other information (e.g., title blocks, legends, annotations, etc.) (not shown).

[0003] A typical plant or infrastructure project can have hundreds of related P&IDs that have been created over the course of many years. These P&IDs can often be available in image-only format (e.g., as a graphics file such as JPG or PNG, or as an image-only PDF, etc.) that lacks machine-readable information (e.g., metadata) about the symbols, text boxes, and connections represented therein. Sometimes the image-only P&IDs originate from a scan of a printed document and are of poor quality, have low resolution, visual artifacts, unclear or blurry portions, etc.

[0004] In image-only format, the information in the P&IDs is difficult to verify and difficult to use. There is an increasing desire to create digital twins of plants and infrastructures, and the information in the P&IDs is often useful in creating such models. However, since the information in image-only format is largely inaccessible to design and modeling applications, obtaining such information often involves lengthy manual inspection and data entry. Updating the P&IDs themselves is difficult in the case of image-only format.

[0005] One possible approach to address the problem of P&IDs in only image format is to manually recreate each P&ID in a design application. A human operator looks at the image-only P&ID and manually recreates each textbox, symbol, connection, etc. that he or she sees using tools provided by the design application. The result is a new P&ID that includes machine-readable information describing the textboxes, symbols, and connections, etc. Depending on the complexity of the P&ID, the manual recreation can take hours or tens of hours, rendering it impractical in many use cases. Furthermore, the manual recreation is inherently error-prone due to its detailed but repetitive nature. Substantial additional time and effort can be required to review and verify the manually recreated P&ID to ensure an acceptable level of accuracy.

[0006] Another possible approach to address the problem of P&IDs in only image format is to attempt to use pattern recognition algorithms to determine the information in each P&ID. The pattern recognition algorithm can search the P&ID for exact (pixel-by-pixel identical) copies of a set of clip images from an image library. Upon finding an exact copy, the information associated with the clip image in the image library can be associated with the exact copy. While in theory this approach can build a set of machine-readable information for P&IDs in only image format, in practice this approach is largely infeasible. To operate successfully, the pattern recognition algorithm requires a library of clip images identical to everything in the P&ID. Since P&IDs often contain variations on how to represent textboxes, symbols, connections, etc., in practice a new image library can need to be custom created for each P&ID or nearly each P&ID. Furthermore, the pattern recognition algorithm can be easily confused when the P&ID has low resolution, includes visual artifacts, has unclear or blurry portions, or is otherwise of low quality. In such cases, images that ideally should match equally can not match equally in practice, which can result in missed recognitions.

[0007] Accordingly, there is a need for improved techniques for extracting machine-readable information from P&IDs in only image format. SUMMARY

[0008] In various embodiments, techniques are provided for using machine learning to extract machine-readable labels for textboxes and symbols in P&IDs in only image format (e.g., JPG, PNG, image-only PDF, etc.). The information in each label is predicted with a corresponding confidence level. The machine learning algorithm is adapted and extended to address the challenges posed by P&IDs.

[0009] In one embodiment, the P&ID data extraction application uses an optical character recognition (OCR) algorithm to predict bounding boxes for text boxes in a P&ID in image-only format. The P&ID data extraction application uses a first machine learning algorithm to detect symbols in the P&ID, which returns a predicted bounding box and a predicted class of equipment for each symbol. One or more of the predicted bounding boxes can be pruned by non-maximum suppression to avoid overlapping detections. The P&ID data extraction application uses a second machine learning algorithm to infer attributes of each detected symbol with a remaining predicted bounding box. For each detected symbol with a remaining predicted bounding box, the P&ID data extraction application stores the predicted bounding box and a label including the predicted class of equipment and the inferred attributes in a machine-readable format.

[0010] It should be understood that various other features and alternatives can be implemented in addition to those discussed in the SUMMARY. The SUMMARY is intended to serve only as a brief introduction to the reader and does not mean to limit or suggest all that is included by the examples mentioned herein, or that is essential or fundamental to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are described below, refer to example implementations of the present disclosure and, as such, do not limit its scope in any way. The drawings include the following figures:

[0012] Figure 1 is a view of a portion of an example P&ID;

[0013] Figure 2 is a high-level block diagram of an example P&ID data extraction application;

[0014] Figure 3 is a high-level sequence of steps for extracting machine-readable labels for text boxes and symbols in a P&ID in image-only format;

[0015] Figure 4 is a view of a portion of an example pre-processed P&ID that can be displayed in a user interface of a P&ID data extraction application;

[0016] Figure 5 is a diagram of an example region-based convolutional neural network object detection architecture (e.g., a Faster R-CNN architecture) that can be adapted to process P&ID characteristics;

[0017] Figure 6 is a flowchart of an example sequence of steps for a modified non-maximum suppression algorithm;

[0018] Figure 7 is a diagram illustrating the use of a deep neural network architecture adapted for attribute classification;

[0019] Figure 8is a view of a portion of an example marked-up P&ID that can be displayed in a user interface of a P&ID data extraction application; and

[0020] Figure 9 is a flowchart of an example sequence of steps for training a neural network of a P&ID data extraction application to detect symbols. DETAILED DESCRIPTION

[0021] Figure 2 is a high-level block diagram of an example P&ID data extraction application. The P&ID data extraction application 200 can be a standalone software application or a component of a larger software application, such as a design and modeling software application. The software can be divided into local software 210 that executes on one or more computing devices (collectively, “local devices”) that are local to an end user, and cloud-based software 212 that, in some cases, executes on one or more computing devices (collectively, “cloud computing devices”) that are remote from the end user and accessible via a network (e.g., the Internet). Each computing device can include a processor, memory / storage, a display screen, and other hardware (not shown) for executing software, storing data, and / or displaying information. The local software 210 can include a front-end client 220 and one or more back-end clients 230 that operate on the local devices. In some cases, the cloud-based software 212 can include one or more back-end clients 230 that operate on the cloud computing devices. The front-end client 220 can provide user interface functionality as well as perform certain non-processing-intensive operations. The back-end client(s) 230 can perform certain more processing-intensive operations (e.g., OCR operations, machine learning operations, etc.). The front-end client 220 and the back-end client(s) 230 can operate on different tasks at the same time, such that a user can utilize the user interface of the P&ID data extraction application 200 to perform a task while one or more back-end clients 230 are performing a different task without having to wait for it to complete.

[0022] Figure 3 is a high-level sequence of steps 300 for extracting machine-readable markings for text boxes and symbols in a P&ID in an image-only format. At step 310, the P&ID data extraction application 200 loads a P&ID in an image-only format (e.g., JPG, PNG, image-only PDF, etc.) that lacks machine-readable information about text boxes, symbols, and connections. In some cases, the P&ID is derived from a scan of a printed document.

[0023] At step 320, the P&ID data extraction application 200 pre-processes the P&ID to rasterize, resize, and / or binarize the P&ID. Rasterization can involve decompression, conversion, and / or extraction operations to produce a rasterized P&ID. Resizing can involve changing the resolution (e.g., dots per inch (DPI)) to a resolution that is easier to process (e.g., by OCR algorithms, machine learning algorithms, etc.). Binarization can involve reducing color or grayscale information in the P&ID to black and white (i.e., binary information) using adaptive thresholding. As an additional part of step 320, the P&ID data extraction application 200 can also apply noise reduction and image geometric correction. Noise reduction can involve applying filters to remove noise (e.g., that can have been introduced by scanning a paper document) from the P&ID without sacrificing excessive amounts or true details. Image geometric correction can involve correcting for cropping, rotation, or other types of distortion (e.g., that can have been introduced by scanning a paper document).

[0024] At step 330, which can be optional, the P&ID data extraction application 200 can display the pre-processed P&ID within a window of its user interface while extracting machine-readable markings in subsequent steps. Figure 4 is a view of a portion of an example pre-processed P&ID 400 that can be displayed in the user interface of the P&ID data extraction application 200. At this stage, text boxes and symbols are not easily extractable from the underlying image.

[0025] At step 340, the P&ID data extraction application 200 employs an OCR algorithm to predict bounding boxes that enclose each text box in the P&ID and the text within each text box in the P&ID. The OCR algorithm can be implemented by an OCR framework that is suitable for the characteristics of the P&ID (e.g., Form Recognizer of Microsoft Azure Cognitive Services). Generally, the operation of the OCR algorithm can be divided into three main stages: a physical layout analysis stage, a text recognition stage, and a language modeling stage. In the physical layout stage, the OCR algorithm divides the image into non-text regions and text lines and places bounding boxes around the text lines. In the text recognition stage, the OCR algorithm identifies the text contained within each text line and represents possible recognition alternatives as a hypothesis graph. Recognition can involve segmenting each text line into individual characters, extracting features from the individual character images, and classifying the individual character images based on the extracted features. In the language modeling stage, the OCR algorithm can make selections among the recognition alternatives based on knowledge about language, vocabulary, and grammar.

[0026] The language modeling stage can use statistical language models such as dictionaries, n-gram patterns, stochastic grammars, etc. The language modeling stage can be adapted to P&ID characteristics. First, instead of using a general dictionary, a domain-specific dictionary can be used that can be used to bias predictions towards words and codes expected in P&IDs. The domain-specific dictionary can be an industry-wide dictionary based on general knowledge of words and codes used in P&IDs for a given industry, or can be a user-specific dictionary based on a specific list of words and codes used by the user in their P&IDs. Second, domain-specific patterns for n-grams can be used to prevent biasing towards words and codes expected in P&IDs but uncommon in general text (e.g., a number "0" and a letter together, which can often be misinterpreted as the letter "O" under general text pattern conditions, a number "1" and a letter together, which can often be misinterpreted as the letter "l" under general text pattern conditions, etc.). The domain-specific patterns for n-grams can be custom coded or extracted from P&IDs that have been successfully processed and validated.

[0027] At step 350, the P&ID data extraction application 200 employs a machine learning algorithm to detect symbols in the P&ID and produce a set of predicted bounding boxes and predicted equipment classes. The machine learning algorithm can involve a region-based convolutional neural network object detection architecture adapted to handle P&ID characteristics (e.g., a Faster Region-CNN (R-CNN) architecture). An object detection framework (e.g., the Detectron2 framework) can implement the architecture.

[0028] Figure 5 is a diagram 500 of an example region-based convolutional neural network object detection architecture (e.g., a Faster R-CNN architecture) 500 that can be adapted to handle P&ID characteristics. Generally, the architecture takes a pre-processed P&ID as an input image 510 and produces a set of predicted bounding boxes, predicted classes, and confidences as an output 520. The architecture 500 includes three main parts: a feature pyramid network 530, a region proposal network (RPN) 540, and a box head 550. The feature pyramid network 530 extracts feature maps from the input image at different scales. The RPN 540 detects object regions from the multi-scale feature maps. The box head 550 clips regions of the feature maps into multiple fixed-size features using the proposal boxes, obtains bounding box locations and classification results, and filters them to produce the final output of bounding boxes and classes.

[0029] More specifically, the input image 510 is set to a predetermined image size. In general applications, the minimum image size can be set to 600 to 800 pixels. To have enough resolution to "see" all the symbols in the P&ID, the minimum image size can be increased to a larger value (e.g., 1200 pixels).

[0030] The feature pyramid network 530 includes a residual neural network (Resnet) having stem blocks and stages, each stage containing multiple bottleneck blocks. Each bottleneck block includes multiple convolutional layers. The layers can also include an input layer and an output layer. Each layer can include a predetermined number of filters. To better handle P&ID characteristics, the number of filters in the input layer can be increased from the number of filters used in general applications, while the number of filters in each output layer can be decreased from the number of filters used in general applications. The bottleneck blocks and their layers generally operate to concatenate high-to-low resolution convolutions. To better detect both very large objects (e.g., that can occupy 20-30% of the total P&ID) and very small objects (e.g., that only occupy a few pixels), the number of resolutions can be expanded (e.g., adding at least one resolution above and at least one resolution below) beyond the range of resolutions used in general applications.

[0031] The output of the feature pyramid network 530 is a set of multi-scale feature maps with different receptive fields received by the RPN 540. The RPN 540 includes an RPN head that processes the feature maps to produce an objectness map and an anchor delta map. Ultimately, up to a predetermined number of proposal bounding boxes are selected. Typically, this is done by applying the proposed anchor deltas to the corresponding anchors, sorting the proposal bounding boxes by objectness score, and selecting the top scoring boxes from each feature level. In general applications, the predetermined number of proposal bounding boxes is typically 1000. To better handle P&IDs that can have hundreds of symbols, the number can be increased (e.g., to 2500).

[0032] The set of multi-scale feature maps from the feature pyramid network 530 and the proposal bounding boxes from the RPN 540 are received by the box head 550. Region of interest (ROI) pooling processes clip the regions of the feature maps specified by the proposal bounding boxes into a number of fixed-size features. The clipped regions are then fed to a head network having layers that classify the objects in the regions and fine-tune the bounding box positions and shapes. Up to a predetermined number of predicted bounding boxes and predicted classes are inferred, along with a confidence. In general applications, the predetermined number of predicted bounding boxes is typically 100. To better handle P&IDs that can have hundreds of symbols, the number can be increased (e.g., to 784).

[0033] Return Figure 3At step 360, the P&ID data extraction application 200 employs a modified non-maximum suppression algorithm to extract any predicted bounding boxes determined to be incorrectly overlapped. The non-maximum suppression algorithm can be optimized to better handle P&IDs. Figure 6 is a flowchart of an example sequence of steps for the modified non-maximum suppression algorithm. The modified non-maximum suppression algorithm takes as input a set of predicted bounding boxes, associated classes, and confidences. At step 610, the algorithm weights the confidence of each bounding box based on a measure of importance of the associated class and its location. The measure of importance can be predefined based on typical patterns in P&IDs in the industry or in the P&IDs of a particular user. At step 620, the modified non-maximum suppression algorithm finds the bounding box with the highest weighted confidence in the set of predicted bounding boxes, removes it from the set, and places it in an output set. At step 630, the modified non-maximum suppression algorithm computes a dynamic overlap of the other bounding boxes in the set of predicted bounding boxes with the bounding box with the highest weighted confidence. In contrast to traditional overlap computations, the dynamic overlap takes into account the shape of the symbols enclosed within each bounding box (e.g., a circular symbol within a rectangular bounding box) and adjusts the computed overlap based on the shape. This adjustment can be based on a computation of the actual overlap given the shape of each bounding box, or based on various heuristics or approximations of the overlap given the shape. At step 640, the modified non-maximum suppression algorithm removes all bounding boxes from the set of predicted bounding boxes for which the computed overlap is greater than a predetermined threshold, provided that such predicted bounding box is not associated with a symbol of a class designated as a container (i.e., a symbol whose class indicates that it is intended to include other symbols). At step 650, the modified non-maximum suppression algorithm determines whether there are any remaining bounding boxes in the set of predicted bounding boxes (i.e., that have not been placed in the output set or removed). If so, a loop is performed to step 620. If not, at step 660, the output set is returned and the extraction of bounding boxes with incorrect overlap is completed.

[0034] At step 370, the P&ID data extraction application 200 employs a machine learning algorithm to infer attributes of each symbol with a predicted bounding box that remains after the bounding boxes with incorrect overlap have been extracted. The machine learning algorithm can be based on a deep neural network architecture suitable for attribute classification implemented by a framework (e.g., the PyTorch framework). The framework can implement features such as n-dimensional tensor computation with acceleration and automatic differentiation. Figure 7is a diagram 700 illustrating the use of a deep neural network architecture suitable for attribute classification. The deep neural network architecture receives as an input image 710 a portion of the P&ID detected as a symbol by step 350. The input image is applied to a dense convolutional neural network (e.g., DenseNet backbone) 720. While a convolutional network typically has one connection between each layer and its subsequent layers, each layer of a dense convolutional neural network can receive the feature maps of all previous layers as input and provide its own feature maps as input to all subsequent layers. The output of the dense convolutional neural network 720 is provided to a common decoding layer 730, which can be a fully connected layer. The common decoding layer 730 produces an output module 740, which includes a variable number of attributes depending on the class, as well as a variable number of possible values for each attribute and a confidence for each possible value. For example, for the class “valve,” there can be 8 different attributes (e.g., type, opening, mode of operation, number of inlets, regulation mode, angle, fault type, and connection type). Each of these 8 different attributes can have a number of possible values ranging from 2 to 11. Each value has an associated confidence that it is the correct value. Assuming that the number of possible choices is finite, the confidence always sums to 100%. For example, assuming an attribute has 3 possible values, the confidence can be 79.6%, 19.5%, and 0.9%. While the deep neural network architecture can return all possible values, only those values with a confidence exceeding a predetermined threshold can be utilized.

[0035] At step 380, the P&ID data extraction application 200 constructs labels from the predicted class of equipment and the inferred attributes and stores the bounding boxes and labels in a machine-readable format in memory. The machine-readable format can be separate from the image-only format, e.g., a JSON file related to but separate from a JPG file, a PNG file, or an image-only PDF file including the P&ID image. Alternatively, the machine-readable format can be integrated into the file storing the P&ID image.

[0036] Depending on the use case, the machine-readable format can be used in various ways. At step 390, which can be optional, the P&ID data extraction application 200 can display the label-extracted P&ID within a window of its user interface. Figure 8 is a view of a portion of an example label-extracted P&ID 800 that can be displayed in the user interface of the P&ID data extraction application 200. The bounding boxes of text boxes and symbols are visually represented by an overlay (e.g., a colored highlight). The labels can be displayed in a secondary window of the user interface (not shown) for user confirmation or correction. The machine-readable format can additionally or alternatively be provided to a design and modeling application (e.g., for use in building a model / digital twin or for other purposes).

[0037] Before using the neural network described above in connection with steps 350 and 370, the neural network needs to be trained on a number of labeled samples. The labeled samples can be produced in various ways. In some cases, the labeled samples can be generated by a special labeling application that includes automation and support for efficiency in repetitive tasks common to labeling P&IDs. In other cases, the labeled samples can be produced using conventional techniques. Figure 9 is a flowchart of an example sequence of steps 900 for training a neural network of a P&ID data extraction application 200 to detect symbols. At step 910, a batch of labeled samples (e.g., 4) is received and applied to a current model that attempts to detect symbols. In some cases, the labeled samples can be subject to random clipping to avoid showing the same content to the neural network when performing iterative operations. At step 920, the output (predictions) are compared to the labels of the samples that represent ground truth. A set of loss values are computed to indicate how close or far the model is from the desired output. In general applications, less common categories are given less importance. To better handle P&IDs in which some symbols are much more prevalent than others, the loss values can be altered to give higher weight to samples that are part of less common categories. At step 930, the errors produced by the model are analyzed and backpropagated. The model parameters are adjusted to avoid the observed errors. At step 940, the performance of the model is computed using another labeled sample (that has not been seen during training) and the performance is compared to a predetermined level of performance. Additionally, the run time is compared to a predetermined time limit. If neither condition is met, then a loop is performed to step 910 and training continues with a new batch of labeled samples. If either condition is met, then it is considered a stopping criterion and training ends and the model is output at step 950.

[0038] It should be appreciated that various adjustments and modifications can be readily made to the above to suit various implementations and environments. While many aspects of the technology are discussed above as being implementable by particular software processes executing on particular hardware, it should be appreciated that some or all of the technology can also be implemented by different software on different hardware. In addition to general purpose computing devices, the hardware can include specially configured logic circuits and / or other types of hardware components. Most importantly, it should be appreciated that the above description is meant to be taken only by way of example.

Claims

1. A method for extracting machine-readable labels of text boxes and symbols in a piping and instrumentation diagram (P&ID) in a picture-only format, comprising: 1) predicting bounding boxes and labels of text indicating text boxes in a P&ID in a picture-only format by an optical character recognition (OCR) algorithm of a P&ID data extraction application executing on a computing device; 2) detecting symbols in the P&ID by a first machine learning algorithm of the P&ID data extraction application, the detection returning for each symbol a predicted bounding box and a predicted class of equipment, the first machine learning algorithm being based on a deep neural network architecture; 3) pruning one or more of the predicted bounding boxes to avoid overlapping detections; 4) inferring a variable number of attributes of each detected symbol with remaining predicted bounding boxes by a second machine learning algorithm of the P&ID data extraction application, the second machine learning algorithm being based on a different deep neural network architecture adapted for attribute classification, wherein each attribute has a variable number of attribute values; 5) storing, for each detected symbol with remaining predicted bounding boxes, the predicted bounding boxes and labels including the predicted class of equipment and inferred attributes in a machine-readable format in a memory of the computing device; and 6) displaying the predicted bounding boxes and labels of the text boxes and the symbols in a user interface.

2. The method of claim 1, wherein the detection step 2) produces a confidence for each predicted bounding box and associated predicted label, and the inference step 4) produces a confidence for each attribute, and the storing step 5) stores each confidence.

3. The method of claim 1, further comprising: preprocessing, by the P&ID data extraction application, the P&ID to at least one of rasterize, resize, or binarize the P&ID.

4. The method of claim 1, wherein the OCR algorithm uses a domain-specific dictionary in language modeling, the domain-specific dictionary being based on knowledge of words and codes used in P&IDs in a given industry, or based on a list of words and codes known to be used by users in P&IDs.

5. The method of claim 1, wherein the OCR algorithm uses a domain-specific pattern of n-grams in language modeling to prevent biasing toward words and codes expected in P&IDs.

6. The method of claim 1, wherein the first machine learning algorithm of the P&ID data extraction is based on a region-based convolutional neural network object detection architecture.

7. The method of claim 6, wherein the region-based convolutional neural network object detection architecture is a faster region-based convolutional neural network (R-CNN) architecture.

8. The method of claim 6, wherein the region-based convolutional neural network object detection architecture is adapted to use an enlarged image size, use an increased number of filters in the input layer and a decreased number of filters in the output layer, consider an extended number of resolutions to give higher weight to samples that are part of an uncommon class in the loss computation, or to use an increased number of proposal and predicted bounding boxes.

9. The method of claim 1, wherein the step of extracting 3) uses a non-maximum suppression algorithm.

10. The method of claim 9, wherein the non-maximum suppression algorithm comprises operations to: weight the confidence of each bounding box based on the importance of the associated class and a measure of the location; compute a dynamic overlap of other bounding boxes with the bounding box having the highest weighted confidence; and delete the bounding boxes where the computed dynamic overlap is greater than a predetermined threshold.

11. The method of claim 1, wherein the machine-readable format is a JavaScript Object Notation (JSON) file associated with the P&ID.

12. A computing device comprising: a display screen; a processor; and a memory coupled to the processor and configured to store a piping and instrumentation diagram (P&ID) in an image-only format and a P&ID data extraction application that, when executed, is operable to: 1) predict, using optical character recognition (OCR), bounding boxes in the P&ID and labels of text indicating text boxes, 2) detect, using a first machine learning algorithm, symbols in the P&ID and return, for each detected symbol, a predicted bounding box and a predicted class of equipment, the first machine learning algorithm being based on a deep neural network architecture, 3) infer, using a second machine learning algorithm, a variable number of attributes of each detected symbol, the second machine learning algorithm being based on a different deep neural network architecture adapted for attribute classification, wherein each attribute has a variable number of attribute values, and 4) store, in a machine-readable format, the predicted bounding box and the labels including the predicted class of equipment and the inferred attributes for each detected symbol in the memory.

13. The computing device of claim 12, wherein the P&ID data extraction application, when executed, is further operable to produce a confidence of each predicted bounding box and associated predicted label, produce a confidence of each attribute, and store each confidence in the memory.

14. A non-transitory computing device-readable medium having instructions stored thereon that, when executed by one or more computing devices, are operable to: 1) predict, using optical character recognition (OCR), bounding boxes in a piping and instrumentation diagram (P&ID) in an image-only format and labels of text indicating text boxes; 2) detect, using a first machine learning algorithm, symbols in the P&ID and return, for each symbol, a predicted bounding box and a predicted class of equipment, the first machine learning algorithm being based on a deep neural network architecture; ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 3) Pruning one or more of the predicted bounding boxes to avoid overlapping detections; 4) Using a second machine learning algorithm to infer a variable number of attributes for each detected symbol with remaining predicted bounding boxes, the second machine learning algorithm being based on a different deep neural network architecture adapted for attribute classification, wherein each attribute has a variable number of attribute values; 5) Storing, in a machine readable format, for each detected symbol with remaining predicted bounding boxes, the predicted bounding boxes and labels comprising predicted classes and inferred attributes of the equipment; and 6) Displaying, in a user interface, the text boxes and the predicted bounding boxes and labels of the symbols.

15. The non-transitory electronic device readable medium of claim 14, wherein the instructions are further operable when executed to: produce a confidence for each predicted bounding box and associated predicted label, produce a confidence for each attribute, and store each confidence.

16. The non-transitory electronic device readable medium of claim 14, wherein the instructions are further operable when executed to: pre-process the P&ID to at least one of rasterize, resize, or binarize the P&ID.

17. The non-transitory electronic device readable medium of claim 14, wherein the OCR uses at least one of a domain-specific dictionary or a domain-specific pattern of n-grams in language modeling, the first machine learning algorithm used in step 2) of detecting symbols employs a region-based convolutional neural network object detection architecture, and the pruning step 3) uses a non-maximum suppression algorithm.

Images