Method and apparatus for assigning machine-readable markings to piping and instrumentation diagrams

Abstract

In example embodiments, techniques are provided for efficient tagging, checking, and correction of predictions of P&IDs in image only format. To tag text boxes in a P&ID, a tagging application executes an OCR algorithm to predict a bounding box around each text box and machine-readable text within each text box, and displays these predictions in its user interface. The tagging application provides functionality for receiving user confirmation or correction of each predicted bounding box and predicted machine-readable text. To tag symbols in a P&ID, the tagging application receives user input to draw a bounding box around a symbol and assign the symbol to a category of equipment. In the case of multiple occurrences of a particular symbol, the tagging application provides functionality for copying and automatically detecting and assigning bounding boxes and categories. To tag connections in a P&ID, the tagging application receives user input to define connection points at corresponding symbols, and creates a connection between the connection points.

Description

Technical Field

[0001] This disclosure generally relates to the use of piping and instrumentation diagrams (P&IDs) in plant and infrastructure projects, and more specifically to techniques for marking, inspecting, and correcting predictions of P&IDs in image-only format. Background Technology

[0002] Functional diagrams are used in many engineering fields. Unlike physical models that represent the specific location, size, and shape of components, functional diagrams are independent of location, size, and shape, focusing instead on the flow of processes. In the fields of plant and infrastructure design and maintenance, a common type of functional diagram is the P&ID (Product and Indicator Diagram). Figure 1 This is a view of a portion of example P&ID 100. P&ID includes symbols 110 representing components (e.g., valves, pumps, containers, instruments, etc.), text boxes 120 providing descriptions of the components (e.g., names, codes, attributes, etc.), connections 130 representing connections between components (e.g., pipes, wires, etc.), and other information (e.g., title boxes, legends, notes, etc.) (not shown).

[0003] A typical factory or infrastructure project may have hundreds of related P&IDs created over many years. These P&IDs are often available in image-only formats (e.g., as graphic files such as JPG or PNG, or as image-only PDFs, etc.) lacking machine-readable information (e.g., metadata) about the symbols, text boxes, and connections represented therein. Sometimes, image-only P&IDs originate from scans of printed documents and are of poor quality, with low resolution, visual artifacts, unclear or blurry parts, etc.

[0004] In image-only format, the information in a P&ID is difficult to verify and use. There is a growing desire to create digital twins of factories and infrastructure, and the information in a P&ID is often useful in creating such models. However, because image-only information is largely unavailable for design and modeling applications, obtaining such information typically involves lengthy manual checks and data entry. Even updating the P&ID itself using image-only format is difficult. Often, changes require manually recreating the entire P&ID to represent the information in a more editable, machine-readable format.

[0005] One possible approach to solving the problem of image-only P&IDs is to use machine learning to automatically extract information from P&IDs and store it in a machine-readable format. However, the deployment of machine learning algorithms for P&ID data extraction presents supporting technical challenges. Typically, machine learning requires large amounts of annotated data from which artificial intelligence can infer machine learning models. For P&ID data extraction, this annotated data can take the form of labeled example P&IDs, which include depictions of text boxes, symbols, connections, etc. (e.g., raster images) and corresponding machine-readable tags. However, such labeled example P&IDs are rare, and creating specifically labeled example P&IDs for the purpose of training machine learning algorithms using existing software applications and workflows is a daunting task. A typical P&ID can include thousands of text boxes, symbols, and connections. However, existing software applications and workflows often lack any significant automation or support for improving the efficiency of repetitive tasks. While it may be workable when only a few dozen elements exist, it proves impractical when the number of elements scales up to thousands.

[0006] Similar technical challenges arise when processing the output of machine learning algorithms used for P&ID data extraction. The output may take the form of labeled P&IDs, where the machine learning algorithm has correctly predicted most of the labels, but a few errors exist. It might be desirable to allow users to inspect the predicted labels and correct the errors. However, this is a daunting task using existing software applications and workflows. Similar to the lack of significant automation or support for improving the efficiency of repetitive tasks when creating datasets of labels for training, the inspection and correction of predicted labels is hindered, rendering the workflow impractical.

[0007] Therefore, there is a need for improved techniques for predicting P&IDs in image-only formats for labeling, inspection, and correction. Summary of the Invention

[0008] In example embodiments, techniques are provided for efficiently labeling, inspecting, and correcting predictions of P&IDs in image-only formats (e.g., JPG, PNG, image-only PDF, etc.). The labeling application loads the image-only P&IDs and preprocesses them to rasterize, resize, and / or color-separate the P&IDs. To label text boxes within the P&IDs, the labeling application performs an OCR algorithm to predict the bounding box around each text box and the machine-readable text within each text box, and displays these predictions in its user interface. The labeling application provides functionality for receiving user confirmation or correction for each predicted bounding box and predicted machine-readable text. To label symbols within the P&IDs, the labeling application receives user input to draw bounding boxes around the symbols and assigns the symbols to a type of equipment. In cases where a particular symbol appears multiple times, the labeling application provides functionality for copying and automatically detecting and assigning bounding boxes and types. In copying, the labeling application copies the bounding box around a given symbol to another symbol and copies the type to be assigned to the other symbol. In automatic detection and assignment, the tagging application automatically detects other identical symbols corresponding to a given symbol, places bounding boxes around them, and automatically assigns them the same type as the given symbol. To tag connections in P&ID, the tagging application receives user input to define connection points at corresponding symbols and creates connections between those points.

[0009] In addition to those discussed in this summary, various additional features can be implemented by the tagging application to provide automation, improve workflow efficiency, or provide other benefits. This summary is intended simply as a brief introduction to the reader and does not indicate or imply that the examples mentioned herein cover all aspects of this disclosure, or are necessary or essential aspects of this disclosure. Attached Figure Description

[0010] The accompanying drawings, which relate to exemplary embodiments, are described, in which:

[0011] Figure 1 This is a view that is part of the example P&ID;

[0012] Figure 2 It is a high-level block diagram of example tagging applications that can be used to tag, inspect tags, and correct predictions for P&ID in image-only formats (e.g., JPG, PNG, image-only PDF, etc.);

[0013] Figures 3A-3B It is a sequence of high-level steps for assigning machine-readable tags to P&IDs in image-only formats;

[0014] Figure 3C This is a sequence of steps used to demonstrate a collision detection algorithm;

[0015] Figure 4 This is a screenshot showing the user interface of an example rasterized P&ID tagging application in the main window.

[0016] Figure 5 This is a screenshot of the user interface of an example tag application, showing the correctly predicted bounding box of the first text box and the correctly predicted machine-readable text.

[0017] Figure 6 This is a screenshot of an example tag application's user interface showing the bounding box of the second text box's error prediction and the machine-readable text of the error prediction;

[0018] Figure 7 This is a screenshot of the user interface of an example markup application showing the correction of the bounding box of the incorrect prediction of the second text box;

[0019] Figure 8 This is a screenshot of the user interface of an example tag application that shows the correction of machine-readable text that was incorrectly predicted for the second text box.

[0020] Figure 9 This is a screenshot of the user interface of an example marker application that shows the bounding box of the drawn symbol;

[0021] Figure 10 This is a screenshot of the user interface of an example tag application for selecting equipment types;

[0022] Figure 11 This is a screenshot of the user interface of an example tag application that demonstrates the copy operation;

[0023] Figure 12 This is a screenshot showing the user interface of an example tag application that automatically assigns bounding boxes and categories to symbols;

[0024] Figure 13 This is a screenshot of the user interface of an example tag application, showing the first attribute of the tag symbol;

[0025] Figure 14 This is a screenshot of the user interface of an example tag application, showing the second attribute of the tag symbol;

[0026] Figure 15 This is a screenshot showing the user interface of an example tag application that defines connection points and creates connections;

[0027] Figure 16 Here are screenshots of an example tag application's user interface illustrating the attributes of the tag connection; and

[0028] Figure 17 This is a screenshot showing the user interface of an example tag application that forms text associations. Detailed Implementation

[0029] Figure 2 This is a high-level block diagram of an example tagging application 200 that can be used to tag, inspect, and correct predictions for P&ID in image-only formats (e.g., JPG, PNG, image-only PDF, etc.). Tagging application 200 can be a standalone software application or a component of a larger software application (e.g., a design and modeling software application). The software can be divided into local software 210, which executes on one or more computing devices (collectively, "local devices") located on the end user's premises, and, in some cases, cloud-based software 212, which executes on one or more computing devices (collectively, "cloud computing devices") accessible via a network (e.g., the Internet) remote from the end user. Each computing device may include a processor, memory / storage device, display screen, and other hardware (not shown) for executing the software, storing data, and / or displaying information. Local software 210 may include a front-end client 220 and one or more back-end clients 230 operating on the local device. In some cases, cloud-based software 212 may include one or more back-end clients 230 operating on the cloud computing device. Front-end client 220 can provide user interface functionality and perform certain non-processing-intensive operations related to assigning machine-readable tags to image-only P&IDs, and in some cases, checking and correcting predicted machine-readable tags for image-only P&IDs. The predicted machine-readable tags may be provided by a machine learning algorithm of a P&ID data extraction application (not shown). The P&ID data extraction application may be a standalone software application or another component of the same larger software application, of which tagging application 200 is a part. One or more back-end clients 230 can perform certain more processing-intensive operations (e.g., optical character recognition (OCR) operations) related to assigning machine-readable tags to image-only P&IDs and correcting predicted machine-readable tags. Front-end client 220 and one or more back-end clients 230 can operate on different tasks simultaneously, allowing a user to perform a task using the user interface of tagging application 200 while one or more back-end clients 230 are performing different tasks, without waiting for them to complete.

[0030] Figures 3A-3BThis is a sequence of high-level steps 300 for assigning machine-readable tags to a P&ID in an image-only format. Such steps can be performed as part of a dataset that generates tags that are used as input to a machine learning algorithm for training a P&ID data extraction application. In this case, when step 300 is initially performed, there may be no existing machine-readable tags in the P&ID for text boxes, symbols, and connections. Alternatively, at least some of steps 300 can be performed as part of tag correction errors for predictions output by the machine learning algorithm of the P&ID data extraction application. In this case, when the steps are performed, there may be existing machine-readable tags in the P&ID for at least some text boxes, symbols, and connections, and the steps can be used to correct such predictions by assigning corrected tags.

[0031] At step 310, the tagging application 200 loads a P&ID in an image-only format (e.g., JPG, PNG, image-only PDF, etc.) that lacks machine-readable information (e.g., metadata) describing text boxes, symbols, links, etc. In some cases, the P&ID originates from a scan of a printed document.

[0032] At step 320, the tag application 200 preprocesses the P&ID to rasterize the P&ID, resize the P&ID, and / or color-separate the P&ID. Rasterization may involve decompression, conversion, and / or extraction operations to produce a rasterized P&ID. Resizing may involve changing the resolution (e.g., dots per inch (DPI)) to a resolution that is easier to display and / or process (e.g., via OCR algorithms). Color separation may involve separating color channels to facilitate easier processing (e.g., text of different colors can overlap, which could interfere with OCR algorithms if not separated).

[0033] At step 330, the tag application 200 displays rasterized (and resized and color-separated) P&IDs within the main window of its user interface. Figure 4 This is a screenshot 400 of the user interface of an example tagging application 200 for rasterized P&ID in the main window 410. The user interface also includes a command menu 420, a category menu 430 listing possible tags for symbols and connections, a text detection results box 440 for displaying predicted machine-readable text for the currently selected text box, and a magnified portion window 450 for displaying a magnified view of a portion of the P&ID.

[0034] At step 340, the tagging application 200 tags the text boxes in the P&ID by performing several sub-steps. These sub-steps can vary depending on whether the tags are part of the dataset used to generate tags that are input to a machine learning algorithm for training the P&ID data extraction application, or as part of correcting errors in the tags predicted by the machine learning algorithm output by the P&ID data extraction application. If the tags are part of the dataset used to generate the tags, at sub-step 342, the tagging application 200 performs an OCR algorithm that predicts the bounding box surrounding each text box in the P&ID and the text within each text box. Alternatively, if the tags are part of correcting errors in the predicted tags, sub-step 342 can be skipped, and the predicted bounding boxes loaded based on data associated with the P&ID (e.g., a Java Description Language Object Notation (JSON) file) are output by the machine learning algorithm of the P&ID data extraction application.

[0035] At substep 344, the marker application 200 displays the predicted bounding box and the predicted machine-readable text for each text box. The predicted bounding boxes (e.g., for those text boxes in the view) may be displayed in the main window 410, while the predicted machine-readable text (e.g., for the currently selected text box) may be displayed in the detection results box 440. Some predictions may be correct, while others may include errors. Figure 5 This is a screenshot 500 of the user interface of an example tag application 200, showing the correctly predicted bounding box and the correctly predicted machine-readable text of the first text box. In this example, the currently selected text box is for reactor trip. The bounding box 510 is highlighted, while the predicted machine-readable text "reactor trip" is shown in the detection result box 440. Figure 6 This is a screenshot 600 of the user interface of an example tag application 200, showing the incorrectly predicted bounding box and incorrectly predicted machine-readable text of the second text box. In this example, the currently selected text box is for the valve. The bounding box 610 is highlighted, while the predicted machine-readable text "Note 1" is shown in the detection result box 440. As can be seen, the bounding box 610 does not completely surround the text box (the part of "5" is cut off), and the predicted machine-readable text is missing a character (the "5" is missing).

[0036] At substep 346, for each item, the marking application 200 receives confirmation that the predicted bounding box or predicted machine-readable text is correct, or receives a correction for the predicted bounding box or predicted machine-readable text. Each text box may be marked with a flag until the corresponding confirmation or correction is received. To facilitate quick checking, users may be allowed to use shortcuts (e.g., arrow keys on the keyboard) to scroll through all text boxes or only the marked text boxes.

[0037] In substep 348, which can occur concurrently with other substeps 342-346, when the area of ​​one bounding box is contained within another bounding box by more than a predefined percentage, the markup application 200 detects a conflict between the text box bounding boxes, automatically removes one bounding box, and merges the associated machine-readable text into the machine-readable text of the other bounding box. Such conflict detection can resolve OCR problems associated with multi-line text (whose OCR algorithms tend to separate rather than group as typically expected) and otherwise accelerate markup.

[0038] Figure 3C This is a sequence of steps 301 of an example collision detection algorithm that can be used in sub-step 348 (and sub-step 359, which is further discussed below). At step 311, the collision detection algorithm receives the current set of bounding boxes for P&ID and an indication of the visible portion of the P&ID in the main window 410 for which tagging 200 is applied. The x-axis and y-axis coordinates of the bounding boxes can be maintained in a classification list. At step 321, the collision detection algorithm performs a bisection (e.g., left bisection) on the x-axis and y-axis coordinates to return potential candidates for collisions. The x-axis and y-axis bisections can be performed in parallel with speed processing. To do this, the collision detection algorithm can keep the updated list of bounding boxes for each coordinate in order when any modifications are made. At step 331, the collision detection algorithm merges the bisection sets for the x-axis and y-axis coordinates and retains only potential candidates for collisions that fit both cases. To avoid unnecessary computation, the merging can focus on the visible portion of the P&ID in the main window 410. At step 341, the collision detection algorithm manages overlapping bounding boxes. Some items may overlap (e.g., text boxes inside symbols). In this case, the collision detection algorithm can select the smallest bounding box. At step 351, diagonal bounding boxes can be handled specifically. In step 311, bisection on the x-axis and y-axis assumes a horizontal or vertical rectangular bounding box. To accommodate temporary bounding boxes arranged diagonally, potential collisions with them can be specifically examined to avoid false collision detections. At step 361, a custom user filter can be applied to further reduce potential collision candidates. Finally, at step 371, the remaining bounding boxes (if any) are returned.

[0039] Figure 7 This is a screenshot 700 of the user interface of an example marker application 200 illustrating the correction of an incorrectly predicted bounding box for a second text box. In this example, the user manipulates a pointer in the main window 410 to drag a corner of the bounding box 610 so that it completely surrounds the text box (including the entire "5"). Alternatively, the user can drag a corner in the zoomed-out portion of the window 450 for a similar correction. Figure 8 This is a screenshot 800 of the user interface of an example tag application 200 illustrating the correction of erroneously predicted machine-readable text in the second text box. In this example, the user manipulates the text cursor in the detection result box 440 to correct the text (adding a missing "5"). In some cases, the OCR algorithm (running as a background process) can detect the corrected bounding box and automatically correct the predicted machine-readable text. If such automatic correction itself does not contain errors, then correction in the detection result box 440 may not be necessary.

[0040] At step 350, the tagging application 200 tags the symbols in the P&ID by performing several sub-steps. These sub-steps may vary depending on whether the tagging is part of a dataset that generates tags that are used as input to a machine learning algorithm for training the P&ID data extraction application, or as part of correcting errors in the tagging predictions output by the machine learning algorithm of the P&ID data extraction application. At sub-step 352, in response to user input in the user interface, the tagging application 200 draws bounding boxes around the corresponding symbols of at least some of the symbols in the P&ID. If sub-step 352 is performed as part of a dataset that generates tags, no exit bounding boxes may exist around the symbols, and the drawing may create new bounding boxes. Alternatively, if sub-step 352 is performed as part of correcting errors in the predicted tagging, existing predicted bounding boxes may exist around the symbols, and the drawing may correct the predicted bounding boxes by changing their size or shape. Figure 9 This is a screenshot 900 of the user interface of an example marker application 200, showing the bounding box of a drawn symbol. In this example, the symbol is a valve. The user manipulates pointer 910 in the main window 410 to drag the corner of the bounding box 920 so that it surrounds the valve. Alternatively, the user can drag the corner in the zoomed-out portion of the window 450.

[0041] At substep 354, in response to user input in the user interface, the tagging application 200 assigns a type of equipment to a given symbol. If substep 354 is performed as part of the dataset used to generate the tags, there may be no exiting types, and the step can still assign types to symbols. Alternatively, if performing substep 354 is part of correcting errors in the predicted tags, there may be existing predicted types, and the step can correct those predicted types. Figure 10 This is a screenshot 1000 of the user interface of an example labeling application 200 for selecting equipment types. In this example, the user selects the type "Valve" 1010 from the category menu 430. The symbol is then labeled as Valve. Various user interface mechanisms can be used to facilitate efficient selection. For example, different types of equipment can be displayed in different colors for easy identification.

[0042] In many P&IDs, a particular symbol appears more than once in the attached figures, and each instance of individually labeling such a symbol may be highly repetitive. This can occur when the label is part of a dataset that generates labels that are used as input to a machine learning algorithm for training a P&ID data extraction application, or as part of correcting errors in predicted labels output by the machine learning algorithm of the P&ID data extraction application (e.g., requiring the same correction to be performed multiple times). Such repetition can be addressed in various ways. At substep 356, in response to user input in the user interface, the labeling application 200 copies the bounding box around a given symbol to another symbol and copies the type of equipment to be assigned to the other symbol. Figure 11 This is a screenshot 1100 of the user interface of an example tag application 200 illustrating the copy operation. In this example, the user selects the bounding box 920 and the category "valve" and copies it (e.g., by pressing button 1110 in the command menu 420) and drags the representation 1120 to set it during each instance of the valve.

[0043] Alternatively, at substep 358, in response to user input in the user interface, the marking application 200 uses an image detection algorithm to automatically detect other identical symbols in the P&ID that correspond to the given symbol, places bounding boxes around the other symbols, and automatically assigns them the same category as the given symbol. Instead of tediously drawing bounding boxes around each other and selecting a category for every other symbol, the user only needs to perform a quick check. The other symbols can each be marked with a flag until the user receives confirmation or correction regarding the correspondence between the automatically placed bounding boxes and the automatically selected categories. To facilitate quick checking, the user can be allowed to use shortcuts (e.g., arrow keys on the keyboard) to scroll through the marked symbols. Figure 12This is a screenshot 1200 of the user interface of an example tagging application 200 that automatically assigns bounding boxes and categories to symbols. In this example, the user selects a bounding box 920 and a category “valve” and triggers automatic assignment (e.g., by pressing button 1210 in command menu 420). The image detection algorithm of tagging application 200 automatically detects symbols 1220 and 1230, automatically draws bounding boxes around them, and assigns a category “valve” to them. Some symbols 1220 are detected correctly and may be confirmed by the user. Symbol 1230 is detected incorrectly and may be corrected by the user.

[0044] In substep 359, which can occur simultaneously with other substeps 352-348, when the area of ​​one bounding box is contained within another bounding box by more than a predefined percentage, a conflict is detected between the bounding boxes marked with a 200 detection symbol, and the bounding box is automatically deleted. (The above is in...) Figure 3C The steps described can be used to perform collision detection for symbols. Some larger symbols may include smaller symbols. Collision detection ensures that larger symbols will be labeled using a single bounding box and category applied to the entire symbol.

[0045] At step 360, the tagging application 200 tags the attributes of the symbols in the P&ID. This step can vary depending on whether the tags are part of the dataset of tags generated as input to a machine learning algorithm used to train the P&ID data extraction application, or as part of correcting errors in predicted tags output by the P&ID data extraction application. If step 360 is performed as part of the dataset used to generate tags, the symbol exit attribute may not exist. When assigning a category to a symbol, the user can select a value for each of several attributes in a list of attributes for that category. Each attribute may have a default value used if the user does not actively make a selection. Alternatively, if step 360 is performed as part of correcting errors in predicted tags, existing predicted attributes may exist, and this step may correct the predicted attributes. User-selected attribute values ​​for attributes in the attribute list may override predicted attribute values. Figure 13 This is a screenshot 1300 of the user interface of an example tagging application 200, showing the first attribute of the tagging symbol. In this example, the user selects the category "Valve" 1010 from the category menu 430. The symbol enclosed by the bounding box 920 is then labeled as a valve. A valve can have multiple attributes, including, among others, type attributes (e.g., gate valve, ball valve, check valve, etc.) and open attributes (e.g., open, locked open, normally closed, locked closed, etc.). Here, the user selects the value "spherical" 1330 for the type attribute 1320 in the attribute list 1310. Figure 14This is a screenshot 1400 of the user interface of an example tag application 200, showing the second attribute of the tag symbol. In this example, the user selects the value "normally closed" 1330 for the open attribute 1410 in the attribute list 1310.

[0046] The attributes of a given symbol can also be copied or applied when the tag application 200 copies the bounding box and type of a given symbol for use by other symbols, or when the tag application 200 automatically detects the bounding box and type of a given symbol and applies them to other symbols. In this way, users are spared from repeatedly entering the attributes of the same symbol in P&ID.

[0047] At step 370, the tagging application 200 tags connections in the P&ID by performing multiple sub-steps. These sub-steps can vary depending on whether the tags are part of a dataset used to generate tags that are input to a machine learning algorithm for training the P&ID data extraction application, or as part of correcting errors in predicted tags output by the machine learning algorithm of the P&ID data extraction application. At sub-step 372, in response to user input in the user interface, the tagging application 200 defines connection points at corresponding symbols. If sub-step 372 is performed as part of a dataset for generating tags, no exiting connection points may exist, and new connection points can be defined. Alternatively, if sub-step 372 is performed as part of correcting errors in predicted tags, existing predicted connection points may exist, and existing connection points can be moved or replaced with different connection points. The tagging application 200 can automatically adjust the position of connection points to be at the boundaries of symbols without user input. At sub-step 374, the tagging application 200 creates connections between connection points. The tagging application 200 can automatically associate connections with the symbol of each connection point. Figure 15 This is a screenshot 1500 of the user interface of an example tagging application 200 that shows the definition of connection points and the creation of connections. In this example, the user has indicated two connection points 1510 and 1520. In response, the tagging application 200 has created a connection 1530 between them.

[0048] At step 380, the tagging application 200 tags the attributes of the connections in P&ID. This step can vary depending on whether the tags are part of the dataset that generates the tags and is used as input to a machine learning algorithm for training the P&ID data extraction application, or as part of correcting errors in the predicted tags output by the machine learning algorithm of the P&ID data extraction application. If step 380 is performed as part of the dataset that generates the tags, the connection exit attribute may not exist. Alternatively, if step 380 is performed as part of correcting errors in the predicted tags, existing predicted attributes may exist, and this step can correct the predicted attributes. The user can select the attribute value from the list of connection attributes. Figure 16 This is a screenshot 1600 of the user interface of an example tag application 200, showing the attributes of the tag connection. In this example, the user has selected the value "Electrical" 1610 from the list of connection attributes 1620.

[0049] At step 390, the tagging application 200 associates at least some of the text boxes with corresponding symbols or connections in P&ID, thereby establishing text associations. This step may vary depending on whether the tags are part of a dataset that generates tags that are used as input to a machine learning algorithm for training a P&ID data extraction application, or as part of correcting errors in predicted tags output by the machine learning algorithm of the P&ID data extraction application. If step 390 is performed as part of a dataset that generates tags, there may be no need to exit the association. Alternatively, if step 390 is performed as part of correcting errors in predicted tags, existing predicted associations may exist, and this step may correct the predicted associations. The user can select a symbol or connection and then select one or more text boxes to associate with it, thus linking them together. When a symbol, connection, or text box is selected, the associated item can be highlighted to allow for quick checking of the association. A text box can be associated with multiple symbols or connections, and a symbol or connection can be associated with multiple text boxes. To speed up the association process, the tagging application 200 may automatically associate text boxes with symbols, where their corresponding bounding boxes overlap. Figure 17 This is a screenshot 1700 showing the user interface of an example tag application 200 that forms text associations. In this example, the user has selected the symbol 1710 for the valve and then indicates that it should be associated with two text boxes 1720 and 1730 by pressing a key on the keyboard and using the pointer to select the corresponding text box.

[0050] At step 395, the tagging application 200 stores machine-readable tagging for text boxes, symbols, and connections, and the associations therebetween, in a machine-readable format. The machine-readable format can be separate from image-only formats, such as a separate JSON file associated with a JPG file containing P&ID images, a PNG file, or an image-only PDF file. Alternatively, the machine-readable format can be integrated into the file storing the P&ID images. Depending on the use case, the machine-readable format can be provided as input to machine learning algorithms used to train P&ID data extraction applications, or it can be provided to design and modeling applications (e.g., for building models / digital twins).

[0051] It should be understood that although steps 300 are shown sequentially, they can be performed in various different orders. Some steps 300 may be performed only partially at a given time, and such partial execution may be interleaved with the execution of other steps or parts thereof. For example, some text boxes in P&ID may be marked, then some symbols may be marked, then some more text boxes may be marked, then some more symbols may be marked, and so on. Furthermore, some steps 300 may be omitted. For example, there may not be any marking for the text boxes that need correction, and correction sub-step 324 may be omitted.

[0052] Generally, it should be understood that the above description can be readily adapted and modified to suit various implementation methods and environments. While many aspects of the technologies discussed above can be implemented by specific software processes executing on specific hardware, it should be understood that some or all of the technologies described can also be implemented by different software on different hardware. In addition to general-purpose computing devices, the hardware may include specially configured logic circuits and / or other types of hardware components. Most importantly, it should be understood that the above description is intended to be taken by way of example only.

Claims

1. A method for assigning machine-readable tags to piping and instrumentation diagrams (P&IDs) in image-only format, comprising: One or more text boxes in the P&ID are marked at least in part by: The predicted bounding box and predicted machine-readable text of each text box are displayed in the user interface of the tagging application running on the computing device. For at least one of the one or more text boxes, the user interface receives a correction for the predicted bounding box or the predicted machine-readable text, and updates the predicted bounding box or the predicted machine-readable text. One or more symbols in the P&ID are marked at least in part by the following: Draw a bounding box around a given symbol in response to user input in the user interface. The type of equipment is assigned to the given symbol in response to user input in the user interface. Based on the drawn bounding box from the user input and the type of equipment, automatically detect one or more other symbols in the P&ID corresponding to the given symbol, place the bounding box around each of the one or more other symbols, and assign the type of equipment to each of the one or more other symbols. For at least one of the other symbols, the user interface receives a correction for the automatically placed bounding box or the type of the assigned equipment, and updates the automatically placed bounding box or the type of the assigned equipment. One or more connections in the P&ID are marked at least in part by the following: Define the connection points at the corresponding symbols, and Create connections between the connection points; Associate one or more of the text boxes with the corresponding symbols or connections in the P&ID; as well as The tags used for the one or more text boxes, symbols, and connections, and the associations therebetween, are stored in the memory of the computing device in a machine-readable format.

2. The method of claim 1, wherein labeling one or more text boxes, labeling one or more symbols, and labeling one or more connections are performed as part of a dataset of labels for generating a machine learning algorithm for training a P&ID data extraction application, and the method further comprises: The machine-readable format is provided as training input to the machine learning algorithm of the P&ID data extraction application.

3. The method of claim 1, wherein labeling one or more text boxes, labeling one or more symbols, and labeling one or more connections are performed as part of the error correction of the predicted labeling output of the machine learning algorithm for the P&ID data extraction application, and the machine-readable format is the correction output of the P&ID data extraction application.

4. The method of claim 1, wherein the text boxes marking one or more of the P&ID further include: The predicted bounding box and predicted machine-readable text for each of the text boxes are determined using an optical character recognition (OCR) algorithm.

5. The method of claim 1, wherein the text boxes marking one or more of the P&ID further include: The user interface receives confirmation that one or more of the predicted bounding box or predicted machine-readable text are correct. Each text box is marked with a flag until a corresponding correction to the predicted bounding box or predicted machine-readable text is received, or confirmation that the predicted bounding box or predicted machine-readable text is a correct correspondence is received.

6. The method of claim 1, wherein the marking of one or more symbols in the P&ID further comprises: In response to user input in the user interface to manipulate the pointer to drag the corner of the bounding box, the bounding box is drawn around the symbol.

7. The method of claim 1, wherein the marking of one or more symbols in the P&ID further comprises: The user interface receives confirmation that the automatically detected bounding box or automatically assigned equipment type is correct for at least one of the one or more other symbols. Each of the one or more other symbols is marked with a flag until a corresponding correction for the automatically detected bounding box or automatically assigned equipment type is received, or confirmation that the predicted bounding box or predicted machine-readable text is a correct correspondence is received.

8. The method of claim 1, wherein the marking of one or more symbols in the P&ID further comprises: In response to user input in the user interface, the bounding box around the given symbol is copied to the other symbol, and the type of equipment to be assigned is copied to the other symbol.

9. The method according to claim 1, further comprising: The attributes of one or more symbols in the P&ID are marked at least in part by the following: In response to user input in the user interface, select the value of an attribute from the attribute list for each of the one or more symbols.

10. The method according to claim 1, further comprising: The attributes of the connections in the P&ID are marked at least in part by the following methods: In response to user input in the user interface, select the value of each attribute on one or more attribute lists in the connection, where each attribute is initially set to a default value.

11. The method according to claim 1, further comprising: A conflict is detected when a first bounding box exceeding a predefined percentage is contained within a second bounding box used for text boxes or symbols. as well as The first bounding box is automatically deleted in response to the detection of the conflict.

12. The method of claim 1, wherein the association further comprises: The bounding box of the text box was detected to be surrounded by the bounding box of a specific symbol; as well as In response to the detection of the enclosing, the text box is automatically associated with the specific symbol.

13. The method of claim 1, wherein the user interface comprises a main window displaying a view of the P&ID and a magnified portion window displaying an enlarged view of the P&ID, wherein user input is available in either the main window or the magnified portion window.

14. A computing device, comprising: Display screen; processor; as well as A memory coupled to the processor and configured to store pipeline and instrumentation diagrams (P&ID) in image-only format and a tagging application, the tagging application being operable when executed to: Load the P&ID; Mark one or more text boxes in the P&ID. One or more symbols in the P&ID are marked in response to user input that draws a bounding box around a given symbol in the view of the P&ID and user input that assigns a type of equipment to the given symbol. The automatic detection of one or more other symbols in the P&ID corresponding to the given symbol, wherein the automatic detection of one or more other symbols in the P&ID corresponding to the given symbol places a bounding box around each of the one or more other symbols, and assigns the type of equipment to each of the one or more other symbols. Mark one or more connections in the P&ID. Associate one or more of the text boxes with the corresponding symbols or connections in the P&ID. The tags used for the one or more text boxes, symbols, and connections, and the associations therebetween, are stored in the memory in a machine-readable format.

15. The computing device of claim 14, wherein the tagging application is further operable to tag the one or more text boxes in the P&ID by: The predicted bounding box and predicted machine-readable text for each element in the text box are determined using an optical character recognition algorithm. The predicted bounding box and predicted machine-readable text for each of the text boxes are displayed on the screen. For at least one of the one or more text boxes, receive a correction for the predicted bounding box or the predicted machine-readable text, and update the predicted bounding box or the predicted machine-readable text.

16. The computing device of claim 14, wherein the tagging application is further operable to tag one or more connections in the P&ID by: Define the connection points at the corresponding symbols, and Create connections between the connection points.

17. A non-transitory computing device readable medium having instructions stored thereon, the instructions being operable, when executed by one or more computing devices, to: Mark one or more text boxes in the piping and instrumentation diagram (P&ID); One or more symbols in the P&ID are marked at least in part by the following: In response to user input, draw a bounding box around a given symbol in the view of the P&ID. The type of equipment is assigned to the given symbol in response to user input. Based on the drawn bounding box from the user input and the type of equipment, automatically detect one or more other symbols in the P&ID corresponding to the given symbol, place the bounding box around each of the one or more other symbols, and assign the type of equipment to each of the one or more other symbols. For at least one of the other symbols, receive from the user a correction for the automatically placed bounding box or the type of the assigned equipment, and update the automatically placed bounding box or the type of the assigned equipment. Mark one or more connections in the P&ID; Associate one or more of the text boxes with the corresponding symbols or connections in the P&ID; as well as The tags used for the one or more text boxes, symbols, and connections, and the associations therebetween, are stored in a machine-readable format.

18. The non-transitory computing device readable medium of claim 17, wherein the instructions, when executed, are further operable to: The machine-readable format is provided as training input to P&ID data extraction applications that use machine learning algorithms.

19. The non-transitory computing device readable medium of claim 17, wherein the instructions, when executed, are further operable to: The machine-readable format is provided as the corrected output for P&ID data extraction applications using machine learning algorithms.

Images