Data-Efficient Object Detection of Engineering Blueprint Symbols

Abstract

The method includes obtaining (902) an engineering blueprint (102) including a plurality of symbols (300, 302) and connections including the symbols, where different ones of the symbols on the engineering blueprint represent different types of equipment. The method also includes identifying (904) visual features on the engineering blueprint. The method further includes processing (906) the visual features using at least one trained machine learning model (106, 108, 110) to (i) identify boundaries (204) around the symbols on the engineering blueprint and (ii) classify the symbols on the engineering blueprint into a plurality of classifications (206), where the symbols in the different classifications are associated with different types of symbols.

Description

[Technical field]

[0001] The present disclosure is directed generally to machine learning systems and, more specifically, to data-efficient object detection of engineering blueprint symbols. [Background technology]

[0002] Engineering drawings typically include many symbols, which may represent sensors or other equipment in a manufacturing facility or other facility or location. Engineering drawings also typically include various connections between the symbols, which may represent material flows or other relationships between equipment at a location. In some cases, connections between symbols on engineering drawings may be required to identify and understand complex relationships between equipment. However, this process is often accomplished using manual efforts that are time consuming, costly, and potentially inaccurate. Automated methods for performing this process would typically require a large number of annotated engineering drawings, which may not be available, thus hindering the use of these automated methods. Summary of the Invention

[0003] The present disclosure relates to data-efficient object detection of engineering blueprint symbols.

[0004] In a first embodiment, a method includes obtaining an engineering blueprint including a plurality of symbols and connections associated with the symbols, where different ones of the symbols in the engineering blueprint represent different types of equipment. The method also includes identifying visual features in the engineering blueprint. The method further includes processing the visual features using at least one trained machine learning model to (i) identify boundaries around the symbols in the engineering blueprint and (ii) classify the symbols in the engineering blueprint into a plurality of classifications, where the symbols in the different classifications are associated with different types of symbols.

[0005] In a second embodiment, the equipment includes at least one processing device configured to obtain an engineering blueprint including a plurality of symbols and connections associated with the symbols, where different ones of the symbols in the engineering blueprint represent different types of equipment. The at least one processing device is further configured to identify visual features in the engineering blueprint. The at least one processing device is further configured to process the visual features using the at least one trained machine learning model to (i) identify boundaries around the symbols in the engineering blueprint, and (ii) classify the symbols in the engineering blueprint into a plurality of classifications, where the symbols in the different classifications are associated with different types of equipment.

[0006] In a third embodiment, a non-transitory computer readable medium has computer readable program code stored thereon that, when executed, causes one or more processors to obtain an engineering blueprint including a plurality of symbols and connections associated with the symbols, where different ones of the symbols in the engineering blueprint represent different types of equipment. The medium further has computer readable program code stored thereon that, when executed, causes the one or more processors to identify visual features in the engineering blueprint. The medium has computer readable program code stored thereon that, when executed, causes the one or more processors to process the visual features using at least one trained machine learning model to (i) identify boundaries around the symbols in the engineering blueprint, and (ii) classify the symbols in the engineering blueprint into a plurality of classifications, where the symbols in the different classifications are associated with different types of symbols.

[0007] Other technical features will be readily apparent to those skilled in the art from the following figures, descriptions and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0009] [Figure 1] 1 illustrates an example architecture for data-efficient object detection of engineering blueprint symbols in accordance with the present disclosure.

[0010] [Diagram 2] 2 illustrates an example of data processing within the architecture of FIG. 1 according to the present disclosure.

[0011] [Diagram 3] 2 illustrates example shapes and aspect ratios of engineering blueprint symbols that can be detected using the architecture of FIG. 1 in accordance with the present disclosure.

[0012] [Figure 4] 2 illustrates examples of machine learning pathways for different types of engineering blueprint symbols within the architecture of FIG. 1 in accordance with the present disclosure.

[0013] [Diagram 5] 1 illustrates a first exemplary technique for training data augmentation according to this disclosure.

[0014] [Figure 6] 14 illustrates a second exemplary technique for training data augmentation according to this disclosure.

[0015] [Figure 7] 13 illustrates a third exemplary technique for training data augmentation according to this disclosure.

[0016] [Figure 8] 1 illustrates an example apparatus that supports training of data-efficient object detection and / or data augmentation of engineering blueprint symbols in accordance with the present disclosure.

[0017] [Figure 9] 1 illustrates an example method for data-efficient object detection of engineering blueprint symbols in accordance with the present disclosure.

[0018] [Figure 10]1 illustrates a first exemplary method for training data augmentation according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] 1 through 10 described below and the various embodiments used to illustrate the principles of the present disclosure are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any type of suitably arranged device or system.

[0020] As discussed above, engineering drawings typically include many symbols, which may represent sensors or other equipment in a manufacturing facility or other facility or location. Engineering drawings also typically include various connections between symbols, which may represent material flows or other relationships between equipment at a location. In some cases, connections between symbols on engineering drawings may be required to identify and understand complex relationships between equipment. However, this process is often accomplished using manual efforts that are time-consuming, costly, and potentially inaccurate. Automated methods for performing this process typically require a large number of annotated engineering drawings, which may not be available, thus hindering the use of these automated methods.

[0021] Electronic analysis of engineering drawings generally involves generating a digital representation of an engineering system based on non-digital engineering blueprints (also known as engineering drawings). Drawing analysis generally involves using one or more computer vision models to detect target symbols in the engineering drawing, determine connections between the detected symbols, and output a digital representation of the engineering drawing in a structured format (e.g., using structured data).

[0022] The present disclosure provides various techniques for data-efficient object detection of symbols in engineering drawings, and various techniques for training data augmentation associated with engineering drawings. The techniques for data-efficient object detection are based on the use of one or more machine learning models that can be trained to recognize symbols in engineering drawings and, optionally, to recognize connections between the recognized symbols. The one or more machine learning models can be trained using various information, such as training diagrams developed from one or more domain-specific engineering symbol libraries, or other training data collected or generated in any suitable manner. Thus, the techniques for data-efficient object detection can be used to perform efficient and accurate symbol detection using engineering drawings, resulting in a more accurate digital representation of the engineering drawings. Furthermore, the techniques for data-efficient object detection can provide improved model efficiency, i.e., these techniques can reduce the time it takes to perform symbol identification.

[0023] Techniques for training data augmentation support the use of one or more mechanisms that can be used to augment training data associated with engineering drawings. For example, techniques for training data augmentation can perform resizing of one or more labeled engineering drawings using random scaling and random cropping and modification of the resized labeled engineering drawings to create at least one composite drawing. Techniques for training data augmentation can further or alternatively include cropping one or more target symbols and randomly placing them on a background of one or more engineering drawings to create at least one composite drawing. Techniques for training data augmentation can further or alternatively include adding text to one or more symbols from at least one symbol library and randomly placing them on a background of one or more engineering drawings to create at least one composite drawing. One, some, or all of these approaches can be used to generate additional labeled training data that can be used to train one or more machine learning models. Thus, these techniques can be used to simplify the collection or generation of training data used to train machine learning models, reducing the time and costs associated with training machine learning models.

[0024] It is noted here that both (i) the techniques for data-efficient object detection and (ii) the techniques for training data augmentation are described as being performed using the same device or system. As a result, one or more techniques for training data augmentation can be used to generate training data, which can be used to train one or more machine learning models, such that the trained machine learning models can be used to perform data-efficient object detection. However, it is also possible to use these techniques separately. Thus, the techniques for data-efficient object detection can be used without training data augmentation, and the techniques for training data augmentation can be used without data-efficient object detection. Also, even when both techniques are used together, they do not have to be performed using the same device or system, and can be implemented together or separately, as desired.

[0025] FIG. 1 illustrates an example of an architecture 100 for data-efficient object detection of engineering blueprint symbols in accordance with the present disclosure. As shown in FIG. 1, the architecture 100 generally operates to receive and process at least one input engineering blueprint 102. Each input engineering blueprint 102 generally includes a number of symbols representing sensors or other equipment and a number of connections to, from, and between the symbols. Each input engineering blueprint 102 may also include notations or other markings, such as markings around one or more particular portions of the input engineering blueprint 102, each of which includes an associated piece of equipment. Each input engineering blueprint 102 may be obtained from any suitable source, such as a camera, scanner, database, user device, or other source of input engineering blueprints.

[0026] Here, the input engineering blueprint 102 is provided to a feature extractor 104, which generally operates to extract visual features of the input engineering blueprint 102. The feature extractor 104 may use any suitable technique to identify visual features of the input engineering blueprint 102. For example, the feature extractor 104 may represent a trained machine learning model, such as a convolution neural network (CNN) or other type of machine learning model, trained to process the input engineering blueprint 102 using various convolutional, pooling, or other layers to generate a feature vector or other extracted visual features based on the input engineering blueprint 102. As a particular example, the feature extractor 104 may represent an implementation of a VGG or EfficientNet feature extractor.

[0027] The extracted visual features are provided to a feature pyramid network 106, which processes the extracted visual features to generate feature maps related to the features of the input engineering blueprint 102. Different ones of the feature maps generated by the feature pyramid network 106 may include features of the input engineering blueprint 102 at different scales. This enables the architecture 100 to later identify symbols of the input engineering blueprint 102 having various sizes, such as by identifying symbols having significantly different sizes. The feature pyramid network 106 may use any suitable technique to analyze the visual features and generate feature maps at different scales. For example, the feature pyramid network 106 may represent a trained machine learning model, such as a CNN or other type of machine learning model, that has been trained to process feature vectors or other extracted visual features and generate feature maps at different scales.

[0028] The feature maps of different scales are provided to the region proposal network 108 and the region of interest network 110. The region proposal network 108 generally operates to use the feature maps to identify potential symbols in the input engineering blueprint 102 and to identify bounding boxes or other possible boundaries around the potential symbols in the input engineering blueprint 102. For example, the region proposal network 108 may use different dedicated symbol prediction heads or other logic to generate class-independent bounding boxes around potential symbols in the input engineering blueprint 102. The bounding boxes are referred to herein as "class-independent" because they may be generated without reference to the particular types of symbols contained in the input engineering blueprint 102. The region proposal network 108 may use any suitable technique to identify potential symbols and boundaries around the potential symbols in the input engineering blueprint 102. For example, the region proposal network 108 may represent a trained machine learning model, such as a CNN or other type of machine learning model, that has been trained to process the feature maps and identify potential symbols and boundaries around the potential symbols.

[0029] The region of interest network 110 generally operates to process feature maps of different scales and boundaries of potential symbols in the input engineering blueprint 102 to identify actual symbols in the input engineering blueprint 102 and classify those symbols. For example, the region of interest network 110 may use machine learning to classify each potential symbol into different classes, with the different classes being associated with different types of engineering symbols (such as different classes of symbols representing sensors, vessels, valves, actuators, or other components). The region of interest network 110 may use any suitable technique to identify and classify symbols in the input engineering blueprint 102. For example, the region of interest network 110 may represent a trained machine learning model, such as a CNN or other type of machine learning model with region of interest pooling, trained to process feature maps and symbol boundaries to identify and classify different symbols.

[0030] In this manner, architecture 100 generally supports a multi-step approach for symbol detection that includes (i) identifying class-independent boundaries of potential symbols, and (ii) identifying and classifying actual symbols based on the identified boundaries. Through proper training of the machine learning models used in architecture 100, architecture 100 can be used to effectively and accurately identify engineering symbols in blueprints. This can be done much more quickly and accurately compared to manual approaches, and this can be done without the need for a large number of annotated engineering drawings. The identified engineering symbols can be used in any suitable manner, such as to create a digital representation of the engineering blueprint (which itself can be used in any suitable manner).

[0031] It should be noted that the functionality illustrated in or described in connection with FIG. 1 may be implemented in any suitable manner in an electronic device, such as a computing device. For example, in some embodiments, at least some of the functionality illustrated in or described in connection with FIG. 1 may be implemented or supported using one or more software applications or other software instructions executed by one or more processing units of a desktop computer, laptop computer, tablet computer, server computer, or other device. In other embodiments, at least some of the functionality illustrated in or described in connection with FIG. 1 may be implemented or supported using dedicated hardware components. In general, the functionality illustrated in or described in connection with FIG. 1 may be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0032] Although FIG. 1 illustrates one example of an architecture 100 for data-efficient object detection of engineering blueprint symbols, various modifications may be made to FIG. 1. For example, although various functions illustrated in or described with respect to FIG. 1 are described as being implemented or performed using machine learning models, any number of machine learning models may be used to implement architecture 100. In some cases, different machine learning models may be trained to perform different functions of architecture 100. In some cases, a single machine learning model may be trained to perform multiple functions of architecture 100. Additionally, components may be added, omitted, combined, further subdivided, duplicated, or arranged in any other suitable configuration within architecture 100 according to particular needs.

[0033] FIG. 2 illustrates an example of data processing within the architecture 100 of FIG. 1 according to the present disclosure. As shown in FIG. 2, the input engineering design 102 is processed using the feature extractor 104 and the feature pyramid network 106, which generates various extracted feature maps 202, an example of which is shown here. The feature maps 202 may include low-level and high-level visual features of the input engineering design 102 or may otherwise identify visual features of the input engineering design 102 at different scales. The region proposal network 108 processes the feature maps 202 to identify one or more potential symbols in the input engineering design 102 and a boundary 204 (such as a box or other polygon) around each potential symbol in the input engineering design 102. In some cases, the region proposal network 108 may perform bounding box regression to identify the boundaries 204 of the potential symbols in the input engineering design 102.

[0034] The region of interest network 110 processes the feature map 202 and potential symbol boundaries 204 to identify actual symbols in the input engineering design 102 and generate classifications 206 of the actual symbols in the input engineering design 102. For example, the region of interest network 110 may generate a classification 206 for each identified boundary 204, where the classification 206 identifies the type of symbol contained within the identified boundary 204 (or does not identify the type of symbol if the region of interest network 110 determines that the identified boundary 204 does not contain an actual symbol). Thus, each classification 206 may identify any type of symbol detected in the input engineering design 102. In some cases, the region of interest network 110 may identify a classification target for a symbol in the input engineering design 102, where the classification target represents a particular type of symbol expected in a given domain or other application.

[0035] Engineering symbols are 2D objects that have different characteristics from 3D objects (people, animals, vehicles, etc.) in terms of visual features. Unlike object detection in 3D space, where machine learning models need to learn effective feature extraction across various viewing angles and different distances, 2D object detection models only need to learn effective feature extraction for limited scaling and limited shifts of 2D objects. Furthermore, engineering drawings often need to follow domain-specific standard symbol templates, which means that the variability of engineering symbols across different engineering blueprints tends to be low compared to common 3D objects. To take advantage of the 2D-specific characteristics of engineering symbols and their low variability, scale-specific prediction heads or other types of parallel prediction heads can be used in the region proposal network 108 to improve model prediction quality, as described in more detail below. Also, training data augmentation based on operations such as random "copy and paste" data augmentation can be used to improve data efficiency for training the region proposal network 108, as described in more detail below.

[0036] In some embodiments, the region proposal network 108 and the region of interest network 110 can be trained to jointly optimize their model gradients and weights using a regression loss (associated with generating the boundary 204) and a classification loss (associated with generating the classification 206). As a specific example, the region proposal network 108 and the region of interest network 110 can be trained to minimize the following loss function that combines the regression loss (right hand side of the sum) and the classification loss (left hand side of the sum):

number

[0037] Any suitable technique may be used to implement the functionality of one or more components of the architecture 100 to train the one or more machine learning models. For example, training data including engineering blueprints associated with known boundaries 204 and known classifications 206 may be used to train the region proposal network 108 and the region of interest network 110. In some embodiments, techniques such as backpropagation may be used to adjust model gradients and weights of the one or more machine learning models until the one or more machine learning models identify the known boundaries 204 and known classifications 206 of the engineering blueprints in the training data with a desired degree of accuracy. In certain embodiments, training may continue until a loss value (which may be calculated using the equation above or any other suitable method) falls below a certain threshold or until some other one or more criteria are met (such as after a certain training time or a certain number of training iterations).

[0038] Although Figure 2 illustrates one example of data processing within the architecture 100 of Figure 1, various modifications may be made to Figure 2. For example, the particular feature maps 202, boundaries 204, and classifications 206 may vary based on the particular input engineering blueprints 102 processed by the architecture 100.

[0039] For effective feature extraction of three-dimensional objects to be performed, a machine learning model often needs to be trained to identify features of a three-dimensional object regardless of the distance of the three-dimensional object from the camera. That is, the machine learning model should be able to effectively identify features of a three-dimensional object regardless of whether the three-dimensional object is close to or far from the camera. However, during detection of an engineering symbol, each engineering symbol may typically be represented in a limited range of shapes and aspect ratios. FIG. 3 shows an example of shapes and aspect ratios of engineering blueprint symbols 300 and 302 that can be detected using the architecture 100 of FIG. 1 according to the present disclosure. As shown in FIG. 3, the symbol 300 is formed using a circle, and text may optionally be present within the circle. Also, the symbol 302 is formed using a circle within a square, and text may optionally be present within the square and the circle. These two types of symbols 300 and 302 may typically have an aspect ratio of 1:1. It should be noted that other symbols typically have other shapes (such as lines, ellipses, triangles, diamonds, or other regular or irregular shapes) or combinations of shapes and / or other aspect ratios (e.g., higher or wider aspect ratios).

[0040] Based on the knowledge that each engineering symbol may typically have a known or predefined shape and aspect ratio, separate pathways may be created and trained within the architecture 100, where the different pathways are used to predict the boundaries 204 and classifications 206 of different engineering symbols having different shapes and / or aspect ratios. FIG. 4 illustrates an example of machine learning pathways for different types of engineering blueprint symbols within the architecture 100 of FIG. 1 in accordance with the present disclosure. As shown in FIG. 4, the feature extractor 104 may extract visual features from the input engineering blueprint 102, and k different pathways may be used to process the extracted visual features to identify boundaries and classifications of different types of symbols (e.g., symbols having different shapes and / or aspect ratios) that may be included in the input engineering blueprint 102. For example, different portions 106a-106k of the feature pyramid network 106, different portions 108a-108k of the region proposal network 108, and different portions 110a-110k of the region of interest network 110 may be trained or used in different ways to process different types of symbols. In some cases, the different portions 110a-110k of the region of interest network 110 may be trained to recognize different symbols, and each portion 110a-110k of the region of interest network 110 may be configured to output a true or false indicator that identifies whether a symbol identified in the input engineering design 102 is the particular symbol associated with that portion 110a-110k of the region of interest network 110.

[0041] This approach allows various paths to learn useful features specific to different types of symbols, such as symbols having different shapes and / or aspect ratios. As a result, different paths can be trained to effectively identify different types of symbols. Furthermore, since this approach separates the gradient and weight updates along each path, different paths can be trained separately or independently from each other. Thus, while training the architecture 100, different symbols in the training data can be used to update gradients and weights only along paths associated with these particular types of symbols. There is no need to update paths associated with other types of symbols, simplifying and speeding up the training process. Furthermore, if certain types of symbols have low quality annotations in the training data, the adverse effects associated with a lack of high quality training data for these particular types of symbols are limited to those particular paths, and other paths associated with symbols having higher quality annotations in the training data are not affected.

[0042] FIGURE 3 illustrates an example of shapes and aspect ratios of engineering blueprint symbols 300 and 302 that may be detected using architecture 100 of FIGURE 1 in accordance with the present disclosure, although various modifications may be made to FIGURE 3. For example, other engineering blueprint symbols may have any other suitable shapes and / or any other suitable aspect ratios. FIGURE 4 illustrates an example of machine learning paths for different types of engineering blueprint symbols within architecture 100 of FIGURE 1 in accordance with the present disclosure, although various modifications may be made to FIGURE 4. For example, architecture 100 may support a desired number of parallel paths to identify a desired number of engineering blueprint symbol types.

[0043] Training a complex deep learning model or other machine learning model typically requires a large amount of labeled training data. To support training of one or more machine learning models related to engineering symbols (e.g., one or more machine learning models used in architecture 100 or other machine learning models used for other functions related to engineering symbols or blueprints), various effective data augmentation approaches can be implemented by taking advantage of the low variability of engineering symbols between drawings (at least within a particular domain, such as a particular industry). Using these data augmentation approaches, the number of annotated training drawings required to train one or more machine learning models can be significantly reduced. The following describes three examples of data augmentation approaches that can be used to generate training data.

[0044] FIG. 5 illustrates a first exemplary technique for training data augmentation according to the present disclosure. This exemplary approach includes resizing one or more labeled engineering blueprints using at least one random scale, and randomly cropping and modifying the resized labeled engineering blueprints to create at least one composite blueprint. As shown in FIG. 5, for example, a labeled engineering blueprint 500 represents annotated training data. For example, the labeled engineering blueprint 500 may include symbols with known boundaries 204 and known classifications 206. A first exemplary approach for training data augmentation may resize the labeled engineering blueprint 500 using one or more random scales (such as a larger or smaller scale), and the resized engineering blueprint may be cropped to include only a particular region 502. Additionally, one or more modifications may be applied to the cropped region 502, such as by applying color shifts, line thickening / thinning, enlargement, and / or the addition of noise pixels to the cropped region 502. This results in a new labeled engineering design 504, which represents a modified and rescaled portion of the labeled engineering design 500. Because the new design 504 is generated using the contents of the labeled engineering design 500, the new design 504 represents complete labeled training data that can be used to train a machine learning model. For example, the new design 504 can include known symbols with known boundaries 204 and known classifications 206.

[0045] It should be noted that any number of resizing scales can be applied to each labeled engineering blueprint 500, and any number of overlapping and / or non-overlapping regions 502 can be defined for each rescaled version of each engineering blueprint 500. Also, it should be noted that each region 502 for each rescaled version of each engineering blueprint 500 can be used to generate any number of new blueprints 504, such as when different correction types or combinations of correction types are applied to each region 502, or when different levels of correction are applied to each region 502. As a result, a very large amount of annotated training data can be generated based on a limited number of labeled engineering blueprints 500. Furthermore, it should be noted that when one or more machine learning models are used to analyze engineering blueprints in a particular domain (such as a particular industry), the labeled engineering blueprints 500 used for data augmentation can be limited to engineering blueprints in that particular domain.

[0046] FIG. 6 illustrates a second example technique for training data augmentation according to the present disclosure. This example approach involves cropping one or more symbols and randomly placing the symbols in the background of one or more engineering drawings to create at least one composite drawing. As shown in FIG. 6, for example, a labeled engineering drawing 600 represents the annotated training data. For example, the labeled engineering drawing 600 may include symbols with known boundaries 204 and known classifications 206. A second example technique for training data augmentation may crop one or more symbols from the labeled engineering drawing 600, optionally modify at least one of the cropped symbols, and insert at least one of the cropped (possibly modified) symbols into the background of the same engineering drawing 600 and / or at least one additional engineering drawing. Examples of modifications may include the same types of modifications discussed above with respect to FIG. 5, such as color shifting, line thickness / thinning, enlargement, and / or adding noise pixels. This may generate a new labeled engineering drawing 602. Because the new schematic 602 is generated using the content of one or more labeled schematics, the new schematic 602 represents fully labeled training data that can be used to train a machine learning model, i.e., the new schematic 602 can include known symbols with known boundaries 204 and known classifications 206.

[0047] Also, any individual selected potentially altered symbol, or any combination of selected potentially altered symbols, may be identified and inserted into any number of engineering blueprints. Furthermore, the insertion location of the cropped potentially altered symbols may vary significantly. As a result, a very large amount of annotated training data may be generated based on a limited number of labeled engineering blueprints 600. Furthermore, it should be noted that when one or more machine learning models are used to analyze engineering blueprints in a particular domain (such as a particular industry), the labeled engineering blueprints 600 used for data augmentation may be limited to engineering blueprints in that particular domain.

[0048] This second approach to data augmentation is sometimes said to represent a "copy and paste" data augmentation process, since one or more symbols are effectively copied from one or more engineering blueprints and pasted onto the same or one or more additional engineering blueprints. However, it is noted here that engineering blueprints do not generally permit duplication of engineering symbols, due to standard practices or otherwise. As a result, the pasting of the copied, potentially modified symbol onto an engineering blueprint is said to occur in the "background" of that blueprint. This means that the symbol is pasted onto the engineering blueprint in a position where the pasted symbol does not overlap with another symbol.

[0049] FIG. 7 illustrates a third exemplary technique for training data augmentation according to the present disclosure. This exemplary technique may additionally or alternatively include adding text to one or more symbols from at least one symbol library and randomly placing the selected symbols in the background of one or more engineering drawings to create at least one composite drawing. For example, as shown in FIG. 7, one or more libraries 700 of known symbols may be used, such as in an engineering drawing application that may be used by one or more users to create or edit engineering drawings. The one or more libraries 700 may be randomly sampled to select one or more known symbols from the one or more libraries 700. The selected one or more symbols may be modified as needed. Examples of modifications may include the same types of modifications described above in FIG. 5, such as color shifting, line thickness / thinness, enlargement, and / or adding noise pixels. Additional variations may include randomly creating text for one or more selected symbols and / or randomly changing the size of one or more selected symbols (the aspect ratio of each symbol is generally maintained). It should be noted that various limits can be placed on the random resizing of selected symbols, such as by limiting the amount of resizing so that the relative size ratios of the different symbols are maintained. An example of this might include ensuring that the size of a sensor's symbol is no larger than the associated device's symbol.

[0050] One or more modified versions of a symbol selected from the library 700 can be inserted into at least one engineering blueprint 702. Note that each modified version of a symbol from the library 700 represents annotated training data since the boundary 204 and classification 206 of that symbol are known. In some cases, the engineering blueprint 702 can represent a labeled engineering blueprint including known symbols, and one or more modified versions of one or more symbols selected from one or more libraries 700 can be inserted into the background of the engineering blueprint 702 (e.g., in a non-overlapping manner). In other cases, the engineering blueprint 702 can represent an empty blueprint into which multiple modified versions of multiple symbols selected from one or more libraries 700 can be inserted. Overall, this approach can be used to emulate an actual engineering drawing process in which an engineer or other user drags a symbol from one or more symbol libraries, adds an identifier string to the symbol, and drops the symbol onto the drawing.

[0051] It should be noted that different versions of the same symbol from library 700 can be generated, such as by applying different modification types, combinations of modification types, levels of modification, text, and / or resizing to the symbol. Also, any individual selected potentially modified symbol, or any combination of selected potentially modified symbols, can be identified and inserted into any number of engineering blueprints. Furthermore, the insertion location of the selected potentially modified symbols can be widely varied, and the particular library or libraries 700 used can be varied. As a result, very large amounts of annotated training data can be generated based on the library or libraries 700 of symbols. Furthermore, it should be noted that when using one or more machine learning models to analyze engineering blueprints in a particular domain (such as a particular industry), the library or libraries 700 used for data augmentation can be limited to symbols or libraries of symbols in that particular domain.

[0052] 5-7 illustrate example techniques for training data augmentation, although various modifications can be made to FIGs. 5-7. For example, other suitable engineering blueprints, symbols, symbol modifications, and symbol libraries can be used to generate annotated training data for one or more machine learning models. Also, although three techniques for training data augmentation are described above, any single technique or combination of techniques can be used for training data augmentation in any particular implementation.

[0053] 8 illustrates an example apparatus 800 that supports training of data-efficient object detection and / or data augmentation of engineering blueprint symbols in accordance with the present disclosure. One or more instances of apparatus 800 may be used to implement, for example, the object detection functionality described with respect to Figures 1-4 and / or the training data augmentation functionality described with respect to Figures 5-7. However, the object detection functionality and / or the training data augmentation functionality may be implemented in any other suitable manner.

[0054] As shown in Figure 8, device 800 illustrates a computing device or system including at least one processing unit 802, at least one storage unit 804, at least one communication unit 806, and at least one input / output (I / O) unit 808. Processing unit 802 is capable of executing instructions loadable into memory 810. Processing unit 802 includes any suitable number and type of processors or other processing devices in any suitable arrangement. Exemplary types of processing units 802 include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0055] Memory 810 and persistent storage 812 are examples of storage 804 and represent any structure capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information, either temporary or permanent). Memory 810 may represent random access memory or any other suitable volatile or non-volatile storage. Persistent storage 812 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0056] The communications unit 806 supports communication with other systems or devices. For example, the communications unit 806 may include a network interface card or a wireless transceiver that facilitates communication over a wired or wireless network. The communications unit 806 may support communication over any suitable physical or wireless communication link.

[0057] The I / O unit 808 allows for the input and output of data. For example, the I / O unit 808 may provide a connection for user input via a keyboard, mouse, keypad, touch screen, or other suitable input device. The I / O unit 808 may also send output to a display, printer, or other suitable output device. It should be noted, however, that if device 800 does not require local I / O, such as if device 800 represents a remotely accessible server or other device, then the I / O unit 808 may be omitted.

[0058] In some embodiments, instructions may be loaded into at least one storage device 804 and executed by at least one processing device 802 to perform data-efficient object detection of engineering blueprint symbols. For example, the instructions, when executed, may cause the at least one processing device 802 to obtain an input engineering blueprint 102, identify visual features (e.g., feature maps) of the input engineering blueprint 102, and process the visual features using at least one trained machine learning model. The at least one trained machine learning model may be trained to identify boundaries 204 around potential symbols in the input engineering blueprint 102 and classify the symbols in the input engineering blueprint 102 into appropriate classifications 206. Each classification 206 may identify a type of symbol detected in the input engineering blueprint 102, and different classifications 206 may be associated with different types of symbols. In this manner, the device 800 may be used to identify types of symbols contained in an engineering blueprint.

[0059] Additionally or alternatively, in some embodiments, instructions can be loaded into at least one storage device 804 and executed by at least one processing device 802 to perform training of the data augmentation. For example, the instructions, when executed, can cause the at least one processing device 802 to obtain training data including at least one training engineering design 102, such as one including known symbols (possibly symbols having known boundaries 204 and known classifications 206). The instructions, when executed, can also cause the at least one processing device 802 to augment the training data with at least one additional training engineering design 102, such as by using one or more of the approaches described above. Each additional training engineering design 102 can include a composite design including at least one additional known symbol (possibly each additional symbol having known boundaries 204 and known classifications 206). The training engineering design and the additional training engineering design can be used to train at least one machine learning model. In this manner, the device 800 can be used to augment training data based on or including engineering designs.

[0060] Although Figure 8 illustrates one example of an apparatus 800 supporting data-efficient object detection and / or training data augmentation of engineering blueprint symbols, various modifications can be made to Figure 8. For example, computing and communication devices and systems come in a wide variety of configurations, and Figure 8 is not intended to limit this disclosure to any particular computing or communication device or system.

[0061] 9 illustrates an example of a method 900 for data-efficient object detection of engineering blueprint symbols according to the present disclosure. For ease of explanation, the method 900 may be described as being performed using the architecture 100 of FIG. 1, which may be implemented using one or more instances of the device 800. However, the method 900 may be performed using any other suitable device and any suitable system.

[0062] As shown in FIG. 9 , in step 902, an input engineering blueprint is obtained. This may include, for example, the processing unit 802 of the device 800 obtaining the input engineering blueprint 102 from the storage device 804 or other suitable source. In step 904, visual features are extracted based on the input engineering blueprint. This may include, for example, the processing unit 802 of the device 800 using the feature extractor 104 to generate a feature vector or other extracted visual features associated with the input engineering blueprint 102. In step 906, the visual features are processed using a feature pyramid network to generate a feature map associated with the input engineering blueprint. This may include, for example, the processing unit 802 of the device 800 using the feature pyramid network 106 to process the feature vector or other extracted visual features to generate feature maps 202 at different scales.

[0063] At step 908, the feature map is processed using a region proposal network to identify potential symbols and boundaries around the potential symbols. This may include, for example, the processing unit 802 of the device 800 using the region proposal network 108 to identify boundaries 204 around potential symbols in the input engineering blueprint 102 based on the feature map 202. As a particular example, the region proposal network 108 may perform bounding box regression to identify boundaries 204 around potential symbols in the input engineering blueprint 102. At step 910, the feature map and the identified boundaries are processed using a region of interest network to identify classifications of the identified symbols. This may include, for example, the processing unit 802 of the device 800 using the region of interest network 110 to generate classifications 206 that identify types of symbols in the input engineering blueprint 102.

[0064] At step 912, the identified classifications and boundaries of the symbols in the input engineering plan can be stored, output, or used. This may include, for example, the processing unit 802 of the device 800 storing the identified symbols and their boundaries 204 and classifications 206 in a structured format, such as by generating a digital representation of the input engineering plan 102 using the identified boundaries 204 and the identified classifications 206. This may also include the processing unit 802 of the device 800 using the identified symbols and their boundaries 204 and classifications 206 to perform one or more functions related to the facility or other location that includes the equipment represented by the input engineering plan 102. In general, the identified symbols and their boundaries 204 and classifications 206 may be used in any suitable manner and for any suitable purpose.

[0065] Although Figure 9 illustrates one example of a method 900 for data-efficient object detection of engineering blueprint symbols, various modifications may be made to Figure 9. For example, although shown as a series of steps, the various steps in Figure 9 may overlap, be performed in parallel, be performed in a different order, or be performed any number of times.

[0066] 10 illustrates an example method 1000 for training data augmentation according to the present disclosure. For ease of explanation, the method 1000 may be described as being performed using the architecture 100 of FIG. 1, which may be implemented using one or more instances of the device 800. However, the method 1000 may be performed using any other suitable device and any suitable system.

[0067] 10, one or more training engineering designs are obtained at step 1002. This may include, for example, the processing unit 802 of the device 800 obtaining one or more input engineering designs 102 from a storage unit 804 or other suitable source. In some cases, each input engineering design 102 may have a known symbol with a known boundary 204 and a known classification 206.

[0068] Using one technique for data augmentation, at least one training engineering diagram may be resized and cropped, and one or more modifications may optionally be applied to the cropped regions of the training engineering diagram to generate first augmented training data, in step 1004. This may include, for example, the processing unit 802 of the device 800 rescaling and cropping each of the one or more engineering diagrams 500 to generate one or more regions 502, and applying one or more modifications to the regions 502 to generate one or more new diagrams 504.

[0069] In step 1006, another technique for data augmentation may be used to crop, optionally modify, and insert one or more symbols from at least one training engineering blueprint into the background of the one or more training engineering blueprints to generate second augmented training data. This may include, for example, the processing unit 802 of the device 800 cropping one or more symbols from the one or more engineering blueprints 600, modifying the cropped one or more symbols as necessary, and inserting the one or more symbols into the one or more engineering blueprints to generate one or more new circuit diagrams 602.

[0070] Further techniques for data augmentation may be used to select, optionally modify, and insert one or more symbols from at least one training library into the background of one or more training engineering blueprints to generate third augmented training data, step 1008. This may involve, for example, the processing unit 802 of the device 800 selecting symbols from one or more libraries 700, modifying the selected one or more symbols as necessary, and inserting the one or more symbols into one or more engineering blueprints to generate one or more new schematics 702.

[0071] At step 1010, one or more machine learning models may be trained using the one or more training engineering blueprints and the augmented training data. This may include, for example, the processing unit 802 of the device 800 performing back-propagation or other training techniques to modify gradients and weights of the one or more machine learning models (such as one or more models implementing the region proposal network 108 and the region of interest network 110) to reduce a loss function (such as that defined above) below a threshold. Ideally, the one or more machine learning models may be trained to be more effective at identifying one or more characteristics of symbols in the input engineering blueprints 102, such as the boundaries 204 and classifications 206 of the symbols, as compared to training without the augmented training data.

[0072] Although Figure 10 illustrates one example of a method 1000 for training data augmentation, various modifications may be made to Figure 10. For example, although shown as a series of steps, the various steps in Figure 10 may overlap, occur in parallel, occur in a different order, or occur any number of times. Also, although three training data augmentation techniques are used in Figure 10, one or any combination of the training data augmentation techniques may be used in any given implementation.

[0073] An exemplary embodiment of the present disclosure that implements data-efficient object detection of engineering blueprint symbols is described below, however, other embodiments may be used in accordance with the teachings of the present disclosure.

[0074] In a first embodiment, a method includes obtaining an engineering blueprint including a plurality of symbols and connections associated with the symbols, where different ones of the symbols in the engineering blueprint represent different types of equipment. The method also includes identifying visual features in the engineering blueprint. The method further includes processing the visual features using at least one trained machine learning model to (i) identify boundaries around the symbols in the engineering blueprint and (ii) classify the symbols in the engineering blueprint into a plurality of classifications, where the symbols in the different classifications are associated with different types of symbols.

[0075] In a second embodiment, the equipment includes at least one processing device configured to obtain an engineering blueprint including a plurality of symbols and connections associated with the symbols, where different ones of the symbols in the engineering blueprint represent different types of equipment. The at least one processing device is further configured to identify visual features in the engineering blueprint. The at least one processing device is further configured to process the visual features using the at least one trained machine learning model to (i) identify boundaries around the symbols in the engineering blueprint, and (ii) classify the symbols in the engineering blueprint into a plurality of classifications, where the symbols in the different classifications are associated with different types of equipment.

[0076] In a third embodiment, a non-transitory computer readable medium has computer readable program code stored thereon that, when executed, causes one or more processors to obtain an engineering blueprint including a plurality of symbols and connections associated with the symbols, where different ones of the symbols in the engineering blueprint represent different types of equipment. The medium further has computer readable program code stored thereon that, when executed, causes the one or more processors to identify visual features in the engineering blueprint. The medium has computer readable program code stored thereon that, when executed, causes the one or more processors to process the visual features using at least one trained machine learning model to (i) identify boundaries around the symbols in the engineering blueprint, and (ii) classify the symbols in the engineering blueprint into a plurality of classifications, where the symbols in the different classifications are associated with different types of symbols.

[0077] In the first, second or third embodiment, any one or any suitable combination of the following features may be used: At least one trained machine learning model is (i) a region proposal network configured to identify boundaries around symbols in engineering blueprints based on visual features; (ii) a region of interest network configured to classify symbols in the engineering blueprint into a plurality of classifications based on visual features and identified boundaries; The visual features of the engineering blueprint may be identified using a feature extractor, and the feature pyramid network may generate feature maps at different scales based on the visual features. The at least one trained machine learning model may include a plurality of machine learning paths, and different ones of the machine learning paths may be trained to identify boundaries around symbols of different types. The at least one trained machine learning model is trained using training data associated with one or more symbols of the different types of symbols by modifying gradients or weights of one or more of the machine learning paths, and not modifying gradients or weights of one or more other ones of the machine learning paths associated with one or more other symbols of the different types of symbols. The different ones of the machine learning paths may be trained to identify boundaries around symbols having different shapes and / or different aspect ratios. A digital representation of the engineering blueprint may be generated using the identified boundaries around the symbols in the engineering blueprint and the classification of the symbols in the engineering blueprint.

[0078] In some embodiments, various functions described in this patent document are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium accessible by a computer, such as read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer readable medium includes media that can store data permanently, and media that can store data and later be overwritten, such as rewritable optical disks or erasable storage devices.

[0079] It is advantageous to establish definitions of certain words and phrases used throughout this patent document. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives include both direct and indirect communication. The terms "include" and "comprise" and its derivatives mean inclusion without limitation. The term "or" is inclusive and / or. The term "related" and its derivatives can mean include, be contained within, interconnect, contain, be contained within, connect, connect or connect, be able to communicate, cooperate, connect, juxtapose, be adjacent, couple or connect, have a property of, or have a relationship, and the like. The phrases "at least one" and "one or more," when used in conjunction with a list of items, mean that different combinations of one or more of the listed items can be used, and only one item in the list may be required. For example, "at least one of A, B, C" includes any combination of A, B, C, A and B, A and C, B and C, and A and B and C.

[0080] The description in this disclosure should not be construed as implying that a particular element, step, or function is an essential or critical element that must be included in the scope of the claims. The scope of the patent is defined solely by the allowed claims. Moreover, no claim invokes 35 U.S.C. §112(f) with respect to any of the appended claims or claim elements, unless the precise words "means" or "step" are expressly used in a particular claim, followed by a participial phrase identifying the function. The use of terms such as, but not limited to, "mechanism," "module," "apparatus," "unit," "component," "element," "member," "device," "machine," "system," "processor," or "controller" in the claims is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or enhanced by the features of the claims themselves, and is not intended to invoke 35 U.S.C. §112(f).

[0081] While this disclosure has described certain embodiments and generally associated methods, modifications and permutations of these embodiments and methods will be apparent to those skilled in the art. Accordingly, the above description of exemplary embodiments does not define or limit the disclosure. Other modifications, substitutions, and alterations are possible without departing from the spirit and scope of the disclosure, as defined by the following claims.

Claims

1. 1. A method comprising: obtaining an engineering drawing comprising a plurality of symbols and connections associated with the symbols, different ones of the symbols representing different types of equipment; identifying visual features of the engineering design; processing the visual features using at least one trained machine learning model to (i) identify boundaries around symbols in the engineering blueprint; and (ii) classify symbols in the engineering blueprint into a plurality of classifications, different ones of the classifications being associated with different types of symbols; The method includes:

2. The at least one trained machine learning model: a region proposal network configured to identify boundaries around symbols in the engineering blueprint based on the visual features; a region of interest network configured to classify symbols in the engineering blueprint into the plurality of classifications based on the visual features and the identified boundaries; The method of claim 1 , comprising:

3. wherein identifying visual features of the engineering design comprises using a feature extractor; The method of claim 1 , further comprising: generating feature maps at different scales based on the visual features using a feature pyramid network.

4. 2. The method of claim 1, wherein the at least one trained machine learning model comprises a plurality of machine learning paths, different ones of the machine learning paths being trained to identify boundaries around the different types of symbols.

5. 5. The method of claim 4, wherein the at least one trained machine learning model is trained using training data associated with one or more symbols of the different types of symbols by modifying gradients or weights of one or more of the machine learning paths, and not modifying gradients or weights of one or more other of the machine learning paths associated with one or more other symbols of the different types of symbols.

6. The method of claim 4 , wherein different ones of the machine learning paths are trained to identify boundaries around symbols having at least one of different shapes and different aspect ratios.

7. 10. The method of claim 1 , further comprising: generating a digital representation of the engineering blueprint using the identified boundaries around the symbols in the engineering blueprint and a classification of the symbols in the engineering blueprint.

8. An apparatus comprising at least one processing unit adapted to carry out the method according to any one of claims 1 to 7.

9. A non-transitory computer readable medium having computer readable program code stored thereon, the computer readable program code, when executed, causing one or more processors to perform the method of any one of claims 1 to 7.

Images