Data Augmentation for Training Data of Engineering Schematics Used with Machine Learning Models

Abstract

The method includes obtaining training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The method also includes augmenting the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The method further includes training at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

Description

[Technical field]

[0001] The present disclosure is directed generally to machine learning systems and, more specifically, to data augmentation for training data of engineering schematics used with machine learning models. [Background technology]

[0002] An engineering diagram typically includes many symbols, which may represent sensors or other pieces of equipment in a manufacturing facility or other facility or location. Engineering diagrams also typically include various connections between the symbols, which may represent material flows or other relationships between equipment at the location. In some instances, the symbols in the engineering diagram and the connections between the symbols are needed to identify and understand complex relationships between equipment. However, this process is often accomplished using manual labor, which can be time-consuming, expensive, and inaccurate. Automated approaches to perform this process typically require a large number of annotated engineering diagrams, which may not be available and thus may hinder the use of these automated approaches. Summary of the Invention

[0003] The present disclosure relates to data augmentation for training data of engineering schematics used with machine learning models.

[0004] In a first embodiment, a method includes obtaining training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The method also includes augmenting the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The method further includes training at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

[0005] In a second embodiment, an apparatus includes at least one processing device. The at least one processing device is configured to obtain training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The at least one processing device is also configured to augment the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The at least one processing device is also further configured to train at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

[0006] In a third embodiment, a non-transitory computer readable medium has computer readable program code stored thereon that, when executed, causes one or more processors to obtain training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The medium also has computer readable program code stored thereon that, when executed, causes one or more processors to augment the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The medium further has computer readable program code stored thereon that, when executed, causes one or more processors to train at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

[0007] Other technical features may become readily apparent to those skilled in the art from the following figures, description, and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates one exemplary architecture for data-efficient object detection on engineering schematic symbols according to this disclosure. [Diagram 2] FIG. 2 illustrates an example data processing in the architecture of FIG. 1 according to the present disclosure. [Diagram 3] FIG. 3 illustrates example shapes and aspect ratios of symbols in an engineering schematic that may be detected using the architecture of FIG. 1 in accordance with this disclosure. [Figure 4] FIG. 4 illustrates example machine learning pathways for different types of engineering schematic symbols in the architecture of FIG. 1 in accordance with this disclosure. [Diagram 5] FIG. 5 illustrates a first exemplary technique for training data augmentation according to this disclosure. [Figure 6] FIG. 6 illustrates a second exemplary technique for training data augmentation in accordance with this disclosure. [Figure 7] FIG. 7 illustrates a third exemplary technique for training data augmentation in accordance with this disclosure. [Figure 8] FIG. 8 illustrates an example device supporting data-efficient object detection and / or training data augmentation on symbols of engineering schematics according to this disclosure. [Figure 9] FIG. 9 illustrates one exemplary method for data-efficient object detection on engineering schematic symbols according to this disclosure. [Figure 10] FIG. 10 illustrates one exemplary method for training data augmentation according to this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1-10, described below, and the various embodiments used to illustrate the principles of the present disclosure, are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure in any way. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any type of suitably arranged device or system.

[0010] As mentioned above, engineering diagrams typically include many symbols, which may represent sensors or other pieces of equipment in a manufacturing facility or other facility or location. Engineering diagrams also typically include various connections between the symbols, which may represent material flows or other relationships between equipment at the location. In some instances, the symbols in the engineering diagram and the connections between the symbols are needed to identify and understand complex relationships between equipment. However, this process is often accomplished using manual labor, which can be time-consuming, expensive, and inaccurate. Automated approaches to perform this process typically require a large number of annotated engineering diagrams, which may not be available and thus may hinder the use of these automated approaches.

[0011] Electronic parsing of engineering diagrams generally involves generating a digital representation of an engineering system based on a non-digital engineering schematic (also known as an engineering diagram). Diagram parsing generally involves using one or more computer vision models to detect target symbols in the engineering diagram, determine connections between the detected symbols, and output a digitized representation of the engineering diagram in a structured format (e.g., by using structured data).

[0012] The present disclosure provides various techniques for data-efficient object detection of symbols in engineering schematics and for training data augmentation related to engineering schematics. The techniques for data-efficient object detection are based on the use of one or more machine learning models that can be trained to recognize symbols in engineering schematics and, optionally, recognize connections between the recognized symbols. The one or more machine learning models can be trained using various information, such as training diagrams developed from one or more domain-specific engineering symbol libraries, or other training data collected or generated in any suitable manner. The techniques for data-efficient object detection can thus be used to perform efficient and accurate symbol detection using engineering schematics, which can result in a more accurate digital representation of the engineering schematic. Furthermore, the techniques for data-efficient object detection can provide improved model efficiency, meaning that these techniques take less time to perform symbol identification.

[0013] Techniques for training data augmentation support the use of one or more mechanisms that can be used to augment training data associated with an engineering schematic. For example, techniques for training data augmentation can perform resizing of one or more labeled engineering schematics using a random scale and random cropping and modification of the resized labeled engineering schematic to create at least one synthetic diagram. Techniques for training data augmentation can also or alternatively include cropping one or more target symbols and randomly placing them into one or more engineering schematic backgrounds to create at least one synthetic diagram. Techniques for training data augmentation can also or alternatively include randomly selecting text from at least one symbol library, optionally adding text onto one or more symbols, and randomly placing them into one or more engineering schematic backgrounds to create at least one synthetic diagram. One, some, or all of these approaches can be used to generate additional labeled training data that can be used to train one or more machine learning models. These techniques may therefore be used to simplify the collection or generation of training data used to train machine learning models, which can reduce the time and costs associated with training machine learning models.

[0014] Note that both (i) techniques for data-efficient object detection and (ii) techniques for training data augmentation are described herein as being performed using the same device or system. As a result, one or more techniques for training data augmentation may be used to generate training data, which is then used to train one or more machine learning models, such that the trained machine learning models may be used to perform data-efficient object detection. However, these techniques can be used separately. Thus, techniques for data-efficient object detection may be used without training data augmentation, and techniques for training data augmentation may be used without data-efficient object detection. Also, note that even when both techniques are used together, these techniques need not be performed using the same device or system, and may be implemented together or separately, as needed or desired.

[0015] FIG. 1 illustrates one exemplary architecture 100 for data-efficient object detection on symbols of an engineering schematic according to the present disclosure. As shown in FIG. 1, the architecture 100 generally operates to receive and process at least one input engineering schematic 102. Each input engineering schematic 102 generally includes a number of symbols representing sensors or other equipment, and a number of connections to, from, and between the symbols. Each input engineering schematic 102 may also include notations or other markings, such as markings around one or more particular portions of the input engineering schematic 102 that each include associated equipment. Each input engineering schematic 102 may be obtained from any suitable source, such as a camera, a scanner, a database, a user device, or other source of input engineering schematics.

[0016] The input engineering schematic 102 is now provided to a feature extractor 104, which generally operates to extract visual features of the input engineering schematic 102. The feature extractor 104 may use any suitable technique to identify visual features of the input engineering schematic 102. For example, the feature extractor 104 may represent a trained machine learning model, such as a convolutional neural network (CNN) or other type of machine learning model, trained to process the input engineering schematic 102 using various convolution, pooling, or other layers to generate a feature vector or other extracted visual features based on the input engineering schematic 102. As a particular example, the feature extractor 104 may represent an implementation of a VGG or EfficientNet feature extractor.

[0017] The extracted visual features are provided to a feature pyramid network 106, which processes the extracted visual features to generate feature maps associated with the features of the input engineering schematic 102. Different ones of the feature maps generated by the feature pyramid network 106 may include features of the input engineering schematic 102 at different scales. This enables the architecture 100 to subsequently identify symbols in the input engineering schematic 102 that have various sizes, such as by identifying symbols that have significantly different sizes. The feature pyramid network 106 may use any suitable technique to analyze the visual features and generate feature maps at different scales. For example, the feature pyramid network 106 may represent a trained machine learning model, such as a CNN or other type of machine learning model, that has been trained to process feature vectors or other extracted visual features and generate feature maps at different scales.

[0018] The feature maps at different scales are provided to a region proposal network 108 and a region of interest network 110. The region proposal network 108 generally operates to identify potential symbols in the input engineering schematic 102 and then use the feature maps to identify bounding boxes or other possible boundaries around the potential symbols in the input engineering schematic 102. For example, the region proposal network 108 may use different specialized symbol prediction heads or other logic to generate class-agnostic bounding boxes around the potential symbols in the input engineering schematic 102. The bounding boxes are referred to herein as “class-agnostic” because they may be generated without reference to the particular types of symbols contained in the input engineering schematic 102. The region proposal network 108 may use any suitable technique to identify potential symbols in the input engineering schematic 102 and boundaries around the potential symbols. For example, the region proposal network 108 may represent a trained machine learning model, such as a CNN or other type of machine learning model, that is trained to process feature maps and identify potential symbols and boundaries around the potential symbols.

[0019] The region of interest network 110 generally operates to process feature maps at different scales and boundaries of potential symbols in the input engineering schematic 102 to identify actual symbols in the input engineering schematic 102 and classify those symbols. For example, the region of interest network 110 can use machine learning to classify each of the potential symbols into different classes, where the different classes are associated with different types of engineering symbols (such as different classes of symbols representing sensors, vessels, valves, actuators, or other components). The region of interest network 110 can use any suitable technique to identify and classify the symbols in the input engineering schematic 102. For example, the region of interest network 110 can represent a trained machine learning model, such as a CNN with region of interest pooling or other type of machine learning model, trained to process feature maps and symbol boundaries to identify and classify different symbols.

[0020] In this manner, architecture 100 generally supports a multi-stage approach for symbol detection that includes (i) identification of class-agnostic boundaries of potential symbols, and (ii) identification and classification of actual symbols based on the identified boundaries. Through proper training of the machine learning models used in architecture 100, architecture 100 can be used to effectively and accurately identify engineering symbols within a schematic. This can be done much more quickly and accurately compared to manual approaches, and this can be done without the need for large numbers of annotated engineering schematics. The identified engineering symbols can be used in any suitable manner, such as to create a digital representation of the engineering schematic, which itself can be used in any suitable manner.

[0021] Note that the functionality shown or described with respect to FIG. 1 may be implemented in an electronic device, such as a computing device, in any suitable manner. For example, in some embodiments, at least some of the functionality shown or described with respect to FIG. 1 may be implemented or supported using one or more software applications or other software instructions executed by one or more processing units of a desktop computer, laptop computer, tablet computer, server computer, or other device. In other embodiments, at least some of the functionality shown or described with respect to FIG. 1 may be implemented or supported using dedicated hardware components. In general, the functionality shown or described with respect to FIG. 1 may be performed using any suitable hardware or any suitable combination of hardware instructions and software / firmware instructions.

[0022] Although FIG. 1 illustrates one example of an architecture 100 for data-efficient object detection of symbols on engineering schematics, various modifications may be made to FIG. 1. For example, although various functions shown or described with respect to FIG. 1 are described as being implemented or performed using machine learning models, any number of machine learning models may be used to implement architecture 100. In some cases, different machine learning models may be trained to perform different functions of architecture 100. In other cases, a single machine learning model may be trained to perform multiple functions of architecture 100. Components may also be added, omitted, combined, further subdivided, duplicated, or arranged in any other suitable configuration within architecture 100 according to particular needs.

[0023] FIG. 2 illustrates an example data processing in the architecture 100 of FIG. 1 according to the present disclosure. As illustrated in FIG. 2, the input engineering schematic 102 is processed using the feature extractor 104 and the feature pyramid network 106, which generates various extracted feature maps 202 (one example of which is shown here). The feature maps 202 may include low-level and high-level visual features of the input engineering schematic 102 or may otherwise identify visual features of the input engineering schematic 102 at different scales. The region proposal network 108 processes the feature maps 202 to identify one or more potential symbols in the input engineering schematic 102 and a boundary 204 (such as a box or other polygon) around each potential symbol in the input engineering schematic 102. In some cases, the region proposal network 108 may perform bounding box regression to identify the boundary 204 of the potential symbols in the input engineering schematic 102.

[0024] The region of interest network 110 processes the feature map 202 and potential symbol boundaries 204 to identify actual symbols in the input engineering schematic 102 and generate classifications 206 of the actual symbols in the input engineering schematic 102. For example, the region of interest network 110 may generate a classification 206 for each identified boundary 204, where the classification 206 identifies the type of symbol contained within the identified boundary 204 (or does not identify the type of symbol if the region of interest network 110 determines that no actual symbol is contained within the identified boundary 204). Thus, each classification 206 may identify one type of symbol detected in the input engineering schematic 102. In some cases, the region of interest network 110 may identify a classification target for a symbol in the input engineering schematic 102, where the classification target represents a particular type of symbol expected in a given domain or other application.

[0025] Engineering symbols are two-dimensional objects that have different characteristics from three-dimensional objects (such as people, animals, or vehicles) in terms of visual features. Unlike object detection in three-dimensional space, where machine learning models need to learn feature extraction that works for various view angles and across different distances, two-dimensional object detection models may only need to learn feature extraction that works for limited scaling and limited shifts of two-dimensional objects. Furthermore, engineering schematics often need to follow domain-specific standard symbol templates, which means that the variability of engineering symbols across different engineering schematics tends to be lower compared to generic three-dimensional objects. To take advantage of the two-dimensional inherent characteristics of engineering symbols and their low variability, scale-specific prediction heads or other types of parallel prediction heads may be used in the region proposal network 108 to improve model prediction quality, as described in more detail below. Also, to improve data efficiency for training the region proposal network 108, training data augmentation based on operations such as random “copy and paste” data augmentation may be used, as described in more detail below.

[0026] In some embodiments, the region proposal network 108 and the region of interest network 110 may be trained to jointly optimize their model gradients and weights using the regression loss (associated with generating the boundary 204) and the classification loss (associated with generating the classification 206). As a particular example, the region proposal network 108 and the region of interest network 110 may be trained to minimize the following loss function, which combines the regression loss (right side of the sum) and the classification loss (left side of the sum):

number

[0027] Any suitable technique may be used to train one or more machine learning models to implement the functionality of one or more components of the architecture 100. For example, training data including engineering schematics associated with known boundaries 204 and known classifications 206 may be used to train the region proposal network 108 and the region of interest network 110. In some embodiments, techniques such as back propagation may be used to adjust model gradients and weights of the one or more machine learning models until the one or more machine learning models accurately identify, to a desired degree, the known boundaries 204 and known classifications 206 of the engineering schematics in the training data. In particular embodiments, training may continue until a loss value (which may be calculated using the formula above or in any other suitable manner) falls below a particular threshold or until some other criterion or criteria are met (such as a specified training time has elapsed or a specified number of training iterations have occurred).

[0028] Although Figure 2 illustrates one example of data processing within the architecture 100 of Figure 1, various modifications may be made to Figure 2. For example, the particular feature maps 202, boundaries 204, and classifications 206 may vary based on the particular input engineering schematic 102 being processed by the architecture 100.

[0029] To perform effective feature extraction of a three-dimensional object, a machine learning model often needs to be trained to identify features of the three-dimensional object regardless of the distance of the three-dimensional object from the camera. That is, the machine learning model should be able to effectively identify features of the three-dimensional object regardless of whether the three-dimensional object is close to or far from the camera. However, during engineering symbol detection, each engineering symbol may typically be represented in a limited range of shapes and aspect ratios. An example of this is shown in FIG. 3, which shows exemplary shapes and aspect ratios of engineering schematic symbols 300 and 302 that may be detected using the architecture 100 of FIG. 1 according to the present disclosure. As shown in FIG. 3, the symbol 300 is formed using a circle, and text may optionally be present within the circle. Also, the symbol 302 is formed using a circle within a square, and text may optionally be present within the square and the circle. These two types of symbols 300 and 302 may typically have a 1:1 aspect ratio. Note that other symbols will typically have other shapes (such as lines, ellipses, triangles, diamonds, or other regular or irregular shapes) or combinations of shapes and / or other aspect ratios (such as higher or wider aspect ratios).

[0030] Based on the knowledge that each engineering symbol may typically have a known or predetermined shape and aspect ratio, separate paths can be created and trained within the architecture 100, where the different paths are used to predict boundaries 204 and classifications 206 for different engineering symbols having different shapes and / or aspect ratios. FIG. 4 illustrates example machine learning paths for different types of engineering schematic symbols in the architecture 100 of FIG. 1 in accordance with the present disclosure. As shown in FIG. 4, the feature extractor 104 can extract visual features from the input engineering schematic 102, and k different paths can be used to process the extracted visual features to identify boundaries and classifications for different types of symbols (such as symbols having different shapes and / or aspect ratios) that may be included in the input engineering schematic 102. For example, different portions 106a-106k of the feature pyramid network 106, different portions 108a-108k of the region proposal network 108, and different portions 110a-110k of the region of interest network 110 may be trained or otherwise used to process different types of symbols. In some instances, the different portions 110a-110k of the region of interest network 110 may be trained to recognize different symbols, and each portion 110a-110k of the region of interest network 110 may be configured to output a true or false indicator that identifies whether an identified symbol in the input engineering schematic 102 is or is not a particular symbol associated with that portion 110a-110k of the region of interest network 110.

[0031] This approach allows various paths to learn useful features specific to different types of symbols, such as symbols with different shapes and / or aspect ratios. As a result, different paths can be trained to effectively discriminate between different types of symbols. Furthermore, this approach separates the gradient and weight updates along each path, such that different paths can be trained separately or independently from one another. Thus, while training the architecture 100, different symbols in the training data can be used to update gradients and weights only along paths associated with those particular types of symbols. Paths related to other types of symbols may not need to be updated, which can simplify and speed up the training process. Additionally, when certain types of symbols have low quality annotations in the training data, the negative effects associated with a lack of quality training data for those particular types of symbols may be limited to those particular paths, while other paths associated with symbols having higher quality annotations in the training data may not be affected.

[0032] Although Fig. 3 illustrates one example of shapes and aspect ratios of engineering schematic symbols 300 and 302 that may be detected using architecture 100 of Fig. 1, various modifications may be made to Fig. 3. For example, other engineering schematic symbols may have any other suitable shapes and / or any other suitable aspect ratios. Although Fig. 4 illustrates one example of machine learning paths for different types of engineering schematic symbols within architecture 100 of Fig. 1, various modifications may be made to Fig. 4. For example, architecture 100 may support any desired number of parallel paths to identify any desired number of engineering schematic symbol types.

[0033] Training a complex deep learning model or other machine learning model typically requires a large amount of labeled training data. To support training of one or more machine learning models related to engineering symbols (such as one or more machine learning models used in the architecture 100 or other machine learning models used for other functions related to engineering symbols or schematics), various effective data augmentation approaches can be implemented by taking advantage of the low variability of engineering symbols across diagrams (at least within a given domain, such as a given industry). These data augmentation approaches can be used to reduce the number of annotated training diagrams required to train one or more machine learning models. The following describes three exemplary data augmentation approaches that can be used to generate training data.

[0034] FIG. 5 illustrates a first exemplary technique for training data augmentation according to the present disclosure. This exemplary technique includes resizing one or more labeled engineering schematics using at least one random scale, and performing random cropping and modification of the resized labeled engineering schematic to create at least one composite view. As shown in FIG. 5, for example, a labeled engineering schematic 500 represents annotated training data, such as when the labeled engineering schematic 500 includes symbols with known boundaries 204 and known classifications 206. The first exemplary technique for training data augmentation can resize the labeled engineering schematic 500 using one or more random scales (such as a larger or smaller scale), and the resized engineering schematic can be cropped to include only a particular region 502. One or more modifications may also be applied to the cropped region 502, such as by applying color shifting, line thickening / thinning, dilation, and / or adding noise pixels to the cropped region 502. This results in a new labeled engineering schematic 504 that represents the modified and resized portion of the labeled engineering schematic 500. Because the new schematic 504 is generated using the contents of the labeled schematic 500, the new schematic 504 represents fully labeled training data that may be used for machine learning model training. For example, the new schematic 504 may include known symbols with known boundaries 204 and known classifications 206.

[0035] Note that any number of resizing scales may be applied to each labeled engineering schematic 500, and any number of overlapping and / or non-overlapping regions 502 may be defined for each rescaled version of each engineering schematic 500. Also note that each region 502 for each resized version of each engineering schematic 500 may be used to generate any number of new schematics 504, such as when different modification types or combinations of modification types are applied to each region 502, or when different levels of modification are applied to each region 502, etc. As a result, it is possible to generate very large amounts of annotated training data based on a limited number of labeled engineering schematics 500. Additionally, note that if one or more machine learning models are used to analyze engineering schematics in a particular domain (such as a particular industry), the labeled engineering schematics 500 used for data augmentation may be limited to engineering schematics for that particular domain.

[0036] FIG. 6 illustrates a second exemplary technique for training data augmentation according to the present disclosure. This exemplary technique includes cropping one or more symbols and randomly placing them on the background of one or more engineering diagrams to create at least one composite diagram. As shown in FIG. 6, for example, a labeled engineering schematic 600 represents annotated training data, such as when the labeled engineering schematic 600 includes symbols with known boundaries 204 and known classifications 206. The second exemplary technique for training data augmentation can crop one or more symbols from the labeled engineering schematic 600, optionally modify at least one of the cropped symbols, and insert at least one of the cropped (and possibly modified) symbols into the background of the same engineering schematic 600 and / or at least one additional engineering schematic. The exemplary modification can include the same types of modifications described above with respect to FIG. 5. The modifications may include color shifting, thickening / thinning of lines, dilation, and / or adding noise pixels. This may result in the creation of a new labeled engineering schematic 602. Because the new schematic 602 is generated using the content of one or more labeled schematics, the new schematic 602 represents fully labeled training data that may be used for machine learning model training. This is due to the new schematic 602 including known symbols with known boundaries 204 and classifications 206.

[0037] Note that different versions of the same symbol may be generated during this augmentation process, such as by cropping the same symbol in different ways and / or applying different modification types, combinations of modification types, and / or levels of modification to the cropped symbols. Any individual cropped and potentially modified symbol, or any combination of cropped and potentially modified symbols, may also be identified and inserted into any number of engineering schematics. Furthermore, the insertion locations of the cropped and potentially modified symbols may vary widely. As a result, it is possible to generate very large amounts of annotated training data based on a limited number of labeled engineering schematics 600. Additionally, note that if one or more machine learning models are used to analyze engineering schematics in a particular domain (such as a particular industry), the labeled engineering schematics 600 used for data augmentation may be limited to engineering schematics for that particular domain.

[0038] This second approach to data extension may be said to represent a "copy and paste" data extension process, since one or more symbols are effectively copied from one or more engineering schematics and then pasted into the same engineering schematic or into one or more additional engineering schematics. Note, however, that overlapping of engineering symbols is typically not permitted in engineering schematics, such as due to standard practice. As a result, pasting the copied and potentially modified symbols into an engineering schematic is said to occur in the "background" of that schematic. This means that the symbols are pasted into the engineering schematic in a location where the pasted symbols do not overlap another symbol.

[0039] FIG. 7 illustrates a third exemplary technique for training data augmentation according to the present disclosure. This exemplary technique includes randomly selecting one or more symbols from at least one symbol library, optionally adding text, and randomly placing the selected symbols into the background of one or more engineering diagrams to create at least one composite diagram. As shown in FIG. 7, for example, one or more libraries 700 of known symbols may be available for use in an engineering diagram application, such as an application that may be used by one or more users to create or edit engineering diagrams. The one or more libraries 700 may be randomly sampled to select one or more known symbols from the library 700. The one or more selected symbols may be optionally modified. Exemplary modifications may include the same types of modifications described above in FIG. 5, such as color shifting, line thickening / thinning, dilation, and / or adding noise pixels. Additional modifications may include randomly creating text for one or more selected symbols and / or randomly resizing one or more selected symbols (while generally maintaining the aspect ratio of each symbol). Note that various limitations may be placed on the random resizing of the selected symbols, such as by limiting the amount of resizing so that the relative size ratios of different symbols are maintained. A particular example of this may include ensuring that sensor symbols are not resized to be larger than their associated equipment symbols.

[0040] One or more modified versions of a selected symbol from library 700 may be inserted into at least one engineering schematic 702. Note that each modified version of a symbol from library 700 represents annotated training data since the boundary 204 and classification 206 of that symbol are known. In some cases, engineering schematic 702 may represent a labeled engineering schematic with known symbols, and one or more modified versions of one or more selected symbols from library 700 may be inserted into the background of engineering schematic 702 (such as in a non-overlapping manner). In other cases, engineering schematic 702 may represent a blank schematic into which multiple modified versions of multiple selected symbols from one or more libraries 700 may be inserted. Overall, this approach may be used to emulate an actual engineering drawing process in which an engineer or other user drags a symbol from one or more symbol libraries, adds an identifier string to the symbol, and drops the symbol into a diagram.

[0041] Note that different versions of the same symbol from library 700 may be generated, such as by applying different correction types, combinations of correction types, levels of correction, text, and / or resizing to the symbol. Any individual selected and potentially modified symbol, or any combination of selected and potentially modified symbols, may also be identified and inserted into any number of engineering schematics. Furthermore, the insertion location of the selected and potentially modified symbols may vary widely, and the particular library 700 used may vary. As a result, it is possible to generate very large amounts of annotated training data based on one or more libraries 700 of symbols. Additionally, note that if one or more machine learning models are used to analyze engineering schematics in a particular domain (such as a particular industry), the one or more libraries 700 used for data augmentation may be limited to symbols or symbol libraries for that particular domain.

[0042] 5-7 show examples of techniques for training data augmentation, various modifications may be made to FIGS. 5-7. For example, any other suitable engineering schematics, symbols, symbol modifications, and symbol libraries may be used to generate annotated training data for one or more machine learning models. Also, although three techniques for training data augmentation are described above, in any particular implementation, any single technique or combination of techniques may be used for training data augmentation.

[0043] 8 illustrates an example device 800 supporting data-efficient object detection and / or training data augmentation of engineering schematic symbols in accordance with the present disclosure. One or more instances of device 800 may be used to implement, for example, the object detection functionality described with respect to FIGS. 1-4 and / or the training data augmentation functionality described with respect to FIGS. 5-7. However, the object detection functionality and / or the training data augmentation functionality may be implemented in any other suitable manner.

[0044] As shown in Figure 8, device 800 represents a computing device or system including at least one processing unit 802, at least one storage unit 804, at least one communication unit 806, and at least one input / output (I / O) unit 808. Processing unit 802 can execute instructions that can be loaded into memory 810. Processing unit 802 includes any suitable number and type of processors or other processing devices in any suitable configuration. Exemplary types of processing units 802 include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0045] Memory 810 and persistent storage 812 are examples of storage 804 and represent any structure capable of storing information (such as data, program code, and / or other suitable information, either temporary or persistent) and facilitating retrieval of information. Memory 810 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 812 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0046] The communications unit 806 supports communication with other systems or devices. For example, the communications unit 806 may include a network interface card or a wireless transceiver that facilitates communication over a wired or wireless network. The communications unit 806 may support communication over any suitable physical or wireless communication link.

[0047] The I / O unit 808 allows for the input and output of data. For example, the I / O unit 808 may provide a connection for user input via a keyboard, mouse, keypad, touch screen, or other suitable input device. The I / O unit 808 may also send output to a display, printer, or other suitable output device. Note, however, that if the device 800 does not require local I / O, such as if the device 800 represents a server or other device that may be accessed remotely, the I / O unit 808 may be omitted.

[0048] In some embodiments, instructions may be loaded into at least one storage device 804 and executed by at least one processing device 802 to perform data-efficient object detection of engineering schematic symbols. For example, the instructions, when executed, may cause the at least one processing device 802 to obtain an input engineering schematic 102, identify visual features (such as a feature map) of the input engineering schematic 102, and process the visual features using at least one trained machine learning model. The at least one trained machine learning model may be trained to identify boundaries 204 around potential symbols in the input engineering schematic 102 and classify symbols in the input engineering schematic 102 into appropriate classifications 206. Each classification 206 may identify a type of symbol detected in the input engineering schematic 102, and different classifications 206 may be associated with different types of symbols. In this manner, the device 800 may be used to identify types of symbols included in an engineering schematic.

[0049] Also, or alternatively, in some embodiments, instructions may be loaded into at least one storage device 804 and executed by at least one processing device 802 to perform training data augmentation. For example, the instructions, when executed, may cause the at least one processing device 802 to obtain training data including at least one training engineering schematic 102, such as one including known symbols (perhaps with known boundaries 204 and known classifications 206 of the symbols). The instructions, when executed, may also cause the at least one processing device 802 to augment the training data with at least one additional training engineering schematic 102, such as by using one or more of the approaches described above. Each additional training engineering schematic 102 may include a composite schematic including at least one additional known symbol (perhaps with known boundaries 204 and known classifications 206 for each additional symbol). At least one machine learning model may be trained using the training engineering schematic and the additional training engineering schematic. In this manner, device 800 can be used to augment training data that is based on or includes engineering schematics.

[0050] Although Figure 8 illustrates one example of a device 800 supporting data-efficient object detection and / or training data augmentation of engineering schematic symbols, various modifications may be made to Figure 8. For example, computing and communication devices and systems may be provided in a wide variety of configurations, and Figure 8 does not limit the disclosure to any particular computing or communication device or system.

[0051] 9 illustrates one exemplary method 900 for data-efficient object detection on symbols of engineering schematics in accordance with the present disclosure. For ease of explanation, the method 900 may be described as being performed using the architecture 100 of FIG. 1, which may be implemented using one or more instances of device 800. However, the method 900 may be performed in any suitable system using any other suitable device.

[0052] As shown in FIG. 9 , in step 902, an input engineering schematic is obtained. This may include, for example, the processing unit 802 of the device 800 obtaining the input engineering schematic 102 from the storage unit 804 or other suitable source. In step 904, visual features are extracted based on the input engineering schematic. This may include, for example, the processing unit 802 of the device 800 using the feature extractor 104 to generate feature vectors or other extracted visual features associated with the input engineering schematic 102. In step 906, the visual features are processed using a feature pyramid network to generate feature maps associated with the input engineering schematic. This may include, for example, the processing unit 802 of the device 800 using the feature pyramid network 106 to process the feature vectors or other extracted visual features and generate feature maps 202 at different scales.

[0053] At step 908, the feature map is processed using a region proposal network to identify potential symbols and boundaries around the potential symbols. This may include, for example, the processing unit 802 of the device 800 using the region proposal network 108 to identify boundaries 204 around potential symbols in the input engineering schematic 102 based on the feature map 202. As a particular example, the region proposal network 108 may perform bounding box regression to identify boundaries 204 around potential symbols in the input engineering schematic 102. At step 910, the feature map and the identified boundaries are processed using a region of interest network to identify classifications of the identified symbols. This may include, for example, the processing unit 802 of the device 800 using the region of interest network 110 to generate classifications 206 that identify types of symbols in the input engineering schematic 102.

[0054] At step 912, the identified classifications and boundaries of the symbols in the input engineering schematic may be stored, output, or used. This may include the processing unit 802 of the device 800 storing the identified symbols and their boundaries 204 and classifications 206 in a structured format, such as by generating a digital representation of the input engineering schematic 102 using the identified boundaries 204 and the identified classifications 206. This may also include the processing unit 802 of the device 800 using the identified symbols and their boundaries 204 and classifications 206 to perform one or more functions related to a facility or other location that includes the equipment represented by the input engineering schematic 102. In general, the identified symbols and their boundaries 204 and classifications 206 may be used in any suitable manner and for any suitable purpose.

[0055] Although Figure 9 illustrates one example of a method 900 for data-efficient object detection on engineering schematic symbols, various modifications may be made to Figure 9. For example, although shown as a series of steps, various steps in Figure 9 may overlap, may be performed in parallel, may be performed in a different order, or may be performed any number of times.

[0056] 10 illustrates one exemplary method 1000 for training data augmentation in accordance with the present disclosure. For ease of explanation, the method 1000 may be described as being performed using the architecture 100 of FIG. 1, which may be implemented using one or more instances of device 800. However, the method 1000 may be performed using any other suitable device and in any suitable system.

[0057] 10, in step 1002, one or more training engineering schematics are obtained. This may include, for example, a processing unit 802 of a device 800 obtaining one or more input engineering schematics 102 from a storage unit 804 or other suitable source. In some instances, each input engineering schematic 102 may have known symbols with known boundaries 204 and known classifications 206.

[0058] Using one technique for data augmentation, at least one training engineering schematic may be resized and cropped to generate first augmented training data, in step 1004, and one or more modifications may optionally be applied to the cropped regions of the training engineering schematic. This may include, for example, processing unit 802 of device 800 rescaling and cropping each of one or more engineering schematics 500 to generate one or more regions 502, and applying one or more modifications to regions 502 to generate one or more new schematics 504.

[0059] Using another technique for data augmentation, in step 1006, one or more symbols may be cropped, optionally modified, and inserted into the context of one or more training engineering schematics to generate second augmented training data. This may include, for example, processing unit 802 of device 800 cropping one or more symbols from one or more engineering schematics 600, optionally modifying one or more cropped symbols, and inserting one or more symbols into one or more engineering schematics (such as in a non-overlapping manner) to generate one or more new schematics 602.

[0060] Using yet another technique for data augmentation, one or more symbols may be selected from at least one symbol library, optionally modified, and inserted into the context of one or more training engineering schematics to generate third augmented training data, step 1008. This may include, for example, the processing unit 802 of the device 800 selecting symbols from one or more libraries 700, optionally modifying one or more of the selected symbols, and inserting one or more symbols into one or more engineering schematics to generate one or more new schematics 702.

[0061] At step 1010, one or more machine learning models may be trained using the one or more training engineering schematics and the augmented training data. This may include, for example, the processing unit 802 of the device 800 performing backpropagation or other training techniques to modify gradients and weights of the one or more machine learning models (such as one or more models implementing the region proposal network 108 and the region of interest network 110) to reduce a loss function (such as that defined above) below a threshold. Ideally, the one or more machine learning models may be trained to be more effective at identifying one or more characteristics of symbols in the input engineering schematic 102, such as symbol boundaries 204 and classifications 206, compared to training without the augmented training data.

[0062] Although Figure 10 illustrates one example of a method 1000 for training data augmentation, various modifications may be made to Figure 10. For example, although shown as a series of steps, the various steps of Figure 10 may overlap, may be performed in parallel, may be performed in a different order, or may be performed any number of times. Also, although three training data augmentation techniques are used in Figure 10, any given implementation may use one or any combination of the training data augmentation techniques.

[0063] Below, we describe an example embodiment of the present disclosure that performs data augmentation for training data of engineering schematics used with machine learning models, although other embodiments may be used in accordance with the teachings of the present disclosure.

[0064] In a first embodiment, a method includes obtaining training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The method also includes augmenting the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The method further includes training at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

[0065] In a second embodiment, an apparatus includes at least one processing device configured to obtain training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The at least one processing device is also configured to augment the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The at least one processing device is further configured to train at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

[0066] In a third embodiment, a non-transitory computer readable storage medium has computer readable program code stored thereon that, when executed, causes one or more processors to obtain training data including at least one training engineering schematic including a plurality of known symbols, where different ones of the known symbols represent different types of equipment. The medium also has computer readable program code stored thereon that, when executed, causes the one or more processors to augment the training data with at least one additional training engineering schematic including at least one composite schematic, where the at least one composite schematic includes additional known symbols. The medium further has computer readable program code stored thereon that, when executed, causes the one or more processors to train at least one machine learning model using the training data including the at least one training engineering schematic and the at least one additional training engineering schematic.

[0067] Any one of the following features, or any suitable combination, may be used with the first, second, or third embodiment. The at least one machine learning model may include (i) a region proposal network configured to identify boundaries around the known symbols and the additional known symbols, and (ii) a region of interest network configured to classify the known symbols and the additional known symbols into a plurality of classifications, where different ones of the classifications are associated with different types of symbols. The at least one machine learning model may be trained by jointly optimizing gradients or weights of the at least one machine learning model using a regression loss and a classification loss. The training data may be augmented with the at least one additional training engineering schematic by cropping a designated one of the at least one training engineering schematic to include a scaled region of the designated training engineering schematic, and modifying the scaled region of the designated training engineering schematic to generate one of the at least one composite schematic. The training data may be augmented with at least one additional training engineering schematic by cropping a designated one of the known symbols from a designated one of the at least one training engineering schematic and inserting the cropped designated known symbol into a background of the designated training engineering schematic or into a background of another of the at least one training engineering schematic to generate one of the at least one composite schematic. The training data may be augmented with at least one additional training engineering schematic by selecting one or more symbols from at least one symbol library and inserting one or more selected symbols from the at least one symbol library into a background of the at least one composite schematic.The at least one machine learning model may include multiple machine learning paths, and different ones of the machine learning paths may be trained to identify boundaries around symbols having different shapes and / or different aspect ratios. The at least one machine learning model may be trained by modifying gradients or weights in one or more of the machine learning paths using training data associated with one or more of the different shapes or aspect ratios without modifying gradients or weights in one or more other of the machine learning paths associated with the one or more other of the different shapes or aspect ratios.

[0068] In some embodiments, various functions described in this patent document are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that carry transitory electrical or other signals. Non-transitory computer readable medium includes media on which data can be stored permanently and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable storage devices.

[0069] It may be advantageous to explain definitions of certain words and phrases used throughout this specification. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof that are adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "include" and "comprise" and their derivatives mean inclusive without limitation. The term "or" is inclusive and / or. The phrase "associated with" and its derivatives may mean include, contained in, interconnected with, contain, included in, connected to, coupled with, in communication with, cooperate with, interleaved, juxtaposed, adjacent to, connected to, having, having a characteristic of, relating to, or the like. The phrases "at least one of" and "one or more of," when used in conjunction with a list of items, mean that different combinations of one or more of the listed items may be used and that only one item in the list may be required. For example, "at least one of: A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.

[0070] The statements in this disclosure should not be read as suggesting that any particular element, step, or function is an essential or critical element that must be included in the scope of the claims. The scope of the patented subject matter is defined solely by the allowed claims. Moreover, none of the claims invoke 35 U.S.C. § 112(f) with respect to any of the appended claims or claim elements, unless the precise words "means for" or "step for" are expressly used in a particular claim followed by a participle phrase that identifies the function. The use of terms such as (but not limited to) "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller" in the claims are understood and intended to refer to structures known to those skilled in the art, as further modified or enhanced by the features of the claims themselves, and are not intended to invoke 35 U.S.C. § 112(f).

[0071] While this disclosure has described certain embodiments and generally associated methods, modifications and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain the disclosure. Other modifications, substitutions, and alterations are also possible without departing from the spirit and scope of the disclosure, as defined by the following claims.

Claims

1. 1. A method comprising: obtaining training data including at least one training engineering schematic including a plurality of known symbols; different ones of the known symbols represent different types of equipment; Steps and augmenting the training data with at least one additional training engineering schematic including at least one composite schematic; the at least one composite schematic includes additional known symbols; Steps and training at least one machine learning model using the at least one training engineering schematic and the training data including the at least one additional training engineering schematic; A method comprising:

2. The at least one machine learning model: (i) a region proposal network configured to identify a boundary around the known symbol and the additional known symbol; (ii) a region of interest network configured to classify the known symbols and the additional known symbols into a plurality of classifications, different ones of the classifications being associated with different types of symbols; Training the at least one machine learning model comprises: using the regression loss and the classification loss to jointly optimize gradients or weights of the at least one machine learning model. The method of claim 1.

3. augmenting the training data with at least one additional training engineering schematic comprises: cropping a designated one of the at least one training engineering schematic, the designated one including a scaled region of the training engineering schematic; modifying the scaled region of the designated training engineering schematic to generate one of the at least one composite schematic; The method of claim 1 , comprising:

4. augmenting the training data with at least one additional training engineering schematic comprises: cropping a designated one of the known symbols from a designated one of the at least one training engineering schematic; inserting the cropped designated known symbol into a background of the designated training engineering schematic or into a background of another of the at least one training engineering schematic to generate one of the at least one composite schematic; The method of claim 1 , comprising:

5. augmenting the training data with at least one additional training engineering schematic comprises: selecting one or more symbols from at least one symbol library; inserting the one or more selected symbols from the at least one symbol library into a background of the at least one composite schematic; The method of claim 1 , comprising:

6. the at least one machine learning model includes a plurality of machine learning paths; Different ones of the machine learning paths are trained to identify boundaries around symbols having at least one of different shapes and different aspect ratios. The method of claim 1.

7. Training the at least one machine learning model comprises: using training data associated with one or more of the different shapes or aspect ratios to modify gradients or weights in one or more of the machine learning paths without modifying gradients or weights in other one or more of the machine learning paths associated with other one or more of the different shapes or aspect ratios; The method of claim 6, comprising:

8. 1. An apparatus comprising at least one processing unit, The at least one processing device comprises: acquiring training data including at least one training engineering schematic including a plurality of known symbols, different ones of the known symbols representing different types of equipment; augmenting the training data with at least one additional training engineering schematic including at least one composite schematic, the at least one composite schematic including additional known symbols; and training at least one machine learning model using the at least one training engineering schematic and the training data including the at least one additional training engineering schematic; The apparatus is configured to:

9. The at least one machine learning model: (i) a region proposal network configured to identify a boundary around the known symbol and the additional known symbol; (ii) a region of interest network configured to classify the known symbols and the additional known symbols into a plurality of classifications, different ones of the classifications being associated with different types of symbols; To train the at least one machine learning model, the at least one processing device comprises: jointly optimizing gradients or weights of the at least one machine learning model using the regression loss and the classification loss; The apparatus of claim 8 , configured as follows:

10. To augment the training data with the at least one additional training engineering schematic, the at least one processing device further comprises: cropping a designated one of the at least one training engineering schematic to include a scaled region of the designated training engineering schematic; and modifying the scaled region of the designated training engineering schematic to generate one of the at least one composite schematic; The apparatus of claim 8 , configured as follows:

11. To augment the training data with the at least one additional training engineering schematic, the at least one processing device further comprises: cropping a designated one of the known symbols from a designated one of the at least one training engineering schematic; and inserting the cropped designated known symbol into a background of the designated training engineering schematic or into a background of another of the at least one training engineering schematic to generate one of the at least one composite schematic. The apparatus of claim 8 , configured as follows:

12. To augment the training data with the at least one additional training engineering schematic, the at least one processing device further comprises: Selecting one or more symbols from at least one symbol library, and inserting the one or more selected symbols from the at least one symbol library into the background of the at least one composite schematic; The apparatus of claim 8 , configured as follows:

13. the at least one machine learning model includes a plurality of machine learning paths; Different ones of the machine learning paths are trained to identify boundaries around symbols having at least one of different shapes and different aspect ratios.

9. The apparatus of claim 8.

14. To train the at least one machine learning model, the at least one processing device comprises: using training data associated with one or more of the different shapes or aspect ratios to modify gradients or weights in one or more of the machine learning paths without modifying gradients or weights in other one or more of the machine learning paths associated with other one or more of the different shapes or aspect ratios; The apparatus of claim 13 , configured as follows:

15. A non-transitory computer readable storage medium having computer readable program code stored thereon, The computer readable program code, when executed, causes one or more processors to: acquiring training data including at least one training engineering schematic including a plurality of known symbols, different ones of the known symbols representing different types of equipment; augmenting the training data with at least one additional training engineering schematic including at least one composite schematic, the at least one composite schematic including additional known symbols; and training at least one machine learning model using the at least one training engineering schematic and the training data including the at least one additional training engineering schematic; A non-transitory computer-readable storage medium.

16. The at least one machine learning model: (i) a region proposal network configured to identify a boundary around the known symbol and the additional known symbol; (ii) a region of interest network configured to classify the known symbols and the additional known symbols into a plurality of classifications, different ones of the classifications being associated with different types of symbols; The computer readable program code that, when executed, causes the one or more processors to train the at least one machine learning model, comprises: When executed, the one or more processors are using the regression loss and the classification loss to jointly optimize gradients or weights of the at least one machine learning model; 16. The non-transitory computer-readable storage medium of claim 15.

17. The computer readable program code that, when executed, causes the one or more processors to augment the training data with the at least one additional training engineering schematic, comprises: When executed, the one or more processors are cropping a designated one of the at least one training engineering schematic to include a scaled region of the designated training engineering schematic; and modifying the scaled region of the designated training engineering schematic to generate one of the at least one composite schematic.

16. The non-transitory computer-readable storage medium of claim 15.

18. The computer readable program code that, when executed, causes the one or more processors to augment the training data with the at least one additional training engineering schematic, comprises: When executed, the one or more processors are cropping a designated one of the known symbols from a designated one of the at least one training engineering schematic; and inserting the cropped designated known symbol into a background of the designated training engineering schematic or into a background of another of the at least one training engineering schematic to generate one of the at least one composite schematic.

16. The non-transitory computer-readable storage medium of claim 15.

19. The computer readable program code that, when executed, causes the one or more processors to augment the training data with the at least one additional training engineering schematic, comprises: When executed, the one or more processors are Selecting one or more symbols from at least one symbol library, and inserting the one or more selected symbols from the at least one symbol library into a background of the at least one composite schematic.

16. The non-transitory computer-readable storage medium of claim 15.

20. the at least one machine learning model includes a plurality of machine learning paths; Different ones of the machine learning paths are trained to identify boundaries around symbols having at least one of different shapes and different aspect ratios.

16. The non-transitory computer-readable storage medium of claim 15.

21. The computer readable program code that, when executed, causes the one or more processors to train the at least one machine learning model, comprises: When executed, the one or more processors are using training data associated with one or more of the different shapes or aspect ratios to modify gradients or weights in one or more of the machine learning paths without modifying gradients or weights in other one or more of the machine learning paths associated with other one or more of the different shapes or aspect ratios; 21. The non-transitory computer-readable storage medium of claim 20.

Images