Synthetic data generation and building information model (BIM) feature extraction from floor plan drawings using machine learning

Abstract

A method and system provide the ability to generate and use synthetic data to extract elements from a floor plan drawing. Generate a room layout. Use the room descriptions to generate and place synthetic instances of symbol elements in each room. Obtain a floor plan drawing and pre-process it to determine a drawing area. Based on the synthetic data, detect symbols in the floor plan drawing. Also detect an orientation of the detected symbols. Based on the detected symbols and orientation, obtain and place building information model (BIM) elements in the floor plan drawing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. Section 119(e) of the following co-pending and commonly assigned U.S. Provisional Patent Applications, which are incorporated herein by reference:

[0003] Provisional Application Serial No. 62 / 937,049, filed on November 18, 2019, entitled “System to Extract BIM Elements from Floor Plan Drawing Using Machine Learning,” inventors are Simranjit SinghKohli, Graceline Caladiao Regala, Yan Fu, Manuel Martinez Alonso, Keith Alfaro, and Emmanuel Gallo, and attorney docket number 30566.0586USP1; and

[0004] Provisional application serial number 62 / 937,053, filed on November 18, 2019, titled “Synthetic Data Generation Method for Machine Learning Tasks on Floor Plan Drawing” and inventors are Simranjit Singh Kohli, Manuel Martinez Alonso, Keith Alfaro, Emmanuel Gallo, Yan Fu and Graceline Caladiao Regala, and is assigned attorney docket number 30566.0587USP1. Background Art 1. Technical Field

[0006] The present invention relates generally to Building Information Modeling (BIM), and in particular, to methods, apparatus, systems, and articles of manufacture for generating synthetic data and extracting BIM elements from floor plan drawings using machine learning.

[0007] 2. Description of Related Technology

[0008] (Note: This application references several different publications, as indicated throughout this specification by reference names / titles enclosed in parentheses, e.g., [Jones]. A list of these different publications, sorted by these reference names / titles, can be found below in the section entitled "References". Each of these publications is incorporated herein by reference.)

[0009] Automatic conversion of graphic documents to BIM drawings has been a popular research topic. Most prior art systems focus on raster-to-vector technology. However, many graphic documents are actually in the format of PDF (Portable Document Format) files, which contain richer parsing information. Although graphic drawings may be of different types, some of them have a consistent pattern that can be used to automatically convert into BIM elements.

[0010] An exemplary graphic drawing is an electrical design drawing. In the field of architectural design, floor plan electrical design tasks are often outsourced to professional electrical design companies. The outsourced company's delivery usually consists of PDF files. Although the PDF file contains vectorized information of electrical symbols, the symbols are not grouped together, and there is no semantic information on the vector image (i.e., PDF). Therefore, architectural designers must manually draft and recreate electrical symbols in a building information modeling (BIM) application (e.g., REVIT application available from the assignee of the present application) while following the electrical drawing PDF file.

[0011] To better understand the problems of the prior art, a further description of prior art systems and methods for generating floor plans may be useful.

[0012] There are some prior art methods for procedural modeling of floor plan generation. Such prior art systems can be used as candidates for floor plan outline generation methods, but they do not address the process of generating the interior design and / or electrical design of the floor plan. Such prior art systems also do not control the number of categories or match the design of a specific drawing type ([Camozzato][Lopes]).

[0013] Other prior art methods have been proposed, but are limited in that they do not solve the problem or provide an effective and comprehensive solution to the embodiments of the present invention. In this regard, Young ([Young]) captures MEP (mechanical, electrical, plumbing) symbols from PDF and provides a document processing platform that replaces the prior art systems and helps enterprises extract key information from their documents using artificial intelligence technology. Such systems can use industry-specific taxonomies and domain knowledge to extract the correct fields from scanned documents and analyze larger PDF format reports ([Spot Intelligence]). PDFtron ([PDFtron]) extracts tables and text from PDF files into XML and HTML by parsing the structure of PDF files. Pdfplumber ([PdfPlumber]) provides an open source implementation that can be used to extract text, lines, rectangles, and tables. Stahl ([Stahl]) uses deep learning and image analysis to create a more accurate PDF to text extraction tool. ScienceBeam ([ScienceBeam]) uses computer vision to extract PDF data using tools to analyze PDF structure and then convert PDF to xml files. However, these prior art systems fail to provide the advantages and capabilities of the embodiments of the present invention as described below.

[0014] In addition to the above, some existing techniques include PDF parsers that are related to PDF analysis but focus mainly on text, paragraph content and tables ([Rahul]), or can be based on scanned paper electrical drawings without using deep learning techniques [Sebastien])

[0015] Additional prior art relates to logic circuit diagrams, including symbol recognition in electrical diagrams using probabilistic graph matching [Groen] or structural recognition of perturbed symbols using discrete relaxation [Habacha]. For engineering drawings, prior art methods for symbol recognition primarily use hand-crafted rules or use rules based on domain knowledge (see [Collin], [Dori], [Ablameyko], [Dosch], and [Ablameyko2]). Other prior art relates to musical scores, as they have a standardized structure and representation, and include extracting stave lines and then recognizing notes using neural networks or feature vector distances (see [Anquetil], [Miyao], [Yadid-Pecht], [Armand], [Randriamahefa], and [Fahmy]). Prior art systems related to architectural drawings have many problems and / or limitations. For example, because there is no standardized representation / symbol that appears to be embedded in the document, it is difficult to separate segmentation from recognition. Therefore, pattern matching can be used (see [Llad], [Valveny], [Ah-Soon], and [Aoki]).

[0016] Some additional academic research involves logo recognition, which is based on extracting signatures from images in terms of contour coding or connected component annotation, and then matching unknown logos to models in a database using different types of distances or neural networks (see [Bunke], [Yu], and [Kasturi]). Alternative research has investigated formula recognition, which uses feature vectors to identify individual symbols and syntactic methods to verify the structure of formulas (see [Lavirotte], [Lee], and [Ramel]).

[0017] However, all of the above-mentioned prior art products / research fail to solve the problem of the present invention and / or provide the solution described herein. Summary of the invention

[0018] To reduce the tedious work of drawing design drawings, embodiments of the present invention provide a pipeline for automatically parsing design drawings and converting drawings into building information model (BIM) elements using deep learning technology. First, the system automatically identifies drawing areas to exclude certain information (e.g., captions, notes, and other areas) so that subsequent pipelines can focus recognition and modeling on drawing areas. Then, the system not only recognizes floor plan drawing (e.g., electrical) symbols, but also extracts the geometric and semantic information of the symbols, such as the symbol label, orientation, and size of the elements, so that they can be automatically placed in the BIM model (e.g., REVIT application) later.

[0019] In addition to the above, data scarcity and data privacy are major limitations of algorithms involving machine learning strategies. The quality and accuracy of the predictions of machine learning tasks are directly related to the size and relevance of the training data set on which the training is based. Therefore, the data set requires a large amount of training data / samples to successfully learn the target task. The number of categories in the data set needs to be balanced, and the training samples must be relevant. In an embodiment of the present invention, an innovative architecture generates a synthetic data set (e.g., used as the basis / foundation of a machine learning model). The architecture can be used for any machine learning task related to BIM drawings, and can solve problems caused by data scarcity and privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Referring now to the drawings, wherein like reference numerals designate corresponding parts throughout:

[0021] Figure 1 An example of a composite floor plan electrical design drawing generated in accordance with one or more embodiments of the present invention is shown;

[0022] Figure 2 shows a logic flow for generating a composite floor plan according to one or more embodiments of the present invention;

[0023] Figure 3 shows an exemplary layout / floor plan that has been generated / obtained according to one or more embodiments of the present invention;

[0024] Figure 4 According to one or more embodiments of the present invention, Figure 2 An overview of the algorithms and processed data flows;

[0025] Figure 5 shows a histogram showing the frequency distribution of different symbols that could potentially be produced by random selection;

[0026] Figure 6 shows a histogram obtained after balancing according to one or more embodiments of the present invention;

[0027] Figure 7 shows exemplary tiles that may be generated from the resulting generated composite floor plan 102 in accordance with one or more embodiments of the present invention;

[0028] Figure 8 shows a logic flow for generating floor plan drawings using machine learning according to one or more embodiments of the present invention;

[0029] Fig. 9 illustrates segmenting an entire graphic drawing page into different sections according to one or more embodiments of the present invention;

[0030] Fig. 10A A graphical drawing PDF file showing one or more embodiments of the present invention;

[0031] Fig. 10B shows grouping of vectorized elements according to one or more embodiments of the present invention;

[0032] Fig. 10C shows the resulting extracted vectorized features with bounding boxes according to one or more embodiments of the present invention;

[0033] Fig.11 shows an enlarged view of a tiled image in accordance with one or more embodiments of the present invention;

[0034] Fig.12 shows a prediction of an exemplary electrical symbol according to one or more embodiments of the present invention;

[0035] Fig.13 shows a resulting floor plan drawing with placed symbols displayed to a user in accordance with one or more embodiments of the present invention;

[0036] Fig.14 is an exemplary hardware and software environment for implementing one or more embodiments of the present invention; and

[0037] Fig.15 A typical distributed / cloud-based computer system according to one or more embodiments of the present invention is schematically illustrated. DETAILED DESCRIPTION

[0038] In the following description, reference is made to the accompanying drawings which form a part hereof and in which are shown by way of illustration several embodiments of the invention. It should be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

[0039] Composite floor plan drawing generation

[0040] Embodiments of the present invention may utilize model-based machine learning when attempting to extract features from drawings. However, the scarcity of data for training such machine learning tasks / models is problematic (e.g., for floor plan drawings or other types of drawings). More specifically, data scarcity is a major limitation of algorithms involving machine learning strategies. The quality and accuracy of predictions for machine learning tasks are directly related to the size and relevance of the training dataset on which the training is based. Therefore, a large amount of training sample data is required for the dataset to successfully learn the target task. In addition, the number of categories in the dataset needs to be balanced, and the training samples must be relevant.

[0041] Synthetic data generation for other machine learning applications may exist in the prior art. However, the prior art fails to address the issues associated with floor plan drawings. For example, [Cinnamon] U.S. Patent Application Publication 20190057520 generates 2D images by reprojecting 3D objects. However, [Cinnamon] is unable to recreate data that is closely related to customer data, nor is it able to control the number of categories present in the dataset. Additionally, as described above, there are some prior art procedural modeling methods for floor plan generation. While such methods may be used as candidates for floor plan outlines, they do not address the process of generating the interior design and / or electrical design of a floor plan. Furthermore, such methods are unable to control the number of categories or match the design of a specific drawing type.

[0042] To solve the problems of the prior art, embodiments of the present invention programmatically generate (synthetic) datasets. Such synthetic datasets may consist of any building floor plan elements (such as electrical symbols, HVAC, furniture, lighting, etc.). The output of the synthetic dataset generation framework may be a floor plan layout or image (e.g., with elements) in a vector format (e.g., 2D vector space). Figure 1 An example of a generated composite floor plan electrical design drawing 102 is shown in accordance with one or more embodiments of the present invention.

[0043] Figure 2 A logic flow for generating a composite floor plan according to one or more embodiments of the present invention is shown.

[0044] At step 202, the algorithm begins generating a room layout / floor plan in a 2D vector space. The room layout consists of a set of 2D positions corresponding to the start and end points of all walls and a set of room descriptions that define the semantics of the room. The room layout can be derived from an existing CAD drawing, generated exhaustively (all possible outcomes) [Per Galle], generated by generative design, or created by a machine learning algorithm such as a GAN (Generative Adversarial Network) [Zheng] or with shape grammars and reinforcement learning [Ruiz-Montiel]. In other words, at step 202, a room layout / floor plan is obtained for one or more rooms of a floor plan drawing (e.g., via one or more different methods). Furthermore, in the room layout, the room descriptions define the semantics of one or more of the rooms.

[0045] Figure 3An exemplary layout / floor plan that has been generated / obtained according to one or more embodiments of the present invention is shown. The room layout description of the layout / floor plan 302 can be stored as a JSON (JavaScript Object Notation) file. The file can consist of a set of floor plans indexed by the name of each floor plan (e.g., "scene_02205"). Each floor plan can have two blocks: (1) a set of all 2D points of the floor plan; and (2) a room description, which consists of an index of the 2D points and a semantic label for the room. Table A is a sample floor plan description:

[0046] Table A

[0047]

[0048] In Table A, the floor plan "Scene_02205" has first the indices of all points, followed by the semantic labels of the two (2) rooms: "Undefined" and "Exterior Wall" (where the numbers represent the index of each point about / in that room).

[0049] return Figure 2 , in step 204, symbol element information of the symbol element is obtained. The symbol element information consists of an enumeration of rooms in which the symbol element may appear. In this regard, symbols may generally appear in a limited subset of room types. For example, a bathtub may generally only be located in a room type / room marked as "bathroom" and not in a kitchen. The enumeration specifies / identifies the room type. In addition, the symbol may generally be located in a specific location in the room or may be constrained to a specific location (for example, a toilet or power plug is placed against a wall and is not generally located in the middle of a bathroom). The attributes / parameters of each symbol identify such attributes of the symbol and can be captured in a configuration file. Therefore, for each design, a configuration file can be used to store all parameters. The configuration file consists of several / multiple blocks. Each block is indexed with a unique identifier. Each block corresponds to the rendering of a different symbol element and the associated attributes of the element. Table B provides an exemplary "bathing" (i.e., bathtub) configuration file block that can be utilized according to one or more embodiments of the present invention.

[0050] Table B

[0051]

[0052] The configuration file identifies / is composed of the following elements:

[0053] the symbol itself (i.e. the "path" to the location of the symbol's vector image), consisting of lines, points, arcs, and possibly other 2D geometric shapes,

[0054] The scale (i.e., “size”) of the symbol / symbol element. The scale / size corresponds to the pixel size of the longest side,

[0055] The orientation relative to the defined geometry of the symbol / symbol element (i.e., the "angle"). The angle is the orientation relative to the original symbol,

[0056] Thickness of 2D geometries (not specified in Table B),

[0057] An enumeration / label of the rooms in which the symbol may appear (i.e., "Rooms"). In one or more embodiments, a "Room" may be a living room, kitchen, bathroom, bedroom, undefined, etc.

[0058] location within the room (attached to a "wall", in the "center" of the room, or "randomly" positioned) (i.e., "location"),

[0059] the maximum number of instances (i.e. “multiples”) of the symbol / symbol element that may appear in the room (this could be on one wall or on all walls),

[0060] Background attributes (i.e., "background") that identify whether a symbol / symbol element belongs to (or is part of) the background or foreground. The background and foreground may be painted different colors, and conflicts between elements may only occur for elements of the same type,

[0061] • The associated label of the symbol / symbol element / block (ie, the "symbol"). Multiple blocks may have the same label, and the label may be exported as part of the configuration file along with information about the bounding box of the label.

[0062] At step 206, based on the description of the room layout, the algorithm generates symbol element instances (also referred to as symbol elements) in the room (based on the room layout and symbol element information), and creates a style for the data set that can match the customer's design. As described above, the configuration file can store symbol element information.

[0063] At step 208, the algorithm generates a floor plan drawing (randomly) based on the instances (i.e., instances that match the descriptions stored by the configuration file). In this regard, each instance may be randomly calculated until the instance conforms to the symbol feature information / attributes. For example, the position of each feature will be randomly calculated until it meets the description parameters and avoids conflicts with other features. If the position of the feature needs to be attached to a wall, the feature is rotated to be perpendicular to the wall. In addition, because the categories may need to be balanced, the probability of occurrence of each feature may be maintained and updated each time the feature occurs. The features will be generated with possible data augmentation (such as, scale, cropping, color changes, and different line thicknesses).

[0064] In optional step 210, the algorithm may add additional information (e.g., some random text, titles, measurements, and links between certain elements) to improve the machine learning model. Other symbols that are not relevant to the detection but are similar in appearance to the target symbol may also be added as noise symbols so that the machine learning model can learn to distinguish between the target symbol and the noise symbols.

[0065] Figure 4 According to one or more embodiments of the present invention, Figure 2 In particular, the algorithm / data processing module 402 receives the room layout description 302 and the configuration file 404 as input, performs Figure 2 The steps of are performed and the resulting synthetic floor plan 102 is output. The output 102 may be a floor plan in vector format or an image. In addition, the parameters / symbols may be found manually or randomly and may be determined using a machine learning algorithm (such as a GAN) that determines whether the output design matches the customer data.

[0066] Balance symbol

[0067] Depending on the configuration of the symbols, if the symbols are randomly placed in the floor plan 102, it may cause unbalanced data problems. Figure 5 A histogram is shown that illustrates the frequency distribution of different symbols that could potentially be generated by random selection. As shown, the "Other" symbol has a significantly higher frequency than the frequencies of the remaining symbols (e.g., Data, TV2, TV1, Triple, Quadruple, GFI, Duplex, Floor Q, Speaker, and Special). Therefore, the floor plan drawing generation step 208 may also include determining that instances of symbol elements in the floor plan drawing are unbalanced.

[0068] To address this issue, the floor plan drawing may be triangulated (i.e., divided into one or more triangles), and random uniform sampling may then be performed within the triangles to ensure that symbols are evenly placed within the floor plan. Additionally, within the triangles, the density of symbols or the maximum number of symbols may be controlled to avoid overcrowding in one area. To ensure that the symbol occurrences in the synthetic dataset are balanced, a histogram of symbol instances may be maintained, and a cumulative distribution transform (CDT) may be used to control the probability of symbol occurrences. This balance means that if one type of symbol is over-represented (or under-represented), the weight of this type of symbol may be reduced (or increased), resulting in an increased (or decreased) chance of the under-represented symbol being selected during symbol placement. In other words, by adjusting the weight of the symbol type, the weight affects how many instances of the symbol type are placed in the floor plan drawing. Through these mechanisms, the symbols in the synthetic dataset are more balanced than if the symbols were placed purely randomly in the floor plan. Additionally, a balanced dataset may also result in improved / increased object detection accuracy. Figure 6 The resulting histogram after balancing is shown in accordance with one or more embodiments of the present invention.

[0069] Tile-based processing

[0070] In one or more embodiments of the invention, rather than processing the entire dataset at once, the floor plans may be broken down into tiles so that the machine learning model is trained one tile at a time. Figure 7 102. An exemplary tile that may be generated from the resulting generated composite floor plan 102 according to one or more embodiments of the present invention is shown. Table C shows the corresponding CSV file, which represents the symbols in one tile "Scene_00000" (identified in the "Image Name" column):

[0071] Table C

[0072]

[0073]

[0074] As shown, the model can be trained one tile at a time. Alternatively (or additionally), all tiles can be processed in parallel. Different values ​​provide the position / location (xmin, xmax, ymin, ymax, xposition, and yposition) and orientation of the symbol in the row within the tile.

[0075] Workflow for generating floor plan drawings using machine learning

[0076] As mentioned above, in the field of architectural design, floor plan design tasks for specific fields (e.g., electrical) are often outsourced to specialized third parties (e.g., electrical design companies). The deliverables from the outsourcing company are usually PDF files. Although the PDF file contains vectorized information of the drawing (e.g., electrical) symbols, the symbols are not grouped together, and there is no semantic information on the vector map. Architects must recreate the design symbols (in 3D building information modeling applications) according to the drawing PDF files of the outsourcing company during drawing. To reduce the tedious work of design drawing, an embodiment of the present invention utilizes a pipeline that uses deep learning technology to automatically parse the drawing PDF files and convert the drawings into BIM elements.

[0077] In the workflow, the system first automatically identifies drawing areas to exclude certain information (e.g., captions, notes, and other areas) so that the pipeline can focus on identifying and modeling drawing areas. Then, the implementation not only recognizes floor plan drawing symbols, but also extracts the geometric and semantic information of the symbols, such as symbol labels, orientation, and size of symbol elements, for later automatic placement via BIM applications.

[0078] Figure 8 1 shows a logical flow for generating floor plan drawings using machine learning according to one or more embodiments of the present invention. Steps 802 to 808 involve the generation and use of synthetic data as described above. In particular, the synthetic data set generator step 802 performs Figure 2 Steps 202 to 210 are performed to generate synthetic data.

[0079] At step 804, an object detection machine learning (ML) model is trained based on the synthetic data, and the ML model / inference graph may be stored 806. In addition, based on the symbolic labels / information updated at step 808, the model may be retrained / updated.

[0080] The workflow for parsing a design drawing begins at step 810 .

[0081] Step 812, prepare / preprocess the PDF drawing. Because the subsequent steps of the workflow are based on object detection in the image, the input PDF file may be preprocessed to remove non-interesting parts of the drawing and rasterize the PDF into an image for further processing (i.e., the PDF file is converted into a raster image that may be in the same format as the above-mentioned composite floor plan drawing). Therefore, steps 812 to 814 provide for obtaining a raster image of the floor plan drawing, and may include one or more of the following steps:

[0082] Parse PDF content and remove all text information;

[0083] Remove background grid lines using a rule-based approach;

[0084] Use color information to remove background features; and

[0085] Rasterize the PDF file into an image using the aspect ratio information to determine the target resolution.

[0086] Automatically identify drawing areas by parsing PDF content

[0087] In steps 812 to 814, the (electrical) drawing area to be checked is extracted / determined. In this regard, the drawing area can be automatically / autonomously identified / determined / extracted by parsing the PDF content. As explained herein, although the drawings and text may refer to electrical drawing processing, embodiments of the present invention are not limited to electrical drawings and may include processing of any type of design drawings with symbols.

[0088] In design drawings, there may be some fixed patterns on the layout of the drawings, such as borders, captions, notes, titles, and actual drawing areas. The embodiments of the present invention are only interested in the drawing areas. Therefore, to focus on the drawing areas and reduce computational overhead, a machine learning model can be trained to segment the rasterized PDF image into multiple segments, and only segments with specific patterns that reflect the desired drawing area / drawing type (e.g., electrical drawings) will be passed through the pipeline for processing / recognition. Fig. 9 The entire graphic drawing page is divided into different sections according to one or more embodiments of the present invention. As shown, the drawing 900 has been divided into a border line 902, a drawing area 904, a title area 906, a note area 908 and a description text area 910. Only the drawing area 904 will be passed to the pipeline for further processing. The remaining segmented areas 902 and 906 to 910 are filtered out from the processing.

[0089] To summarize the step 814 of determining / extracting drawing regions, an ML model may be used to segment the raster image into a plurality of sections, and the ML model identifies a fixed pattern of the layout of the floor plan drawing. Thereafter, one or more sections are selected that include / define the design drawing region.

[0090] Group elements in the drawing area of ​​the area of ​​interest

[0091] Unlike rasterized images, PDF contains more meaningful information. By parsing the PDF file, elements such as text, lines, arcs, and rectangles can be obtained from the PDF file. After identifying the drawing area 904, the vectorized elements in the area are extracted. The vectorized elements are grouped to form candidate areas for symbol recognition (i.e., candidate areas of candidate elements). Candidates / candidate elements can be filtered out by analyzing (i.e., based on) the size of their bounding boxes. FIG. 10A to FIG. 10CAn exemplary group extraction from a graphic drawing PDF file according to one or more embodiments of the present invention is shown. Each group may be a different color. In this regard, Fig. 10A A PDF file showing the graphic drawings is shown. Fig. 10B , vectorized elements 1002, 1004, 1006, 1008, etc. are grouped. Fig. 10C The resulting extracted vectorized features are shown with bounding boxes 1110 (only some bounding boxes are shown for illustration purposes).

[0092] Object recognition and classification using models trained with synthetic drawings

[0093] return Figure 8 , using the synthetic dataset method described above, a synthetic floor plan design drawing dataset (e.g., stored in the inference graph 806) can be generated / obtained using the symbol library and the synthetic floor plan dataset. Because the design drawings are generated by a program, the ground truth of the labels and the positions of the symbols are known, thereby reducing the work for tedious and erroneous manual annotation processes. In other words, the synthetic symbol labels, synthetic symbol positions, and synthetic symbol orientations of the synthetic data in the synthetic floor plan design drawing dataset are known.

[0094] Typically, electrical design drawings are large, but popular object detection models only take small images as input, such as 224*224. As described above, in order to recognize small symbols in a large drawing, embodiments of the present invention can tile the synthesized electrical drawing image into small tiles of a fixed size (i.e., such as Figure 7 ), so that when the image is scaled to 224*224 or other low-resolution sizes, the symbols within the tile can be recognized. Moreover, to ensure that symbols located at the tile borders can also be detected, the implementation scheme can ensure that there is overlap between adjacent tiles. The output of the object detection model includes the coordinates of the bounding box, the category / label of the symbol, and the confidence score of the detection. Fig.11 A tiled small image with a tile size of 300*300 (ie, Figure 7 Such tiles can be used for object detection model training as described herein. Thus, the synthetic floor plan design drawing dataset can be tiled into multiple tiles, each of which is processed independently (and possibly in parallel).

[0095] Electrical symbol detection using trained object detection models

[0096] At step 816, objects in the PDF drawing plan are detected. As described above, symbol detection will only be applied to the design (e.g., electrical) drawing area 904. To ensure successful detection of the symbols of the target drawing, it is necessary to ensure that the size of the symbols in the image to be detected is close to the symbol size shown in the training data set. Therefore, the rasterized image will also be tiled into small tiles with overlap. At step 818, the trained object detection model (i.e., ML model / inference graph 806) is used to traverse all small tiles to detect all symbols (i.e., perform object detection with the ML model).

[0097] As described above, vector features in the graphic drawing have been extracted as regions of interest (ie, at step 820). Symbols generated by the object detection model that do not overlap the regions of interest will be filtered out. Detections with too low confidence scores may also be filtered out.

[0098] In addition, as part of object detection 816, detection (bounding) boxes may be merged by retaining only the most reliable detections (boxes) to ensure that there are no overlapping detected symbols in the result. Because the drawing area is cropped from the original PDF file (i.e., the drawing area is extracted at step 814), the symbol detection results also need to be mapped back to the original PDF using the scale and offset information of the cropped information.

[0099] In view of the above, step 816 includes detecting symbols represented by the vectorized elements (wherein the symbols / symbol representations include / consist of symbol labels) based on the vectorized elements and the synthetic floor plan design drawing dataset. In addition, a confidence level of the symbol may be determined and evaluated to determine whether the confidence level is above a threshold confidence level / below a threshold confidence level / within a threshold confidence level. Symbols with confidence levels below the threshold are filtered out, while those symbols with confidence levels above the threshold (i.e., second symbols) are retained for further processing.

[0100] Determine the orientation of the symbol

[0101] The object detection model (generated by object detection 820) only gives the size and type of the bounding box, but still lacks the orientation of the symbol, which makes automatic symbol placement difficult. Step 822 provides for determining the orientation (based on the synthetic floor plan design drawing dataset).

[0102] From object detection 816, there are already many symbol instances within the floor plan images generated by the synthetic dataset (in the reasoning graph 806) and the orientation of the synthetic symbols is known, and this orientation information can be used for orientation information learning. Therefore, at step 830, with the known symbol orientation (in the synthetic floor plan design drawing dataset), another machine learning model 832 (i.e., a symbol classification and orientation ML model) is trained to predict the orientation of the detected symbols. The training of step 830 utilizes the symbols from the symbol legend 828 to generate the symbol classification and orientation ML model, which is then stored in the database / reasoning graph 832 and accessed via it.

[0103] Once the model has been generated / trained, step 822 may be performed to actually use the model to determine the orientation of the symbol in the drawing. At step 834, because the orientation of a symbol is typically aligned with the wall direction, the symbol has a limited orientation, such as four (4) directions (left, right, up, down) or a more detailed 360 degree orientation. Therefore, it is sufficient to use the classification model (at step 834) as well as the object orientation model (at step 836) for symbol orientation prediction. For example, the nearest wall of the detected symbol may also be queried in the floor plan drawing, and the direction of the wall may be used to further validate the predicted orientation. Thus, at step 822, the orientation of the object symbol instance is determined based on the ML model trained in step 830.

[0104] Both the object detection of step 816 and the object orientation of step 822 are based on one or more ML models (e.g., a detection ML model and a symbol classification and orientation ML model [also referred to as a symbol orientation classification model]). However, such ML models may produce outputs with different confidence levels. For example, an ML model may detect a particular symbol or orientation with a 25% confidence / accuracy level or another symbol with a 75% confidence / accuracy level (e.g., based on unknown graphics, missing data, or other problems during detection / orientation).

[0105] At step 838, BIM symbol elements may be filtered. Such filtering may be based on domain knowledge and / or errors / low confidence in the detection / orientation outputs. For example, domain knowledge may be used to determine that it does not make sense if a bathtub is located in a kitchen or a duplex / light switch is not attached to the wall. Thus, at step 838, such erroneous or low confidence predictions may be filtered out. In this regard, confidence threshold levels may be defined and used to determine a tolerable confidence / accuracy level (e.g., a user may adjust the threshold level or the threshold level may be predefined within the system). Notably, the confidence thresholds may be set at multiple locations (e.g., at Figure 8 Filter step 838 is performed at the current position in the process, or after performing other steps.

[0106] User interaction for providing feedback and improving object detection models

[0107] After the orientation is determined (i.e., at step 822) and the symbol element filtering is performed at step 838, user interaction may be provided at step 840 to obtain feedback and improve the object detection model. In this regard, a floor plan drawing with placed BIM elements may be presented to the user. For example, the filtered and detected symbols may be presented to the user in different colors to obtain different confidence levels. Then, user feedback is received. In addition, the user may adjust the confidence threshold to filter out some detections. Based on the user feedback at step 840, the label, orientation, or other information may be corrected as needed at step 842. In this regard, the user may also fine-tune the bounding box of the detected symbol and correct the incorrect label of some symbols. The user feedback may also update the symbol orientation. The user's feedback (i.e., the updated symbol label 808 and the updated symbol orientation 848) may be considered as ground truth and may be used to retrain the object detection model (i.e., at step 804) and the symbol orientation classification model (i.e., at step 830).

[0108] After determining the orientation, at step 844 , BIM elements (eg, electrical BIM elements) are automatically / autonomously extracted / acquired based on the symbolic object tags. Fig.12 A prediction of an exemplary electrical symbol with four (4) labels (also referred to as symbol orientation classification) is shown in accordance with one or more embodiments of the present invention. As shown, the four (4) labels are label=0, label=1, label=2, and label=3. The acquisition step 844 of the BIM elements essentially provides for acquiring BIM 3D elements corresponding to the detected, oriented, filtered, and adjusted (i.e., at steps 816, 822, 838, and 842) 2D symbols. In this regard, based on the classification of the symbol (determined at step 834), there is a one-to-one (1-to-1) mapping to the BIM elements of that category. Therefore, based on the symbol label / classification of step 844, the appropriate BIM elements can be acquired.

[0109] At step 846, the (electrical) symbols are automatically / autonomously placed in the floor plan drawing (eg, based on the extracted size and orientation information). Fig.13A resulting floor plan drawing with placed symbols is shown displayed to a user in accordance with one or more embodiments of the present invention. As shown, adjacent to the graphic / icon of each placed symbol is a label for the symbol and the accuracy / confidence level for the symbol (e.g., duplex: 99% 1302, data 89% 1304, etc.). Labels for different categories of elements / symbols may be displayed in different colors (or in distinguishable patterns) (e.g., duplex may be displayed in green, while data may be displayed in light blue). Additional user feedback 840 may also be obtained at this stage and used to retrain the model as described above.

[0110] Hardware Environment

[0111] Fig.14 1400 (referred to as a computer-implemented system and / or a computer-implemented method) for implementing one or more embodiments of the present invention. The hardware and software environment includes a computer 1402 and may include peripheral devices. The computer 1402 may be a user / client computer, a server computer, or may be a database computer. The computer 1402 includes a hardware processor 1404A and / or a dedicated hardware processor 1404B (hereinafter collectively referred to as a processor 1404, alternatively) and a memory 1406, such as a random access memory (RAM). The computer 1402 may be coupled to and / or integrally formed with other devices, including input / output (I / O) devices, such as a keyboard 1414, a cursor control device 1416 (e.g., a mouse, a pointing device, a pen and tablet, a touch screen, a multi-touch device, etc.), and a printer 1428. In one or more embodiments, the computer 1402 may be coupled to or may include a portable or media viewing / listening device 1432 (e.g., an MP3 player, an iPod, a NOOK, a portable digital video player, a cellular device, a personal digital assistant, etc.). In another embodiment, the computer 1402 may include a multi-touch device, a mobile phone, a gaming system, an Internet-enabled television, a television set-top box, or other Internet-enabled device executing on a variety of platforms and operating systems.

[0112] In one embodiment, computer 1402 operates by hardware processor 1404A executing instructions defined by computer program 1410 (e.g., a computer-aided design [CAD] application) under the control of operating system 1408. Computer program 1410 and / or operating system 1408 may be stored in memory 1406 and may interface with users and / or other devices to accept input and commands and, based on such input and commands and the instructions defined by computer program 1410 and operating system 1408, provide output and results.

[0113] The output / result may be presented on the display 1422 or provided to another device for presentation or further processing or action. In one embodiment, the display 1422 includes a liquid crystal display (LCD) having a plurality of individually addressable liquid crystals. Alternatively, the display 1422 may include a light emitting diode (LED) display having a red, green, and blue diode cluster driven together to form a full-color pixel. Each liquid crystal or pixel of the display 1422 changes to an opaque or translucent state in response to the processor 1404 according to the data or information generated by applying the instructions of the computer program 1410 and / or the operating system 1408 to the input and command to form a part of the image on the display. The image may be provided by a graphical user interface (GUI) module 1418. Although the GUI module 1418 is depicted as a separate module, the instructions for executing the GUI functions may reside in or be distributed in the operating system 1408, the computer program 1410, or be implemented with a dedicated memory and processor.

[0114] In one or more embodiments, the display 1422 is integrally formed with / integrated into the computer 1402 and includes a multi-touch device having a touch sensing surface (e.g., a track pad or touch screen) that is capable of recognizing the presence of two or more points of contact with the surface. Examples of multi-touch devices include mobile devices (e.g., IPHONE, NEXUS S, DROID devices, etc.), tablet computers (e.g., IPAD, HP TOUCHPAD, SURFACE devices, etc.), portable / handheld game / music / video player / console devices (e.g., IPOD TOUCH, MP3 players, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc.), touch tables and walls (e.g., where images are projected through acrylic and / or glass and then illuminated with LED backlights).

[0115] Some or all of the operations performed by the computer 1402 according to the computer program 1410 instructions may be implemented in the special purpose processor 1404B. In this embodiment, some or all of the computer program 1410 instructions may be implemented via firmware instructions stored in a read-only memory (ROM), a programmable read-only memory (PROM), or a flash memory in the special purpose processor 1404B or the memory 1406. The special purpose processor 1404B may also be hardwired by circuit design to perform some or all operations to implement the present invention. In addition, the special purpose processor 1404B may be a hybrid processor that includes a dedicated circuit for performing a subset of functions, and other circuits for performing more general functions such as responding to computer program 1410 instructions. In one embodiment, the special purpose processor 1404B is an application specific integrated circuit (ASIC).

[0116] The computer 1402 may also implement a compiler 1412 that allows an application or computer program 1410 written in a programming language such as C, C++, assembly, SQL, PYTHON, PROLOG, MATLAB, RUBY, RAILS, HASKELL, or other language to be translated into processor 1404 readable code. Alternatively, the compiler 1412 may be an interpreter that directly executes instruction / source code, translates source code into an intermediate representation that is executed, or executes stored precompiled code. Such source code may be written in a variety of programming languages ​​such as JAVA, JAVASCRIPT, PERL, BASIC, etc. Once completed, the application or computer program 1410 uses the relationships and logic generated using the compiler 1412 to access and manipulate data received from the I / O devices and stored in the memory 1406 of the computer 1402.

[0117] Computer 1402 also optionally includes external communication means, such as a modem, satellite link, Ethernet card, or other means for accepting input from other computers 1402 and providing output to the computer.

[0118] In one embodiment, the instructions implementing the operating system 1408, computer program 1410, and compiler 1412 are tangibly embodied in a non-transitory computer-readable medium (e.g., data storage device 1420), which may include one or more fixed or removable data storage devices, such as a zip drive, a floppy disk drive 1424, a hard drive, a CD-ROM drive, a tape drive, etc. In addition, the operating system 1408 and computer program 1410 consist of computer program 1410 instructions that, when accessed, read, and executed by the computer 1402, cause the computer 1402 to perform the steps necessary to implement and / or use the present invention or load the instruction program into the memory 1406, thereby creating a special data structure, thereby causing the computer 1402 to operate as a specially programmed computer to perform the method steps described herein. The computer program 1410 and / or operating instructions may also be tangibly embodied in the memory 1406 and / or the data communication device 1430, thereby making a computer program product or article of manufacture according to the present invention. For this purpose, the terms "article of manufacture," "program storage device," and "computer program product" as used herein are intended to encompass a computer program accessible from any computer-readable device or media.

[0119] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used with the computer 1402 .

[0120] Fig.15 A typical distributed / cloud-based computer system 1500 is schematically shown, which uses a network 1504 to connect a client computer 1502 to a server computer 1506. A typical combination of resources may include: a network 1504, which includes the Internet, a LAN (local area network), a WAN (wide area network), an SNA (system network architecture) network, etc.; a client 1502, which is a personal computer or workstation (such as Fig.14 ); and server 1506, which is a personal computer, workstation, minicomputer or mainframe computer (such as Fig.14 ). However, it may be noted that different networks, such as a cellular network (e.g., GSM [Global System for Mobile Communications] or other), a satellite-based network, or any other type of network, may be used to connect the client 1502 and the server 1506 according to embodiments of the present invention.

[0121] A network 1504, such as the Internet, connects the client 1502 to the server computer 1506. The network 1504 may utilize Ethernet, coaxial cable, wireless communication, radio frequency (RF), etc. to connect the client 1502 and the server 1506 and provide communication therebetween. In addition, in a cloud-based computing system, the client 1502, the server computer 1506, and users may share resources (e.g., storage devices, processors, applications, memory, infrastructure, etc.) in the client 1502 and the server computer 1506 across one or more networks. Resources may be shared by multiple users and may be dynamically reallocated based on demand. In this regard, cloud computing may be referred to as a model for supporting access to a shared pool of configurable computing resources.

[0122] The client 1502 can execute a client application or web browser and communicate with a server computer 1506 executing a web server 1510. Such a web browser is typically a program such as MICROSOFT INTERNET EXPLORER / EDGE, MOZILLAFIREFOX, OPERA, APPLE SAFARI, GOOGLE CHROME, etc. In addition, the software executed on the client 1502 can be downloaded from the server computer 1506 to the client computer 1502 and installed as a plug-in or ACTIVEX control of the web browser. Therefore, the client 1502 can use the ACTIVEX component / component object model (COM) or distributed COM (DCOM) component to provide a user interface on the display of the client 1502. The web server 1510 is typically a program such as MICROSOFT'S INTERNET INFORMATION SERVER, etc.

[0123] Web server 1510 may host an Active Server Page (ASP) or Internet Server Application Programming Interface (ISAPI) application 1512 that may be executing a script. The script calls an object (called a business object) that executes the business logic. The business object then manipulates the data in a database 1516 through a database management system (DBMS) 1514. Alternatively, the database 1516 may be part of the client 1502 or directly connected to the client, rather than communicating / obtaining information from the database 1516 across the network 1504. When developers encapsulate business functionality into objects, the system may be referred to as a component object model (COM) system. Therefore, the script executed on the web server 1510 (and / or application 1512) calls the COM object that implements the business logic. In addition, server 1506 may utilize MICROSOFT'S TRANSACTION SERVER (MTS) to access the required data stored in the database 1516 via interfaces such as ADO (Active Data Objects), OLE DB (Object Linking and Embedding Database), or ODBC (Open Database Connectivity).

[0124] In general, these components 1500 to 1516 include logic and / or data embodied in or retrievable from a device, medium, signal, or carrier (e.g., a data storage device, a data communication device, a remote computer, or a device coupled to a computer via a network or via another data communication device, etc.) Further, this logic and / or data, when read, executed, and / or interpreted, causes the steps necessary to implement and / or use the present invention to be performed.

[0125] Although the terms "user computer," "client computer," and / or "server computer" are referred to herein, it should be understood that such computers 1502 and 1506 may be interchangeable and may also include thin client devices with limited or full processing capabilities, portable devices (such as cell phones), notebook computers, pocket computers, multi-touch devices, and / or any other device with suitable processing, communication, and input / output capabilities.

[0126] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used with computers 1502 and 1506. Embodiments of the present invention are implemented as a software / CAD application on a client 1502 or server computer 1506. Furthermore, as described above, the client 1502 or server computer 1506 may comprise a thin client device or a portable device with a multi-touch based display.

[0127] in conclusion

[0128] The description of the preferred embodiment of the present invention ends here. Some alternative embodiments for implementing the present invention are described below. For example, any type of computer (such as, a mainframe computer, a minicomputer or a personal computer) or computer configuration (such as, a time-sharing mainframe computer, a local area network or an independent personal computer) can be used together with the present invention.

[0129] The foregoing description of the preferred embodiments of the present invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. It is intended that the scope of the present invention is not limited by this specific embodiment, but is limited by the appended claims.

[0130] References

[0131] [Per Galle]Per Galle. "An algorithm for exhaustive generation ofbuilding floor plans". In: Communications of the ACM 24.12(1981), pp.813–825.

[0132] [Zheng]Hao Zheng,Weixin Huang.2018. "Architectural DrawingsRecognition and Generation through Machine Learning".Cambridge,MA,ACADIA.

[0133] [Ruiz-Montiel]Manuela Ruiz-Montiel, Javier Boned, Juan Gavilanes, Eduardo Jiménez, Lawrence Mandow, José-Luis Pérez-de-la-Cruz: Design with shapegrammars and reinforcement learning. Advanced Engineering Informatics 27(2):230-245(2013).

[0134] [Camozzato] Camozzato, Daniel. "A METHOD FOR GROWTH-BASED PROCEDURALFLOOR PLAN GENERATION." (2015).

[0135] [Lopes]Lopes,Tutenel,Smelik,deKraker,and Bidarra“A Constrained GrowthMethod for Procedural Floor Plan Generation”,Proceedings of GAME-ON 2010(Nov.2010).

[0136] [Young]www.linkedin.com / posts / brett-young-sf_quantitytakeoffs-machinelearning-mep-activity-6584839340420149249-IM_Z LinkedIn Video by BrettYoung(2019).

[0137] [Spot Intelligence]www.spotintelligence.com.

[0138] [PDFtron]“Extract Tables from Your PDFs”,PDFTron SDK,2020.

[0139] [PDFplumber]“pdfplumber”,Github 2020.

[0140] [Stahl]Stahl,Christopher G.,Young,Steven R.,Herrmannova,Drahomira,Patton,Robert M.,and Wells,Jack C.DeepPDF:A Deep Learning Approach toExtracting Text from PDFs.United States:N.p.,2018.Web.

[0141] [Ecer]Ecer,Daniel,and Maciocci,Giuliano.ScienceBeam–using computervision to extract PDF data.(August 4,2017).

[0142] [Rahul]Rahul,R.,Paliwal,S.,Sharma M.,and Vig,L.Automatic InformationExtraction from Piping and Instrumentation Diagrams(Jan 28,2019).

[0143] [Sebastien]Sébastien Adam,Jean-Marc Ogier,Claude Cariou,Rémy Mullot,Jacques Labiche,et al..Symbol and character recognition:application toengineering drawings.IJDAR,2000,3(2),pp.89-101.hal00439443.

[0144] [Groen]Groen,F.,Sanderson,A.,Schlag,F.:Symbol recognition inelectrical diagrams using a probabilistic graph matching.PRL 3(1985)343–350.

[0145] [Habacha]Habacha,A.:Structural recognition of disturbed symbols usingdiscrete relaxation.In:Proceedings of 1st.Int.Conf.on Document Analysis andRecognition.(1991)170–178 Saint Malo,France.

[0146] [Collin]Collin,S.,Colnet,D.:Syntactic analysis of technical drawingdimensions.Int.Journal of Pattern Recognition and Artificial Intelligence 8(1994)1131–1148.106,107,112,126.

[0147] [Dori]Dori,D.:Asyntactic / geometric approach to recognition ofdimensions in engineering machine drawings.Computer Vision,Graphics and ImageProcessing 47(1989)271–291.107,112,126.

[0148] [Ablameyko]Ablameyko,S.:An Introduction to Interpretation of GraphicImages.SPIE Optical Engineering Press(1997).

[0149] [Dosch]Dosch,P.,Masini,G.,Tombre,K.:Improving arc detection ingraphics recognition.In:Proceedings of 15th.Int.Conf.on PatternRecognition.Volume 2.(2000)243–246 Barcelona,Spain.106.

[0150] [Ablameyko2]Ablameyko,S.,Bereishik,V.,Frantskevich,O.,Homenko,M.,Paramonova,N.:Knowledge-based recognition of crosshatched areas inengineering drawings.In Amin,A.,Dori,D.,Pudil,P.,Freeman,H.,eds.:Advances inPattern Recognition.Vol.1451 of LNCS(1998)460–467.

[0151] [Anquetil]Anquetil,E.,Co¨uasnon,B.,Dambreville,F.:A symbol classifierable to reject wrong shapes for document recognition systems.In Chhabra,A.,Dori,D.,eds.:Graphics Recognition-Recent Advances.Springer,Berlin(2000)209–218 Vol.1941 of LNCS.

[0152] [Miyao]Miyao,H.,Nakano,Y.:Note symbol extraction for printed pianoscores using neural networks.IEICE Trans.Inf.and Syst.E79-D(1996)548–553.

[0153] [Yadid-Pecht]Yadid-Pecht,O.,Gerner,M.,Dvir,L.,Brutman,E.,Shimony,U.:Recognition of handwritten musical notes by a modified neocognitron.MachineVision and Applications 9(1996)65–72.107,110,128.

[0154] [Armand]Armand,J.:Musical score recognition:a hierarchical andrecursive approach.In:Proceedings of Second IAPR Int.Conf.on DocumentAnalysis and Recognition,ICDAR’93.(1993)906–909Tsukuba,Japan.

[0155] [Randriamahefa]Randriamahefa,R.,Cocquerez,J.,Fluhr,C.,P′epin,F.,Philipp,S.:Printed music recognition.In:Proceedings of Second IAPRInt.Conf.on Document Analysis and Recognition,ICDAR’93.(1993)898–901 Tsukuba,Japan.

[0156] [Fahmy]Fahmy,H.,Blonstein,D.:Agraph grammar programming style forrecognition of music notation.Machine Vision and Applications 6(1993)83–99.

[0157] [Llad]Llad′os,J.,S′anchez,G.,Mart′1,E.:A string-based method torecognize symbols and structural textures in architectural plans.In Tombre,K.,Chhabra,A.,eds.:Graphics Recognition:Algorithms and Systems.Springer,Berlin(1998)91–103 Vol.1389 of LNCS.

[0158] [Valveny]Valveny,E.,Mart′1,E.:Hand-drawn symbol recognition ingraphic documents using deformable template matching and a bayesianframework.In:Proceedings of 15th.Int.Conf.on Pattern Recognition.Volume 2.(2000)239–242 Barcelona,Spain.

[0159] [Ah-Soon]Ah-Soon,C.,Tombre,K.:Architectural symbol recognition usinga network of constraints.PRL 22(2001)231–248.

[0160] [Aoki]Aoki,Y.,Shio,A.,Arai,H.,Odaka,K.:A prototype system forinterpreting handsketched floor plans.In:Proceedings of 13th.Int.Conf.onPattern Recognition.(1996)747–751 Vienna,Austria.

[0161] [Bunke]Bunke,H.:Attributed programmed graph grammars and theirapplication to the schematic diagram interpretation.IEEE Trans.on PAMI 4(1982)574–582.

[0162] [Yu]Yu,Y.,Samal,A.,Seth,C.:A system for recognizing a large class ofengineering drawings.IEEE Trans.on PAMI 19(1997)868–890.

[0163] [Kasturi]Kasturi,R.,Bow,S.,El-Masri,W.,Shah,J.,Gattiker,J.,Mokate,U.:A system for interpretation of line drawings.IEEE Trans.on PAMI 12(1990)978–992.

[0164] [Lavirotte]Lavirotte,S.,Pottier,L.:Optical formula recognition.In:Proceedings of 4th Int.Conf.on Document Analysis and Recognition.(1997)357–361 Ulm,Germany.

[0165] [Lee]Lee,H.,Lee,M.:Understanding mathematical expressions usingprocedure-oriented transformation.PR 27(1994)447–457.

[0166] [Ramel]Ramel,J.,Boissier,G.,Emptoz,H.:A structural representationadapted to hand-written symbol recognition.In Chhabra,A.,Dori,D.,eds.:Graphics Recognition:Recent Advances.Springer-Verlag,Berlin(2000)228–237Vol.1941 of LNCS.

[0167] [Cinnamon]United States Patent Application PublicationNo.20190057520,filed on October 31,2017 with inventors Ian Cinnamon,BrunoBrasil Ferrari Faviero,and Simanta Gautam entitled“Generating Synthetic ImageData”.

Claims

1. A computer-implemented method for utilizing a composite floor plan drawing, comprising: generating a room layout of one or more rooms of the composite floor plan drawing, wherein the room layout includes a room description defining semantics of one or more of the rooms; obtaining symbol element information for a symbol element, wherein the symbol element information includes an enumeration of room types in which the symbol element can appear and locations within the one or more rooms in which the symbol element can appear; programmatically generating composite symbol data including instances of the symbol elements within one or more of the rooms, wherein the instances populate the room layout based on the room layout and the symbol element information; as well as The composite floor plan drawing is programmatically generated based on the composite symbol data and the room layout, wherein: The ground truth of the labels and the positions of the symbols in the synthetic floor plan drawing are known; training an object detection model using the synthetic floor plan drawing, wherein the ground truth is used for orientation information learning; and The object detection model is used to autonomously identify and extract symbol elements in different floor plan drawings, wherein: The object detection model segments the different floor plan drawings into a plurality of sections and identifies fixed patterns; The object detection model selects one or more of the plurality of sections that define a design drawing area; and The object detection model recognizes the symbol elements in selected one or more of the plurality of segments.

2. The computer-implemented method of claim 1, wherein the room layout comprises: A set of two-dimensional (2D) locations corresponding to the start and end points of all walls.

3. The computer-implemented method of claim 1 , wherein said generating an instance of said symbol element further comprises: A pattern of the data set is generated that can match one of the one or more designs, wherein for each of the one or more designs, the configuration file stores the symbol element information.

4. The computer-implemented method of claim 1, wherein: Storing the symbol element information in a configuration file; The configuration file includes a plurality of blocks; indexing each block in the plurality of blocks with a unique identifier; and Each block of the plurality of blocks corresponds to the rendering of a different symbol element.

5. The computer-implemented method of claim 4, wherein: Each block of the plurality of blocks includes the symbol element information; The symbol element information includes: the proportions of the elements of the symbol; the orientation of the symbol elements; The maximum number of instances; A background attribute that identifies whether the symbol element belongs to the background or the foreground; and The associated label of the symbol element.

6. The computer-implemented method of claim 1, wherein: The generating the composite floor plan drawing includes randomly generating the composite floor plan drawing by randomly calculating each instance until the instance conforms to the symbol element information.

7. The computer-implemented method of claim 6, further comprising: determining that said instance of a symbol element in said composite floor plan drawing is unbalanced; triangulating the composite floor plan drawing into one or more triangles; as well as The symbol elements are evenly placed in the composite floor plan drawing using random uniform sampling within the one or more triangles.

8. The computer-implemented method of claim 7, further comprising: Based on the determination that the instances of symbol elements are unbalanced, a weight of a symbol type is adjusted, wherein the weight affects how many instances of the symbol type are placed into the composite floor plan drawing.

9. The computer-implemented method of claim 1, further comprising: Random additional information is added to the synthetic floor plan drawings to improve the machine learning model.

10. A computer-implemented system for generating a composite floor plan drawing, comprising: (a) a computer having a memory; (b) a processor executing on said computer; (c) the memory storing a set of instructions, wherein the set of instructions, when executed by the processor, causes the processor to perform operations comprising: (i) generating a room layout of one or more rooms of the composite floor plan drawing, wherein The room layout includes a room description defining semantics of one or more of the rooms; (ii) obtaining symbol element information for a symbol element, wherein the symbol element information includes an enumeration of room types in which the symbol element can appear and locations within the one or more rooms in which the symbol element can appear; (iii) programmatically generating, within one or more of the rooms, composite symbol data comprising instances of the symbol elements, wherein the instances populate the room layout based on the room layout and the symbol element information; and (iv) programmatically generating the composite floor plan drawing based on the composite symbol data and the room layout, wherein: (A) the ground truth of the labels and the positions of the symbols in the synthetic floor plan drawing are known; (v) training an object detection model using the synthetic floor plan drawings, wherein the ground truth is used for orientation information learning; and (vi) autonomously identifying and extracting symbolic elements in different floor plan drawings using the object detection model, wherein: (A) the object detection model segments the different floor plan drawings into a plurality of segments and identifies fixed patterns; (B) the object detection model selects one or more of the plurality of segments that define a design drawing area; and (C) The object detection model recognizes the symbol elements in one or more selected segments of the plurality of segments.

11. The computer-implemented system of claim 10, wherein the room layout comprises: A set of two-dimensional (2D) locations corresponding to the start and end points of all walls.

12. The computer-implemented system of claim 10, wherein said generating an instance of said symbol element further comprises: A pattern of the data set is generated that can match one of the one or more designs, wherein for each of the one or more designs, the configuration file stores the symbol element information.

13. The computer implemented system of claim 10, wherein: Storing the symbol element information in a configuration file; The configuration file includes a plurality of blocks; indexing each block in the plurality of blocks with a unique identifier; and Each block of the plurality of blocks corresponds to the rendering of a different symbol element.

14. The computer implemented system of claim 13, wherein: Each block of the plurality of blocks includes the symbol element information; The symbol element information includes: the proportions of the elements of the symbol; the orientation of the symbol elements; The maximum number of instances; A background attribute that identifies whether the symbol element belongs to the background or the foreground; and The associated label of the symbol element.

15. The computer implemented system of claim 10, wherein: The generating the composite floor plan drawing includes randomly generating the composite floor plan drawing by randomly calculating each instance until the instance conforms to the symbol element information.

16. The computer implemented system of claim 15, wherein the operations further comprise: determining that said instance of a symbol element in said composite floor plan drawing is unbalanced; triangulating the composite floor plan drawing into one or more triangles; as well as The symbol elements are evenly placed in the composite floor plan drawing using random uniform sampling within the one or more triangles.

17. The computer implemented system of claim 16, wherein the operations further comprise: Based on the determination that the instances of symbol elements are unbalanced, a weight of a symbol type is adjusted, wherein the weight affects how many instances of the symbol type are placed into the composite floor plan drawing.

18. The computer implemented system of claim 10, wherein the operations further comprise: Random additional information is added to the synthetic floor plan drawings to improve the machine learning model.

Images