Machine learning techniques for extracting floor plan elements from architectural drawings

Abstract

One embodiment of the present invention proposes a technique for extracting data from architectural drawings. The technique includes performing one or more operations via one or more machine learning models to extract a first image of a floor plan region from the architectural drawings. The technique also includes generating boundary segmentation based on the first image of the floor plan region, wherein the boundary segmentation includes one or more boundary types for one or more portions of the floor plan region.

Description

Background of the Invention Technical Field

[0002] The embodiments disclosed herein relate generally to computer science and machine learning, and more specifically to machine learning techniques for extracting floor plan elements from architectural drawings. Background Technology

[0004] A typical architectural floor plan is a two-dimensional (2D) drawing of the physical layout of rooms, spaces, or other physical features of the structure. For example, an architectural floor plan may include scaled-down drawings showing views from above a room, a floor in the building, and / or the entire building. An architectural floor plan may include depictions of walls, windows, doorways, staircases, appliances, fixtures, materials, and / or other components or areas in the floor. An architectural floor plan may also include title blocks, labels, notes, or other textual annotations that identify or describe the floor plan and / or its various components.

[0005] Because floor plans illustrate the layout, relative sizes, and relationships between other physical features of rooms, spaces, or structures, they typically play a central role in the design, construction, or alteration of structures. For example, an architect or designer may generate and iteratively update one or more floor plans for a building under construction or renovation until they conform to client preferences and meet planning guidelines, building codes, and / or other requirements or constraints. In another example, floor plans can serve as the basis for designing furniture layouts, wiring systems, and other aspects of rooms or buildings.

[0006] While computer-based tools or programs can speed up the creation of floor plans, they are often stored or represented in ways that hinder the efficient retrieval and analysis of different types of information within them. For example, users can create or update floor plans by interacting with computer-aided design (CAD) programs to draw or resize lines representing walls and other external or internal boundaries in a building layout; add windows and doors along the boundaries; and / or drag and drop appliances, fixtures, furniture, and / or materials into the building layout. Users can also create architectural drawings that include floor plans and then add dimensions, notes, or other text-based annotations to them. Finally, users can store architectural drawings in a repository that includes a large number of other architectural documents. The problem is that a user, or another user, often has to manually browse and open documents in the repository to find a floor plan within the architectural drawings or identify other floor plans similar to those included in the stored architectural drawings.

[0007] To reduce the time and overhead associated with retrieving and analyzing floor plans, techniques have been developed to parse and extract information from images or documents containing floor plans. One method for extracting information from floor plans involves binarizing the image of the floor plan into black and white pixels, using line extraction techniques to extract boundary lines, and applying a set of rules to the boundary pixels to identify walls, windows, doors, and / or other features. However, a drawback of this approach is that a given set of rules can only be used to identify boundaries within the floor plan if the floor plan conforms to a specific depiction of walls, windows, doors, and / or other features based on that specific depiction. As a result, additional rules or sets of rules must be created to identify boundaries in the floor plan that otherwise depict these types of features or include different or other feature types.

[0008] Other methods for extracting information from floor plans involve using machine learning models to semantically segment the floor plan into walls, windows, doors, room types, and / or other features. However, these methods are limited to predicting a fixed set of features within the floor plan and cannot be generalized to consider other parts or features that might be outside the fixed set of features in the floor plan. These machine learning methods typically involve training the machine learning model to perform multiple tasks, which can increase the overall training and inference overhead. For example, a deep learning model may include two different neural network branches performing two different tasks. The first branch can use a set of features extracted from an input image of the floor plan to predict pixels in the image representing walls, doors, windows, and / or other boundaries. The second branch can use the same features and attention weights generated by the first branch to predict pixels representing bedrooms, bathrooms, living rooms, hallways, balconies, stairs, closets, and / or other room types in the floor plan. However, if the training data used for the deep learning model does not include labels for specific boundaries or room types, the model will be unable to predict those specific boundaries or room types. Furthermore, the multi-task nature of deep learning models typically increases the amount of training data required to train the deep learning model, requires additional training iterations to reach convergence, and increases resource consumption during the execution of the deep learning model.

[0009] As mentioned above, there is a need in the art for more efficient techniques to automatically access the features and elements of building floor plans when using computer-aided design programs. Summary of the Invention

[0010] One embodiment of the present invention proposes a technique for extracting data from architectural drawings. The technique includes performing one or more operations via one or more machine learning models to extract a first image of a floor plan region from the architectural drawings. The technique also includes generating boundary segmentation based on the first image of the floor plan region, wherein the boundary segmentation includes one or more boundary types for one or more portions of the floor plan region.

[0011] One technical advantage of the disclosed technology over existing technologies is that it allows for the efficient identification and extraction of one or more floor plans from a vast collection of stored architectural documents. Therefore, the disclosed technology is more scalable, efficient, and less resource-intensive than traditional methods that involve manually browsing, searching, or viewing the entire collection of stored architectural documents to identify and extract floor plans within those documents. Another technical advantage of the disclosed technology is the use of multiple smaller machine learning models to extract different types of semantic information from those floor plans. Thus, the disclosed technology is able to extract a wider variety of semantic information from floor plans compared to existing methods, and it also reduces the overhead of training and executing different machine learning models compared to conventional methods that involve training and executing a single, more complex machine learning model. These technical advantages provide one or more technical improvements over existing methods. Attached Figure Description

[0012] To gain a detailed understanding of the features described above in the various embodiments, reference can be made to several embodiments to describe the inventive concept briefly outlined above in more detail, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings only show typical embodiments of the inventive concept and should not be considered as limiting the scope, and that other equally effective embodiments exist.

[0013] Figure 1 A computing device configured to implement one or more aspects of various implementation schemes is shown.

[0014] Figure 2 Including according to various implementation schemes Figure 1 More detailed illustrations of the analysis engine, boundary segmentation engine, and room segmentation engine.

[0015] Figure 3 Exemplary images of floor plan areas extracted from architectural drawings according to various implementation schemes are shown.

[0016] Figure 4 Exemplary boundary segmentation of images for floor plan areas according to various embodiments is shown.

[0017] Figure 5A Exemplary vectorized boundary segmentation of images for floor plan areas according to various embodiments is shown.

[0018] Figure 5B The following are shown according to various implementation schemes. Figure 5A An example room segmentation generated by vectorized boundary segmentation.

[0019] Figure 6 A flowchart of the steps for automatically extracting data from architectural drawings according to various implementation schemes is presented. Detailed Implementation

[0020] In the following description, many specific details are set forth to provide a more thorough understanding of the various implementations. However, it will be apparent to those skilled in the art that the inventive concepts can be practiced without one or more of these specific details.

[0021] System Overview

[0022] Figure 1 A computing device 100 configured to implement one or more aspects of various embodiments is illustrated. The computing device 100 may be a desktop computer, laptop computer, smartphone, personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments of the present invention. The computing device 100 is configured to run an analytics engine 122, a boundary segmentation engine 124, and a room segmentation engine 126 residing in memory 116. It should be noted that the computing device described herein is illustrative, and any other technically feasible configuration falls within the scope of the invention. For example, multiple instances of the analytics engine 122, the boundary segmentation engine 124, and the room segmentation engine 126 may execute on a set of nodes in a distributed and / or cloud computing system to implement the functionality of the computing device 100. In another example, the analytics engine 122, the boundary segmentation engine 124, and / or the room segmentation engine 126 may be implemented by the same or different hardware and / or software components. In the third example, the various hardware and / or software components of the computing device 100 may be combined, rearranged, omitted, and / or replaced with other components that have similar or different functionalities.

[0023] In one embodiment, computing device 100 includes, but is not limited to, interconnects (buses) 112 connecting one or more processors 102, input / output (I / O) device interfaces 104 coupled to one or more input / output (I / O) devices 108, memory 116, storage devices 114, and network interfaces 106. Processor 102 can be any suitable processor implemented as a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), artificial intelligence (AI) accelerator, any other type of processing unit, or a combination of different processing units (such as a CPU configured to operate in conjunction with a GPU). Generally, processor 102 can be any technically feasible hardware unit capable of processing data and / or executing software applications. Furthermore, in the context of this disclosure, the computing elements shown in computing device 100 may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executing within a computing cloud.

[0024] In one embodiment, I / O device 108 includes means for receiving input, such as a keyboard, mouse, touchpad, and / or microphone, and means for providing output, such as a display device and / or speaker. Additionally, I / O device 108 may include means for receiving input and providing output, such as a touchscreen, a Universal Serial Bus (USB) port, etc. I / O device 108 may be configured to receive various types of input from end users of computing device 100 (e.g., designers) and also to provide various types of output to end users of computing device 100, such as displayed digital images or digital video or text. In some embodiments, one or more of I / O devices 108 are configured to couple computing device 100 to network 110.

[0025] In one implementation, network 110 is any technically feasible type of communication network that allows the exchange of data between computing device 100 and external entities or devices, such as a web server or another networked computing device. For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (Wi-Fi) network, and / or the Internet, etc.

[0026] In one embodiment, storage device 114 includes non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 may be stored in storage device 114 and loaded into memory 116 during execution.

[0027] In one embodiment, memory 116 includes a random access memory (RAM) module, a flash memory cell, or any other type of memory cell or a combination thereof. Processor 102, I / O device interface 104, and network interface 106 are configured to read data from memory 116 and write data to memory. Memory 116 includes various software programs executable by processor 102 and application data associated with said software programs, analytics engine 122, boundary segmentation engine 124, and room segmentation engine 126.

[0028] Analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 include the functionality to extract semantic information from architectural floor plans via a multi-stage process. More specifically, analysis engine 122 identifies architectural drawings including floor plans and performs image cropping of the floor plans from the architectural drawings. Boundary segmentation engine 124 generates boundary segments that assign labels representing various types of boundaries to pixels in each image of the floor plan. Room segmentation engine 126 converts the boundary segments into a vectorized form and uses the boundary segments to perform room segmentation, dividing the corresponding floor plan into regions representing different rooms in the floor plan.

[0029] In one or more embodiments, the analysis engine 122, the boundary segmentation engine 124, and / or the room segmentation engine 126 employ various machine learning and / or image processing techniques to perform floor plan cropping, boundary segmentation, boundary vectorization, and / or room segmentation. As described in further detail below, these techniques can be used to identify and analyze floor plans within large or diverse architectural documents, thereby reducing the manual and computational overhead associated with browsing and / or searching large architectural document repositories to obtain floor plans. These techniques can also be used to extract a wider variety of semantic information compared to conventional image processing or deep learning techniques used to identify floor plan elements within floor plans.

[0030] Machine learning techniques for building floor plan segmentation

[0031] Figure 2 Including according to various implementation schemes Figure 1 More detailed illustrations of the analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 are provided below. Each of these components is described in more detail below.

[0032] Analysis engine 122 performs preprocessing, which identifies architectural drawings 202, including floor plans, from a potentially large number of architectural drawings, documents, and / or other types of files. For example... Figure 2As shown, the analysis engine 122 can apply one or more filters 214 to identify a subset of files as architectural drawings. For example, the analysis engine 122 can retrieve the file set from a catalog, database, file system, and / or another repository owned, maintained, or used by the entity involved in creating or using the architectural drawings. The analysis engine 122 can also use a Page Description Language (PDL) interpreter, an Optical Character Recognition (OCR) tool, and / or another technology to extract text 230 from each file. The analysis engine 122 can use filters 214 to compare the extracted text 230 with a pre-defined list of keywords typically found in architectural drawings containing floor plans. If the text 230 from the file does not contain any keywords in the list, the analysis engine 122 can determine that the file is unlikely to contain a floor plan. If the text 230 from the file contains one or more keywords in the list, the analysis engine 122 can add the file to a set of “candidate” architectural drawings with a reasonable probability of containing a floor plan.

[0033] For each “candidate” architectural drawing identified by filter 214, analysis engine 122 inputs data related to one or more views 228 and / or text 230 in architectural drawing 202 into classifier 204. Classifier 204 then generates an output indicating the drawing type 210 of the input data. For example, analysis engine 122 and / or another component may train a multi-label convolutional neural network (CNN), decision tree, random forest, gradient boosting tree, logistic regression model, support vector machine, Bayesian network, ensemble model, hierarchical model, and / or implement classifier 204 as another type of machine learning model to predict a set of classes associated with the set of architectural drawings based on text 230 in the architectural drawings, images of individual views 228 or forms in the architectural drawings, and / or other representations of the architectural drawings. These classes may include drawing types of different types of floor plans, such as (but not limited to) floor plans of single-family homes, multi-family buildings, hospitals, offices, retail spaces, airports, museums, and / or schools. These classes may also, or alternatively, include a “general” class representing any type of floor plan, one or more classes representing non-floor plan drawing types, and so on. For example (elevations, sections, projections, etc.), and / or the "default" class for documents that do not include any type of architectural drawings.

[0034] Continuing the example above, after training classifier 204, analysis engine 122 can input new forms, views, text 230, and / or another part of candidate architectural drawings 202 into classifier 204, and classifier 204 can output a set of scores ranging from 0 to 1, corresponding to a set of classes. Each score indicates that candidate architectural drawings 202 belongs to the corresponding class (…). For exampleThe probability of (including the drawing type represented by the class). When the score output by classifier 204 exceeds a threshold ( For example When the value is 0.7, 0.8, etc., the analysis engine 122 can determine the corresponding drawing type 210 applicable to the candidate architectural drawings 202.

[0035] When a predicted drawing type 210 of a given architectural drawing 202 corresponds to a floor plan, the analysis engine 122 uses the detection model 206 to generate floor plan bounding boxes 212 for the floor plans within the image. In some embodiments, the detection model 206 includes a fully convolutional neural network (FCNN) that generates features from the entire input image and uses said features to predict the coordinates and / or dimensions of one or more bounding boxes for one or more floor plans in the input image. During training, the parameters of the FCNN are updated to reduce the error between the bounding boxes output by the FCNN and the labeled “region of interest” representing the area occupied by the floor plans in the input image. The FCNN may also generate a confidence score for each bounding box, where the confidence score represents the confidence level that the bounding box includes a floor plan and / or the confidence level of the accuracy of the bounding box. For example, the confidence score can be calculated as the probability that the bounding box contains a floor plan multiplied by the probability that the bounding box contains the floor plan and the corresponding ground reality (…). Right now Intersection over Union (IoU) between marked regions of interest.

[0036] Continuing the example above, after training the FCNN, the analysis engine 122 can input an image of an architectural drawing 202 identified by the classifier 204 as having a floor plan drawing type 210 into the FCNN. The FCNN can then output the coordinates and / or dimensions of one or more bounding boxes in the image. As the FCNN outputs multiple bounding boxes of the image, the analysis engine 122 can filter bounding boxes with confidence scores that do not meet numerical, percentile, and / or other types of thresholds. The analysis engine 122 can also, or alternatively, use non-maximum suppression techniques to filter bounding boxes that highly overlap with each other. As a result, the analysis engine 122 can generate a given floor plan bounding box 212 from the output of the detection model 206, which is associated with a high confidence score from the detection model 206 and does not significantly overlap (or not overlap at all) with other bounding boxes output by the detection model 206.

[0037] By using filter 214 and classifier 204 to identify architectural drawings and determine the drawing type 210 of each architectural drawing 202, the analysis engine 122 can analyze the drawings from (…). For exampleThe analysis engine 122 efficiently extracts a set of architectural drawings with floor plans from a large collection of documents or files in the repository. This smaller set of extracted architectural drawings can then be fed into a more resource-intensive detection model 206 to accurately generate floor plan bounding boxes 212 for each floor plan in the extracted architectural drawings. Therefore, the analysis engine 122 verifies that a document may contain floor plans before generating the floor plan bounding boxes 212 for the floor plans using the relatively resource-intensive detection model 206, thereby avoiding the increased execution time and resource overhead that would result from using the detection model 206 to detect and locate floor plans in all documents or files.

[0038] After generating a floor plan bounding box 212 for a given floor plan in architectural drawing 202, analysis engine 122 and / or boundary segmentation engine 124 extract a floor plan image 234 defined by the floor plan bounding box 212 from the corresponding architectural drawing 202. For example, analysis engine 122 and / or boundary segmentation engine 124 can generate the floor plan image 234 by performing a cropping operation to remove pixels falling outside the floor plan bounding box 212 from the image of architectural drawing 202. The following is about... Figure 3 It describes the floor plan bounding boxes and floor plan images of the architectural drawings in more detail.

[0039] Boundary segmentation engine 124 uses segmentation model 208 to generate boundary segments 236 from floor plan image 234, which associate each pixel in floor plan image 234 with labels representing various types of boundaries in the floor plan. For example, segmentation model 208 can be implemented using a deep learning model including: an encoder that extracts features from input floor plan image 234; and a decoder that converts the features into segmentations of pixels in input floor plan image 234 to background, walls, doors, and / or other labels used to train the deep learning model. The following is about... Figure 4 The boundary segmentation of the floor plan image is described in more detail.

[0040] Room segmentation engine 126 converts the raster-based boundary segmentation 236 from boundary segmentation engine 124 into vectorized boundary segmentation 240, which includes vector representations of boundary pixels in boundary segmentation 236. For example, room segmentation engine 126 can use a CNN to convert boundary segmentation 236 into a vector-based shape of the boundaries of a floor plan. The CNN can use asymmetric convolutions running at multiple resolutions and can detect linear features with low brightness (…). example like(The boundaries in boundary segmentation 236). For each pixel determined to contain linearity, resolution participates in weighted optimization to compute the resulting vectorized output. Typically, the room segmentation engine 126 can use various machine learning models, image processing tools, heuristics, and / or other techniques to convert boundary segmentation 236 into vectorized boundary segmentation 240.

[0041] Room segmentation engine 126 also generates room segments 242 that divide the floor plan into discrete rooms based on vectorized boundaries in vectorized boundary segmentation 240. For example, room segmentation engine 126 can extend some or all of the vectorized boundaries by a fixed or variable amount to "close" gaps that may be caused by incomplete vectorization of boundary segmentation 236. Room segmentation engine 126 can then identify spaces in the floor plan completely enclosed by the extended vectorized boundaries as different rooms within the floor plan. The following section discusses… Figures 5A to 5B Vectorized boundary segmentation and room segmentation are described in more detail.

[0042] Therefore, the analysis engine 122, the boundary segmentation engine 124, and the room segmentation engine 126 extract different types of semantic information from the given architectural drawing 202. The analysis engine 122 identifies the drawing type 210 associated with the architectural drawing 202 and generates floor plan bounding boxes 212 for any floor plan in the architectural drawing 202. The boundary segmentation engine 124 generates boundary segments 236 that identify the type and location of the boundaries within each floor plan, and the room segmentation engine 126 performs vectorized boundary segments 240 and room segments 242 to divide each floor plan into areas representing different rooms.

[0043] The semantic information extracted by the analysis engine 122, the boundary segmentation engine 124, and the room segmentation engine 126 can be further used to organize, compare, retrieve, or otherwise manage the corresponding architectural drawings. For example, drawing type 210, floor plan bounding box 212, floor plan image 234, boundary segmentation 236, vectorized boundary segmentation 240, room segmentation 242, and / or other semantic information generated by the analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 can be stored in a repository along with the corresponding architectural drawing 202 and / or used to annotate architectural drawing 202. The semantic information from multiple architectural drawings can then be used to search for or retrieve architectural drawings that possess the following characteristics: floor plan, floor plan associated with a given building or project, floor plan associated with a specific drawing type 210, etc. For example Floor plans of single-family homes, hospitals, retail spaces, etc., floor plans including a certain number of rooms, floor plans including certain types of boundaries, and / or floor plans that are structurally or otherwise similar to the floor plan selected by the user.

[0044] Semantic information can also, or alternatively, be used to perform additional analyses and / or design generation related to the corresponding architectural drawings. For example, drawing type 210, floor plan bounding box 212, floor plan image 234, boundary segmentation 236, vectorized boundary segmentation 240, room segmentation 242, and / or other semantic information generated by analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 can be used to calculate room area ratios, wall or window ratios, or other types of metrics or statistics related to the floor plan; For example Create variations of floor plans by adding, deleting, or adjusting walls, rooms, windows, doors, or other elements of the floor plan; generate three-dimensional (3D) models from floor plans; and / or identify trends or patterns associated with floor plans related to various locations, building types, drawing types, architects, clients, and / or other attributes.

[0045] In one or more implementations, filter 214, classifier 204, detection model 206, segmentation model 208, and / or other components use semantic information generated by analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 to train, retrain, or otherwise update. These components may also, or alternatively, use synthetic data generated based on and / or independently of semantic information to train, retrain, or update.

[0046] For example, the value of drawing type 210 generated by classifier 204 can be verified by obtaining user-generated tags for the corresponding document. The verified value and / or user-generated tags can then be used to retrain classifier 204 and / or update filter 214, such that filter 214 includes a more comprehensive or accurate list of keywords associated with architectural drawings and / or that classifier 204 outputs a more accurate prediction of drawing type 210 for the corresponding document.

[0047] In another example, the generalized class set output by classifier 204 can be used ( For example This includes architectural drawings with floor plans, architectural drawings without floor plans, and documents that are not architectural drawings. Document collections can be grouped or filtered, and "subclasses" of these classes (including architectural drawings with floor plans, architectural drawings without floor plans, and documents that are not architectural drawings) can be defined. For example Different floor plans (or building types including floor plans) are given additional user-generated labels. These additional labels and corresponding documents can then be used to train a new version of the classifier 204, enabling it to distinguish between these subclasses. One or more versions of the detection model 206 can then be trained or retrained to generate bounding box predictions for the floor plan (or other object) associated with each of the subclasses.

[0048] In the third example, synthetic training data can be generated for detection model 206 by combining or arranging bounding boxes previously output by detection model 206, user-generated labels representing regions of interest in floor plan areas within architectural drawings, and / or other components of architectural drawings and / or forms into a synthetic set of architectural drawings. The synthetic architectural drawings can then be used to retrain detection model 206, thereby allowing detection model 206 to identify and predict bounding boxes for floor plans across various architectural drawing layouts.

[0049] In the fourth example, a 2D or 3D model of a building with defined rooms, walls, doors, windows, stairs, and / or other types of objects can be rasterized into color-coded images, where each color represents a different type of object. These rasterized images can then be used as training data for segmentation model 208.

[0050] Figure 3 An exemplary extraction of image 304 of a floor plan area extracted from architectural drawing 302 is shown according to various embodiments. Figure 3 As shown, architectural drawing 302 includes a floor plan area as well as a title block, annotations, boundaries, and / or other elements not directly related to the floor plan area. One or more filters 214 can be applied to the text in architectural drawing 302 to identify architectural drawing 302 as a document or file that is a “candidate” for having one or more floor plans. Images or other content representations in architectural drawing 302 can then be input into classifier 204 to generate a prediction of drawing type 210 for architectural drawing 302. As described above, drawing type 210 can represent one or more types of floor plans, one or more types of architectural drawings that do not include floor plans, and / or one or more categories of documents or files that are not architectural drawings.

[0051] If drawing type 210 indicates that architectural drawing 302 includes floor plans, then an image of architectural drawing 302 is input into detection model 206 to generate a floor plan bounding box 212 for the floor plan area. Image 304 is then generated by cropping the portion of architectural drawing 302 outside the floor plan bounding box 212.

[0052] Figure 4 An exemplary boundary segmentation 236 for an image of a floor plan area according to various embodiments is shown. Figure 4As shown, the exemplary boundary segmentation 236 includes multiple labels for different pixels in the image. Pixels marked as non-boundaries are shown in white, pixels marked as walls are shown as solid black lines, pixels marked as windows are shown as dashed lines representing closely spaced solid lines of exterior walls in the floor plan area, and pixels marked as interior doors are shown as dashed lines representing wider spaced solid lines of interior walls in the floor plan area.

[0053] As described above, boundary segmentation 236 can be generated by inputting an image of the floor plan area into segmentation model 208. The encoder in segmentation model 208 extracts features from the image, while the decoder in segmentation model 208 converts the features into boundary segmentation 236.

[0054] Additionally, segmentation model 208 can be adapted to detect different types of boundaries and / or portions of floor plan regions in an image. For example, one or more versions of segmentation model 208 can be trained to predict different types of rooms, stairs, elevators, and / or other spaces in an image. In another example, one or more versions of segmentation model 208 can be trained to identify appliances, fixtures, furniture, and / or materials in an image.

[0055] Figure 5A An exemplary vectorized boundary segmentation 240 for an image of a floor plan area according to various embodiments is shown. Figure 5A As shown, the exemplary vectorized boundary segmentation 240 includes a plurality of line segments 502 to 512, which are generated from pixels marked with boundary markers in the corresponding boundary segmentation 236. For example, line segments 502, 504, and 510 can be generated from three lines representing the outer boundary of a building in a floor plan, while line segments 506, 508, and 510 can be generated from three lines representing the inner boundary of a building in a floor plan.

[0056] Figure 5B The following are shown according to various implementation schemes. Figure 5A The exemplary room segmentation 242 generated by the vectorized boundary segmentation 240. (Example: ...) Figure 5B As shown, room partition 242 includes an interior portion of the floor plan, which has been divided into discrete areas representing rooms in the building. Therefore, room partition 242 includes line segments 502 to 512 representing vectorized boundaries in the floor plan, and shaded areas 514 to 518 defined by lines representing rooms surrounded by the vectorized boundaries.

[0057] As described above, room segmentation 242 can be generated by extending some or all of the vectorized boundaries in vectorized boundary segmentation 240 to "close" gaps that may be caused by incomplete vectorization of boundary segmentation 236. For example, each of line segments 502 to 512 and the other line segments in the exemplary vectorized boundary segmentation 240 can be extended by a fixed amount, a proportion of the original length of each vector, and / or another amount. The extension of line segments 502 to 512 closes the gaps between line segments 502 and 504, between line segments 502 and 506, and between line segments 510 and 512.

[0058] As an extension Figure 5A All line segments in the vectorized boundary segmentation 240 are replaced or supplemented by closing gaps between line segments 502 and 504, between line segments 502 and 506, and between line segments 510 and 512. Each gap can be detected as a space below a threshold between two vectors. The threshold may include (but is not limited to) a certain number of pixels in the original boundary segmentation 236; a proportion of the minimum, maximum, or average line segment length in the vectorized boundary segmentation 240; and / or another value. When such a gap is detected, the gap is "closed" by extending one of the line segments. Thus, the gap between line segments 502 and 504 can be closed by extending line segment 504 to the left, the gap between line segments 502 and 506 can be closed by extending line segment 506 to the left, and the gap between line segments 510 and 512 can be closed by extending line segment 512 downwards.

[0059] exist Figure 5A After the gaps between line segments in the vectorized boundary segmentation 240 have been closed, the spaces completely enclosed by the extended vectorized boundary in the floor plan can be identified as different rooms within the floor plan. For example, area 514, enclosed by the intersection of line segments 502 and 504, line segments 502 and 506, line segments 504 and 512, and line segments 506 and 512, can be identified as the first room. Area 516, enclosed by the intersection of line segments 502 and 506, line segments 502 and 508, line segments 506 and 512, and line segments 508 and 512, can be identified as the second room. Area 518, enclosed by the intersection of line segments 502 and 508, line segments 502 and 510, line segments 508 and 512, and line segments 510 and 512, can be identified as the third room.

[0060] Figure 6 Flowcharts of method steps for automatically extracting data from architectural drawings, based on various implementation schemes, are presented. (Although references are provided...) Figures 1 to 2 The system described herein describes the method steps, but those skilled in the art will understand that in other embodiments, any system can be configured to perform the method steps in any order.

[0061] As shown in the figure, the analysis engine 122 applies a set of keyword filters to the 602 document set to identify a candidate set of architectural drawings. For example, the analysis engine 122 can retrieve a set of architectural documents from a repository. The analysis engine 122 can then identify the candidate set of architectural drawings as a subset of architectural documents that include one or more keywords typically found in architectural drawings with floor plans.

[0062] Next, the analysis engine 122 executes a 604 classifier, which identifies a set of architectural drawings with floor plans from the candidate architectural drawings. For example, an image of each candidate architectural drawing, text within each candidate architectural drawing, and / or another representation of each candidate architectural drawing can be input into the classifier. In response to the input, the classifier can output a score representing the probability of each class to which a candidate architectural drawing belongs. When a given score for a candidate architectural drawing exceeds a threshold, the architectural drawing can be determined to belong to the corresponding class. Therefore, if the score output by the classifier indicates that a candidate architectural drawing belongs to a class corresponding to a floor plan or the type of floor plan, then the candidate architectural drawing can be identified as an architectural drawing with one or more floor plans.

[0063] Analysis engine 122 then executes a detection model 606, which generates bounding boxes for floor plan regions within architectural drawings identified by a classifier. For example, the detection model may include an FCNN that predicts the coordinates and / or dimensions of the bounding boxes based on an image of the architectural drawing.

[0064] Analysis engine 122 and / or boundary segmentation engine 124 extract images of the 608th floor plan area from the architectural drawings based on bounding boxes. For example, analysis engine 122 and / or boundary segmentation engine 124 can extract images by cropping the area of ​​the architectural drawings that is outside the bounding boxes.

[0065] The boundary segmentation engine 124 also generates 610 boundary segments based on the floor plan image. For example, the boundary segmentation engine 124 can use an encoder to generate a feature set from the image and use a decoder to convert the feature set into boundary segments. Boundary segments include pixels and / or other portions of the floor plan labeled with one or more boundary types, such as walls, windows, interior doors, exterior doors, railings, and / or stairs.

[0066] Finally, the room segmentation engine 126 generates room segments for the 612 floor plan regions based on the vectorized representation of the boundary segments. For example, the room segmentation engine 126 can use one or more vectorization techniques to convert the boundary segments into a vectorized representation. The room segmentation engine 126 can also adjust the endpoints of one or more line segments to close the gaps between pairs of line segments in the vectorized representation. The room segmentation engine 126 can then determine one or more regions representing one or more rooms in the floor plan based on the intersections associated with the adjusted line segments.

[0067] Analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 can repeat operations 606 to 612 for any remaining architectural drawings 614. For example, analysis engine 122, boundary segmentation engine 124, and room segmentation engine 126 can generate floor plan images, boundary segments, and / or room segments for all architectural drawings with floor plans identified in operation 604. Analysis engine 122, boundary segmentation engine 124, and / or room segmentation engine 126 can also store or associate floor plan images, boundary segments, and / or room segments with the corresponding architectural drawings. Floor plan images, boundary segments, and / or room segments can then be used to search architectural drawings and / or floor plans and / or perform additional analyses or design generation related to floor plans as discussed above.

[0068] In summary, the disclosed technology utilizes various machine learning and image processing techniques to extract semantic information related to floor plans from architectural drawings. One or more keyword filters and / or classifiers are used to identify architectural drawings as having floor plans. A detection model is used to generate bounding boxes for floor plans within architectural drawings, and these bounding boxes are used to extract images of floor plans from the architectural drawings. A segmentation model is applied to the images of the floor plans to generate boundary segments, which include pixels in the image labeled with different boundary types. The boundary segments are vectorized, and the intersections of line segments representing boundaries in the vectorized boundary segments are used to divide the floor plan into regions representing different rooms.

[0069] Semantic information extracted from multiple architectural drawings can be further used to organize, compare, or otherwise manage architectural drawings and / or corresponding floor plans. For example, floor plan images, boundary segments, and / or room segments of a large number of architectural drawings can be stored in a repository. Floor plan images, boundary segments, and / or room segments can be used to identify architectural drawings with floor plans, compare floor plans across projects, and / or retrieve floor plans similar to a given floor plan. Semantic information can also, or alternatively, be used to perform additional analyses and / or design generation related to the corresponding architectural drawings or floor plans. For example, semantic information can be used to generate floor plans similar to a given floor plan or set of floor plans, generate variations of floor plans, and / or calculate metrics or statistics related to the floor plans.

[0070] One technical advantage of the disclosed technology over existing technologies is that it allows for the efficient identification and extraction of one or more floor plans from a vast collection of stored architectural documents. Therefore, the disclosed technology is more scalable, efficient, and less resource-intensive than traditional methods that involve manually browsing, searching, or viewing the entire collection of stored architectural documents to identify and extract floor plans within those documents. Another technical advantage of the disclosed technology is the use of multiple smaller machine learning models to extract different types of semantic information from those floor plans. Thus, the disclosed technology is able to extract a wider variety of semantic information from floor plans compared to existing methods, and it also reduces the overhead of training and executing different machine learning models compared to conventional methods that involve training and executing a single, more complex machine learning model. These technical advantages provide one or more technical improvements over existing methods.

[0071] [Combination of claims added before filing]

[0072] Any and all combinations of any element of any claim and / or any element described in this application, in any manner, fall within the scope of the invention and protection.

[0073] The descriptions of the various embodiments have been presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be readily apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0074] Aspects of this embodiment may be embodied as a system, method, or computer program product. Therefore, aspects of this disclosure may take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may be collectively referred to herein as a “module,” “system,” or “computer.” Furthermore, any hardware and / or software technology, process, function, component, engine, module, or system described in this disclosure may be implemented as a circuit or a set of circuits. Additionally, aspects of this disclosure may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code implemented thereon.

[0075] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0076] The foregoing description of aspects of this disclosure refers to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine. When executed via a processor of the computer or other programmable data processing apparatus, the instructions enable the performance of functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, code segment, or portion comprising one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions indicated in the blocks may appear in an order different from that shown in the drawings. For example, two blocks shown consecutively may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or performs a combination of dedicated hardware and computer instructions.

[0078] Although the foregoing relates to embodiments of this disclosure, other and additional embodiments of this disclosure may be contemplated without departing from the basic scope of this disclosure, the scope of which is defined by the appended claims.

Claims

1. A computer-implemented method for extracting data from architectural drawings, the method comprising: At least a portion of a first image of an architectural drawing, including a floor plan area, is identified by one or more machine learning models by performing one or more of the following operations: determining whether text extracted from the architectural drawing includes at least one keyword from a pre-specified list, the pre-specified list including multiple keywords typically found in architectural drawings with floor plan areas; in response to determining that the text includes at least one keyword from the pre-specified list, executing a classifier that determines a drawing type associated with the architectural drawing based on a second image of the architectural drawing; and, when the drawing type corresponds to a floor plan, executing a detection model that generates a bounding box of the floor plan area based on the second image of the architectural drawing. In response, the first image of the floor plan area is extracted from the architectural drawings; as well as A boundary segmentation is generated based on the first image of the floor plan region, wherein the boundary segmentation includes one or more boundary types for one or more portions of the floor plan region.

2. The computer-implemented method according to claim 1, further comprising generating room segmentation of the floor plan area by dividing the floor plan area into multiple areas representing multiple rooms, based on the vectorized representation of the boundary segmentation.

3. The computer-implemented method according to claim 2, wherein the room division for generating the floor plan area further comprises: Adjusting multiple endpoints of multiple line segments included in the vectorized representation of the boundary segmentation to generate multiple adjusted line segments; as well as The plurality of regions are determined based on a plurality of intersections associated with the plurality of adjusted line segments.

4. The computer-implemented method of claim 1, wherein executing the classifier comprises: The classifier is applied to the first image of the floor plan area to generate a set of scores; as well as A threshold is applied to the set of scores to determine the drawing type.

5. The computer-implemented method according to claim 1, further comprising: A training dataset is generated based on one or more outputs generated by the one or more machine learning models; as well as Update multiple parameters of the one or more machine learning models based on the training dataset.

6. The computer-implemented method of claim 5, wherein generating the training dataset comprises combining the first image of the floor plan region with one or more images extracted from one or more supplementary architectural drawings to generate a synthetic floor plan.

7. The computer-implemented method according to claim 1, wherein generating the boundary segmentation comprises: The encoder is applied to the first image to generate a feature set; as well as The decoder is applied to the feature set to generate the boundary segmentation.

8. The computer-implemented method of claim 1, wherein the one or more boundary types include at least one of a wall, window, interior door, exterior door, railing, or staircase.

9. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: At least a portion of a first image of an architectural drawing, including a floor plan area, is identified by one or more machine learning models by performing one or more of the following operations: determining whether text extracted from the architectural drawing includes at least one keyword from a pre-specified list, the pre-specified list including multiple keywords typically found in architectural drawings with floor plan areas; in response to determining that the text includes at least one keyword from the pre-specified list, a classifier is applied to a second image of the architectural drawing to determine the drawing type associated with the architectural drawing; And when the drawing type corresponds to a floor plan, the detection model is applied to the second image of the architectural drawing to generate a bounding box for the floor plan area; In response, the first image of the floor plan area is extracted from the architectural drawings; as well as A boundary segmentation is generated based on the first image of the floor plan region, wherein the boundary segmentation includes one or more boundary types for one or more portions of the floor plan region.

10. The one or more non-transitory computer-readable media of claim 9, wherein the instructions further cause the one or more processors to perform the step of: generating room segmentation of the floor plan region based on a vectorized representation of the boundary segmentation by dividing the floor plan region into multiple regions representing multiple rooms.

11. The one or more non-transitory computer-readable media of claim 10, wherein the room division for generating the floor plan area further comprises: Adjusting multiple endpoints of multiple line segments included in the vectorized representation of the boundary segmentation to generate multiple adjusted line segments; as well as The plurality of regions are determined based on a plurality of intersections associated with the plurality of adjusted line segments.

12. The one or more non-transitory computer-readable media of claim 10, wherein the instructions further cause the one or more processors to perform the step of: retrieving at least one of the first image, the boundary segmentation, or the room segmentation in response to a search associated with the architectural drawing.

13. The one or more non-transitory computer-readable media of claim 9, wherein the instructions further cause the one or more processors to perform the following steps: A training dataset is generated based on one or more outputs generated by the one or more machine learning models; and The parameters of the one or more machine learning models are updated based on the training dataset.

14. One or more non-transitory computer-readable media according to claim 13, wherein generating the training dataset comprises combining the first image of the floor plan region with one or more images extracted from one or more additional architectural drawings to generate a synthetic floor plan.

15. The one or more non-transitory computer-readable media of claim 9, wherein performing the one or more operations via the one or more machine learning models further comprises applying one or more keyword filters to a collection of documents to identify the architectural drawings.

16. One or more non-transitory computer-readable media according to claim 9, wherein the one or more boundary types include at least one of a wall, window, interior door, exterior door, railing, or staircase.

17. A system comprising: One or more memories, wherein the one or more memories store instructions; as well as One or more processors, said one or more processors being coupled to said one or more memories and configured to: when executing said instructions At least a portion of a first image of an architectural drawing, including a floor plan area, is identified by one or more machine learning models by performing one or more of the following operations: determining whether text extracted from the architectural drawing includes at least one keyword from a pre-specified list, the pre-specified list including multiple keywords typically found in architectural drawings with floor plan areas; and, in response to determining that the text includes at least one keyword from the pre-specified list, applying a classifier to a second image of the architectural drawing to determine the drawing type associated with the architectural drawing. And when the drawing type corresponds to a floor plan, the detection model is applied to the second image of the architectural drawing to generate a bounding box for the floor plan area; In response, the first image of the floor plan area is extracted from the architectural drawings; Boundary segmentation is generated based on the first image of the floor plan region, wherein the boundary segmentation includes one or more boundary types for one or more portions of the floor plan region; and The room segmentation of the floor plan area is generated based on the vectorized representation of the boundary segmentation, wherein the room segmentation includes dividing the floor plan area into multiple regions representing multiple rooms.

Images