Prediction of the internal model of the structure

Abstract

Methods and systems are provided for improved prediction and generation of structure interiors. In one aspect, a method is provided that includes receiving an exterior image of a structure and using a machine learning model to determine an exterior surface of the structure. The exterior surface may surround an exterior portion of the structure. The machine learning model may further determine exterior features of the structure and may determine an interior model of the structure based on the exterior surface of the exterior features. Based on the exterior surface and the interior model, a three-dimensional representation of the interior and exterior portions of the structure may be generated. TIFF2025503485000002.tif111159

Description

[Technical field]

[0001] Claiming priority This application claims priority to and benefit of U.S. patent application Ser. No. 17 / 559,297, filed Dec. 22, 2021, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] background In various situations and applications, it may be advantageous to know the interior layout of a structure before entering the structure, for example, the interior layout may be useful in emergency response scenarios, demolition analysis, military scenarios, etc. Summary of the Invention

[0003] overview The present disclosure presents a novel and innovative system and method for improved prediction and generation of structure interiors. In a first aspect, a method is provided that includes receiving an exterior image of a structure and using a machine learning model to determine an exterior surface of the structure. The exterior surface may surround an exterior portion of the structure depicted in the exterior image. The machine learning model may also determine exterior features of the structure based on the exterior image and / or the exterior surface. The machine learning model may also determine an interior model of the structure based on the exterior surface and the exterior features. The method may also include generating a three-dimensional representation of the interior portion of the structure and the exterior portion of the structure based on the exterior surface and the interior model.

[0004] In a second aspect according to the first aspect, the exterior feature includes at least one of a door, a window, a structural support element, a corner, a roof, and / or a utility system of the structure.

[0005] In a third aspect according to either the first or second aspect, the structure has a plurality of floors, and the step of determining the internal model includes determining a plurality of internal models for the plurality of floors.

[0006] In a fourth aspect according to the third aspect, multiple levels of the structure are identified based on an exterior characteristic of the structure.

[0007] In a fifth aspect according to the fourth aspect, multiple floors are identified based on multiple levels of windows at multiple heights within the structure.

[0008] In a sixth aspect according to any of the first to fifth aspects, the method further includes receiving training data for the plurality of structures prior to receiving the exterior images. The training data may include exterior images of the plurality of structures, predicted exterior surfaces for the plurality of structures, predicted exterior features for the plurality of structures, and predicted internal models for the plurality of structures. The first machine learning model may be trained to generate predicted exterior surfaces and predicted external features of at least a subset of the plurality of structures based on at least (i) the exterior images of the subset of the plurality of structures and (ii) the predicted exterior surfaces of the subset of the plurality of structures. The first machine learning model may also be trained to predict an internal model of at least a subset of the plurality of structures based on at least the predicted exterior surfaces and predicted external features.

[0009] In a seventh aspect according to the sixth aspect, the method further includes, prior to the step of receiving the training data, receiving a plurality of architectural plans for the plurality of structures, and generating, using the second machine learning model, a predicted internal model for the plurality of structures based on the plurality of architectural plans.

[0010] In an eighth aspect according to any of the sixth and seventh aspects, training a first machine learning model to predict the outer contour and external features includes receiving a first external image of a first structure from the plurality of structures, and predicting, with the first machine learning model, a first outer surface and a first external feature of the first structure based on the first external image. One or more differences between (i) the first outer surface of the first structure and the expected outer surface of the first structure, and / or (ii) between the first external feature of the first structure and the expected external feature may be detected. Based on the one or more differences, one or more parameters of the first machine learning model may be adjusted.

[0011] In a ninth aspect according to any of the sixth to eighth aspects, the step of training the first machine learning model to predict the internal model includes receiving an exterior surface and an exterior feature of the first structure from the plurality of structures, and predicting a first internal model of the first structure based on an outer contour of the first structure using the first machine learning model. One or more differences between the first internal model of the first structure and the expected internal model of the first structure may be detected. Based on the one or more differences, one or more parameters of the first machine learning model may be adjusted.

[0012] In a tenth aspect according to any of the first to ninth aspects, the structure includes at least one of a building, a vehicle, an infrastructure component, a watercraft, a spacecraft, an aircraft, a tank, and / or an appliance.

[0013] In an eleventh aspect, a method is provided that includes receiving training data for a plurality of structures. The training data may include exterior images of the plurality of structures, predicted exterior surfaces for the plurality of structures, predicted exterior features for the plurality of structures, and predicted interior models for the plurality of structures. The first machine learning model may be trained to predict predicted exterior surfaces and predicted exterior features of at least a subset of the plurality of structures based on at least (i) the exterior images of the subset of the plurality of structures and (ii) the predicted exterior surfaces of the subset of the plurality of structures. The first machine learning model may also be trained to predict interior models of at least a subset of the plurality of structures based on at least the predicted exterior surfaces and predicted exterior features. The first machine learning model may be arranged to predict outer contours and interior models for additional structures distinct from the plurality of structures.

[0014] In a twelfth aspect according to the eleventh aspect, the method further includes, prior to the step of receiving the training data, receiving a plurality of architectural plans for the plurality of structures, and generating, using the second machine learning model, a predicted internal model of the plurality of structures based on the plurality of architectural plans.

[0015] In a thirteenth aspect according to any of the eleventh and twelfth aspects, training a first machine learning model to predict the outer contour and external features includes receiving a first external image of a first structure from the plurality of structures, and predicting, using the first machine learning model, a first outer surface and a first outer feature of the first structure based on the first outer image. One or more differences between (i) the first outer surface of the first structure and the expected outer surface of the first structure, and / or (ii) the first outer feature of the first structure and the expected outer feature may be determined. Based on the one or more differences, one or more parameters of the first machine learning model may be adjusted.

[0016] In a fourteenth aspect according to any of the eleventh to thirteenth aspects, the step of training the first machine learning model to predict the internal model includes receiving an exterior surface and an exterior feature of the first structure from the plurality of structures, and predicting a first internal model of the first structure based on the exterior surface of the first structure using the first machine learning model. One or more differences between the first internal model of the first structure and the predicted internal model of the first structure may be determined. Based on the one or more differences, one or more parameters of the first machine learning model may be adjusted.

[0017] In a fifteenth aspect according to the fourteenth aspect, the exterior surface of the first structure is one of an expected exterior surface of the first structure included in the training data and / or a predicted exterior surface of the first structure generated by the first machine learning model. The exterior feature of the first structure may be one of an expected exterior feature of the first structure included in the training data and / or a predicted exterior feature of the first structure generated by the first machine learning model.

[0018] In a sixteenth aspect according to any of the eleventh to fifteenth aspects, the step of training the first machine learning model to predict the interior model includes training the first machine learning model to generate an interior model that conforms to at least one of: (i) exterior spatial constraints; and / or (ii) typical construction methods and structural design requirements.

[0019] In a seventeenth aspect according to the sixteenth aspect, at least one of (i) exterior spatial constraints and / or (ii) general construction methods and structural design requirements are represented in an objective function for the first machine learning model.

[0020] In an eighteenth aspect according to any of the eleventh to seventeenth aspects, the step of disposing the first machine learning model further includes receiving an exterior image of the structure, and using the machine learning model to determine an exterior surface, exterior features, and an interior model of the structure. Based on the exterior surface and the interior model, a three-dimensional representation of an interior portion of the structure and an exterior portion of the structure can be generated.

[0021] In a nineteenth aspect according to the eighteenth aspect, the step of determining the outer contour, external features, and internal model of the structure further includes using the machine learning model to determine an exterior surface of the structure that surrounds an exterior portion of the structure depicted in the exterior image, and using the machine learning model to determine the external features of the structure based on the exterior image and / or the exterior surface. The internal model of the structure may also be determined based on the outer contour and external features using the machine learning model.

[0022] In a twentieth aspect according to any of the thirteenth to nineteenth aspects, the exterior feature includes at least one of a door, a window, a structural support element, a corner, and / or a utility system of the structure.

[0023] The features and advantages described herein are not all-inclusive, and many additional features and advantages will become apparent to those skilled in the art, particularly upon consideration of the figures and description.Furthermore, it should be noted that the language used herein has been selected primarily for ease of reading and explanation, and not to limit the scope of the disclosed subject matter. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates a system for predicting an internal model of a structure according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 2 illustrates a machine learning model processing flow according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 3 illustrates features according to an exemplary embodiment of the present disclosure. [Figure 4]FIG. 4 illustrates an architectural plan according to an exemplary embodiment of the present disclosure. [Diagram 5] FIG. 5 illustrates a system for generating internal model training data according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates a method for predicting an internal model of a structure according to an exemplary embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates a method for training a machine learning model to predict an internal model, according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS For a particular building, it may be possible to access and examine architectural plans (blueprints, floor plans, etc.) that depict or otherwise represent the interior layout of the building. In such a scenario, it may be possible to reconstruct the interior layout of the building based on these plans. For example, U.S. Patent Application Serial No. 17 / 487,838 describes various techniques for extracting and constructing the interior layout of a building based on the architectural plans of the building.

[0026] However, in certain scenarios, it may not be possible to locate copies of architectural plans for a particular structure. For example, in a combat scenario, an individual may not have access to architectural plans for a building prior to entry. In an emergency response scenario, there may not be time to locate copies of architectural plans for a building. It will be further appreciated that architectural plans for various structures may not be readily accessible for a variety of other reasons (plans not digitized, plans stored in private or siloed databases, lost plans, etc.). Thus, it may be necessary to quickly reconstruct the interior layout of a structure using other means.

[0027] In many situations, external imagery of a structure may be more readily available. For example, overhead imagery (e.g., captured by a satellite or UAV) and / or other imagery (e.g., captured by an individual located within view of the building) may be used to help determine a plausible interior layout of the structure. A structural engineer may then use specialized software to analyze the exterior imagery and create representative models of the structure based on standard construction templates and common practices. These representative models (interior and exterior models) may then be used for various types of structural analysis. However, these techniques are time-consuming and cumbersome, and often cannot be done quickly enough to be used in emergency scenarios. Thus, there is a need to expedite and automate this process so that the exterior imagery can be used to predict the interior layout in emergency scenarios.

[0028] One solution to this problem is to analyze the exterior image using one or more machine learning models. In one embodiment, the machine learning model may analyze the exterior image to determine the exterior surface of the structure depicted in the exterior image. The exterior image and / or the exterior surface may then be used by the model to identify one or more exterior features of the structure. In certain cases, these exterior features may be common to both the interior and exterior of the structure. The exterior surface and the exterior features may then be used to predict an interior model of the structure that includes one or more interior features. These interior features may include structural features (support beams, wall assemblies, etc.) and / or functional features for the occupants of the structure (doors, windows, plumbing, HVAC components, etc.). The machine learning model may be configured to position the interior features of the structure such that the interior features align with any associated exterior features and fit within the exterior surface of the structure. In certain scenarios, the machine learning model may be trained based on the output of other machine learning models. For example, a second machine learning model may be used to generate a training dataset of expected interior layouts for the structure based on architectural blueprints of the structure. In such cases, exterior images of multiple structures can be combined with known interior layouts for those structures to form a unique training dataset for a machine learning model.

[0029] 1 illustrates a system 100 for predicting an interior model of a structure, according to an exemplary embodiment of the present disclosure. The system 100 may be configured to predict an interior model of an interior portion of one or more structures 108. In particular, the system 100 may be configured to receive an exterior image 112 of the structure 108 and generate a three-dimensional representation 122 of the structure 108. The three-dimensional representation 122 may include a three-dimensional model (computer model, CAD model, three-dimensional GIS model) of the exterior and interior of the structure 108.

[0030] Although depicted as a building in FIG. 1, in practice, the structure 108 may include any type of structure (e.g., a man-made structure). For example, the structure 108 may include any structure having at least a partially enclosed interior space. As specific examples, the structure may include a single story building, a multi-story building, a warehouse structure, infrastructure facilities, outdoor structures (pavilions, observation decks, decks, bridges, dams, etc.), or combinations thereof. As further examples, the infrastructure facilities may include interior and exterior structures such as dams, storm drains, sewer pipes, tunnels (e.g., access tunnels, automobile tunnels), waterways, utility stations (e.g., pumping stations), conduits (e.g., electrical conduits), etc. In further embodiments, the structure may include parts of other products or devices (vehicle parts, aircraft parts, artillery parts, weapon parts) or other components (mechanical parts, chemical components, electrical components). Thus, when a building is referred to herein, it should be understood to apply to any type of structure as well. Similarly, in this disclosure, the terms "blueprint," "architectural plan," and "plan" (and similar terms) are used to refer to plans for buildings and other structures. In fact, those skilled in the art will appreciate that these documents may be referred to using different terms in other instances. For example, such documents may be referred to as "site plans," "facility plans," or other similar terms. As further examples, plans discussed herein may include one or more floor plans, elevations, circuit board layouts, product plans, and the like. As one specific example, the structure may include an aircraft engine, and the plan may include a product plan for the engine. As another specific example, the structure may include an artillery weapon, and the plan may include a multi-view structural plan or product plan for the artillery weapon.

[0031] The exterior of the structure 108 may include any portion of the structure 108 that is visible from outside the structure 108 (e.g., visible in an exterior image of the structure 108). Additionally or alternatively, the exterior of the structure 108 may include a building surface that faces the exterior environment of the structure 108 (such as the outdoor environment surrounding the structure 108). The interior of the structure 108 may include any portion of the structure 108 that is not visible from outside the structure 108 (e.g., visible in an exterior image of the structure 108). In certain cases, portions of the interior of the structure 108 may be visible in an exterior image (e.g., through a window of the structure 108). Additionally or alternatively, the interior of the structure 108 may include any portion of the structure 108 that is contained within the exterior of the structure 108. In certain cases, the interior of the structure may include portions of the exterior walls or support system of the structure 108. For example, the exterior of the structure 108 may include the exterior surface of a concrete exterior wall, and the interior of the structure 108 may include the interior portion of the concrete wall (e.g., not in direct contact with the outdoor area surrounding the structure 108) and other materials within the brick exterior wall (support beams, interior support materials for the concrete wall, etc.). In various embodiments, the interior of the structure 208 may include one or more of the structural components (frame of the structure 108, load-bearing walls / beams / other systems of the structure 108), the interior / exterior construction materials of the structure, material properties (strength under tension / compression, impact resistance, dimensions, etc.), structural details of the structure 108, and any other elements that contribute to the overall structural strength of the structure 108. In certain embodiments, the interior of the structure 108 may include elements that do not contribute to the structural strength of the structure 108, such as non-load-bearing walls, doors, windows, water supply and drainage systems, HVAC systems, fire suppression systems, etc.

[0032] In particular, the system 100 may include a computing device 102 configured to receive an external image 112 and generate a three-dimensional representation 122. For example, the computing device 102 may receive the external image 112 from a database 104 that stores external images 114, 116 of a plurality of different structures. The external images 112, 114, 116 may represent one or more two-dimensional or three-dimensional images depicting the exterior of the structure 108, and in certain cases, the outdoor area surrounding the structure 108. The external image 112 may include any image of the exterior of the structure 108. For example, the external image 112 may include an overhead image (e.g., captured by a satellite / UAV). As another example, the external image 112 may be captured from ground level (e.g., by an individual within view of the structure 108). In certain cases, the external image 112 may be orthorectified such that the image can be used (e.g., by a user or computing process) as the basis for accurate spatial measurements of the structure. In further instances, the exterior image 112 may include a three-dimensional image of the exterior of the structure 108 .

[0033] The computing device 102 may include a machine learning model 124 configured to determine an exterior surface 126 and / or an interior model 134 of the structure 108. For example, the machine learning model 124 may determine the exterior surface 126 of the structure 108 based on the exterior image. The exterior surface 126 may include a three-dimensional representation of the exterior of the structure 108. For example, the exterior surface 126 may include a three-dimensional envelope or contour that approximates a shape of the exterior of the structure 108. The machine learning model 124 may further identify one or more exterior features 132 depicted in the exterior image 112 and / or based on the exterior surface 126. In certain implementations, the exterior features 132 may include one or more functional or aesthetic features of the structure 108 that are externally visible on the structure 108. In certain implementations, at least a subset of the exterior features 132 may be common to the exterior of the structure and the interior of the structure. For example, exterior features 132 may include windows, doors, exterior vents, exterior wall materials, visible / exposed support elements, and the like.

[0034] Based on the exterior surface 126 and the exterior model 132, the machine learning model 124 may determine the interior model 134. The interior model 134 may represent a predicted layout or structural design of the interior portion of the structure 108. For example, the interior model 134 may represent a two-dimensional and / or three-dimensional representation of the interior portion of the structure 108. In particular, the interior model 134 may be generated to conform to one or more spatial constraints of the exterior surface 126, as well as common construction practices and structural design requirements (such as constraints representing well-established common construction practices, standard construction templates and structural design rules of thumb). For example, the interior model 134 may be generated by combining multiple interior features common to the structure (such as structures of the same type as the structure 108). For example, if the exterior features 132 indicate that the structure 108 has concrete exterior walls, the interior features 140 may include a common concrete wall assembly selected based on the size of the walls (such as length and height). As another example, if the exterior features 132 indicate that the structure 108 has a glass exterior wall, the interior features 140 may include support structures (e.g., support frames) common to buildings with glass exterior walls. In that case, the interior layout features may be combined and positioned to fit one or more constraints indicated by the exterior surface 126 and / or the exterior features 132. In particular, the interior features 140 may be generated to match one or more of the exterior features 132. As a particular example, the interior model 134 may be generated to include rooms that match certain exterior features 132, such as windows and exterior air conditioning units. Additionally or alternatively, the interior features 140 may be generated to fit within the exterior surface 126. For example, the exterior surface 126 may define a boundary for the exterior walls of the interior model 134. Thus, the rooms in the interior model 134 (and the interior features 140 contained therein) may be generated to fit within the boundaries indicated by the exterior surface 126. In certain embodiments, to match the exterior surface 126 and the external features 132, the machine learning model 124 may generate a number of internal features 140 based on common construction techniques.The machine learning model 124 may then combine and position these internal features 140 to fit within the exterior surface 126 and to match one or more external features 132 that are common to both the exterior and interior of the structure 108.

[0035] In certain implementations, the machine learning model 124 may determine the exterior surface 126, the external features 132, and the interior model 134 in a particular order based on the received exterior image 112. For example, FIG. 2 illustrates a machine learning model process flow 200 according to an exemplary embodiment of the present disclosure. In the machine learning model process flow 200, the machine learning model 124 receives the exterior image 112. Based on the exterior image 112, the machine learning model 124 may identify the exterior surface 126 of the structure 108. In particular, the machine learning model 124 may construct a three-dimensional boundary surface of the structure 108 depicted in the exterior image 112. For example, the machine learning model 124 may align multiple images of the structure 108 based on various visual / spatial features and extract an approximate exterior surface encompassing the depicted portion of the structure 108 based on the aligned images. The machine learning model 124 may then identify the external features 132 based on the exterior image 112 and / or the exterior surface 126. For example, the machine learning model 124 may include a neural network (e.g., a convolutional neural network, a recurrent neural network) configured to identify a particular type of predetermined feature depicted in the external image 112 informed by associated contours of the features in the exterior surface 126. The machine learning model 124 may then determine an interior model 134 of the structure 108 based on the exterior surface 126 and the external features 132 using the techniques described above.

[0036] The actual exterior features 132 and interior features 140 generated by the machine and model 124 may include one or more different types of information. For example, FIG. 3 illustrates features 302, 304 according to an exemplary embodiment of the present disclosure. In particular, FIG. 3 depicts an interior feature 302, which may be an exemplary embodiment of one of the interior features 140, and an exterior feature 304, which may be an exemplary embodiment of one of the exterior features 132. Each of the features 302, 304 includes a label 306, 308, physical dimensions 310, 312, materials 314, 316, and one or more adjacent elements 318, 320. The labels 306, 308 may indicate a feature title or type for the features 302, 304. For example, the label 306 of the interior feature 302 may identify the feature 302 as an "interior door." As another example, the label 308 of the exterior feature 304 may identify the feature 304 as a "window." In additional or alternative embodiments, the labels 306, 308 may provide further information about the features 302, 304. For example, the features 302, 304 may include information about material properties of the materials used to construct the features (e.g., strength under tension / compression, impact resistance, assembly techniques). In certain embodiments, the labels 306, 308 may include a unique identifier for the feature 302, 304 (such as a unique alphanumeric identifier) ​​that may be used to uniquely reference the particular feature 302, 304 elsewhere within the internal model 134.

[0037] The physical dimensions 310, 312 may refer to one or more physical dimensions (such as a length, width, height, thickness, etc.) for the features 302, 304. For example, the physical dimensions 312 may include the height and width of a window (such as the exterior feature 304). As another example, the physical dimensions 310 may include the height and width of an interior door (such as the interior feature 302). In additional or alternative embodiments, the physical dimensions 310, 312 may include one or more of the length and thickness of a support beam and / or the length and thickness of an interior wall assembly.

[0038] Materials 314, 316 may indicate one or more construction materials used to form features 302, 304. For example, if features 302, 304 correspond to individual structural elements (support beams, windows, doors, drywall, ducts, etc.), materials 314, 316 may indicate the materials from which the individual elements are made. Additionally or alternatively, features 302, 304 may correspond to a particular assembly (e.g., a wall assembly). In such cases, materials 314, 316 indicate a type of assembly, which may specify one or more materials used to build the assembly (size and spacing of support beams, use of brick, concrete, drywall, thickness of any cladding used).

[0039] Additionally or alternatively, features 302, 304 may specify one or more adjacent elements 318, 320. For example, adjacent elements 318, 320 may identify interior and / or exterior features directly adjacent to features 302, 304. For example, if interior feature 302 is a door, adjacent element 318 may include a unique identifier (such as label 306) of a wall adjacent to the door. As another example, if exterior feature 304 is an exterior HVAC unit, adjacent element 320 may identify adjacent structural elements (such as adjacent walls and / or roofs) and / or adjacent HVAC elements (such as a duct connected to the exterior HVAC unit).

[0040] It should be understood that the above examples of labels 306, 308, physical dimensions 310, 312, materials 314, 316, and adjacent elements 318, 320 are merely illustrative. Thus, based on this disclosure, one of ordinary skill in the art may recognize additional or alternative labels, physical dimensions, materials, and / or adjacent elements that may be used to describe the internal and / or external features. Furthermore, one of ordinary skill in the art will understand that features 302, 304 may include additional or alternative information, including omitting one or more of the labels, physical dimensions, materials, and / or adjacent elements, as well as adding one or more additional fields to features 302, 304. All such implementations are hereby deemed to be within the scope of this disclosure.

[0041] Returning to Figure 1, the machine learning model 124 may generate the internal model 134 to include the internal features and / or the external features 132, as described above in connection with Figure 3. However, in addition, the machine learning model 124 may generate the internal model 134 to include a visual or structural representation of the structure 108. In particular, the internal model 134 may include a visual representation of the internal features 140 arranged to match the exterior surface 126 and the external features 132, as described above. In certain implementations, the internal model 134 may be generated as a two-dimensional representation of the structure 108.

[0042] For example, FIG. 4 illustrates a portion of a building plan 400 according to an exemplary embodiment of the present disclosure. The plan 400 may be an exemplary implementation of all or a portion of a two-dimensional interior model 134 that may be generated by the machine learning model 124. The plan 400 includes various elements of a building, some of which are identified using reference numbers for the following discussion. The illustrated plan 400 may be a portion of a floor of a building. The plan 400 includes a depiction of exterior walls 404, 414 and interior walls 402, 406, 410 (only a subset of which are numbered for clarity). The interior walls 402, 406, 410 include two different types of walls: interior partition walls 402, 406 and interior load-bearing walls 410. The plan 400 also includes a depiction of a substructure 408 along with structural connections 412 that connect other portions of the building (such as the interior load-bearing walls 410) to the substructure 408. The exterior walls 404, 414, the load-bearing walls 410, the substructure 408, and the structural connections 412 may all be interior features 140 of the structure.

[0043] In certain implementations, not all of the illustrated features may be necessary to properly determine the interior layout of the building. For example, the substructure 408 and the structural connections 412 may not be necessary to accurately determine the interior layout of the floor. Thus, in certain implementations, the machine learning model 134 may not be trained to include the substructure 408 and / or the structural connections 412 as interior features 140 in the interior model 134. For clarity, the blueprint 400 includes bounding boxes 418, 420, 422, 424 around corresponding elements 202, 204, 208, 216, although such bounding boxes may not be included in the interior model 134 of the corresponding structure 108.

[0044] Returning to FIG. 1 , the computing device 102 may then generate a three-dimensional representation 122 of the structure 108 based on the internal model 134. For example, if the internal model 134 is a two-dimensional representation of the interior of the structure 108, the computing device 102 may extrude individual internal features 140 within the internal model 134 to generate the three-dimensional representation 122. In particular, the internal features 140 may be extruded according to physical dimensions 310 stored in association with the internal features 140. For example, the physical dimensions 310 may specify a height of one or more individual internal features 140. In such a case, the computing device 102 may extrude the individual internal features 140 according to the specified height while generating the three-dimensional representation 122. Furthermore, the computing device 102 may combine adjacent internal features 140 within the three-dimensional representation 122 (e.g., by extruding between adjacent elements). In particular, the adjacent elements may be identified based on corresponding information stored within the internal features 140 (e.g., adjacent elements 318, 320, etc.).

[0045] In certain cases, the structure 108 may have more than one floor or level. For example, the computing device 102 may determine that the structure 108 includes multiple levels based on the exterior dimensions of the structure 108 determined based on the exterior image 112 (e.g., a height greater than a certain predefined threshold, such as 15 feet, 30 feet, 45 feet, etc.). Additionally or alternatively, the computing device 102 may determine that the structure 108 includes multiple levels based on the exterior features 132 identified within the exterior image 112. For example, the computing device 102 may detect the horizontal arrangement of windows at multiple heights (e.g., three different heights). In such cases, the computing device 102 may determine that the structure 108 has multiple levels (e.g., three different levels).

[0046] For a structure 108 having multiple levels, the machine learning model 124 may be configured to generate multiple interior models 134. For example, the machine learning model 124 may generate a separate interior model 134 for each separate level detected in the structure 108. In such cases, when generating the three-dimensional representation 122 of the structure 108, the computing device 102 may combine multiple three-dimensional representations of individual floors or levels of the structure 108. For example, the three-dimensional representation 122 may be formed by combining three separate three-dimensional representations of each of the three floors according to the order of the floors in the structure (e.g., from lowest to highest).

[0047] In certain implementations, the three-dimensional representation may include structural information regarding the structure 108. For example, the three-dimensional representation 122 may store materials 314, 316 of individual elements (individual internal features, individual external features, etc.) within the three-dimensional representation 122. In particular, each individual element or feature within the three-dimensional representation 122 may include corresponding information (label, physical dimensions, material, adjacent elements, etc.) stored in association with a corresponding extruded internal feature 140.

[0048] Returning to the machine learning model 124, FIG. 1 depicts the machine learning model 124 as a single machine learning model. However, in certain implementations, the machine learning model 124 may be depicted as two or more models. For example, in certain implementations, the machine learning model 124 may be implemented as three separate machine learning models: a first model for generating the exterior surface 126, a second model for identifying the exterior features 132, and a third model for generating the interior model 134. Furthermore, it should be understood that although the terms "machine learning model" and "internal model" both include the word "model," the internal model 134 should not be understood to include a machine learning model or other predictive model. In particular, as described above, the internal model 134 should be understood to include a representation (such as a two-dimensional and / or three-dimensional representation) of the interior of the structure 108.

[0049] Further, the machine learning model 124 may be trained to accurately generate the exterior surface 126, the external features 132, and the internal model 134. In particular, the machine learning model 124 may be trained by the computing device 102 and / or another computing device based on data contained in the training database 106. In particular, the training database 106 may include the external images 118, 120 stored in association with the expected exterior surfaces 128, 130, the expected external features 146, 148, and the expected internal models 136, 138. Techniques for training the machine learning model 124 using data from the training database 106 are discussed in more detail below in connection with the method 700 and FIG.

[0050] In certain implementations, multiple structures may be depicted in the external image 112. In such cases, the computing device 102 may detect each of the multiple structures 108 and may repeat the process for each of the structures 108 identified in the image. For example, the machine learning model 124 may detect multiple exterior surfaces 126 for the multiple structures 108 in the external image 112. Upon detecting the multiple structures, the machine learning model 124 may then proceed to identify the exterior surfaces 126 and external features 132 of each of the multiple structures. The machine learning model 124 may also proceed to generate an internal model 134 for each of the individual structures. This processing may be similar to the techniques described above in connection with the structures 108. In certain cases, the multiple structures may be processed one at a time (e.g., in the order in which they are detected in the external image 112). Additionally or alternatively, the structures may be processed at least partially in parallel. For example, the computing device 102 may run multiple instances of the machine learning model 124, with each instance responsible for processing a single structure detected within the external image 112. As another example, the machine learning model 124 may identify the exterior surfaces 126 of each of the identified structures before identifying the external features 132 of each of the multiple structures, and then ultimately generate respective interior models 134 for the multiple structures.

[0051] The computing device 102 also includes a processor 142 and a memory 144. The processor 142 and the memory 144 may implement one or more aspects of the computing device 102. For example, the memory 144 may store one or more instructions that, when executed by the processor 142, may cause the processor 142 to perform one or more operational features of the computing device 102 (e.g., to implement the machine learning model 518). The processor 142 may be implemented as one or more central processing units (CPUs), field programmable gate arrays (FPGAs), and / or graphics processing units (GPUs) configured to execute the instructions stored in the memory 144. Additionally, the computing device 102 may be configured to communicate (e.g., with the database 104 and / or the training database 106) using a network 508. For example, the computing device 102 may communicate with the network 508 using one or more wired network interfaces (such as an Ethernet interface) and / or wireless network interfaces (such as a Wi-Fi, Bluetooth, and / or cellular data interface). In particular cases, the network may be implemented as a local network (such as a local area network), a virtual private network, an L1, and / or a global network (such as the Internet).

[0052] 5 illustrates a system 500 for generating internal model training data according to an exemplary embodiment of the present disclosure. For example, the system 500 may be configured to generate at least a portion of the training data stored in the training database 106 of the system 100. In particular, the system 500 may be configured to generate the predicted internal models 136, 138 described above. The system 500 includes a computing device 502, a database 504, and a training database 506. The training database 506 may be an exemplary implementation of the training database 106. The training database 506 stories the internal models 524, 526, which may be an exemplary implementation of the predicted internal models 136, 138.

[0053] The computing device 502 may receive the architectural plans 510 and generate one or more interior models 522 based on the architectural plans 510. In particular, the architectural plans 510 may include one or more floor plans 516 of a structure, and the computing device 502 may generate the interior model 522 of the structure based on the floor plans 516. The computing device 502 may receive the architectural plans 510 from a database 504 configured to store the architectural plans 512, 514. For example, the database 504 may store blueprints, construction plans, and / or any other architectural plans for a plurality of structures. In particular, in certain cases, the database 504 may be a government database or a commercial database of the architectural plans 512, 514.

[0054] The computing device 502 may generate the interior model 522 using the machine learning model 518. In a preferred embodiment, the machine learning model 518 is separate from the machine learning model 124. The machine learning model 518 may detect one or more internal features in the floor plan 516 and generate the interior model 522 based on the internal features and their relative positions in the floor plan 516. In particular, the machine learning model 518 may be configured to identify which sheets in the architectural plan 510 contain or depict the floor plan 516 and associated structures. The machine learning model 518 may then use one or more image processing techniques to detect and extract individual features in the floor plan 516. Detecting and extracting the internal features 520 may include identifying various types of information about the internal features, including labels, physical dimensions, materials, adjacent elements, similar to the internal features 140, 302, 304 described above. The machine learning model 518 may then build the interior model 522 including the internal features 520 based on the relative positions of the internal features 520 in the floor plan 516. Further details regarding the techniques used to detect and generate internal features 520 and internal models 522 are further described in U.S. patent application Ser. No. 17 / 487,838, entitled “Generating Vector Versions of Structural Plans,” filed Sep. 28, 2021, the entirety of which is incorporated by reference into this specification for all purposes.

[0055] Once the internal model 522 is generated, the internal model 522 may then be used to train other machine learning models. For example, the internal model 522 may be stored in a training database 506 that includes multiple internal models 524, 526. These internal models 522, 524, 526 may be used to train machine learning models for various purposes. In one particular case, the internal models 522, 524, 526 may be stored in the training database 506 in association with external images of the corresponding structures. In such a case, the internal models 522, 524, 526 may be used to train the machine learning model 124. In particular, the internal models 522, 524, 526 may be exemplary implementations of the expected internal models 136, 138 in the training database 106.

[0056] The computing device 502 also includes a processor 528 and a memory 530. The processor 528 and the memory 530 may implement one or more aspects of the computing device 502. For example, the memory 530 may store one or more instructions that, when executed by the processor 528, may cause the processor 528 to perform one or more operational features of the computing device 502 (e.g., to implement the machine learning model 518). The processor 528 may be implemented as one or more central processing units (CPUs), field programmable gate arrays (FPGAs), and / or graphics processing units (GPUs) configured to execute the instructions stored in the memory 530. Additionally, the computing device 502 may be configured to communicate (e.g., with the database 504 and / or the training database 506) using the network 508. For example, the computing device 502 may communicate with the network 508 using one or more wired network interfaces (such as an Ethernet interface) and / or wireless network interfaces (such as a Wi-Fi, Bluetooth, and / or cellular data interface). In certain cases, the network may be implemented as a local network (such as a local area network), a virtual private network, an L1, and / or a global network (such as the Internet).

[0057] In certain cases, all or parts of the systems 100, 500 may be combined. For example, as previously described, the training database 506 may be implemented at least in part by the training database 106. Additionally or alternatively, the computing devices 102, 502 may be implemented by the same computing device. In further embodiments, the systems 100, 500 may be implemented in a distributed computing environment (such as a cloud computing environment). In such cases, each of the systems 100, 500 may be implemented by one or more (e.g., multiple) computing devices in the distributed computing environment (e.g., in one or more clusters of the distributed computing environment).

[0058] FIG. 6 illustrates a method 600 for predicting an internal model of a structure according to an exemplary aspect of the disclosure. In particular, the method 600 may be performed to predict an internal model 134 of a structure 108 based on an external image 112 of the structure. The method 600 may be implemented on a computer system, such as the system 100. For example, the method 600 may be implemented by the computing device 102. The method 600 may also be implemented by a set of instructions stored on a computer-readable medium that, when executed by a processor, causes the computer system to perform the method 600. For example, all or a portion of the method 600 may be implemented by the processor 142 and the memory 144. The following example is described with reference to the flow chart shown in FIG. 6, although many other ways of performing the operations associated with FIG. 6 may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, one or more of the blocks may be repeated, and some of the described blocks may be optional.

[0059] The method 600 may begin with receiving an exterior image of a structure (block 602). For example, the computing device 102 may receive an exterior image 112 of the structure 108. The exterior image 112 may depict one or more exterior surfaces of the structure 108. For example, the exterior image 112 may depict an exterior surface of the structure 108 as seen from above, below, or outside the structure 108. In certain implementations, as described above, the exterior image 112 may depict a three-dimensional appearance of the structure 108.

[0060] An exterior surface of the structure may be determined (block 604). For example, the heating device 102 may determine the exterior surface 126 of the structure 108. In certain implementations, the exterior surface 126 may include a three-dimensional representation of the exterior dimensions of the structure 108. For example, the exterior surface 126 may represent an exterior contour of a building as seen from within the exterior image 112. Thus, the exterior surface 126 may include contours or other representations of various exterior features 132 (decorative features, functional features, structural features, etc.) on the structure 108. The exterior surface 126 may be determined using a machine learning model 124. For example, the machine learning model 124 may be configured to extract the exterior surface 126 from one or more exterior images of the structure 108 included within the exterior image 112. In embodiments in which the external image 112 includes a three-dimensional representation (e.g., a three-dimensional image) of the structure 108, determining the exterior surface 126 may include extracting a three-dimensional contour of the structure 108 from contours of other structures in a surrounding area in the external image 112. In embodiments in which the external image 112 does not include a three-dimensional representation of the structure 108, determining the exterior surface 126 may include combining multiple perspective views of the structure 108 into the exterior surface 126.

[0061] Exterior features of the structure may be determined (block 606). For example, the computing device 102 may determine exterior features 132 of the structure 108. The exterior features 132 may be identified from within the exterior image 112. For example, the computing device 102 may use the machine learning model 124 to identify the exterior features 132. In one particular instance, the machine learning model 124 may be configured to identify doors, windows, visible structural beams, HVAC system components, and exterior plumbing fixtures as the exterior features 132. The exterior features 132 may be based on a visual depiction of the exterior of the structure 108 in the exterior image 112. Additionally, the exterior features 132 may be identified within the exterior surface 126 and / or within a three-dimensional representation of the structure 108 in the exterior image 112 (if included).

[0062] An interior model of the structure may be determined based on the exterior surfaces and the exterior features (block 608). For example, the computing device 102 may determine an interior model 134 of the structure 108 based on the exterior surfaces 126 and the exterior features 132. For example, as described above, the interior model 134 may be generated by predicting one or more interior features 140 contained within the structure 108. The interior features 140 may be predicted according to one or more of the exterior features 132, common construction practices for structures similar to the structure 108, regulatory requirements for structures similar to the structure 108, structural design rules of thumb, and the like. The interior features 140 may then be placed within the interior model 134 based on the exterior surfaces 126 and / or the exterior features 132. For example, if the exterior features 132 include one or more features common to both the interior and exterior of a building (e.g., doors, windows, HVAC ducts, etc.), the interior features 140 may be placed to align with the corresponding exterior features 132 (e.g., interior doors and windows on the exterior walls align with exterior doors and windows). As another example, the internal features 140 may be positioned to fit within dimensions dictated by the exterior surface 126 (e.g., to fit within an interior space of the structure 108 as indicated by the exterior dimensions of the structure 108 within the exterior surface 126). The internal model 134 is then generated as a representation (a three-dimensional representation, a two-dimensional representation, etc.) of the internal features 140.

[0063] A three-dimensional representation of the structure may be generated (block 610). For example, the computing device 102 may generate the three-dimensional representation 122 of the structure 108 based on the external structure 126 and / or the internal model 134. In particular, as described above, the internal features 140 and / or the external features 132 may specify physical dimensions 310, 312 of the individual features 140, 132. Accordingly, the computing device 102 may be configured to extrude the external features 132 and the internal features 140 according to these physical dimensions 310, 312. In additional or alternative embodiments, the internal model 134 may be generated as a three-dimensional representation of the interior of the structure 108. In such instances, the computing device 102 may generate the three-dimensional representation 122 of the structure 108 by combining the three-dimensional exterior surface 126 of the structure 108 with the three-dimensional internal model 134 of the structure 108.

[0064] Thus, method 600 allows a computing device to predict interior models of buildings and other structures without looking inside the structure or analyzing the structure's interior blueprints. The interior models generated according to method 600 may be used for structural or other analysis of the structure. For example, the predicted interior layout represented by the interior model may be used by emergency response teams (e.g., paramedics, firefighters, police officers, etc.) to predict the interior layout of a structure in which an emergency is occurring. Such blueprints may then be used by the emergency response teams in navigating the interior of the structure to locate and resolve the emergency. The interior model may also be useful in combat scenarios. For example, military operatives may utilize the interior model to assist in navigating the interior of a building, similar to the emergency response teams described above. As another example, the interior model may be used for destruction analysis of the structure (e.g., predicting the minimum amount of munitions required to destroy or disable the structure 108 to reduce collateral damage from the structure 108 falling on other nearby structures). As discussed above, previous systems for performing these functions have typically relied on interior views or blueprints of the structure or on manual work by specialized engineers, which may be unavailable or too time consuming to be used in emergency or combat situations.

[0065] FIG. 7 illustrates a method 700 for training a machine learning model to predict an internal model according to an exemplary embodiment of the present disclosure. In particular, the method 700 may be used to train a machine learning model, such as the machine learning model 124, to predict an internal model 134 of a structure 108 based on an external image 112 of the structure 108. The method 700 may be implemented on a computer system, such as the system 500. For example, the method 700 may be implemented by the computing device 502. The method 700 may also be implemented by a set of instructions stored on a computer-readable medium that, when executed by a processor, causes the computer system to perform the method 700. For example, all or a portion of the method 700 may be implemented by the processor 142 and the memory 144. The following example is described with reference to the flowchart shown in FIG. 7, although many other ways of performing the operations associated with FIG. 7 may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, one or more of the blocks may be repeated, and some of the described blocks may be optional.

[0066] The method 700 may begin with receiving training data for a plurality of structures (block 702). For example, the computing device 102 (or another computing device) may receive the training data for a plurality of structures. The training data may be stored in the training database 106. For example, the training data may include exterior images 118, 120 of the plurality of structures. The training data may further include predicted interior models 136, 138 and predicted exterior surfaces 128, 130 of the structures along with predicted exterior features 146, 148 of the structures. In certain implementations, the predicted interior models 136, 138 may be generated by another machine learning model 518. For example, as described in more detail above in connection with the system 500, the predicted interior models may be generated by another machine learning model 518 based on architectural blueprints.

[0067] The first machine learning model may be trained to generate predicted exterior surfaces and predicted external features (block 704). For example, the first machine learning model 124 may be trained to predict the exterior surface 126 and external features 132 of the structure 108 based on the exterior image 112 of the structure 108. Training the machine learning model 124 in this manner may include providing the external image 118 to the machine learning model 124 and having the machine learning model 124 generate one or more predicted exterior surfaces in the external features based on the external images 118, 120, where the predicted exterior surfaces generated by the machine learning model 124 may be compared to the expected exterior surfaces 128, 130 associated with the external images 118, 120 in the training data. Similarly, the predicted external features may be compared to the expected external features 146, 148 associated with the external images 118, 120 in the training data. One or more differences between the predicted exterior surface and the expected exterior surface 128, 130 and / or between the predicted external features and the expected external features 146, 148 may be detected. For example, the predicted exterior surface may include a three-dimensional shape that is different from the shape of the corresponding expected exterior surface. As another example, the predicted external features may include features that are not present in the corresponding expected external features, or may not include features that are present in the expected external features. As a further example, one or more aspects of the predicted external features (such as the metadata of FIG. 3) may differ from aspects of the corresponding expected external features. Based on these differences, one or more parameters of the machine learning model 124 may be adjusted. For example, one or more weights associated with individual features (such as individual spatial features in the external image) may be adjusted. Additionally or alternatively, one or more individual features may be added to or removed from the machine learning model 124.

[0068] The first machine learning model may also be trained to predict the interior model (block 706). For example, the machine learning model 124 may be trained to predict the interior model 134 of the structure. Training the machine learning model 124 in this manner may include comparing the interior model generated by the machine learning model to the expected interior models 136, 138. For example, the computing device 102 (or another computing device) may provide the machine learning model 124 with the external images 118, 120. The machine learning model 124 may generate a predicted interior model of the corresponding structure based on the external images 118, 120. For example, the machine learning model 124 may generate a predicted exterior surface and predicted exterior features for each set of received external images 118, 120. Based on the images 118, 120, the predicted exterior surface, and / or the predicted exterior features, the machine learning model 124 may then generate a predicted interior model of the corresponding structure. These predicted internal models may then be compared to expected internal models 136, 138 corresponding to each external image 118, 120. One or more differences between the predicted internal models and the expected internal models 136, 138 may then be identified. For example, the predicted internal models may include internal features that are not present in the corresponding expected internal models, or may not include internal features included in the corresponding predicted internal models. As another example, one or more aspects of the predicted internal features (such as the metadata of FIG. 3) may differ from aspects of the corresponding expected internal features. As a further example, one or more internal features may be in a different location in the predicted internal model than in the corresponding expected internal model. Based on these differences, one or more parameters of the machine learning model 124 may be adjusted. For example, one or more weights associated with individual features (such as individual spatial features in the external images) may be adjusted. Additionally or alternatively, one or more individual features may be added to or removed from the machine learning model 124. In particular cases, the machine learning model 124 may be trained according to (eg, to maximize) one or more objective functions.In certain cases, these objective functions may be formulated to enforce certain constraints (e.g., to ensure that all interior features fall within corresponding portions of the exterior surface). Other constraints may include, where appropriate, ensuring that interior features match corresponding exterior features, or that the placement of certain interior features (such as structural assemblies) conforms to common construction practices, requirements, and / or rules of thumb.

[0069] The first machine learning model may deploy the machine learning model to predict the exterior surface and the interior model (block 708). For example, the machine learning model 124 may be deployed to predict the exterior surface 126 and the interior model 134. For example, the machine learning model 124 may be deployed within the computing device 102 for use in predicting the exterior surface and the interior model in real time based on an external image of the structure separate from that used in training the machine learning model 124. For example, a user may capture or otherwise identify (e.g., within the database 104) an exterior image 112 of the structure 108. The external image 112 may then be provided to the computing device 102, which may then utilize the machine learning model 124 to predict the exterior surface 126 and the interior model 134 of the structure 108. In certain cases, the exterior surface 126 and the interior model 134 may then be used to generate a three-dimensional representation 122 of the structure 108, as described above.

[0070] Thus, the method 700 enables a computing device to train a machine learning model for use in predicting exterior and interior models of a structure. As further described above, such a representation of a structure may be useful in many scenarios (combat scenarios, emergency response scenarios, etc.). Moreover, in certain implementations, the method 700 relies on unique training data received from a second machine learning model that can generate interior models of buildings based solely on their architectural blueprints. Such a system dramatically increases the available training data for the machine learning model 124 trained in the method 700. Thus, a machine learning model trained in this manner may be significantly more accurate in its interior model predictions than a model that relies on traditionally available training data.

[0071] Moreover, method 700 is flexible enough to be used with different types of model architectures. For example, in certain implementations, machine learning model 124 may be implemented as two or more individual machine learning models. For example, a first machine learning model may be used to predict exterior surface 126 and exterior features 132, and a second machine learning model may be used to predict interior model 134. In such cases, block 704 may be performed to train the first machine learning model, and block 706 may be performed to train the second machine learning model.

[0072] FIG. 8 illustrates an exemplary computer system 800 that may be utilized to implement one or more of the devices and / or components discussed herein, such as computing devices 102, 502, databases 104, 504, and training databases 106, 506. In certain aspects, one or more computer systems 800 perform one or more steps of one or more methods described or illustrated herein. In certain aspects, one or more computer systems 800 provide functionality described or illustrated herein. In certain aspects, software operating on one or more computer systems 800 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Certain aspects include one or more portions of one or more computer systems 800. Reference herein to a computer system may encompass a computing device, where appropriate, and vice versa. Additionally, reference to a computer system may encompass one or more computer systems, where appropriate.

[0073] The present disclosure contemplates any suitable number of computer systems 800. The present disclosure contemplates computer system 800 taking any suitable physical form. By way of example and not limitation, computer system 800 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on module (COM) or a system on module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, computer system 800 may include one or more computer systems 800, may be single or distributed, may span multiple locations, may span multiple machines, may span multiple data centers, or may reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 800 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 800 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. One or more computer systems 800 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations, where appropriate.

[0074] In particular aspects, computer system 800 includes a processor 806, memory 804, storage 808, an input / output (I / O) interface 810, and a communication interface 812. Although this disclosure describes and illustrates particular computer systems having particular numbers of particular components in particular arrangements, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0075] In certain aspects, the processor 806 includes hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, to execute an instruction, the processor 806 may retrieve (or fetch) an instruction from an internal register, an internal cache, memory 804, or storage 808, decode and execute the instruction, and then write one or more results to an internal register, an internal cache, memory 804, or storage 808. In certain aspects, the processor 806 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates the processor 806 including any suitable number of any suitable internal caches, where appropriate. By way of example and not limitation, the processor 806 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 804 or storage 808, and the instruction cache may speed up retrieval of those instructions by the processor 806. The data in the data cache may be a copy of data in memory 804 or storage 808 to be operated on by a computer instruction, a result of a previous instruction executed by the processor 806 accessible to a subsequent instruction or for writing to the memory 804 or storage 808, or any other suitable data. The data cache may speed up read or write operations by the processor 806. The TLB may speed up virtual address translation for the processor 806. In certain aspects, the processor 806 may include one or more internal registers for data, instructions, or addresses. The present disclosure contemplates a processor 806 including any suitable number of any suitable internal registers, where appropriate. The processor 806 may include one or more arithmetic logic units (ALUs), may be a multi-core processor, and may include one or more processors 806, where appropriate. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0076] In certain aspects, memory 804 includes a main memory for storing instructions for processor 806 to execute or data for processor 806 to manipulate. By way of example and not limitation, computer system 800 may load instructions from storage 808 or another source (such as another computer system 800) into memory 804. Processor 806 may then load instructions from memory 804 into an internal register or cache. To execute instructions, processor 806 may retrieve instructions from the internal register or cache and decode them. During or after execution of instructions, processor 806 may write one or more results (which may be intermediate or final results) to an internal register or cache. Processor 806 may then write one or more of those results to memory 804. In certain aspects, the processor 806 executes only instructions in one or more internal registers or caches or in memory 804 (and not in storage 808 or elsewhere) and operates only on data in one or more internal registers or caches or in memory 804 (and not in storage 808 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 806 to the memory 804. The buses may include one or more memory buses, as described in further detail below. In certain aspects, one or more memory management units (MMUs) are present between the processor 806 and the memory 804 to facilitate accesses to the memory 804 requested by the processor 806. In certain aspects, the memory 804 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Further, this RAM may be single-ported RAM or multi-ported RAM, where appropriate. This disclosure contemplates any suitable RAM. Memory 804 may include one or more memories 804, where appropriate.Although this disclosure describes and illustrates particular memory implementations, this disclosure contemplates any suitable memory implementation.

[0077] In certain aspects, storage 808 includes mass storage for data or instructions. By way of example, and not limitation, storage 808 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 808 may include removable or non-removable (or fixed) media, where appropriate. Storage 808 may be internal or external to computer system 800, where appropriate. In certain aspects, storage 808 is non-volatile solid-state memory. In certain aspects, storage 808 includes read-only memory (ROM). Where appropriate, this ROM may be mask program ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates mass storage 808 taking any suitable physical form. Storage 808 may include, where appropriate, one or more storage control units that facilitate communications between processor 806 and storage 808. Where appropriate, storage 808 may include one or more storages 808. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0078] In certain aspects, I / O interface 810 includes hardware, software, or both that provide one or more interfaces for communication between computer system 800 and one or more I / O devices. Computer system 800 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person (i.e., a user) and computer system 800. By way of example and not limitation, an I / O device may include a keyboard, keypad, microphone, monitor, screen, display panel, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I / O device, or a combination of two or more of these. An I / O device may include one or more sensors. Where appropriate, I / O interface 810 may include one or more device drivers or software drivers that enable processor 806 to drive one or more of these I / O devices. I / O interface 810 may include one or more I / O interfaces 810, where appropriate. Although this disclosure describes and illustrates particular I / O interfaces, this disclosure contemplates any suitable I / O interface or combination of I / O interfaces.

[0079] In certain aspects, the communication interface 812 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication, etc.) between the computer system 800 and one or more other computer systems 800 or one or more networks 814. By way of example and not limitation, the communication interface 812 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or any other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a Wi-Fi network. The present disclosure contemplates any suitable network 814 and any suitable communication interface 812 for the network 814. By way of example and not limitation, the network 814 may include one or more of an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 800 may communicate with a wireless PAN (WPAN) (e.g., a Bluetooth® WPAN, etc.), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network, etc.), or any other suitable wireless network, or a combination of two or more of these. Computer system 800 may include any suitable communications interface 812 for any of these networks, where appropriate. Communications interface 812 may include one or more communications interfaces 812, where appropriate. Although this disclosure describes and illustrates particular communications interface implementations, this disclosure contemplates any suitable communications interface implementations.

[0080] Computer system 802 may also include a bus. A bus may include hardware, software, or both and may communicatively couple components of computer system 800 to one another. By way of example and not limitation, a bus may include an Accelerated Graphics Port (AGP) bus or any other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a Low PIN Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB), or another suitable bus, or a combination of two or more of these buses. A bus may include one or more buses, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0081] As used herein, one or more computer-readable non-transitory storage media may include, where appropriate, one or more semiconductor-based or other types of integrated circuits (ICs) (such as field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. A computer-readable non-transitory storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0082] In this specification, "or" is inclusive and not exclusive, unless otherwise expressly indicated or otherwise understood by the context. Thus, in this specification, "A or B" means "A, B, or both," unless otherwise expressly indicated or otherwise understood by the context. Furthermore, "and" is both conjunctive and individual, unless otherwise expressly indicated or otherwise understood by the context. Thus, in this specification, "A and B" means "both A and B, or A and B individually," unless otherwise expressly indicated or otherwise understood by the context.

[0083] The scope of the disclosure encompasses all changes, substitutions, variations, modifications, and alterations to the exemplary aspects described or illustrated herein that would be understood by a person skilled in the art. The scope of the disclosure is not limited to the exemplary aspects described or illustrated herein. Furthermore, although the disclosure describes and illustrates each aspect herein as including certain components, elements, features, functions, operations, or steps, any of these aspects may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person skilled in the art. Furthermore, when the appended claims refer to an apparatus or system or a component of an apparatus or system that is adapted, arranged, capable, configured, enabled, operable, or operates to perform a particular function, it encompasses that apparatus, system, or component, whether or not it or that particular function is activated, turned on, or unlocked, so long as the apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operates. Additionally, although the present disclosure describes or illustrates particular aspects as providing certain advantages, the particular aspects may provide none, some, or all of these advantages.

[0084] All of the disclosed methods and procedures described in this disclosure can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer-readable or machine-readable medium, including volatile and non-volatile memories, such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be provided as software or firmware, and may be implemented in whole or in part in hardware components, such as ASIC, FPGA, DSP, or any other similar device. The instructions may be configured to be executed by one or more processors that, upon executing a series of computer instructions, perform or facilitate the execution of all or a portion of the disclosed methods and procedures.

[0085] It should be understood that various changes and modifications to the examples described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. Accordingly, such changes and modifications are intended to be included within the scope of the appended claims.

Claims

1. receiving an exterior image of the structure; determining an exterior surface of the structure that surrounds an exterior portion of the structure depicted in the exterior image using a machine learning model; determining external characteristics of the structure based on the external image and / or the external surface using the machine learning model; determining an internal model of the structure based on the exterior surface and the external features using the machine learning model; and generating a three-dimensional representation of an interior portion of the structure and an exterior portion of the structure based on the exterior surface and the internal model; A method comprising:

2. The method of claim 1 , wherein the exterior features include at least one of a door, a window, a structural support element, a corner, a roof, and / or a utility system of the structure.

3. 2. The method of claim 1, wherein the structure has multiple floors and the step of determining the internal model comprises determining multiple internal models for the multiple floors.

4. The method of claim 3 , wherein the levels of the structure are identified based on an external characteristic of the structure.

5. The method of claim 4 , wherein the multiple floors are identified based on multiple levels of windows at multiple heights within the structure.

6. Prior to the step of receiving an external image, receiving training data for a plurality of structures, the training data including exterior images of the plurality of structures, predicted exterior surfaces for the plurality of structures, predicted exterior features for the plurality of structures, and predicted internal models for the plurality of structures; training a first machine learning model to generate predicted exterior surfaces and predicted external features of at least a subset of the plurality of structures based on at least (i) exterior images of the subset of the plurality of structures and (ii) expected exterior surfaces of the subset of the plurality of structures; and training the first machine learning model to predict an internal model of at least the subset of the plurality of structures based at least on the predicted exterior surface and the predicted external features.

2. The method of claim 1, further comprising:

7. Prior to the step of receiving the training data, receiving a plurality of architectural plans for the plurality of structures; and generating the predicted internal models for the plurality of structures based on the plurality of architectural plans using a second machine learning model; 7. The method of claim 6, further comprising:

8. Training the first machine learning model to predict outer contours and external features includes: receiving a first exterior image of a first structure from the plurality of structures; predicting a first exterior surface and a first exterior feature of the first structure based on the first exterior image using the first machine learning model; (i) detecting one or more differences between the first exterior surface of the first structure and an expected exterior surface of the first structure, and / or (ii) between the first exterior feature and an expected exterior feature of the first structure; and adjusting one or more parameters of the first machine learning model based on the one or more differences.

7. The method of claim 6, comprising:

9. training the first machine learning model to predict the internal model, receiving an exterior contour and an exterior characteristic of a first structure from the plurality of structures; predicting a first internal model of the first structure based on the outer contour of the first structure using the first machine learning model; detecting one or more differences between the first internal model of the first structure and an expected internal model of the first structure; and adjusting one or more parameters of the first machine learning model based on the one or more differences.

7. The method of claim 6, comprising:

10. The method of claim 1 , wherein the structure comprises at least one of a building, a vehicle, an infrastructure component, a watercraft, a spacecraft, an aircraft, a tank, and / or an appliance.

11. receiving training data for a plurality of structures, the training data including exterior images of the plurality of structures, predicted exterior surfaces for the plurality of structures, predicted exterior features for the plurality of structures, and predicted internal models for the plurality of structures; training a first machine learning model to generate predicted exterior surfaces and predicted external features of at least a subset of the plurality of structures based on at least (i) exterior images of the subset of the plurality of structures and (ii) expected exterior surfaces of the subset of the plurality of structures; training the first machine learning model to predict an internal model of at least the subset of the plurality of structures based at least on the predicted exterior surface and the predicted external features; and deploying the first machine learning model to predict an outer contour and an internal model for an additional structure distinct from the plurality of structures. A method comprising:

12. Prior to the step of receiving the training data, receiving a plurality of architectural plans for the plurality of structures; and generating the predicted internal models for the plurality of structures based on the plurality of architectural plans using a second machine learning model; 12. The method of claim 11, further comprising:

13. Training the first machine learning model to predict outer contours and external features includes: receiving a first exterior image of a first structure from the plurality of structures; predicting a first exterior surface and a first exterior feature of the first structure based on the first exterior image using the first machine learning model; (i) detecting one or more differences between the first exterior surface of the first structure and an expected exterior surface of the first structure, and / or (ii) between the first exterior feature and an expected exterior feature of the first structure; and adjusting one or more parameters of the first machine learning model based on the one or more differences.

12. The method of claim 11, comprising:

14. training the first machine learning model to predict the internal model, receiving an exterior surface and an exterior characteristic of a first structure from said plurality of structures; predicting a first interior model of the first structure based on the exterior surface of the first structure using the first machine learning model; detecting one or more differences between the first internal model of the first structure and an expected internal model of the first structure; and adjusting one or more parameters of the first machine learning model based on the one or more differences.

12. The method of claim 11, comprising:

15. 15. The method of claim 14, wherein the exterior surface of the first structure is one of an expected exterior surface of the first structure included in the training data and / or a predicted exterior surface of the first structure generated by the first machine learning model, and the external feature of the first structure is one of an expected exterior feature of the first structure included in the training data and / or a predicted external feature of the first structure generated by the first machine learning model.

16. 12. The method of claim 11, wherein training the first machine learning model to predict the interior model comprises training the first machine learning model to generate an interior model that conforms to at least one of: (i) spatial constraints of the exterior surface; and / or (ii) typical construction methods and structural design requirements.

17. 17. The method of claim 16, wherein the at least one of (i) the spatial constraints of the exterior surface and / or (ii) the general construction practices and structural design requirements are represented in an objective function for the first machine learning model.

18. The step of deploying the first machine learning model comprises: Receiving external images of the structure; determining an exterior surface, exterior features, and an interior model of the structure using a machine learning model; and generating a three-dimensional representation of an interior portion of the structure and an exterior portion of the structure based on the exterior surface and the interior model; 12. The method of claim 11, further comprising:

19. determining the outer contour, external features, and internal model of the structure; determining, using a machine learning model, an exterior surface of the structure that surrounds an exterior portion of the structure depicted in the exterior image; determining external characteristics of the structure based on the external image and / or the exterior surface using the machine learning model; and determining an internal model of the structure based on the outer contour and the external features using the machine learning model; 20. The method of claim 18, further comprising:

20. The method of claim 13 , wherein the exterior features include at least one of a door, a window, a structural support element, a corner, and / or a utility system of the structure.

Images