Automated three-dimensional building model estimation

Abstract

Automated three-dimensional (3D) building model estimation to predict roof profile, gradient, and height based on images and 3D data is disclosed. In an embodiment, a method includes obtaining an aerial image of a building based on an input address; obtaining three-dimensional (3D) data including the building based on the input address; preprocessing the aerial image and the 3D data; reconstructing a 3D building model from the pre-processed image and 3D data, the reconstructing comprising: predicting a mask of each roof component of the building using instance segmentation; predicting the contour of each roof component by using a first machine learning model taking the mask as an input; predicting the gradient and height of each roof component by using a second machine learning model taking the mask and the contour as input; and rendering the 3D building model based on the predicted profile, gradient, and height of each roof component.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 17,702,723, filed on March 23, 2022. The disclosure of the prior application is considered part of the disclosure of the present application and is incorporated by reference into the disclosure of the present application. Technical Field

[0003] The present disclosure generally relates to estimating three-dimensional (3D) building structures, such as roofs. Background Art

[0004] According to the International Energy Agency, solar energy is the fastest growing energy source in the world. Solar energy works by capturing the sun's energy and converting it into electricity for use in a home or business. The sun's energy is captured using solar panels, which are often installed in areas that can be exposed to the sun to the greatest extent, such as rooftops. Solar panels are composed of multiple solar cells made of silicon, which have positive and negative layers that create an electric field. When photons from the sun hit the solar cells, electrons are released from their atoms. By attaching conductors to the positive and negative sides of the solar cells, an electrical circuit is formed. When the electrons flow through the circuit, direct current (DC) is generated, which is converted into alternating current (AC) by an inverter to power a home or office. Excess power is stored in batteries.

[0005] The number of solar panels required for a solar energy system depends on how much energy the building uses, the available surface area of ​​the roof, the climate and peak sunlight at the building's location, and the wattage and relative efficiency of the solar panels. Multiple solar panels (modules) can be wired together to form a solar array. The peak sunlight hours at the building's location affect the amount of energy the solar array will produce. Also, the size and shape of the roof will affect the size of the solar panels and the number of solar panels used in the solar panel. The most popular solar panels are photovoltaic (PV) solar panels, which are manufactured in a standard size of approximately 65 inches by 39 inches, with some variation between manufacturers. The size and shape of the roof will directly affect the size and number of solar panels to be installed. Because the available roof area is large, larger panels can be installed at a lower cost per panel. However, if the available roof area is limited or partially shaded, fewer, smaller, high-efficiency panels can be installed, but at a higher cost per panel.

[0006] There are many different roof types that complicate solar system design, including but not limited to: Gable, Hip, Masard, Gambrel, Flat, Skillion, Jerkinhead, Butterfly, Bonnet, Saltbox, Sawtooth, Curved, Pyramid, Dome, and any combination of the foregoing. In addition, any structures installed on the roof (such as heating, air conditioning and ventilation (HVAC) equipment, chimneys, vents, etc.) will reduce the available surface area for solar panel installation.

[0007] Existing software solutions for optimizing solar panel installations use aerial imagery to estimate the available surface area of ​​a roof. However, these techniques often require extensive user input, making the design process tedious for the user. What is needed is an automated process that requires minimal user input to estimate the 3D building structure, and in particular, to determine the available area of ​​a 3D roof model with a high degree of accuracy in order to design and simulate a virtual solar system that can output performance data that can be used to design an actual solar system that achieves the user's target energy savings goals and other user goals. Summary of the invention

[0008] An automated three-dimensional (3D) building model estimation system and method is disclosed that predicts roof profile, slope, and height from imagery and 3D data.

[0009] In an embodiment, a method includes: using one or more processors to obtain an aerial image of a building based on an input address; using one or more processors to obtain three-dimensional (3D) data containing the building based on the input address; using one or more processors to preprocess the aerial image and the 3D data; using one or more processors to reconstruct a 3D building model from the preprocessed image and 3D data, the reconstruction including: using a first machine learning model to predict the outline of each roof component; using a second machine learning model to predict the slope and height of each roof component based on the predicted outline; and using one or more processors to render the 3D building model based on the predicted outline, at least one slope and height of each roof component.

[0010] In an embodiment, predicting the profile of each roof component using the first machine learning model also includes: for each roof component in the sequence of roof components, predicting the location of each peripheral edge of the roof component; and for each roof component, predicting the location of each fold in the roof component.

[0011] In an embodiment, locations are predicted by a neural network that outputs a probability distribution over potential locations.

[0012] In an embodiment, the probability distribution is used to guide the search process of estimating how good each prediction will be.

[0013] In an embodiment, the search process explores a specified number of forward steps and compares the resulting rooftop representation for each possible next node or polyline to the output of the instance segmentation network.

[0014] In an embodiment, the output of the instance segmentation network is considered to be a close approximation to the actual two-dimensional (2D) structure of the roof.

[0015] In an embodiment, the search results are used to update a probability distribution predicting the location of the next node or polyline.

[0016] In an embodiment, the search is a Monte Carlo Tree Search (MCTS).

[0017] In an embodiment, the first machine learning model and the second machine learning model are parts of a single neural network.

[0018] In an embodiment, preprocessing the aerial image and 3D data also includes: generating a 3D mesh from the 3D data; generating a digital surface model (DSM) of a building using the 3D mesh; aligning the image and the DSM; generating a building mask from the image; calculating the orientation of each roof surface of the building using the 3D data and the building mask; snapping the orientation of the building to a grid; obtaining the extent of the building using the building mask; and cropping the image so that the building is located in the center of the image and aligned with the grid axes.

[0019] In an embodiment, the method further comprises: predicting a mask for each roof component of the building using instance segmentation; predicting a contour of each roof component using a first machine learning model with the mask as input; and

[0020] A second machine learning model that takes the masks and contours as input is used to predict the slope and height of each roof component.

[0021] Other embodiments include, but are not limited to, systems and computer-readable storage media.

[0022] Certain embodiments disclosed herein provide one or more of the following advantages. An automated solar energy system design tool uses aerial imagery, 3D point clouds (e.g., radar point clouds), machine learning (e.g., neural networks), and shading algorithms to estimate the size and shape of a building's roof and determine the optimal location of solar panels to maximize exposure to sunlight. The disclosed embodiments are fully automated and require minimal user input, such as the user's home address, utility rates, and the user's average monthly energy bill. The output is an estimated 3D building model that is input into an automated design tool that generates a virtual solar energy system design based on the estimated 3D building model.

[0023] A virtual solar energy system is automatically simulated to determine its performance, including, for example, calculating the finances of solar energy production and estimating output power. The automated solar energy system design tool may be accessed by a consumer or professional solar panel installer, for example, via the World Wide Web or via an application programming interface (API).

[0024] The details of the disclosed embodiments are set forth in the accompanying drawings and the description that follows. Other features, objects, and advantages are apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A graphical user interface (GUI) of an automated solar energy system design tool according to an embodiment is illustrated.

[0026] Figure 2 is a flow chart of an automated process for estimating a 3D building model, according to an embodiment.

[0027] Figure 3 is an example input image for a pre-processing pipeline according to an embodiment.

[0028] Figure 4 is example input 3D data for a pre-processing pipeline according to an embodiment.

[0029] Figure 5A and Figure 5B An example pre-processing output is illustrated, wherein an image and a DSM derived from the 3D data are aligned with the grid axes, cropped around the building, and aligned with each other, according to an embodiment.

[0030] Figure 6 is a flow chart of a preprocessing pipeline according to an embodiment.

[0031] Figure 7 Illustrated are the results of a “spike-free” triangulation algorithm applied to 3D data to generate a 3D mesh in accordance with an embodiment.

[0032] Fig. 8Aand Figure 8B A registration process according to an embodiment is illustrated, whereby the DSM and the image are aligned.

[0033] Figure 9A-9C It illustrates the use of semantic segmentation techniques separately on the image and DSM to predict building structures, trees, and background, and then using cross-correlation to calculate where the image and DSM are aligned, according to an embodiment.

[0034] Fig. 10A and Fig. 10B Illustrated is the generation of a building mask from an image according to an embodiment.

[0035] Fig.11A and Fig. 11B Illustrated is the use of a building mask to obtain the extent of a building and crop the image so that the building is centered in the image and aligned with the grid axes, according to an embodiment.

[0036] Fig.12 is a flow chart of a reconstruction process for generating a 3D building model from pre-processed images and a DSM according to an embodiment.

[0037] Fig.13 Illustrated is an output mask generated for each roof plane by a neural network using an instance segmentation technique in accordance with an embodiment.

[0038] Fig.14 Illustrated is the use of heuristics and polygon fitting to convert roof planes into polygons, and then projecting the two-dimensional (2D) polygons into 3D polygons using a DSM, according to an embodiment.

[0039] Fig.15A and Fig. 15B Illustrated is a non-parametric retrieval technique for finding a matching roof type template in a database and then overlaying the template on an image, according to an embodiment.

[0040] Figure 16A-16C Illustrated is the use of a neural network to predict the offset of each node in the roof to adjust the internal structure of the roof template to match the image, according to an embodiment.

[0041] Fig.17 Illustrated is the output of a neural network for predicting edge types for all edges of a 3D polygon, according to an embodiment.

[0042] Fig.18 An alternative technique for wall detection using a neural network is illustrated, which is used in an alternative polygon fitting algorithm, according to an embodiment.

[0043] Fig.19An alternative polygon fitting algorithm is illustrated according to an embodiment, in which detected wall and roof planes are combined and all possible intersection points between them are determined.

[0044] Fig. 20 Illustrated is a synthetic radar technique in which a neural network is used to compute a height map from a 2D image, and then use the height map to convert the 2D image into a rough 3D model of a building, according to an embodiment.

[0045] Fig.21 A technique for detecting obstacles on a rooftop according to an embodiment is illustrated.

[0046] Fig. 22 is a flow chart of an automated 3D building estimation process according to an embodiment.

[0047] Fig.23 According to the embodiment, Fig. 22 Flowchart of the preprocessing process of the 3D building estimation process.

[0048] Fig.24A and Fig. 24B According to the embodiment, Fig. 22 Flowchart of the reconstruction process of the 3D building estimation process.

[0049] Fig.25A and Fig.25B are before and after images illustrating a capture algorithm for removing gaps between roof faces, according to an embodiment.

[0050] Fig.26 is a top view of a 2D mesh overlaid with an image showing a rooftop with no gaps, according to an embodiment.

[0051] Fig. 27 Illustrated is shading via ray tracing for a 3D building model, obstacles, and surroundings, according to an embodiment.

[0052] Fig.28 is used according to the example as reference Figure 1-Figure 27 The generated estimated 3D building model automates the solar energy system design process flow.

[0053] Fig.29 It is used to implement reference Figure 1-Figure 28 A block diagram of a computer architecture that describes the features and processes.

[0054] Fig.30 is a flow chart of an automated 3D building estimation process for predicting roof profiles, slopes, and heights based on aerial imagery and 3D data, according to an embodiment.

[0055] Figures 31A-31KFurther illustrated are steps of an automated 3D building estimation process for predicting roof profiles, slopes, and heights from aerial imagery and 3D data according to an embodiment.

[0056] Fig.32 is a flow chart illustrating a Monte Carlo Tree Search (MCTS) as applied to roof profile prediction according to an embodiment.

[0057] Fig.33 is a flow chart illustrating predicting the slope and height of a roof assembly according to an embodiment.

[0058] Fig.34A and Fig.34B Illustrated is a complete 3D model of a roof generated based on an automated 3D building estimation process that predicts roof profile, slope, and height from imagery and 3D data, according to an embodiment.

[0059] Fig.35 is a flow chart of an automated 3D building estimation process for predicting roof profiles, slopes, and heights from imagery and 3D data, according to an embodiment.

[0060] The use of the same reference numbers in different drawings indicates similar elements. DETAILED DESCRIPTION

[0061] Explanation of terms / diagrams

[0062] In the following detailed description, for the purpose of explanation, numerous specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, it is apparent that the disclosed embodiments can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessary obfuscation of the present invention.

[0063] In the accompanying drawings, for ease of description, a specific arrangement or ordering of schematic elements is shown, such as the arrangement or ordering of representation devices, modules, instruction blocks, and data elements. However, it should be understood by those skilled in the art that the specific ordering or arrangement of schematic elements in the accompanying drawings does not imply a requirement for a specific processing order or sequence, or separation of processes. In addition, the inclusion of schematic elements in the accompanying drawings does not imply that such elements are required in all embodiments or that the features represented by such elements may not be included in some embodiments or may be combined with other elements in some embodiments.

[0064] In addition, in the accompanying drawings that connecting elements (such as solid or dotted lines or arrows) are used to illustrate the connection, relationship or association between two or more other schematic elements, the lack of any such connecting elements does not mean that there may be no connection, relationship or association. In other words, some connections, relationships or associations between elements are not shown in the accompanying drawings to avoid confusing the present disclosure. In addition, for ease of explanation, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, in the case where a connecting element represents the communication of a signal, data or instruction, it will be understood by those skilled in the art that, as required, such element represents one or more signal paths (e.g., bus) to affect communication.

[0065] Several features are described below, each of which can be used independently of each other or in any combination with the other features. However, any individual feature may not solve any of the problems discussed above or may only solve one of the problems discussed above. Some of the problems discussed above may not be fully solved by any of the features described herein. Although headings are provided, information related to a particular heading but not found in the section with that heading may also be found elsewhere in this description.

[0066] As used herein, the term "one or more" includes a function performed by an element, a function performed by more than one element (e.g., in a distributed manner), several functions performed by an element, several functions performed by several elements, or any combination thereof. It will also be understood that, although the terms first, second, etc. are used to describe various elements in this article in some cases, these elements should not be limited by these terms. These terms are only used to distinguish an element from another element. For example, the first contact person can be referred to as the second contact person, and similarly, the second contact person can be referred to as the first contact person, without departing from the scope of the various embodiments described. The first contact person and the second contact person are both contacts, but they are not the same contact person. The terms used in the description of the various embodiments described herein are only used to describe the purpose of a specific embodiment and are not intended to be limited.

[0067] As used in the description of the various disclosed embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that, as used herein, the term "and / or" refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that, when used in this specification, the term "comprising" specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0068] As used herein, the term “if” is alternatively construed to mean “when…” or “upon…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrase “if it is determined that…” or “if [stated condition or event] is detected” is alternatively construed to mean “upon determining…” or “in response to determining…” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event]”, depending on the context.

[0069] Reference will now be made in detail to the disclosed embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various disclosed embodiments may be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail to avoid unnecessarily obscuring aspects of the disclosed embodiments.

[0070] Detailed Description

[0071] Figure 1 A graphical user interface (GUI) 100 of an automated solar energy system design tool according to an embodiment is illustrated. In the example shown, the GUI 100 includes a 3D image 101 of a building 103, the address of which is entered by a user in a text box 102. Navigation controls 105 allow the user to manipulate the viewing angle (e.g., rotate, tilt) of the 3D image. An output pane 104 displays the results of a performance simulation of a solar energy system design based on an estimated 3D building model, as described more fully below. In this example, the results include monthly dollars saved, monthly percentage offset, and system size in kilowatts (kW). GUI affordances 106 allow a user to request a quote for the design and / or installation of a solar energy system.

[0072] Figure 2 2 is a flow chart of an automated process 200 for estimating a 3D building model according to an embodiment. The process 200 begins with a consumption module 202, which collects various data required for 3D building estimation, solar energy system design and simulation. Such data includes, but is not limited to, user input, such as an input address of a building, billing information (e.g., average monthly energy costs for a user), utility rates, and a consumption profile for the building location from a consumption profile database. The consumption profile includes energy consumption in kWh per hour for a year. Utility rates can be looked up in a database. The system uses the energy consumption and utility rates to calculate monthly electricity costs, with hourly usage in kWh.

[0073] The input address of the building is used to obtain geographic data 201 (e.g., images, 3D data) of the building from a geographic database such as a U.S. Geological Survey Geographic Information System (GIS) or a proprietary database. For example, the 3D data can be a point cloud generated by a light detection and ranging (radar) sensor, or a point cloud obtained using photogrammetry and synthetic radar. The 3D data can be in the form of a digital surface model (DSM), which is generated by rasterizing the point cloud data into a 2D grid / image so that it can be pre-processed with 2D image data, as described in further detail below. The pre-processed image and DSM are input into a reconstruction module 203, which estimates the 3D building model and identifies any rooftop obstructions. Next, the estimated 3D building model and rooftop obstructions are input into a shading module 204, which uses simulations to determine the amount of sunlight that the roof must be exposed to. The output of the shading module 204 (e.g., irradiance data) is input into the automated solar energy system design module 205, which automatically builds a virtual solar energy system based on the estimated 3D building model and the shading module output. The virtual solar energy system can then be simulated using the simulation module 206 to determine its performance.

[0074] In a separate processing pipeline, data output by consumption module 202 (e.g., energy consumption profile, utility rates) and performance results from simulation module 206 (e.g., power output) are input into financial simulation 207 for generating various financial data, including but not limited to monthly savings and offsets, such as Figure 1 As shown in GUI 100.

[0075] Figure 3 is an example input image 300 for a preprocessing pipeline according to an embodiment. The goal of the preprocessing pipeline is to take arbitrary images and 3D data and convert them into a standard format that the neural network can easily handle during reconstruction. Image 300 is a 2D aerial image that can be captured using any suitable means, including but not limited to fixed-wing aircraft, helicopters, unmanned aerial vehicles (aka "drones"), satellites, balloons, airships and blimps, rockets, pigeons, kites, parachutes, independently telescopic and vehicle-mounted poles. Images for a particular input address can be stored in an indexed database, such as a GIS or a proprietary database.

[0076] Figure 4is an example input 3D data for a preprocessing pipeline according to an embodiment. The example shown is a radar (LiDAR) point cloud, which is a collection of points representing a 3D shape or feature. In an embodiment, the raw point cloud data can be filtered to remove any outlier points caused by sensor noise before being used for 3D building estimation. Such filtering techniques can be statistical, neighborhood-based, projection-based, signal processing-based, or partial differential equation (PDF)-based. Some example filtering techniques include, but are not limited to: voxel grid (VG) filter, normal-based bilateral filter (NBF), moving least squares (MLS), weighted local optimal projection (WLOP), edge-aware resampling (EAR), and L0 minimization (L0). As previously described, the radar point cloud can be fitted to a 2D grid to produce a DSM or height map.

[0077] Figure 5A and Figure 5B An example pre-processing output according to an embodiment is illustrated, wherein an image and a DSM derived from 3D data are axis-aligned with a grid, cropped around a building, and aligned with each other. As shown, the cropped image and the DSM of the building are both axis-aligned with the grid and with each other.

[0078] Figure 6 6 is a flow chart of a preprocessing pipeline 600 according to an embodiment. In the radar preprocessing path of the preprocessing pipeline 600, a 3D mesh is generated from radar data using a "spike-free" triangulation method, such as described in Anahita Khosravipour et al., Generating spike-free digital surface models using LiDAR raw point clouds: A new approach for forestry applications. International Journal of Applied Earth Observation and Geoinformation, 52: 104-114, June 5, 2016.

[0079] The term "spike-free" refers to the way the method generates a smooth mesh for the trees. The 3D mesh is rasterized into a DSM or height map / image. Because the image and radar data are not aligned at the beginning, the DSM (height map) and aerial image of the building are input into the registration module 602 to align the radar data and aerial image to the grid and to each other. Figure 7 Illustrated are the results of a “spike-free” triangulation algorithm applied to 3D radar data to generate a 3D mesh in accordance with an embodiment. Fig. 8A and Figure 8BThe registration process of DSM and image alignment according to an embodiment is illustrated.

[0080] Concurrently, in the image preprocessing path, the image is input into the building segmentation module 602. The building segmentation module 602 uses known image semantic segmentation techniques to label each pixel of the aerial image as a building or a non-building, thereby generating a building mask, such as Fig. 10A and Fig. 10B As shown in. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation by Liang-Chieh Chen et al. https: / / arxiv.org / abs / 1802.02611v3 describes an example image semantic segmentation algorithm.

[0081] The building mask and aligned DSM / image are then input to the orientation and cropping module 604. The orientation and cropping module 604 uses the radar data within the building mask to calculate the orientation of each roof face. In an alternative embodiment, a neural network is used to predict the roof face orientation. For example, the radar data is used to calculate the dominant orientation of the entire roof, and then that orientation is "snapped" onto a 90 degree grid. The building mask is also used to obtain the basic extents of the building and crop the image so that the building is centered in the image and the axis is aligned, such as Fig.11A and Fig. 11B After preprocessing, the images and DSM are properly formatted to be input into one or more neural networks (such as convolutional neural networks (CNNs) or any other suitable neural networks) for reconstructing a 3D building model, in particular a 3D roof model.

[0082] Figure 9A-9C Further illustrated is the use of semantic segmentation techniques on image 901 and DSM 902 to predict building structures 903, trees 904, and background, respectively. According to an embodiment, cross-correlation is used to calculate where the image and DSM are aligned, such as Fig. 9C , where spot 905 indicates where the pixels of the two images are more closely aligned. An example cross correlation technique is described in Briechle, Kai, and Uwe D. Hanebeck, Template matching using fast normalized cross correlation. Optical Pattern Recognition XII. Vol. 4387. International Society for Optics and Photonics, 2001.

[0083] Alternatively, the image 901 and DSM 902 are fed to a neural network trained to predict the numerical offset between two images, such as described in Learning to Compare ImagePatches Via Convolutional Neural Networks. CVPR. 2015. 7299064 by Sergey Zagoruyko, Nikos Komodakis. Instead of predicting a similarity value, the x / y offset value between the image and the DSM is predicted.

[0084] Fig.12 1 is a flowchart of a reconstruction process for generating a 3D building model from a preprocessed image and DSM according to an embodiment. The preprocessed image / DSM is input into a roof face segmentation module 1203. The roof face segmentation module 1203 generates a mask for each roof face using an instance segmentation using a neural network, such as described in Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth. (https: / / arxiv.org / abs / 1906.11109v1) by Davy Neven et al.

[0085] Fig.13 An example roof face mask 1301-1311 generated by a neural network using an instance segmentation technique is illustrated. After the roof face mask is generated, the mask is input into a polygon fitting module 1204, which converts the mask into a 2D polygon using a basic heuristic and removes discontinuities. The radar data is then used to project the 2D polygons into 3D polygons using RANSAC fitting, as described in Yang, Michael Ying, and Wolfgang, Plane detection in point cloud data. Proceedings of the 2ndintconf on machine control guidance, Bonn. Vol. 1. 2010. In an alternative embodiment, a neural network is used to predict 3D polygons from 2D polygons.

[0086] Gaps can be introduced between roof faces by simply converting the roof face masks into polygons, e.g. Fig.25A By generating a 2D mesh instead of separate polygons, a 2D roof surface with no gaps can be generated, as shown in Fig.25BIn an embodiment, semantic segmentation techniques are used to predict edge and node probabilities for each pixel in the image. Points are then selected uniformly along nodes, edges, and across the image using Poisson disk sampling, as described in Cook, Robert L., StochasticSampling in Computer Graphics. ACM transactions on Graphics, 5, January 1, 1986, pp. 51-72.

[0087] The disk radius is varied to sample more densely at the nodes and edges. Delaunay triangulation is then performed to generate a 2D mesh. Each triangle in the 2D mesh is labeled according to its roof face. By combining all triangles in the 2D mesh with the given roof face labels, polygons are extracted for each roof face 2501-2507 that has no gaps between adjacent roof faces, as shown in FIG. Fig.25B as shown in . Fig.26 is a top view of a 2D mesh overlaid with an image showing a rooftop with no gaps, according to an embodiment.

[0088] Fig.14 The diagram illustrates the use of heuristics and polygon fitting to convert a roof plane into a polygon, and then projecting the 2D polygon into a 3D polygon using 3D data. Once the 3D polygons are generated, the edges of each polygon are selected as "azimuth edges" and are assigned heights and slopes to define the 3D plane. For example, when a 2D polygon is projected into a 3D polygon, the plane is forced to point to one of the four cardinal directions relative to the dominant roof orientation (left, right, up, and down). For each possible direction, determine the slope and height of the plane because the azimuth / direction is fixed. The direction, slope, and height of the best fit are then selected. For a south-facing roof face, there will be an east-west edge that is flat and oriented perpendicular to the azimuth direction of the plane. Any point along the edge can be used to draw a plane with a given height and slope so that the edge is located along the plane.

[0089] Fig.12The last step of the roof face segmentation pipeline shown in is to input the 3D polygon into the edge type detection module 1202. The edge type detection module 1202 uses a neural network and known image semantic segmentation techniques to predict the edge type of all edges of the 3D polygon. Examples of edge types include, but are not limited to: eaves, rakes, ridges, valleys, and ridges. An example image semantic segmentation technique is described by Liang-Chieh Chen et al. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. (https: / / arxiv.org / abs / 1802.02611v3).

[0090] Fig.17 The output of a neural network for predicting edge types for all edges of a 3D polygon according to an embodiment is illustrated. The predicted edge types are used to define wall steps. For example, solar panels should not be placed near the edge of a roof face to allow firefighters to enter, etc.

[0091] In the separated roof face process, the roof template database 1206 is searched to find a matching roof template. In an embodiment, the process includes: 1) axis-aligning the image as described above; 2) computing the embedding of the image; 3) finding a roof template in the database 1206 that is similar to the roof reconstructed based on the embedding; 4) finding the height, width, length, and position of the roof template; 5) overlaying the roof template on the roof image; 6) adjusting the internal structure of the roof template to match the roof image; and 7) checking whether the roof template is more accurate than the roof face generated by the roof face segmentation module 1203. After the checking step, one of the adjusted roof templates or the roof faces generated by the roof face segmentation module 1203 is selected to be included in the estimated 3D roof model. In an embodiment, steps 2 and 3 above use known metric learning techniques to retrieve roof templates, such as those described in FaceNet: A Unified Embedding for Face Recognition and Clustering (https: / / arxiv.org / abs / 1503.03832) by Florian Schroff et al.

[0092] Fig.15A and Fig. 15B The non-parametric retrieval technique described above is illustrated. In step 6, a neural network is used to predict the offset of each node in the roof to adjust the internal structure of the roof template to match the image, as shown in FIG. Figure 16A-16C as shown in .

[0093] In the above process, embedding is an N-dimensional vector (e.g., N=64) produced by a neural network. The neural network is trained so that if two roofs are similar, their embeddings will become close in the embedding space, and if they are not similar, their embeddings will be far apart. In step 4, the size of the template is known and the size of the target roof is estimated using the segmentation / alignment pipeline described previously.

[0094] Fig.18 and Fig.19 An alternative technique for wall detection with a polygon fitting algorithm using a neural network according to an embodiment is illustrated. A neural network is first used to predict each row and column of an image grid containing a roof image, whether or not the row / column has a wall. A polygon fitting algorithm is then used to reconstruct the roof by combining all wall and roof planes and finding all possible intersections between them. Optimization is then used to select the most accurate roof structure. An example polygon fitting algorithm is described in Polyfit: Polygonal surface reconstruction from pointclouds. Proceedings of the IEEE International Conference on Computer Vision. 2017 by Nan, Liangliang, and Peter Wonka.

[0095] Fig. 20 A synthetic radar technique according to an embodiment is illustrated, where a neural network is used to compute a height map from a 2D image, and then the height map is used to convert the 2D image into a rough 3D model of a building. As described in Srivastava, Shivangi, Michele Volpi, and Devis Tuia, Joint height estimation and semantic labeling of monocular aerial images with CNNs. IEEE International Geoscience and Remote Sensing Symposium (IGARSS). 2017, this technique can perform reconstruction from images alone without any 3D data.

[0096] Fig.21 The technology for detecting obstacles on the roof according to an embodiment is illustrated. Fig.12, the image and DSM are input into the obstacle detection module 1204, which outputs the rooftop obstacles. In an embodiment, the obstacle detection module 1204 includes a neural network trained to predict rooftop obstacles using instance segmentation techniques, such as described in Mask R-CNN by He, Kaiming et al. Proceedings of the IEEE International Conference on Computer Vision, 2017. These rooftop obstacles show locations where solar panels cannot be installed and objects that will cast shadows on the solar panels.

[0097] Return to reference Figure 2 , process flow 200 proceeds by automatically generating a virtual solar energy system based on an estimated roof model, shading / irradiance data (the flux of radiant energy per unit area) from a shading / irradiance model, a component database, and user preferences, as described in reference Fig.28 described.

[0098] The component database includes data sheets and price lists for commercial solar equipment and hardware, including but not limited to: solar panels, inverters, monitoring equipment, racks and mounting hardware (e.g., rails, flashers, lugs, mounting brackets, wire clamps, connector kits, braces, end caps, accessories, tilt legs), balancing hardware (e.g., DC / AC disconnects, junction boxes, combiner boxes, circuit breakers, fuses, load centers, rapid shutoffs, surge devices), electrical wiring, charge controllers, batteries, and more.

[0099] The system design is then simulated using performance simulation 206 to determine the electrical performance of the system design. The performance data generated by the performance simulation is used together with the utility rate data and the user consumption profile to determine monthly cost savings, monthly offsets, and other financial data useful to the consumer or professional solar panel installer. The performance simulation 206 uses the reference Fig. 27 The method described calculates irradiance values. Given the amount of irradiance on each panel, the amount of current produced by each panel is estimated. Using circuit modeling, the system generates a combined IV curve for all solar panels connected in a string. The inverter is then simulated by finding the maximum power point and estimating the efficiency of the conversion to AC power to ultimately obtain the energy output per hour (in kWh). For example, this calculation is performed for each hour or N minute increment (e.g., N=15) of the simulated year.

[0100] In an embodiment, the energy consumption profile and utility rates used in the consumption step are used to calculate the energy cost of the building before the solar energy system is installed. The energy consumption is subtracted from the solar power generation to obtain the hourly post-solar energy consumption for the simulated year. A monthly bill for the new consumption value is then calculated. By comparing the two bills, the monthly savings from installing the solar energy system is calculated.

[0101] In an embodiment, further simulations may be run to calculate the return on investment (ROI), net present value (NPV), and annual cash flows under different financing options (eg, cash purchase, loan, and lease).

[0102] Example Process

[0103] Fig. 22 is a flow chart of an automated 3D building estimation process 2200 according to an embodiment. Fig.29 The computer architecture 2900 described implements the process 2200 .

[0104] The process 2200 begins by obtaining a building address, utility rates and billing information (2201) and obtaining 3D data and image data for the building address (2202). In an embodiment, the building address is entered by a user through a GUI of an online automated 3D building design tool. In an alternative embodiment, the address is obtained programmatically, for example, through an API. In an embodiment, the utility rates can be obtained from a utility rate database maintained by a utility company, or from a third party provider (such as Genability, Inc. of San Francisco, California, USA). In an embodiment, the image data and 3D data are obtained from a public or proprietary database of images and 3D data that can be retrieved using the building address of the building. In an embodiment, the 3D data is 3D radar data.

[0105] Process 2200 continues by performing 3D building / roof estimation (2203) using the 3D data and images, and determining the available roof area (2204) based on the 3D building / roof model and detected roof obstacles, as described in reference to FIG. Figure 2-Figure 21 described.

[0106] Process 2200 continues by determining installation locations for solar panels in the available roof area based on the available roof area and the shading / irradiance model (2205), and automatically designing a virtual solar energy system for the installation location (2206).

[0107] Process 2200 continues by performing a simulation of the virtual solar energy system at the installation site to determine performance and generate metrics ( 2207 ).

[0108] These metrics (such as monthly cost savings and reimbursements) can be displayed to the user through the GUI of the automated design tool, or provided in a report to the customer or professional solar panel installer.

[0109] Fig.23 According to the embodiment, Fig. 22 Flow chart of the preprocessing process 2300 of the 3D building estimation process. Fig.29 The computer architecture 2900 described implements the process 2300 .

[0110] Process 2300 begins by generating a DSM from a 3D mesh (2301), as shown in FIG. Figure 7 For example, the DSM can be rasterized into a 2D image using a spike-free 3D grid of radar data.

[0111] The process 2300 continues by aligning the image with the DSM image so that they are aligned with each other (2302), as shown in reference Figure 6 - Figure 9.

[0112] Process 2300 continues by generating a building mask from the image (2303), and using the building mask and 3D data to orient, crop, and align the image and DSM with grid axes to determine the orientation of each roof face (2304), as described with reference to Figures 10-11. The building mask is also used to obtain building extents.

[0113] Fig.24A and Fig. 24B According to the embodiment, Fig. 22 Flowchart of the reconstruction process 2400, 2500 of the 3D building estimation process of FIG. The process 2400, 2500 can use reference Fig.29 The described computer architecture 2900 is implemented.

[0114] refer to Fig.24A , process 2400 begins by performing roof face segmentation to obtain 2D roof faces (2401), and then fitting 3D polygons to the 2D roof faces using the image and 3D data (2402). For example, roof segmentation generates a 2D roof face, which is fitted to a 2D polygon using a heuristic method to remove discontinuities, and then projected into a 3D polygon using RANSAC fitting using the 3D data. Process 2400 continues by performing edge type detection on the 3D polygon using the image and DSM (2404). Examples of edge types include, but are not limited to: eaves, rakes, ridges, valleys, and ridges. Edge type detection can be implemented using semantic segmentation techniques. Process 2400 continues by performing obstacle detection (2404) using the image / DSM.

[0115] refer to Fig. 24B , process 2405 begins by retrieving a roof template (2406) that matches the image / DSM, as shown in reference Fig.14 As described above, either the 3D roof surface or the roof template generated by the segmentation is then selected as the 3D representation of the roof (2407). Processes 2400 and 2500 may be performed in parallel or in series.

[0116] Fig. 27 The diagram illustrates shading by ray tracing of a 3D building model, obstacles, and surroundings in accordance with an embodiment. In an embodiment, shading is determined using ray tracing of a 3D model of the building 2700, rooftop obstacles, and its surroundings. A radar grid (generated from the previous spike-free triangulation step) is used to model the building's surroundings, and shadows can then be projected onto the building. For a given point on a roof or on a solar panel, the system calculates whether the solar panel is in shadow for each hour of the simulated year. In addition, the system calculates irradiance by combining weather data with shading information. Based on the angle of the solar panel surface relative to the sun, and based on whether the solar panel is in shadow, the system calculates the amount of sunlight that strikes the solar panel surface in W / m for each hour of the simulated year. 2 .

[0117] Fig.28 is used according to the example as reference Figure 1-Figure 27 The process flow 2800 for automated solar energy system design of the generated estimated 3D building model.

[0118] Process 2800 begins by determining a grid of all possible panel locations based on the desired panel size and spacing (2801). Process 2800 continues by calculating the irradiance for each solar panel location at each hour of the year (2802) based on weather data and a 3D model of the site (including buildings, rooftop obstacles and its surroundings). Process 2800 continues by estimating how much savings each potential panel will generate at each hour of the year (2803) based on its electrical characteristics and utility rates. Process 2800 continues by calculating the optimal set of panels to minimize costs and maximize savings (2804). For each potential model of the inverter, process 2800 continues by determining the optimal number of inverters and the connection of the solar panels to each other and the inverter (2805). Given a combined panel / inverter system, process 2800 continues by re-evaluating performance and savings (2806). The re-evaluation step reduces errors introduced by simplifying assumptions in earlier steps. The re-evaluation step also evaluates the cost and aesthetics of the layout (e.g., whether the panels are rectangular groups or irregular shapes).

[0119] Each step of process 2800 is run sequentially to generate a single optimal design using integer linear programming. Then, a genetic algorithm is used to make many small modifications at each step and determine which combination overall produces the best design for the customer.

[0120] Example System Architecture

[0121] Fig.29 It is used to implement reference Figure 1-Figure 28 2900. The architecture 2900 may be implemented on any electronic device that runs a software application derived from compiled instructions, including but not limited to a personal computer, server, smart phone, media player, electronic tablet, game console, email device, etc. In some embodiments, the architecture 2900 may include one or more processors 2902, one or more input devices 2904, one or more display devices 2906, one or more network interfaces 2908, and one or more computer-readable media 2910. Each of these components is coupled via one or more buses 2912.

[0122] Display device 2906 may be any known display technology, including but not limited to display devices using liquid crystal display (LCD) or light emitting diode (LED) technology. Processor 2902 may use any known processor technology, including but not limited to graphics processors and multi-core processors.

[0123] Input device 2904 can be any known input device technology, including but not limited to keyboard (including virtual keyboard), mouse, trackball and touch-sensitive pad or display. In some embodiments, input device 2904 can include a microphone that facilitates voice-enabled functions (such as speech to text, speaker recognition, voice replication, digital recording and telephone functions). Input device 2904 can be configured to facilitate processing voice commands, voiceprints and voice authentication. In some embodiments, the audio recorded by input device 2904 is transmitted to an external resource for processing. For example, the voice commands recorded by input device 2904 can be transmitted to a network resource, such as a network server that performs voice recognition on the voice commands.

[0124] Bus 2912 may be any known internal or external bus technology including, but not limited to, ISA, EISA, PCI, PCI Express, NuBus, USB, Serial ATA, or FireWire.

[0125] The computer readable medium 2910 may be any medium that participates in providing instructions to the processor 2902 for execution, including but not limited to non-volatile storage media (e.g., optical disks, magnetic disks, flash drives, etc.) or volatile storage media (e.g., SDRAM, ROM, etc.). The computer readable medium 2910 may include various instructions 2914 for implementing an operating system 2913 (e.g., Mac Linux). The operating system 2913 may be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, etc. The operating system 2913 performs basic tasks including but not limited to: recognizing input from input device 2904; sending output to display device 2906; keeping track of files and directories on the computer readable medium 2910; controlling peripheral devices (e.g., disk drives, printers, etc.) that may be controlled directly or through an I / O controller; and managing traffic on bus 2912. Network communication instructions 2914 may establish and maintain network connections (e.g., software for implementing communication protocols such as TCP / IP, HTTP, Ethernet, etc.).

[0126] Graphics processing system 2915 may include instructions that provide graphics and image processing capabilities. For example, graphics processing system 2915 may implement reference Figure 1 , Fig.21 A and Fig.21 B describes the GUI.

[0127] Application(s) 2916 may be used or implemented as reference Figure 1-Figure 28 Application of the described processes. These processes can also be implemented in the operating system 2913.

[0128] Predict roof profile, slope and height from imagery and 3D data

[0129] Fig.30 is a flow chart of an automated 3D building estimation process 3000 for predicting roof profiles, slopes, and heights based on aerial imagery and 3D data, according to an embodiment.

[0130] In an embodiment, 2D aerial imagery and 3D data (eg, radar data) 3001 are segmented and cropped 3002 to generate a cropped image / DSM 3003, as shown in FIG. Figure 6 The cropped image / DSM 3003 is input into a roof face segmentation network 3004, which generates a plurality of roof faces representing roof components (hereinafter referred to as "roof components"), wherein each roof component includes a set of perimeter edges and fold lines. The perimeter edges together form the outline of the roof. The fold lines are the internal points in the roof component where the slope of the roof face changes. Each perimeter edge and fold line also has an associated height and slope.

[0131] In an embodiment, the process 3000 can be run in two configurations that determine when to predict the height and slope of each perimeter edge or fold line. In the first configuration, the process 3000 first predicts the positions of all perimeter edges and fold lines, and then predicts the height and slope of all predicted edges or fold lines. In the second configuration, the process 3000 alternates between predicting the positions of perimeter edges and fold lines and predicting the height and slope of predicted edges or fold lines, including making adjustments to previous height or slope predictions.

[0132] In an embodiment, a first machine learning model (e.g., a neural network) predicts 3007 the location of each node of a perimeter edge or polyline, which outputs a probability distribution of potential next node or polyline locations. The probability distribution is used to guide a search process 3005 (e.g., a Monte Carlo tree search), which estimates how good each prediction of the next node or polyline (or start or end component 3008) will be. The search process 3005 explores a specified number of forward steps and compares the roof produced by each possible next node or polyline to the output of an instance segmentation network (not shown). The output of the instance segmentation network is considered to be a close approximation of the actual 2D structure of the roof. The results of the search 3005 are used to update the probability distribution to predict where the next node of the perimeter edge or polyline should be.

[0133] The above process continues to iterate outputting nodes / polylines until the searched probability distribution indicates 3009 that the roof is complete 3006. If the first configuration is used, then the slope and height are predicted 3009 after the roof is complete 3006. If the second configuration is used, then the process 3000 alternately predicts the position and height / slope (for polylines or edges), including making adjustments to previous height / slope predictions. After prediction, the roof components are rendered 3010 into 3D models 3011, which are fused together to obtain the final 3D roof model.

[0134] Figures 31A-31K Further illustrated are steps of an automated 3D building estimation process for predicting roof profiles, slopes, and heights from aerial imagery and 3D data according to an embodiment.

[0135] Each step of process 3000 displays a representation of the input (cropped image / DSM) being processed. New points are drawn after each step based on the input, so points drawn in step i will be displayed in step i+1. Process 3000 is repeated in steps 0-4 ( Figures 31A-31E ) draw the perimeter edge of the first roof component. In step 0 ( Fig.31A ), process 3000 draws the lower right corner ( Fig.31A , as it is just a point), and in step 1, process 3000 draws the upper right corner ( Fig.31B, as it is only a point). In step 2, process 300 draws the right edge connecting the upper right corner and the lower right corner ( Fig. 31C ). In step 3, process 3000 draws the upper left corner and the edge connecting the upper left corner and the upper right corner ( Fig.31D ). In step 4, process 3000 draws the lower left corner and the edge connecting the lower left corner and the upper left corner ( Fig.31E ). In step 5, process 3000 draws an edge connecting the lower left corner and the lower right corner ( Fig.31F ), thus completing the outline of the first roof component shown in Step 5 ( Fig.31F ).

[0136] In steps 6-9 ( Figures 31G-31J ), draw the second roof component outline, starting by drawing the upper left corner, which was drawn in step 6 ( Figure 31G ) but is visible in step 7 ( Fig.31H ) shows that another point and edge connecting the upper left and upper right corners have been drawn ( Fig.31H In step 8, process 300 draws the lower left corner and the edge connecting the lower left corner and the upper left corner ( Fig.31I ). In step 9, process 300 draws the lower right corner and the edge connecting the lower left corner and the lower right corner ( Fig.31J ). Process 3000 indicates that process 3000 has been completed at step 10 ( Figure 31K ) is completed in the process 3000. The process 3000 completes two roof component profiles, each having 4 perimeter edges.

[0137] Next, process 3000 uses a second machine learning model (e.g., a neural network) to predict the height and slope of each perimeter edge, as shown in FIG. Fig.33 As described above, a 3D model of each roof component is rendered and the 3D models are combined to produce a complete 3D model of the roof, such as Fig.34A and Fig.34B as shown in .

[0138] Fig.32 3200 is a flow chart illustrating a Monte Carlo Tree Search (MCTS) according to an embodiment. MCTS includes four stages determined by the number of iterations: selection, expansion, simulation, and back propagation. The more iterations there are, the more the tree grows, and the easier it is for the results to converge to meaningful results. A root node is usually provided to start the iteration.

[0139] In the selection phase, the root node goes through the selection phase where the node is selected based on the maximum confidence upper bound (UCB) formula value. The UCB formula attempts to balance the tree based on the constant C using both exploitation and exploration.

[0140] In the expansion phase, if the node has been visited before (i.e., simulated using a neural network), its children (or possible next states) are generated and added to the tree. Otherwise, the search continues to the simulation phase.

[0141] In an embodiment, the probability of each node being explored is a balance of at least three criteria: 1) the number of times the node has been visited (to encourage exploration), 2) the network's estimate of how good the node is (this represents a prior estimate of the node's quality), and 3) the back-propagated value of the node if any paths containing the node have terminated (this incorporates observed evidence to improve estimates of the node's quality). In summary, MCTS will explore nodes it has not seen before, giving priority to nodes that the network indicates are valuable. This process continues until a few promising terminating paths are found, and then the search is narrowed to these paths and prioritized.

[0142] In the back-propagation phase, the values ​​obtained in the simulation phase are propagated from the leaves of the tree to the root of the tree, and the values ​​of the nodes are updated as they are visited.

[0143] refer to Fig.32 , the input roof representation 3201 is processed by a neural network 3206, which predicts the locations of the perimeter edges 3202, 3206 of two different roof components (i.e., different roof faces). The roof representation (cropped image / DSM) with the perimeter edges 3203, 3206 is input to the neural network 3203, 3207, which predicts the perimeter edge locations 3204, 3208, and 3210. This process continues until all perimeter edges of the roof components are located, resulting in the outlines 3205, 3209, and 3211 of the roof components (including the perimeter edges) as shown.

[0144] Note that in Fig.32 In the example of , the root is the roof with no predicted nodes. To the right of the root are the various paths that the search explores, where each additional image / DSM will constitute an expansion of the tree, and these paths are explored based on the neural network's estimate of how good each "action" (node) is. The confidence score on the far right indicates a heuristic assessment of the quality of the finished roof produced by taking a particular path. Whenever a final state is reached, this value is backpropagated along the tree to adjust the network's estimate of how good the actions that led to that state were. Once the specified number of simulations have been run, the most promising action starting from the root is taken by selecting the node, and the process is repeated from the next state.

[0145] In an embodiment, the rendered roof is compared to the face segmentation output to generate a confidence score (e.g., probability). In the example shown, the top branch accurately predicts the outline 3205 of both roof components, so the confidence score is 0.95. The middle branch predicts a single roof component and has a confidence score of 0.56 because it cannot predict the second roof component. The lower branch predicts two roof components, but one roof component has an incorrect edge location, resulting in a confidence score of 0.74.

[0146] Fig.33 is a flow chart illustrating a process 3300 for predicting the slope and height of a roof assembly according to an embodiment. In the example shown, Fig.32 The input roof representation 3205 with two roof components shown in is input into a machine learning model (e.g., a neural network) 3301, which predicts the slope and height of each edge of each component 3302-1, 3302-2...3302-N, as shown in Fig.33 As shown in . In an embodiment, a network is trained using a rooftop image dataset. Features can be created by a backbone CNN, and then features are extracted based on the locations of edges and broken lines in the image. The extracted features are then passed to one or more prediction networks, which output specific slope / height parameters.

[0147] Fig.34A and Fig.34B The diagram shows the Figure 30-33 A complete 3D model of the roof generated by the automated 3D building estimation process described in .

[0148] Fig.35 is a flow chart of an automated 3D building estimation process 3500 for predicting roof profiles, slopes, and heights from aerial imagery and 3D data, according to an embodiment.

[0149] Process 3500 includes the following steps: obtaining an aerial image of a building based on an input address (3501); obtaining three-dimensional (3D) data containing the building based on the input address (3501); preprocessing the aerial image and the 3D data (3502); predicting the contour of each roof component using a first machine learning model that takes the roof surface as input (3503); predicting the slope and height of each roof component using a second machine learning model that takes the roof surface and the contour as input (3504); and rendering a 3D building model based on the predicted contour, slope, and height of each roof component (3505).

[0150] In the context of the present disclosure, the above features and processes may be implemented in whole or in part in a software program including instructions and data stored on a machine-readable medium. A machine-readable medium may be any tangible medium that may contain or store a program or data for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] Computer program codes for performing the disclosed embodiments may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor of the computer or other programmable data processing device, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0152] In addition, although the operations are described in a specific order, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or performing all the operations shown to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of any invention or the content that can be claimed, but rather descriptions of the features of the specific embodiments that can be specific to a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0153] When read in conjunction with the accompanying drawings, in view of the foregoing description, various modifications and adaptations to the foregoing exemplary embodiments disclosed herein will become apparent to those skilled in the relevant art. Any and all modifications will still fall within the scope of the non-limiting and exemplary embodiments of the present invention. In addition, other embodiments not disclosed herein will occur to those skilled in the art because they benefit from the foregoing description and the teachings presented in the accompanying drawings.

[0154] In the foregoing description, embodiments of the invention have been described with reference to many specific details, which may vary from implementation to implementation. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the present invention, and what the applicants intend as the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application, in the specific form in which such claims issue, including any subsequent corrections. Any definitions of terms contained in such claims expressly set forth herein shall govern the meaning of such terms used in the claims. In addition, when we use the term "also including" in the foregoing description or the following claims, what follows the phrase may be additional steps or entities, or sub-steps / sub-entities of the previously enumerated steps or entities.

Claims

1. A method comprising: obtaining, using one or more processors, an aerial image of a building based on an input address; obtaining, using the one or more processors, three-dimensional (3D) data including a building based on an input address; pre-processing the aerial imagery and the 3D data using the one or more processors; Reconstructing a 3D building model from the pre-processed images and the 3D data using the one or more processors, the reconstruction comprising: predicting the profile of each roof component using a first machine learning model; predicting the slope and height of each roof component based on the predicted profile using a second machine learning model; and A 3D building model is rendered using the one or more processors based on the predicted profile, at least one slope, and height of each roof assembly.

2. The method of claim 1, wherein predicting the profile of each roof component using the first machine learning model further comprises: For each roof component in the sequence of roof components, predicting a position of each perimeter edge of the roof component; as well as For each roof component, predict the location of each fold line in that roof component.

3. The method of claim 2, wherein the location is predicted by a neural network that outputs a probability distribution of potential locations.

4. A method as claimed in claim 3, wherein the probability distribution is used to guide a search process that estimates how good each prediction will be.

5. The method of claim 4, wherein the search process explores a specified number of forward steps and compares the roof representation produced for each possible next node or polyline with the output of the instance segmentation network.

6. The method of claim 5, wherein the output of the instance segmentation network is considered to be a close approximation to the actual two-dimensional (2D) structure of the roof.

7. The method of claim 4, wherein the search results are used to update a probability distribution for predicting the location of the next node or polyline.

8. The method of claim 4, wherein the search is a Monte Carlo Tree Search (MCTS).

9. The method of claim 1, wherein the first machine learning model and the second machine learning model are parts of a single neural network.

10. The method of claim 1, wherein preprocessing the aerial image and the 3D data further comprises: Generate 3D mesh from 3D data; Generate a digital surface model (DSM) of the building using the 3D mesh; Align the image with the DSM; Generate building masks from images; Calculate the orientation of each roof face of a building using 3D data with a building mask; Snap the orientation of buildings to the grid; Use the building mask to get the extent of the building; as well as Crop the image so that the building is in the center of the image and aligned with the grid axes.

11. The method of claim 1 , further comprising: Use instance segmentation to predict a mask for each roof component of a building; predicting the profile of each roof component using a first machine learning model with the mask as input; as well as A second machine learning model that takes the mask and the contour as input predicts the slope and height of each roof component.

12. A system comprising: one or more processors; A memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: Obtain aerial images of buildings based on input addresses; Obtaining three-dimensional (3D) data including buildings based on an input address; Pre-processing of aerial images and 3D data; Reconstructing a 3D building model from the pre-processed image and 3D data, the reconstruction comprising: Use instance segmentation to predict a mask for each roof component of a building; predicting the profile of each roof component using a first machine learning model with the mask as input; predicting the slope and height of each roof component using a second machine learning model that takes the mask and the contour as input; and Render a 3D building model based on the predicted profile, slope, and height of each roof component.

13. The system of claim 12, wherein predicting the profile of each roof component using the first machine learning model further comprises: For each roof component in the sequence of roof components, predicting a position of each perimeter edge of the roof component; as well as For each roof component, predict the location of each fold line in the roof component.

14. The system of claim 13, wherein the locations are predicted by a neural network that outputs a probability distribution of potential locations of a node or polyline.

15. The system of claim 14, wherein the probability distribution is used to guide a search process that estimates how good each prediction of a node or polyline will be.

16. The system of claim 15, wherein the search process explores a specified number of forward steps and compares the resulting roof representation for each possible next node or polyline to the output of the instance segmentation network.

17. The system of claim 16, wherein the output of the instance segmentation network is considered to be a close approximation to the actual two-dimensional (2D) structure of the roof.

18. The system of claim 15, wherein the search results are used to update a probability distribution predicting the location of the next node or foldline of the roof component.

19. The system of claim 15, wherein the search is a Monte Carlo Tree Search (MCTS).

20. The system of claim 12, wherein the first machine learning model and the second machine learning model are neural networks.

21. The system of claim 12, wherein pre-processing the aerial image and the 3D data further comprises: Generate 3D mesh from 3D data; Generate a digital surface model (DSM) of the building using the 3D mesh; Align the image with the DSM; Generate building masks from images; Calculate the orientation of each roof face of a building using 3D data with a building mask; Snap the orientation of each roof face to the grid; Use the building mask to get the extent of the building; as well as Crop the image so that the building is in the center of the image and aligned with the grid axes.

Images