Platform, system, and method for classifying asset property and asset feature maintenance management through aerial image analysis

Abstract

To provide a method and a system for automatically classifying a repair condition of an asset property.SOLUTION: A system operational flow 100 for automatically classifying a repair condition of an asset property, comprises the steps of: acquiring an aerial image of a geographic area including an asset; identifying features of the aerial image corresponding to the asset property; analyzing the features to determine an asset property classification; analyzing an area of the aerial image including the asset property to determine a condition classification; and using the asset property classification and the condition classification to determine a risk estimate value of damage to the asset from one or more disasters and / or a cost estimate value for repairing or replacing the asset property.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on “Platform, Systems, and Methods for Identifying Property Characteristics and Property Feature Maintenance Through This application claims priority to U.S. Provisional Patent Application No. 62 / 398,665, filed Sep. 23, 2016, entitled "Aerial Imagery Analysis," which is hereby incorporated by reference in its entirety. [Background technology]

[0002]

[0002] A risk exposure database includes a compilation of as many building assets or characteristics as possible that are relevant for insurance. These assets may include characteristics such as location coordinates, address, slope, and elevation. Other characteristics include building structure type, occupancy type, year of construction and / or renovation, building height, soft stories, number of floors, and floor area. Further characteristics may include roof condition, roof shape, roof covering, roof anchors, roof installations, cladding, and pounding (distance to adjacent buildings). Some of these characteristics may be assessed only by on-site inspection or official documents, while others may be measured using visual images.

[0003]

[0003] The characteristics covered in this disclosure include roof shapes and roof conditions. In one embodiment, roof shapes are divided into five categories: gambrel roof, gable roof, hip roof, square roof, and flat roof. Each roof shape has its own unique response and vulnerability to damage from various hazardous natural phenomena such as earthquakes or wind.

[0004]

[0004] Deep learning involves computational models consisting of multiple processing layers to learn representations of data with multiple levels of abstraction. These models can be considered a way to automate predictive analytics. Representation learning is a set of methods that allow a machine to be fed raw data and automatically find the representations needed for detection or classification. Deep learning applications include speech recognition, motion detection, transformation, and medical diagnosis. Using deep learning algorithms and sample datasets, computers can distinguish and classify a wide range of characteristics with a high level of accuracy, often exceeding the level of human recognition.

[0005]

[0005] One model used in deep learning is the "Network In Network" model described in the paper "Network In Network" by M. Lin et al. and published in the International Conference on Learning Representations (arXiv:1409.1556) in 2014, the contents of which are incorporated herein by reference in their entirety. Using the Network In Network model, multiple layers of artificial perception results are generated using micro-neural networks with complex structures. The artificial perception results are then stacked and averaged to generate a single global average pooling layer for classification.

[0006]

[0006] When applied to visual recognition, deep learning algorithms can classify an observation (e.g., an image) in a number of different ways in order to characterize the features of the observation. In some embodiments, deep learning algorithms can explore an image to classify features, such as a set of intensity values ​​per pixel, or in a more abstract way, as a set of edges, regions of a particular shape. Some representations may perform better than others based on the specific learning task. One of the possibilities of deep learning is to replace human identification of features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction.

[0007]

[0007] The inventors have realised that by applying deep learning methods to a population of risk exposure databases to analyse aerial imagery and automatically extract characteristics of individual assets, it is possible to perform a fast and efficient automatic classification of building types and state of repair. Combining location-based vulnerability with individual asset vulnerability identified in part by a state of repair classification of one or more asset features, the risk of damage from a disaster can be more accurately estimated. Summary of the Invention

[0008]

[0008] The above summary of exemplary embodiments and the following detailed description thereof are merely examples of aspects of the teachings of the present disclosure and are not intended to be limiting.

[0009]

[0009] The systems, methods, and computing system platforms described herein assist in matching aerial image features of one or more assets with corresponding asset conditions (e.g., maintenance levels of the asset features) through machine learning analysis. In a preferred application, the asset condition analysis may be used to estimate risk of damage given one or more disaster conditions, such as a storm. The analysis may further assist in estimating the cost of repairing or replacing each asset, in one embodiment, if a disaster causes the estimated damage. In another embodiment, the analysis may be used to verify that the asset has been repaired.

[0010]

[0010] In one aspect, the disclosure relates to a method for automatically classifying a condition of repair of an asset property, the method including acquiring an aerial image of a geographic area including an asset, identifying features of the aerial image that correspond to the asset property, analyzing the features to determine an asset property classification, analyzing the area of ​​the aerial image that includes the asset property to determine a condition classification, and using the asset property classification and the condition classification to determine a risk estimate of damage to the asset from one or more hazards. The asset property classification may be a shape of a roof top surface. Analyzing the features to determine the asset property may include applying a deep learning analytics model to the features. The deep learning analytics model may be a NIN.

[0011] In some embodiments, analyzing the region of the aerial image including the asset characteristic to determine a condition classification includes applying a machine learning analysis model to image pixels in the region. The machine learning analysis model may include a color histogram analysis model. The condition classification may include a good and bad classification. Determining the risk estimate includes applying a hazard risk profile corresponding to a first hazard and the asset characteristic of the at least one hazard.

[0012] In one aspect, the present disclosure relates to a system for automatically classifying a condition of repair of an asset property, the system including a processing circuit and a non-transitory computer-readable medium having instructions stored thereon. In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to acquire an aerial image of a geographic area including the asset, identify features of the aerial image that correspond to the asset property, analyze the features to determine an asset property classification, analyze the area of ​​the aerial image that includes the asset property to determine a condition classification, and use the asset property classification and the condition classification to determine a replacement cost for replacing the asset property. The aerial image may be a two-dimensional aerial image.

[0013]

[0013] In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to obtain a shape map image including the asset before identifying the feature, overlay the aerial image with the shape map image, and determine whether a boundary of the asset identified by the shape map matches a boundary of the asset shown in the aerial image. The instructions, when executed by the processing circuit, may cause the processing circuit to obtain an alternative aerial image of the asset when it is determined that the shape map does not match a boundary of the corresponding asset. The instructions, when executed by the processing circuit, may cause the processing circuit to assess the orthogonality of the aerial image before identifying the feature. The asset may be a single-family home.

[0014]

[0014] In one aspect, the present disclosure relates to a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processing circuit, cause the processing circuit to receive an identification of an asset and at least one asset characteristic, acquire an aerial image of a geographic region including the asset, and identify individual features of the aerial image corresponding to each of the at least one asset characteristic. In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to analyze, for each asset characteristic, the corresponding features to determine an individual asset characteristic classification, and analyze a region of the aerial image including the individual asset characteristic to determine an individual condition classification. The instructions, when executed by the processing circuit, may cause the processing circuit to determine at least one risk estimate indicative of a risk of damage from a disaster using the asset characteristic classification of each asset characteristic and the condition classification of each asset characteristic.

[0015] In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to determine, prior to acquiring an aerial image of the geographic region, a preferred image type corresponding to each of the at least one asset characteristic based on the at least one asset characteristic. The at least one asset characteristic may include two or more asset characteristics. The instructions, when executed by the processing circuit, may cause the processing circuit to acquire a ground image of the geographic region including the asset in response to determining that the preferred image type corresponding to a first asset characteristic of the at least one asset characteristic is a ground image.

[0016] In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to access one or more known asset characteristics. Determining the at least one risk estimate may further include determining the at least one risk estimate based on the one or more known asset characteristics. The one or more known asset characteristics may include at least one of asset age, asset elevation, asset slope, construction year, renovation year, and building height.

[0017] In some embodiments, receiving an identification to the asset and at least one asset characteristic includes receiving an identification of the asset over a network from a remote computing device. The instructions, when executed by the processing circuit, can cause the processing circuit to provide the at least one risk estimate over the network to the remote computing device in real time in response to receiving the at least one asset characteristic.

[0018]

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate one or more embodiments and, together with the description, explain these embodiments. The accompanying drawings are not necessarily drawn to scale. Any values ​​and dimensions shown in the accompanying graphs and figures are shown for illustrative purposes only and may or may not represent actual or preferred values ​​or dimensions. Where appropriate, some or all features may not be shown to help explain basic features. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a flow diagram of an example of a system operational flow for matching aerial image features with corresponding asset maintenance management levels. [Figure 2A]

[0020] FIG. 1 shows an example of trait classification and corresponding risk profile data. [Figure 2B]

[0021] FIG. 1 illustrates example state characteristics and corresponding example histograms. [Figure 2C]

[0022] FIG. 2 illustrates an example condition profile corresponding to an example asset characteristic type. [Figure 2D]

[0023] 1 is an example graph of error rate when comparing training data with test data when matching aerial image features with corresponding roof top shapes. [Diagram 3]

[0024] FIG. 1 is a block diagram of an example environment for matching topographical features of aerial imagery with corresponding asset maintenance levels. [Figure 4]

[0025] 1 is a flow chart illustrating an example of a method for classifying condition characteristics of an asset based on aerial imagery analysis. [Figure 5A]

[0026] FIG. 2 illustrates an example of a structure geometry map of a geographic region. [Figure 5B]

[0027] FIG. 5B illustrates an example of a two-dimensional aerial image of the geographic area of ​​FIG. 5A. [Figure 5C]

[0028] 5B shows an example of an overlay map image in which a section of the structure geometry map of FIG. 5A is overlaid on a corresponding section of the aerial image of FIG. 5B. [Figure 6]

[0029] FIG. 1 is a block diagram of an example computing system. [Figure 7]

[0030] FIG. 1 is a block diagram of an example of a distributed computing environment, including a cloud computing environment. [Figure 8A]

[0031] 11A-11C are a series of example screenshots of a user interface for reviewing asset characteristics and maintenance level information. [Figure 8B] FIG. 11 is a series of example screenshots of a user interface for reviewing asset characteristics and maintenance level information. [Figure 8C]FIG. 11 is a series of example screenshots of a user interface for reviewing asset characteristics and maintenance level information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020]

[0032] The following description of the accompanying drawings is intended to describe various exemplary embodiments of the disclosed subject matter. Although specific features and functions are described in connection with each exemplary embodiment, it will be apparent to one of ordinary skill in the art that the disclosed embodiments may be practiced without each of those specific features and functions.

[0021]

[0033] Throughout this specification, the phrase "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic shown in connection with an embodiment is included in at least one embodiment of the disclosed subject matter. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Furthermore, the embodiments of the disclosed subject matter are intended to cover such modifications and variations.

[0022]

[0034] It should be noted that, as used herein, singular forms ("a", "an", "the") include plural referents unless the context clearly indicates otherwise. That is, as used herein, the words "a", "an", "the", etc.) have the meaning of "one or more" unless otherwise specified. Furthermore, it should be understood that terms such as "left", "right", "up", "down", "front", "back", "side", "height", "length", "width", "up", "down", "inside", "outside", "inner", "outer", "inside", "outside", etc. that may be used herein merely indicate a point of reference and do not necessarily limit the embodiments of the present disclosure to any particular orientation or configuration. Also, terms such as "first", "second", "third", etc. merely identify one of the many parts, components, steps, operations, functions, and / or points of reference disclosed herein and similarly do not necessarily limit the embodiments of the present disclosure to any particular configuration or orientation.

[0023]

[0035] Additionally, the terms "approximately," "about," "approximately," "within a small margin of error," and similar terms generally refer to ranges that include a particular value within a margin of error of 20%, 10%, or preferably 5% in certain embodiments, and any value therebetween.

[0024]

[0036] All of the features described in connection with one embodiment are intended to be applicable to the additional embodiments described below, unless expressly indicated otherwise or unless the feature or function is incompatible with the additional embodiment. For example, if a given feature or function is expressly indicated in connection with one embodiment but not expressly described in connection with an alternative embodiment, it should be understood that the inventors intend that the feature or function may be developed, utilized, or implemented in connection with the alternative embodiment, unless the feature or function is incompatible with the alternative embodiment.

[0025]

[0037] Referring to FIG. 1, an example of a system operational flow 100 for matching aerial image features with corresponding asset maintenance levels is shown in a flow diagram in which an image of an asset location 102b is acquired (104). In some implementations, initiation of at least a portion of the operational flow 100 by the system may be triggered by an asset classification request submitted by a user, such as an insurance agent or real estate investment broker, on a dashboard interface screen on a computing device connected to the system via a network. For example, as shown in FIG. 8A, a particular asset may be selected in a map view user interface 800, or alternatively, a neighborhood of assets. The operational flow 100 may be used to analyze the condition of each asset feature at each asset location 102b based on the aerial image 102c. In some implementations, the operational flow 100 may assist in estimating, for example, the cost of repair or replacement of each asset feature detected in the aerial image. In another implementation, the operational flow 100 may be used to verify that the asset location 102b has been repaired.

[0026]

[0038] Asset location 102b, in some examples, may represent the location of an asset insured by a particular insurance carrier, the location of an asset recently affected by a disaster such as a tornado, hurricane, earthquake, storm surge, or eruption, the location of an asset in an investment portfolio, or the location of an asset being considered for addition to an investment portfolio. Asset location 102b, in some examples, may be identified by a street address, Global Positioning System coordinates, or other geographic coordinates.

[0027]

[0039] In some implementations, the operational flow 100 obtains (104) an image of each asset location 102b. The image may include an aerial image 102c stored by the system or accessed by the system from a remote networked location. The aerial image 102c may include, for example, a three-dimensional or two-dimensional image of a geographic location that includes at least one of the asset locations 102b. In one implementation, the system may query a remote database to obtain the latest aerial image 102c that includes an image of at least one asset location 102b based on a street address or geographic coordinates. The remote database may include, in some implementations, a database of a private company, such as Google® Earth imagery from Google Inc. (Mountain View, Calif.) or NTT Spatial Information Corporation (Japan). In other implementations, the remote database may include one or more databases of a public agency, such as the Geospatial Information Authority of Japan (GSI), the United States Geological Survey, or the German Federal Office for Mapping and Geodesy. The aerial image may be, in some implementations, stored in a database such as QGIS by the Open Source Geospatial Foundation (OSGeo). The images may be collected from one or more remote network locations using an open source geographic information system (GIS). The format of the images of the asset locations 102b may, in some embodiments, depend on the format received by the various sources available for the aerial imagery 102c. An example of an aerial imagery 510 of multiple asset locations 102b is shown in FIG. 5B. It has been done.

[0028]

[0040] In some embodiments, the system may be configured to query multiple remote database systems to obtain at least two aerial images of a given asset location 102b. The aerial images available in the various databases may vary, for example, in terms of resolution and recency of capture. By collecting more than one image of a particular asset, for example, the system may analyze each image to determine the highest quality image to use in the condition analysis. The condition analysis, in some examples, may include balancing multiple factors such as clarity, completeness, and recency.

[0029]

[0041] Further, in some embodiments, the system may be configured to query multiple remote database systems to obtain both two-dimensional and three-dimensional aerial images. Different asset characteristics may be determined based on whether the aerial images were captured in two-dimensional or three-dimensional format. The two-dimensional aerial images may be used to determine the asset's location coordinates, street name, occupancy type, floor area, skylight presence or absence, chimney presence or absence, roof condition, roof shape, roof covering, roof anchors, roof equipment, and / or pounding, in some examples. The three-dimensional aerial images, when compared to the two-dimensional aerial images, may be used to determine the location coordinates, street name, building structure type, occupancy type, construction year, building height, soft story, number of floors, roof condition, roof shape, roof covering, roof anchors, roof equipment, cladding, and pounding. In some embodiments, when there is an overlap of characteristics identifiable using either the two-dimensional or three-dimensional images, machine learning analysis of both images may be combined to increase confidence in the identification of the individual characteristics.

[0030]

[0042] In some implementations, the system obtains (104) a geometric map image 102a for each of the asset locations 102b. The geometric map image 102a includes a block-shaped layout of the existing assets within a city, town, or village, such as a city planning map used for urban planning and development. The geometric map image 102a may, in another implementation, include a block-shaped layout map used in providing information to a user of a computer-based navigation system. The geometric map image 102a may, in one implementation, be obtained from the Geospatial Information Authority of Japan or Zenrin Corporation of Japan. An example of a city planning map 500 is shown in FIG. 5A (Image obtained from the Geospatial Information Authority of Japan, Digital Map 2500 (Spatial Data Framework)).

[0031]

[0043] The shape map image 102a is used in some embodiments to locate a particular asset. The shape map image 102a of a geographic region may be overlaid with a corresponding aerial image 102c to match assets to the image, for example. An example of an image 520 of the city plan image 500 of FIG. 5A overlaid with the aerial image 510 of FIG. 5B is shown in FIG. 5C. In some embodiments, the shape map image 102a may not match the aerial image 102c in shape or location. As shown in FIG. 5C, for example, the shape map outline (shown in bold) of the location 522 may not be rectangular, but the shape of the rooftops below the outline may be rectangular. In some embodiments, analyzing the multiple aerial images 102c to determine the highest quality image may include identifying one or more aerial images 102c that substantially match the corresponding shape map image 102a. The overlay of the aerial imagery 102c and the shape map image 102a may further be used to aid in cropping the aerial imagery 102c to focus the analysis on a particular asset location 102b. For example, referring to FIGURE 5C, an asset location 524 may be substantially matched with a corresponding city plan image, and the overall outline may be used to aid in cropping the asset location 524. In some examples, an asset location 524 may be identified in the overlaid image 520 if it matches within a predetermined tolerance with an asset in the overlaid image 520 to account for modifications made to the asset 524, such as an asset addition that may change the overall shape of the asset.

[0032]

[0044] Returning to FIG. 1, in some embodiments, the shape map image 102a may be used to correct or compensate for alignment errors or mismatches in the corresponding two-dimensional aerial image 102c. For example, an aerial image showing a standard orthophoto angle may not be based directly on the city plan block. Using the city plan block as a guide, for example, the aerial image may be geometrically corrected to obtain a true orthophoto version of the aerial image.

[0033]

[0045] As illustrated in FIG. 1, in some implementations, the system identifies features of each aerial image 102c to classify asset characteristics (106). Using machine learning for analysis, for example, the system can extract aerial image features of a particular asset location 102b. Groupings of extracted features, such as angles, contours, substantially homogenous areas, and the like, can be used to identify asset features, such as roof tops, swimming pools, chimneys, and skylights. Upon extracting an image-related feature identifiable as an asset feature (e.g., roof top), the extracted feature can be analyzed to determine one or more asset characteristics of the feature (e.g., type of roof top). In one example, a swimming pool can be identified as a pool asset feature (e.g., various shapes of contours of a particular size or larger that bound a substantially homogenous blue area), and then characterized as a particular shape (e.g., rectangular, circular, or kidney-shaped, etc.), type (e.g., above or below ground), and / or size (e.g., approximate area). The machine learning classifier used in the machine learning analysis includes, in some embodiments, a convolutional neural network (CNN) to pre-process the aerial imagery 102c of a particular asset location 102b and to classify the asset features as asset characteristics 110. Using CNN techniques, for example, the biological process of visually inspecting and identifying an object or object feature is mimicked by a feed-forward artificial neural network where multi-layer artificial perception minimizes pre-processing. CNN is a well-established and common machine learning method. An example of a CNN processing model is Alexnet. (See, "ImageNet Classification with Deep Convolutional Neural Networks," Krizhevksy et al., Advances in neural information processing systems. (2012), the entire contents of which are incorporated herein by reference.) In other embodiments, the machine learning classifier may include a Network In Network (NIN), where multiple layers of artificial perception results are generated using micro-neural networks with complex structures.The artificial perception results are then stacked and averaged to generate a single global average pooling layer for classification. NIN has shown better performance results than traditional CNN processing. Furthermore, NIN processing is less storage intensive than CNN processing.

[0034]

[0046] In some implementations, an asset characteristic profile 108 is used to classify the asset features. The asset characteristic profile 108 may be developed, for example, by training a machine learning algorithm using aerial imagery (and in some embodiments, terrestrial imagery 102d) of known asset features. Each asset feature may be separated into multiple classifications, for example. In an example involving classification of terrestrial images, cladding may include stone, brick, stucco, shingles, vertical shingles, horizontal shingles, or metal. The machine learning algorithm may, for example, generate a new aerial image 102c including a gambrel roof top and a characteristic profile 108 for the gambrel roof top. A confidence percentage may be generated for the match with.

[0035]

[0047] Referring to FIG. 2A, in the illustrated embodiment, the extracted angle and contour features of the aerial image 204 may be used to classify the roof shape 202 of a particular asset location 102b. In a first embodiment, the gable roof 202b includes a central peak in a generally rectangular contour as shown in the corresponding aerial image 204b. The gambrel roof 202a includes these characteristics of the gable roof 202b as shown in the aerial image 204a, but the gambrel roof further includes a "skirt-like" region surrounding the central gable feature. The "skirt-like" region may be identified in part based on the corner edges extending from the four corners of the gable feature. The hip roof 202c includes a central peak and two opposing triangular ends extending from either end of the central peak as shown in the aerial image 204c. The central peak forms the top of two trapezoidal portions. In contrast, the rectangular roof 202d does not have a central peak. Instead, as shown in aerial image 204d, a rectangular roof includes four triangular sections that meet at a central point. Finally, a flat roof 202e is represented by a substantially flat surface with a rectangular outline, as shown in aerial image 204e.

[0036]

[0048] 2D, an example error rate graph 240 compares the accuracy of matching training data 242 with matching test data 244 in matching roof top features in aerial images with corresponding roof top types (e.g., shapes). Testing was performed using the NIN deep learning algorithm to classify roof top shapes in 2D aerial images including gambrel, gable, hip, square, and flat roof shapes of various colors. Further testing was performed across different image sizes and mini-batch sizes. In particular, the images were generated using the formula

number

[0037]

[0049] In some embodiments, to improve accuracy, the image may be cropped and / or resized prior to analysis. For example, the image may be cropped to include the asset of interest, or a portion of the asset of interest and its surroundings (e.g., a parcel that includes the asset and / or a portion of the neighborhood proximate to where the asset is located). In another example, resizing to a standard image size may help improve accuracy.

[0038]

[0050] In some embodiments, in addition to roof shape, in some examples, feature analysis may be used to determine additional roof features such as roof coverings, roof anchors, roof fixtures, skylights, observation decks, turrets, towers, dormer windows, and / or chimneys. Furthermore, upon identifying the roof contours, the footprint (e.g., roof size) of the asset location 102b may be calculated based on the scale ratio of the aerial imagery 102c.

[0039]

[0051] Returning to FIG. 1, in some implementations, rather than analyzing only a particular asset location 102b individually, a particular asset location 102b is classified based in part on adjacent structures. For example, based on the dimensions of the asset location 102b, which may include lot dimensions, machine learning may be used to identify additional structures at the asset location 102b, such as a deck, swimming pool, shed, gazebo, detached garage, tennis court, fence, retaining wall, dock, playground equipment, or greenhouse, in some implementations. Additionally, a particular asset location 102b may be classified based in part on the distance (pounding) from adjacent structures.

[0040]

[0052] In some implementations, the terrestrial imagery 102d may be used to classify features that are difficult to recognize using the aerial imagery 102c. The terrestrial imagery 102d may include street view imagery obtained from a street view service or real estate listings in some implementations. In some implementations, street view imagery including a view looking down the street of the property location 102b may be obtained from Google® Street View from Google, Inc., Bing® Maps Streetside from Microsoft Corporation (Redmond, WA), or Mapillary from Mapillary AB (Sweden). For example, using the terrestrial imagery, the system may identify features such as building construction type, cladding, building height, number of soft stories, number of floors, location coordinates, street name, slope, elevation, year built, and / or occupancy type. If characteristics identifiable using the terrestrial imagery 102d overlap with characteristics identifiable using the two-dimensional or three-dimensional aerial imagery 102c, in some implementations, analysis of the terrestrial imagery 102d may be combined with analysis of the aerial imagery to increase confidence in the identification of a particular characteristic or characteristics. For example, house siding features may be more easily detected in the terrestrial image 102d and / or the three-dimensional aerial image 120c than in the two-dimensional aerial image 102c.

[0041]

[0053] In some implementations, the condition of each asset feature may be classified 112 as a corresponding condition characteristic 116. While a new asset may be in good condition, the asset feature condition may deteriorate over time due to normal wear and tear on the asset. Asset features may also be damaged by external forces such as storms and natural disasters. Eventually, the condition of a residential feature may deteriorate to the point that repair and / or replacement may be required. Machine learning algorithms may be used to classify the current condition of each detected asset feature, such as the asset features described above in block 106. Using machine learning for analysis, for example, the system may extract pixel intensity distributions of previously identified asset features in aerial images of a particular asset location 102b. In some implementations, features of a newly constructed asset generally have sharp contrast features that stand out in machine learning image analysis. On the other hand, features of weathered or damaged assets may have smooth edges, fuzzy contrast, and asymmetric wear in places. The machine learning classifiers used in the machine learning condition analysis, in some embodiments, include machine learning analysis to process aerial imagery 102c of a particular asset location 102b and to classify the condition of previously identified asset characteristics 110 as condition characteristics 116. The machine learning analysis may include 2D color histogram analysis or 3D color histogram analysis in some examples. In other embodiments, the machine learning analysis may be performed using pattern recognition algorithms (e.g., determining missing fence posts or missing / misaligned roof top shingles). In other embodiments, the machine learning classifiers include deep learning analysis such as CNN or NIN.

[0042]

[0054] In an illustrated example, referring to FIG. 2B, an example of condition characteristics and corresponding histograms created by machine learning analysis of pixel intensities of aerial images of roof tops of hipped roofs are shown. For example, a set of good condition aerial images 210a-214a are shown along with corresponding good condition histograms 210b-214b. In the illustrated embodiment, the good condition histograms 210b-214b show an indication of the probability of a grayscale color distribution, with each maximum value 210c-214c showing the highest probability of a grayscale color distribution of a roof in good condition. For example, the values ​​can be from 0 (completely black) to 255 (completely white).

[0043]

[0055] Meanwhile, a set of poor condition aerial images 216a-220a are shown along with corresponding poor condition histograms 216b-220b. In the illustrated embodiment, the poor condition histograms 216b-220b show a measure of the probability of the grayscale color distribution, with each individual maximum value 216c-220c showing the highest probability of the grayscale color distribution of the roof in good condition. The maximum values ​​216c-220c corresponding to the poor condition roof tops 216a-220a are significantly lower than the corresponding maximum values ​​210c-214c of the good condition roof tops 210a-214a. Furthermore, the distribution width of the poor condition histograms 216b-220b may be significantly wider than the distribution width of the good condition histograms 210b-214b. As can be seen from the aerial images of roof tops 216a-220a in poor condition, the color distribution is sparse and washed out, and the sharp edges of the roof tops are no longer visible, whereas when looking at the aerial images 210a-214a of roof tops in good condition, the roof tops are substantially more uniform in color and have sharp edges.

[0044]

[0056] Referring to FIG. 2C, a condition profile 230 of the condition of a roof top is shown. In some implementations, pixel value distribution histograms such as histograms 210b-220b may be combined to create an example condition profile 230 corresponding to the condition of a roof top. The condition profile 230 may be included, for example, as one of the condition profiles 114 of FIG. 1. Reference bars 232a, 232b on the condition profile 230 indicate values ​​corresponding to a non-deteriorated hip roof 214a and a deteriorated hip roof 216a, respectively. In some implementations, values ​​of the reference bar 232 closer to 1 on the y-axis indicate a better roof condition, and values ​​closer to 0 indicate a roof top in poor condition. In one implementation, the reference bars 232 of the example condition profile 230 correspond to a test condition sample dataset including 2500 images. In some implementations, the reference bars 232 are combined with a logistic regression to create a roof condition profile graph 236. The accuracy rate of the image histograms is 80% based on several validations.

[0045]

[0057] Although illustrated as one condition profile 230 corresponding to different types of roof tops (e.g., at least hip roof tops 216a and gable roof tops 234), in other embodiments, individual condition profiles 114 may be created for each property characteristic (e.g., a gable roof specific profile, a gambrel roof specific profile, a flat roof specific profile, a hip roof specific profile, a square roof specific profile, etc.). In a further refinement, in certain embodiments, individual condition profiles 114 are created for combinations of property characteristics (e.g., gable shingle roof, gable tile roof, gable metal roof, etc.) to improve accuracy rates based on pixel density corresponding to the combined characteristics. Many combinations of characteristics of property features may be used alone or in combination to create condition profiles 114 designed to accurately identify the condition of the corresponding property features.

[0046]

[0058] Returning to FIG. 1 , in some implementations, the asset characteristics 110 and / or condition characteristics 116 may be analyzed to calculate 120 a risk estimate 122 corresponding to a risk of damage based on one or more types of hazards. The types of hazards may include earthquakes, hurricanes, tornadoes, storm surges, straight winds, or eruptions, in some examples. In some embodiments, the type and estimated severity of the hazard may be based on the particular asset location 102b. For example, assets near active faults may be more susceptible to earthquakes, while coastal assets may be more susceptible to storm surge damage. The risk estimate 122 may include actuarial percentages indicative of the likelihood of damage based on the particular asset characteristics, in some examples. For example, different asset characteristics may be more susceptible to certain types of damage than others. For example, flat roof tops may be more susceptible to damage from excessive snow accumulation, while roof tops with eaves may be more susceptible to high winds.

[0047]

[0059] In some embodiments, one or more disaster risk profiles 118 may be applied based on asset characteristics in calculating the damage risk. Vulnerability to disaster damage may vary by location, elevation, slope, roof top shape, roof top condition, cladding type, cladding condition, and / or pounding, in some examples. In a particular example, as shown in FIG. 2A, the graphical risk profile 206 ranks roof top types by their vulnerability to high winds. Vulnerability to wind damage increases from the lowest vulnerability (flat roof tops 202e) to square roofs 202d, hip roofs 202c, gable roofs 202b, and gambrel roofs 202a with the highest vulnerability to wind damage. Vulnerability may be mathematically modeled based on statistics built from past damage results from various types and / or severities of disasters. Combining location-based vulnerability with vulnerability based on individual asset characteristics, the risk of disaster damage may be more accurately estimated.

[0048]

[0060] Returning to FIG. 1 , the risk estimates 122 may be used in some embodiments in insurance underwriting. For example, the risk estimates 122 may be used to determine whether an asset is adequately insured against potential damage. Additionally, when tracking upcoming natural disasters, insurance companies use the risk estimates 122 to estimate insurance payouts based on the expected path and severity of the storm. Additionally, the risk estimates 122 may be utilized by investors in selecting investment assets for their investment portfolios.

[0049]

[0061] In some implementations, repair or replacement costs may be calculated (126) to determine a cost estimate 128 for placing one or more asset properties in good condition or in a “like new” condition. The cost estimate 128 may be based in part on one or more replacement cost profiles 120 in some embodiments. The replacement cost may vary in some examples based on the type of material (e.g., roof top material, cladding material, etc.), the scale of the work (e.g., estimated area of ​​the roof, estimated length of the fence, etc.), and / or additional asset properties (e.g., contributing to the complexity of the job). The replacement profile 120 may be based in part on location in another example (e.g., different areas have different material and labor costs). In some implementations, the cost estimate 128 for repairing or replacing the asset property may be output to a dashboard interface screen of a remote computing device of a claimant, such as an insurance carrier or real estate investment broker.

[0050]

[0062] In some embodiments, the cost estimates 128 may be used to automatically analyze insurance claims. The cost estimates 128 may also be combined with the risk estimates 122 when analyzing the adequacy of insurance coverage for a particular asset location 102b. The cost estimates 128 may also be used by investors in determining the cost of maintaining / upgrading a particular investment asset.

[0051]

[0063] Although the operational flow is shown as a series of computational steps, other implementations may include more or fewer computational steps. For example, aerial imagery 102c may be analyzed after a claim has been paid to verify that the insured property has been repaired and / or to upgrade the property's estimated value (e.g., if the owner upgraded the property rather than directly repairing / replacing the damaged property features).

[0052]

[0064] Additionally, in other implementations, certain computational process steps may be performed in a different order. For example, cost estimates 128 may be calculated before risk estimates 122. Other modifications of operational flow 100 are also possible.

[0053]

[0065] 3 is a block diagram of an example environment 300 for matching topographical features of aerial imagery with corresponding asset maintenance management levels. The example environment 300 may be used, for example, to implement the operational flow 100 described with respect to FIG. 1. The example environment 300 includes an asset characterization / condition analysis system 302 in communication with a client computing system(s) 306, a number of cost data and / or risk data source computing system(s) 308, and a number of geometry map, aerial imagery, and / or terrestrial imagery source computing system(s) 304. The communication connections may be wired or wireless connections over a variety of network types. Generally, the system 302 obtains asset identifier(s) 340 from the client computing system(s) 306, accesses images of the assets identified by the asset identifier 340 from the image source computing system(s) 304, and analyzes the images to determine asset characteristic(s) 342 and condition characteristic(s) 344. Using cost and / or risk data obtained from the cost / risk data source computing system(s) 308, the system 302 can determine replacement cost estimate data 350 and / or disaster risk estimate data 352 based on the asset characteristics and the condition of the asset.

[0054]

[0066] In some implementations, a user of a particular client computing system 306 submits a request to the system 302 via a graphical user interface provided by the graphical user interface engine 334. The request may include, for example, an identification of at least one asset identifier 340 and one or more asset characteristics 342, and / or a replacement or repair cost estimate, a disaster risk estimate, and a confirmation of repair assessment. The asset identifier 340 may include location information (e.g., an address, geolocation coordinates, parcel boundaries, etc.) in some implementations. The location information may match, for example, the asset location information 102b described with respect to FIG. 1. In other embodiments, the asset identifier 340 may be a unique asset identifier (e.g., a number or string of characters) used to look up the asset location information in the data store 310. In another embodiment, rather than submitting an asset identifier, a user may submit a unique user identifier used to look up one or more assets associated with the user (e.g., assets insured by an insurance client, assets held by an investment client, etc.).

[0055]

[0067] In one example, referring to FIG. 8A , a user may identify the location of one or more assets by selecting the assets in a graphical map interface 800. Many assets are identified in the map view user interface 800 as indicated by associated pins, such as pin 804. The pins may indicate assets that are insured by a particular insurance carrier, types of assets (e.g., commercial, industrial, residential, etc.), and / or ranges of values ​​(e.g., within a range, above a threshold, etc.), in some examples. The pins may be displayed based on filtering options provided by the user or otherwise associated with a user profile (e.g., insurance carrier identification). In a particular example, a user may select three assets associated with three callouts 802. As shown, each callout identifies a replacement cost associated with the asset and invites the user to use a details control to explore details about the selected asset.

[0056]

[0068] In some implementations, the user inputs asset characteristics associated with each selected asset. In other implementations, the characteristics may be accessed upon identification of the asset (e.g., selection of a pin identifying the asset, as shown in FIG. 8A). For example, asset characteristics 342 may be retrieved from data store 310, as shown in FIG. 3. With reference to FIG. 8B, an example asset characteristic user interface 820 presents building area characteristics 822 (e.g., building area 822a, number of stories 822b, and building floor area 824c), and structural characteristics 824 (e.g., construction material 824a, building construction class 824b). As described, different or additional characteristics may vary from implementation to implementation. In some embodiments, the user may edit the building area characteristics 822 and / or structural characteristics 824 via user interface 820, for example, using edit controls 828a or 828b. For example, in the context of recent development or renovation (e.g., addition), the user may modify previously stored property values. For example, the asset characteristics user interface 820 may be presented to a user upon selection of a details control of one of the callouts 802 shown in the user interface of FIG. 8A.

[0057]

[0069] In some implementations, the user interface 820 further presents premium characteristics 826, such as replacement costs. The premium information may relate to, for example, an insurance policy purchased for the property or parameters for insuring the property (e.g., based on upgraded or new characteristics). The user may update the premium information using edit controls 828c, in some embodiments.

[0058]

[0070] In some implementations, upon identifying the asset location information, the system 302 uses the image acquisition engine 336 to access images of the asset from the image source computing system(s) 304. The images may include, in some examples, a geometric map (e.g., geometric map 102a described with respect to FIG. 1), a two-dimensional aerial image (e.g., aerial image 102c described with respect to FIG. 1), a three-dimensional aerial image (e.g., aerial image 102c described with respect to FIG. 1), and / or a terrestrial image (e.g., terrestrial image 102d described with respect to FIG. 1). In some embodiments, the images acquired by the image acquisition engine 336 are based on the asset property(s) identified by the user. For example, certain types of images are better suited to analyze certain types of asset properties. For example, house siding is more easily recognized from a terrestrial image or a three-dimensional aerial image than from a two-dimensional aerial image.

[0059]

[0071] In some implementations, the image quality analysis / pre-processing engine 326 analyzes the acquired image(s) to ensure that the acquired image(s) contain sufficiently clear and detailed images of the asset identified by the asset identifier 340. In some embodiments, the image quality analysis / pre-processing engine 326 crops and / or resizes the acquired image(s). For example, the image quality analysis / pre-processing engine 326 may extract a portion of the acquired image based on the asset (e.g., building size). In some implementations, the image quality analysis / pre-processing engine 326 resizes each acquired image to a standard size. The standard size may be 256×256 pixels per image in a particular example implementation. Resizing to a standard size may, for example, improve the accuracy of subsequent classification analysis. In some embodiments, the image quality analysis / pre-processing engine 326 may be configured to apply corrections to the acquired image(s). For example, the image quality analysis / pre-processing engine 326 may be configured to adjust the aerial imagery from a standard orthophoto angle to a true orthophoto version as described above.

[0060]

[0072] In some implementations, the image quality analysis / pre-processing engine 326 uses the shape contours as part of the analysis. For example, the synthetic image generation engine 318 may overlay an aerial image with a shape map image, as described with respect to FIG. 5C. The image quality analysis / pre-processing engine 326 may use the generated synthetic image to verify that the aerial image matches a city plan or other asset location guide (e.g., navigation map data, etc.). Additionally, the image quality analysis / pre-processing engine 326 may use the overlaid shape map as a guide in determining whether the aerial image needs to be corrected to a true orthophoto angle and / or to what extent to crop the image to capture the asset (and, optionally, the asset's immediate surroundings, such as part of the asset's parcel or neighborhood).

[0061]

[0073] In some implementations, if the image quality analysis / pre-processing engine 326 determines that the acquired image is insufficient, the image quality analysis / pre-processing engine 326 may request a replacement image from the image acquisition engine 336. For example, the image acquisition engine 336 may acquire images based on various factors including, in some examples, recency of capture, resolution, cost, and / or applicability to a particular asset characteristic analysis. If the image quality analysis / pre-processing engine 326 determines that a first acquired image is insufficient, for example, the image acquisition engine 336 may determine the next best source for acquiring an image of the asset.

[0062]

[0074] In some implementations, once an image has been approved (and optionally preprocessed and / or corrected) by the image quality analysis / preprocessing engine 326, the feature identification engine 320 extracts features from the asset image(s) that relate to the identified asset characteristics 342. For example, in the case of a fence, the feature identification engine 320 may identify a perimeter or partial perimeter adjacent to and extending from the asset as indicative of a fence.

[0063]

[0075] In some implementations, once the features are identified by the feature identification engine 320, an asset characteristic classification engine 322 classifies the asset features. For example, the features may be classified using one or more machine learning techniques, as described with respect to computational processing stage 106 (classifying the features of each image) of FIG. 1. The asset characteristic classification engine 322 may generate asset characteristics 342 that represent, for example, a grouping, type, or other characterization of a particular asset feature. In the particular example of a fence, the extracted features may then be analyzed and classified with respect to height and / or material. Once the asset features are classified, the asset characteristic classifications may be stored in the data store 310 as asset characteristics 342 (similar to asset characteristics 110 of FIG. 1).

[0064]

[0076] In some implementations, the asset characteristic classification engine 322 uses characteristic profile(s) 360 in classifying the asset characteristics. The asset characteristic classification engine 322 may, for example, retrieve a particular characteristic profile 360 ​​from the profile data store 314 for each asset characteristic being analyzed. The characteristic profile may be similar to, for example, the characteristic profile 108 described with respect to FIG.

[0065]

[0077] In some implementations, the asset characteristic profile 360 ​​is generated by the characteristic classification learning engine 328. The characteristic classification learning engine 328 may generate the characteristic profile 360 ​​using a sample dataset learning process, for example, as generally described with respect to FIGURE 2A. In particular examples, the characteristic classification learning engine 328 may analyze JPG (a lossy compressed digital image standard developed by the Joint Photographic Experts Group) to recognize various rooftop shapes.

[0066]

[0078] While a particular asset characteristic classification is determined by analysis, in some implementations, one or more characteristic classifications may be obtained from previously stored asset characteristics 342. For example, some basic material and construction information about the asset may be readily available via local and / or remote database system(s), based on tax records, real estate records, etc. In other examples, previously stored asset characteristics 342 may also be obtained from a previous condition assessment performed by system 302 on the asset location.

[0067]

[0079] In some implementations, once the asset characteristic classification engine 322 classifies the asset characteristic(s), the characteristic condition classification engine 324 classifies the condition of each asset characteristic. For example, the asset characteristics may be classified as described with respect to computational process step 112 (classifying the condition of each characteristic) of FIG. 1. The characteristic condition classification engine 324 may generate a condition characteristic 344 that indicates, for example, a rating, qualification, or ranking of the condition of the particular characteristic. In some examples, the characteristic condition classification engine 324 may qualify the characteristic as a rating of (a) "good" or "poor," (b) "like new," "acceptable," "needs repair," or "needs replacement," or (c) a numerical scale such as a scale of 1-5, where 5 may correspond to "like new" and 1 may correspond to "needs replacement." In other implementations, a particular problem may be identified by the characteristic condition classification engine 324, such as, for example, in some examples, a fence feature that is partially collapsed or a roof feature that shows signs of structural damage.

[0068]

[0080] In some implementations, the property condition classification engine 324 uses a condition profile(s) 362 in classifying the asset property condition. The property condition classification engine 324 may, for example, retrieve a particular condition profile 362 from the profile data store 314 for each asset property being analyzed. The condition profile 362 may be similar to the condition profile 114 described with respect to FIG.

[0069]

[0081] In some implementations, the state profile 362 is generated by the state classification learning engine 330. The state classification learning engine 330 may generate the state profile 362 using a sample data set learning process, such as that generally described with respect to FIG. 2B. In certain implementations, the state profile 362 may be generated by identifying peak pixel values ​​of an image in a grayscale. Color histogram analysis can be used to classify the asset condition into a particular category using the calculated peak pixel values.

[0070]

[0082] In some implementations, based on the output of the asset characteristic classification engine 322 and / or the characteristic condition classification engine 324, the risk calculation engine 316 may estimate a damage risk based on one or more types of disaster. For example, disaster risk estimate data 352 may be calculated as described with respect to the calculation process stage 120 (calculate damage risk) of FIG. 1. The disaster risk estimate data 350 may include a percentage of damage probability due to each particular type of risk based on a particular asset characteristic and / or asset condition, as described with respect to the risk estimate 122 of FIG. 1 in one example. Furthermore, in some embodiments, the disaster risk estimate data 350 may include a risk estimate set based on a disaster severity number, a particular track of the disaster, and / or a hazard intensity of the disaster, in some examples. The disaster severity may correspond to a Richter scale of magnitude of seismic activity or a Saffir-Simpson Hurricane Wind Scale category in some examples. The track of the disaster may include a geographic direction, such as northeast, or an impact angle relative to the asset characteristic(s), in some examples. The hazard intensity of a disaster may be related to wind speed or precipitation in some examples. Repair costs may be more accurately estimated if the condition of the roof, for example, is known. In one particular example, poor condition of the roof top may make the roof more susceptible to damage requiring replacement. In another particular example, a roof top in poor condition may increase the likelihood of damage to internal assets (e.g., water damage due to roof cracks) above a set level of disaster severity.

[0071]

[0083] In some embodiments, the risk calculation engine 316 accesses one or more risk profiles 364 from the data store 314 based on asset characteristics and / or condition characteristics. The risk profiles may be similar to the disaster risk profiles 118 described with respect to FIG. 1, for example. The risk profiles 364 may be obtained from one or more data sources 306, for example, by the cost / risk data acquisition engine 338. In other embodiments, the system 302 may generate the risk profile 364 (e.g., using risk data obtained from one or more cost / risk data sources 306 and a risk data analysis engine (not shown)).

[0072]

[0084] The risk estimate data 352, in some embodiments, is shared with one or more clients 306. For example, an insurance provider client 306 may use the risk estimate data when performing insurance underwriting. In another example, a real estate investment broker or firm client 306 may utilize the risk estimate data 352 when selecting investment assets for an investment portfolio.

[0073]

[0085] 8C, an example disaster risk user interface 840 presents risk assessment information based on a risk profile 364 associated with a selected asset (e.g., one of the assets identified by callout 802c in FIG. 8A). User interface 840 may be accessed, for example, by scrolling down from user interface 820. User interface 840 shows a number of disaster risk factors 842 and associated risk estimate data 844 for the selected asset. As shown, disaster risk factors 842 include a fire factor 842a, a wind factor 842b, a flood factor 842c, an earthquake factor 842d, a tsunami factor 842e, and a landslide factor 842f. In some embodiments, only a subset of the risk estimate data is determined using the risk assessment techniques described herein. For example, flood data 844c may be generated based on governmental flood map assessments rather than using aerial imagery assessments, while fire risk 844a may be determined based at least in part on proximity to neighboring assets derived by image analysis and / or asset structure derived by image analysis. Additionally, wind risk data 842b may be derived by roof top analysis, as described in further detail herein.

[0074]

[0086] In some implementations, based on the output of the asset characteristic classification engine 322 and / or the property condition classification engine 324, a cost calculation engine 336 calculates repair or replacement costs to remedy the problems identified by the property condition classification engine 324 to generate replacement cost estimate data 350. For example, the replacement cost estimate data 352 may be calculated as described with respect to calculation process stage 126 (calculate repair or replacement costs) of Figure 1. The replacement cost estimate data 350 may be similar to the cost estimate 128 described with respect to Figure 1, for example.

[0075]

[0087] In some implementations, the cost calculation engine 336 is configured to calculate, in part, 1 ). In some embodiments, the cost calculation engine 336 accesses the one or more cost profiles 366 from the data store 314 based on asset characteristics and / or condition characteristics. The cost profiles 366 may be obtained from one or more cost data sources 306, in some embodiments, by a cost / risk data acquisition engine 338. In other embodiments, the system 302 may generate the cost profiles 366 (e.g., by a cost data analysis engine (not shown) that uses cost data obtained from one or more cost / risk data sources 306).

[0076]

[0088] The cost estimate data 350, in some embodiments, is shared with one or more clients 306. For example, an insurance carrier client 306 may use the cost estimate data in determining the adequacy of insurance coverage for a particular property. In another example, a real estate investment broker or real estate investment firm client 306 may utilize the cost estimate data 350 in determining the cost of maintaining / upgrading a particular investment property.

[0077]

[0089] In some implementations, the cost estimate data 350 and / or the risk estimate data 352 are provided to the requesting client 306 in real time or near real time in response to receiving an identification of one or more assets. For example, the client 306 accessing a dashboard interface may format a request for risk or cost estimates for the assets identified in the dashboard interface and submit the request to the system 302 over a network. In response to the request, the system 302 may perform an analysis as broadly described above and provide a response in the form of a risk or cost analysis to the client 306 in real time or near real time. For example, the risk analysis information is presented in the user interface 840 of FIG. 8C.

[0078]

[0090] In some implementations, the system may use the updated weather data to identify one or more assets in a client real estate portfolio that have been affected by the disaster. In addition to identifying affected assets, the system may proactively generate and provide repair estimates to clients corresponding to the real estate portfolio shortly after the disaster has had an impact. In one example, the weather bureau may update the observation data via a website interface within an hour or so of the observation. In this example, the system may provide repair cost estimates to clients within one to two hours after the disaster occurs.

[0079]

[0091] In other embodiments, rather than sharing cost estimate data 350 and / or risk estimate data 352 directly with clients 306, system 302 may include a report generation engine (not shown) that generates reports regarding the condition, damage, and risk assessment of one or more assets. Additionally, in some embodiments, system 302 may compare current condition characteristics 344 to historical condition characteristics 344 to ascertain whether an asset owner has made repairs to the asset (e.g., based on an insurance claim payment). Other modifications of system 302 are possible.

[0080]

[0092] 4 is a flow chart illustrating an example of a method 400 for classifying condition characteristics of an asset based on aerial imagery analysis. For example, the flow chart may illustrate computational process steps 104-112 as described with respect to FIG 1. Method 400 may be performed, for example, by system 302 of FIG 3.

[0081]

[0093] In some implementations, method 400 begins with accessing (402) a two-dimensional shape map that includes a shape of the asset. The shape map may be accessed by image capture engine 336 from a shape map image source 304, for example, as described with respect to FIGURE 3. The two-dimensional shape map may take the form of, for example, any of the shape maps 102a described with respect to FIGURE 1. In certain implementations, the shape map may be in a form similar to shape map 500 described with respect to FIGURE 5A.

[0082]

[0094] In some implementations, two-dimensional and / or three-dimensional aerial imagery including images of the asset is accessed (404). The two-dimensional and / or three-dimensional aerial imagery may be accessed by the image acquisition engine 336 from an aerial image source 304, for example, as described with respect to FIGURE 3. The two-dimensional and / or three-dimensional aerial imagery may take the form of any of the aerial images 102c described with respect to FIGURE 1. In certain examples, the two-dimensional aerial imagery may be in a form similar to the aerial imagery 510 described with respect to FIGURE 5B.

[0083]

[0095] In some implementations, the aerial image is overlaid with the geometry map (406). The synthetic image generation engine 318 of Figure 3 may, for example, overlay the aerial image with the geometry map to generate a synthetic image. The synthetic image may be, for example, in a form similar to synthetic image 520 described with respect to Figure 5C.

[0084]

[0096] If a shape match is not identified (408), in some implementations, the method 400 accesses (410) an alternative two-dimensional or three-dimensional aerial image. In some embodiments, the image quality analysis / pre-processing engine 326 of FIG. 3 determines whether the composite image shows a sufficient shape match of the asset(s) of interest. If it is determined that the images do not match, the image acquisition engine 336 of FIG. 3 may acquire an alternative two-dimensional or three-dimensional aerial image from, for example, the aerial image source(s) 304. As described with respect to FIG. 3, for example, the image acquisition engine 336 may determine the next best source for acquiring an image of the asset based on one or more elements. However, in other implementations (not shown), the method 400 may proceed without matching the shape map to the aerial image.

[0085]

[0097] If a shape match is identified (408), in some implementations, the aerial image is prepared for formatting (411). In some embodiments, the image is cropped to include the asset of interest or a portion of the asset and its surroundings (e.g., a parcel that includes the asset and / or a portion of the neighborhood proximate to where the asset is located). Cropping may be based, for example, on correlation of the shape map with the asset. As an example, the image may be cropped to include a shape map outline of the asset and a bounded area. In some embodiments, cropping may include cropping to a shape set (e.g., an X×X pixel square, an X×Y rectangle, etc.). In some embodiments, in addition to cropping the image, the image may be resized. For example, based on the resolution of the aerial image, the resolution may be reduced to, for example, fit within an X×X pixel square or an X×Y rectangle. Resizing to a standard image size may contribute to consistency of analysis and improved classification accuracy. In another embodiment, the color depth and / or color mapping of the aerial image may be adjusted. For example, the color depth and color mapping may be consistent across the entire analysis image for consistent color histogram analysis of the image. In certain embodiments, color images may be converted to grayscale for grayscale image analysis. Image adjustments may be performed, for example, by the image quality analysis / preprocessing engine 326 described with respect to FIG.

[0086]

[0098] In some implementations, the orthogonality of the aerial images is assessed (412). For example, the image quality analysis / preprocessing engine 326 may assess the orthogonality of the aerial images. The image quality analysis / preprocessing engine 326 may, in one embodiment, determine that orthogonality correction is desirable. In another embodiment, if the image represents a regular orthophoto rather than a true orthophoto, the image quality analysis / preprocessing engine 326 may notify another module that may correct the capture angle of the aerial images. In other implementations (not shown), the orthogonality may be assessed without the use of a superimposed shape map.

[0087]

[0099] Returning to Figure 4, in some implementations, if the aerial imagery is not orthogonal (414), an orthogonal version of the aerial imagery is generated (416). For example, the image quality analysis / preprocessing engine 326 of Figure 3 may correct the normal orthophoto morphology to generate a true orthophoto morphology. The true orthophoto morphology may, for example, be much more consistent with the shape map.

[0088]

[0100] In some implementations, one or more asset characteristics of the asset are classified (418). In some embodiments, the asset is analyzed based on a portion of the aerial image that is substantially surrounded by a geometric map of the asset. In other embodiments, the asset analysis encompasses the perimeter of the asset (e.g., features proximate to the asset on the asset's parcel, neighboring assets, etc.). The asset characteristic(s) may be classified by the asset characteristic classification engine 322, for example, as described with respect to FIG. 3. In certain examples, a rooftop asset characteristic may be classified as one of various rooftop shapes, as shown in FIG. 2A.

[0089]

[0101] In some embodiments, if an asset property classification is identified (420), a condition of each of the asset properties may be classified (422). The asset property condition(s) may be classified by a property condition classification engine 324, for example, as described with respect to FIG. 3. In particular examples, a classification of a hip roof top feature may be classified as one of good condition and poor condition, as shown in FIG. 2B. In some embodiments, the classification of the condition of each of the asset properties may be output to the claimant's computing device via a dashboard interface screen (424). In some examples, in addition to providing a condition classification for the asset property, the dashboard interface screen may include a replacement cost to replace the asset property, a risk cost estimate for damage from a natural disaster, or an identification of whether repairs have been made to the asset property.

[0090]

[0102] On the other hand, if a classification match is not identified (420), in some implementations, an alternative 2D or 3D aerial image is accessed (410) and method 400 resumes registering the alternative aerial image with the shape map (406). Alternatively, in some implementations, the condition may be assessed based on a default profile (not shown) associated with the asset features even if a classification match is not identified. For example, if the shape of the roof top does not match one of the shapes trained into the system, the condition of the roof top may be further assessed based on a general condition profile trained with various roof top shapes. In some implementations, the condition of the roof top may be assessed by comparing the current roof top condition to historical condition characteristics stored from previous condition assessments for the asset.

[0091]

[0103] Although method 400 is shown as a series of steps, other implementations may include more or fewer steps. For example, in some implementations, terrestrial imagery is acquired and used to classify the same and / or different asset characteristics as described with respect to operational flow 100 of FIG.

[0092]

[0104] Additionally, in other implementations, certain steps may be performed in a different order. For example, in some implementations, asset characteristics and condition characteristics may be classified in parallel (418, 422). Other modifications of operational flow 100 are also possible.

[0093]

[0105] Aspects of the present disclosure may be directed to a computing system for classifying the repair condition of an asset property to provide a dynamic, real-time asset condition assessment in response to a request received from a user, such as an insurance carrier or real estate investment broker, using aerial imagery. The implementations described herein improve upon conventional methods by applying deep learning analytical models to detected asset properties in the acquired imagery to assess whether the condition of the asset property has deteriorated to the point where repair or replacement is necessary and to determine the risk exposure of the asset due to the detected condition of the asset property. The implementations described herein improve the processing efficiency of the system, reduce the time required to perform a condition assessment, and automate the condition assessment process.

[0094]

[0106] Next, a hardware description of a computing device, a mobile computing device, or a server of an exemplary embodiment will be described with reference to FIG. 6. In FIG. 6, the computing device, the mobile computing device, or the server includes a CPU 600 that executes the above-mentioned processes. Process data and instructions may be stored in a memory 602. Furthermore, these processes and instructions may be stored on a storage medium disk 604, such as a hard drive (HDD) or a portable storage medium, or may be stored remotely. Furthermore, the claimed technical advance is not limited to the form of the computer-readable medium on which the instructions of the inventive process are stored. For example, the instructions may be stored on a CD, a DVD, a flash memory, a RAM, a ROM, an EPROM, an EEPROM, a hard disk, or any other information processing device with which the computing device, the mobile computing device, or the server communicates, such as a server or a computer.

[0095]

[0107] Additionally, some of the claimed technological advances may be provided as utility applications, background daemons, or operating system components, or combinations thereof, that run in conjunction with the CPU 600 and an operating system (such as Microsoft Windows 6, UNIX, Solaris, LINUX, Apple MAC-OS, and other systems known to those skilled in the art).

[0096]

[0108] CPU 600 may be an Intel (USA) Xenon or Core processor, or an AMD (USA) Opteron processor, or other processor types recognized by those skilled in the art. Alternatively, CPU 600 may be implemented on an FPGA, ASIC, PLD, or using discrete logic circuitry, as recognized by those skilled in the art. Additionally, CPU 600 may be implemented as multiple processors working together simultaneously to execute instructions of the inventive process described above.

[0097]

[0109] The computing device, mobile computing device, or server of FIG. 6 further includes a network controller 606, such as an Intel Ethernet PRO network interface card from Intel Corporation (USA), for interfacing with a network 6X. As will be appreciated, the network 628 may be a public network, such as the Internet, or a private network, such as a LAN or WAN network, or a combination thereof, and may further include a PSTN or ISDN sub-network. Furthermore, the network 628 may be wired, such as an Ethernet network, or wireless, such as a cellular network, including EDGE, 3G, and 4G wireless cellular systems. The wireless network may further be Wi-Fi, Bluetooth, or any other form of wireless communication known in the art.

[0098]

[0110] The computing device, mobile computing device, or server further includes a display controller 608, such as an NVIDIA GeForce GTX or Quadro graphics adapter from NVIDIA Corporation (USA), for interfacing with a display 610, such as a Hewlett Packard HPL2445w LCD monitor. A general purpose I / O interface 612 interfaces with a keyboard and / or mouse 614, as well as a touch screen panel 616, on or separate from the display 610. The general purpose I / O interface further connects to a variety of peripherals 618, including printers and scanners, such as Hewlett Packard's OfficeJet or DeskJet.

[0099]

[0111] Additionally, a sound controller 620, such as Creative's Sound Blaster X-Fi Titanium, may be provided within the computing device, mobile computing device, or server to interface with a speaker / microphone 622 to provide voice and / or music.

[0100]

[0112] A generic storage controller 624 connects to the storage media disk 604 by a communication bus 626 (which may be ISA, EISA, VESA, PCI, etc.) to interconnect all of the components of the computing device, mobile computing device, or server. A description of the general features and functionality of the display 610, keyboard and / or mouse 614, as well as the display controller 608, storage controller 624, network controller 606, sound controller 620, and generic I / O interface 612 are omitted herein for the sake of brevity, as these features are well known.

[0101]

[0113] One or more processors may be used to implement the various functions and / or algorithms described herein unless otherwise specified. Furthermore, any function and / or algorithm described herein may be executed on one or more virtual processors, on one or more physical computing systems, such as, for example, a computer farm or a cloud drive, unless otherwise specified.

[0102]

[0114] Flowchart and block diagrams of methods, systems, and computer program products according to embodiments of the present disclosure have been described. These aspects are implemented by computer program instructions that can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the function / act specified in the block(s) of the flowchart and / or block diagram.

[0103]

[0115] These computer program instructions may further be stored on a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored on the computer-readable medium create an article of manufacture that includes instruction means that implement the function / act specified in the flowchart and / or block diagram block(s).

[0104]

[0116] These computer program instructions can further be loaded into a computer or other programmable data processing apparatus to cause a computer-implemented process to be generated into a sequence of operational steps to be executed on the computer or other programmable data processing apparatus, whereby the instructions executing on the computer or other programmable data processing apparatus can generate a process for implementing the function / operation specified in the block(s) of the flowcharts and / or block diagrams.

[0105]

[0117] Additionally, the disclosure is not limited to the particular circuit elements described herein, nor is the disclosure limited to the particular sizing and classification of these elements. For example, one skilled in the art may design the circuits described herein based on the sizing and chemistry of the battery or based on the requirements of the backup load being driven.

[0106]

[0118] The functions and features described herein may further be performed by various distributed components of the system. For example, one or more processors may perform these system functions, where the processors may be distributed across multiple components communicating in a network. The distributed components may include one or more client and server machines that may share processing, as shown in FIG. 8, in addition to human interface communication devices (e.g., display monitors, smartphones, tablets, personal digital assistants (PDAs)). The network may be a private network, such as a LAN or WAN, or a public network, such as the Internet. Inputs to the system may be received by direct user input, received remotely, in real time, or as a batch process. Additionally, some implementations may be performed on modules or hardware that are not identical to those described. Thus, other implementations are within the scope that may be claimed.

[0107]

[0119] In some implementations, as illustrated in FIG. 7, the innovations described herein may interface with a Google Cloud Platform™ cloud computing environment 730 to execute at least a portion of the methods or algorithms detailed above. The processes associated with the methods described herein may be executed by a data center 734 on a computational processor, such as a Google Compute Engine. The data center 734 may also include an application processor, such as a Google App Engine, that may be used to interface with the systems described herein, for example, to receive data and output corresponding information. The cloud computing environment 730 may further include one or more databases 738, such as cloud storage and query databases, or other data storage. In some implementations, a cloud storage database 738, such as Google Cloud Storage, may store processed and unprocessed data provided by the systems described herein.

[0108]

[0120] The systems described herein may communicate with a cloud computing environment 730 via a secure gateway 732. In some embodiments, the secure gateway 732 includes a database query interface, such as the Google BigQuery platform.

[0109]

[0121] The cloud computing environment 102 may include a provisioning tool 740 for resource management. The provisioning tool 740 may be connected to computing devices of the data center 734 to facilitate provisioning of computing resources of the data center 734. The provisioning tool 740 may receive requests for computing resources via the secure gateway 732 or the cloud controller 736. The provisioning tool 740 may facilitate connection to a particular computing device of the data center 734.

[0110]

[0122] Network 702 represents one or more networks, such as the Internet, connecting cloud environment 730 to a number of client devices (e.g., in some examples, mobile phone 710, tablet computer 712, mobile computing device 714, and desktop computing device 716). Network 702 may further communicate over wireless networks using various mobile network services 720, such as Wi-Fi, Bluetooth, cellular networks (including EDGE, 3G and 4G wireless cellular systems), or any other form of wireless communication known in the art. In some embodiments, network 702 is independent of local interfaces and networks associated with client devices, and allows for integration of local interfaces and networks configured to perform the processes described herein.

[0111]

[0123] Although specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the present disclosure. Indeed, the novel methods, apparatus, and systems described herein may be embodied in a variety of other forms, and various omissions, substitutions, and changes in the form of the methods, apparatus, and systems described herein may be made without departing from the spirit of the present disclosure. The appended claims and their equivalents are intended to cover such forms or modifications that are within the scope and spirit of the present disclosure.

Claims

1. 1. A system for automatically assessing an asset condition, comprising: a non-transitory computer readable medium storing a plurality of machine learning classifiers, each machine learning classifier being trained to identify at least one asset attribute or at least one asset state; a data store including asset information for a plurality of assets and a plurality of images of said plurality of assets; processing circuitry; The processing circuitry includes: accessing one or more aerial images of the asset from the data store; identifying a footprint of a structure on the asset within at least a first aerial image of the one or more aerial images; applying a first one or more machine learning classifiers of the plurality of machine learning classifiers to at least one of the one or more aerial images to determine a plurality of attributes of the structure, the plurality of attributes including a particular roof material of a plurality of roof materials and a particular roof shape of a plurality of roof shapes; applying a second one or more machine learning classifiers of the plurality of machine learning classifiers to at least one of the one or more aerial images to determine a roof condition of a roof of the structure, the second one or more machine learning classifiers being trained to identify one or more types of problems related to the roof as part of determining the roof condition of the structure; and configured to perform operations including providing, to a remote computing device over a network, at least a portion of at least one of the one or more aerial images of the asset, attribute information corresponding to the plurality of attributes, and status information corresponding to the roof condition, wherein whenever the second one or more machine learning classifiers identify at least one of the one or more types of issues with the roof, the status information includes an identification of one or more issues with the roof. system.

2. The system of claim 1 , wherein the actions further comprise comparing the roof condition to a past roof condition.

3. 2. The system of claim 1 , wherein the operations further include applying a third one or more machine learning classifiers of the plurality of machine learning classifiers to at least one of the one or more aerial images to determine a plurality of asset attributes of one or more features in a vicinity of the structure, the one or more features comprising one or more of a swimming pool or a tennis court.

4. The system of claim 3 , wherein determining the plurality of asset attributes comprises determining at least one of a size or a type of the pool.

5. The system of claim 3 , wherein the operations further comprise determining a distance between the structure and each feature of at least one of the one or more features.

6. The system of claim 1 , wherein the one or more types of problem include structural damage to one or more roof features.

7. The system of claim 6 , wherein the one or more roof features include at least one of a roof shingle, a skylight, or a chimney.

8. The system of claim 1 , wherein the operations further include obtaining geographic coordinates of the asset, and wherein at least one of the one or more aerial images is accessed using the geographic coordinates.

9. The system of claim 1 , wherein the actions further include determining a numerical condition rating qualifying a relative condition of the roof, and the condition information includes the numerical condition rating.

10. 2. The system of claim 1, wherein providing the at least a portion of at least one of the one or more aerial images of the asset, the attribute information corresponding to the plurality of attributes, and the status information corresponding to the roof condition comprises presenting a graphical user interface including the at least a portion of the at least one of the one or more aerial images, the attribute information, and the status information for review by an end user on a display of the remote computing device.

11. The system of claim 1 , wherein at least some of the images are orthophoto images.

12. 1. A method for automatically assessing an asset condition, comprising: obtaining geographic coordinates and / or addresses corresponding to the assets; retrieving one or more aerial images of the asset from at least one database; identifying, by a processing circuitry, a building footprint on the property within at least a first aerial image of the one or more aerial images; applying, by the processing circuitry, a first one or more machine learning models to at least one of the one or more aerial images to classify a plurality of characteristics of the building, the plurality of characteristics including a particular roof material among a plurality of roof materials and a particular roof shape among a plurality of roof shapes; applying, by the processing circuitry, a second one or more machine learning models to at least one of the one or more aerial images to determine a roof condition of a roof of the building, the second one or more machine learning models being trained to identify one or more types of problems related to the roof as part of determining the roof condition of the building; providing, in real time in response to obtaining the geographic coordinates and / or the address, to a remote computing device over a network, at least a portion of at least one of the one or more aerial images of the asset, characteristic information corresponding to the plurality of characteristics, and status information corresponding to the roof condition, wherein whenever the second one or more machine learning models identify at least one of the one or more types of problem with the roof, the status information includes an identification of one or more problems with the roof; A method comprising:

13. The method of claim 12 , wherein the geographic coordinates and / or the address are provided by a user of the remote computing device in association with a risk analysis request.

14. The method of claim 13 , further comprising determining an amount of risk exposure of the asset attributable at least in part to the roof condition.

15. The method of claim 14 , wherein determining the risk exposure comprises applying a condition classification to the asset.

16. The method of claim 12 , wherein the at least one database further comprises property information for a plurality of properties including the property.

17. The method of claim 12 , wherein retrieving the one or more aerial images comprises assessing the one or more aerial images from a remote location over a network.

18. 13. The method of claim 12, further comprising applying, by the processing circuitry, a third one or more machine learning models to at least one of the one or more aerial images to determine a plurality of asset properties of one or more additional structures on the asset.

19. The method of claim 18 , wherein the one or more additional structures include at least one of a swimming pool or a tennis court.

20. 13. The method of claim 12, wherein providing the at least a portion of at least one of the one or more aerial images of the asset, the property information corresponding to the plurality of properties, and the status information corresponding to the roof condition comprises presenting a graphical user interface including the at least a portion of the at least one of the one or more aerial images, the property information, and the status information for review by an end user on a display of the remote computing device.

Images