Platform, system, and method for classifying asset characteristics and asset feature maintenance management through aerial image analysis

Abstract

To provide a method and a system for automatically classifying a repair condition of an asset property.SOLUTION: A system operational flow 100 for automatically classifying a repair condition of an asset property, comprises the steps of: acquiring an aerial image of a geographic area including an asset; identifying features of the aerial image corresponding to the asset property; analyzing the features to determine an asset property classification; analyzing an area of the aerial image including the asset property to determine a condition classification; and using the asset property classification and the condition classification to determine a risk estimate value of damage to the asset from one or more disasters and / or a cost estimate value for repairing or replacing the asset property.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on “Platform, Systems, and Methods for Identifying Property Characteristics and Property Feature Maintenance Through This application claims priority to U.S. Provisional Patent Application No. 62 / 398,665, filed September 23, 2016, entitled "Aerial Imagery Analysis," which is hereby incorporated by reference in its entirety. [Background technology]

[0002]

[0002] A risk exposure database includes a compilation of as many building assets or properties as possible that are relevant to insurance. These assets may include properties such as location coordinates, address, slope, and elevation. Other properties include building structure type, occupancy type, year built and / or remodeled, building height, soft stories, number of floors, and floor area. Further properties may include roof condition, roof shape, roof covering, roof anchors, roof equipment, cladding, and pounding (distance to adjacent buildings). Some of these properties can be assessed only by on-site inspection or official documents, while others can be measured using visual images.

[0003]

[0003] The characteristics addressed in this disclosure include roof shapes and roof conditions. In one embodiment, roof shapes are divided into five categories: gambrel roof, gable roof, hip roof, square roof, and flat roof. Each roof shape has its own unique response and vulnerability to damage from various hazardous natural phenomena, such as earthquakes or wind.

[0004]

[0004] Deep learning involves computational models consisting of multiple processing layers for learning representations of data with multiple levels of abstraction. These models can be thought of as a way to automate predictive analytics. Representation learning is a set of methods that allows a machine to be fed raw data and automatically find the representations needed for detection or classification. Deep learning applications include speech recognition, motion detection, translation, and medical diagnosis. Using deep learning algorithms and sample datasets, computers can distinguish and classify a wide range of characteristics with a high level of accuracy, often exceeding human recognition.

[0005]

[0005] One model used in deep learning is the "Network In Network" model described in the paper "Network In Network" by M. Lin et al., published in the International Conference on Learning Representations (arXiv:1409.1556) in 2014, the contents of which are incorporated herein by reference in their entirety. Using the Network In Network model, multiple layers of artificial perception results are generated using micro-neural networks with complex structures. The artificial perception results are then stacked and averaged to generate a single global average pooling layer for classification.

[0006] When applied to visual recognition, deep learning algorithms can classify observations (e.g., images) in a number of different ways to characterize the features of the observations. In some embodiments, deep learning algorithms can explore images as a set of intensity values ​​per pixel, or in a more abstract way, as a set of edges or regions of a particular shape. Some representations may perform better than others based on specific learning tasks. One of the possibilities of deep learning is to replace human identification of features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction.

[0007]

[0007] The inventors have realized that by applying deep learning methods to a population of risk exposure databases to analyze aerial imagery and automatically extract characteristics of individual assets, it is possible to perform fast and efficient automatic classification of building types and state of repair. When location-based vulnerability is combined with individual asset vulnerabilities identified in part by classification of the state of repair of one or more asset features, the risk of damage from a disaster can be more accurately estimated. Summary of the Invention

[0008]

[0008] The foregoing summary of exemplary embodiments and the following detailed description thereof are merely examples of aspects of the teachings of the present disclosure and are not intended to be limiting.

[0009] The systems, methods, and computing system platforms described herein assist in matching aerial imagery features of one or more assets with corresponding asset conditions (e.g., maintenance levels of the asset features) through machine learning analysis. In a preferred application, the asset condition analysis can be used to estimate risk of damage given one or more disaster conditions, such as a severe storm. The analysis can further, in one example, assist in estimating the cost of repairing or replacing each asset if a disaster causes the estimated damage. In another example, the analysis can be used to verify that the asset has been repaired.

[0010] In one aspect, the present disclosure relates to a method for automatically classifying the condition of repair of an asset property, the method including acquiring aerial imagery of a geographic area including the asset, identifying features of the aerial imagery that correspond to the asset property, analyzing the features to determine an asset property classification, analyzing the area of ​​the aerial imagery that includes the asset property to determine a condition classification, and using the asset property classification and condition classification to determine a risk estimate of damage to the asset from one or more hazards. The asset property classification may be a shape of a roof top surface. Analyzing the features to determine the asset property may include applying a deep learning analytics model to the features. The deep learning analytics model may be a NIN.

[0011] In some embodiments, analyzing a region of the aerial image including the asset characteristic to determine a condition classification includes applying a machine learning analytical model to image pixels in the region. The machine learning analytical model may include a color histogram analytical model. The condition classification may include a good and bad classification. Determining a risk estimate includes applying a hazard risk profile corresponding to a first hazard and the asset characteristic of at least one hazard.

[0012] In one aspect, the present disclosure relates to a system for automatically classifying the condition of repair of an asset property, the system including a processing circuit and a non-transitory computer-readable medium having instructions stored thereon. In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to acquire an aerial image of a geographic area including the asset, identify features of the aerial image that correspond to the asset property, analyze the features to determine an asset property classification, analyze the area of ​​the aerial image that includes the asset property to determine a condition classification, and determine a replacement cost for replacing the asset property using the asset property classification and the condition classification. The aerial image may be a two-dimensional aerial image.

[0013] In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to acquire a shape map image including the asset before identifying the feature, overlay the aerial image with the shape map image, and determine whether the boundary of the asset identified by the shape map matches the boundary of the asset shown in the aerial image. The instructions, when executed by the processing circuit, may cause the processing circuit to acquire an alternative aerial image of the asset when it determines that the shape map does not match the boundary of the corresponding asset. The instructions, when executed by the processing circuit, may cause the processing circuit to assess the orthogonality of the aerial image before identifying the feature. The asset may be a single-family home.

[0014] In one aspect, the present disclosure relates to a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processing circuit, cause the processing circuit to receive an identification of an asset and at least one asset characteristic, acquire an aerial image of a geographic region including the asset, and identify individual features of the aerial image corresponding to each of the at least one asset characteristic. In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to analyze, for each asset characteristic, the corresponding feature to determine an individual asset characteristic classification, and analyze a region of the aerial image including the individual asset characteristic to determine an individual condition classification. The instructions, when executed by the processing circuit, may cause the processing circuit to determine at least one risk estimate indicative of a risk of damage from a disaster using the asset characteristic classification of each asset characteristic and the condition classification of each asset characteristic.

[0015] In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to, prior to acquiring an aerial image of the geographic region, determine, based on the at least one asset characteristic, a preferred image type corresponding to each asset characteristic of the at least one asset characteristic. The at least one asset characteristic may include two or more asset characteristics. The instructions, when executed by the processing circuit, may cause the processing circuit to acquire a ground image of the geographic region including the asset in response to determining that the preferred image type corresponding to a first asset characteristic of the at least one asset characteristic is a ground image.

[0016] In some embodiments, the instructions, when executed by the processing circuit, cause the processing circuit to access one or more known asset characteristics. Determining the at least one risk estimate may further include determining the at least one risk estimate based on the one or more known asset characteristics. The one or more known asset characteristics may include at least one of asset age, asset elevation, asset slope, year built, year remodeled, and building height.

[0017] In some embodiments, receiving an identification of the asset and at least one asset characteristic includes receiving the identification of the asset over a network from a remote computing device. The instructions, when executed by the processing circuit, may cause the processing circuit to provide the at least one risk estimate over the network to the remote computing device in real time in response to receiving the at least one asset characteristic.

[0018]

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate one or more embodiments and, together with the description, explain these embodiments. The accompanying drawings are not necessarily drawn to scale. Any values ​​and dimensions shown in the accompanying graphs and figures are shown for illustrative purposes only and may or may not represent actual or preferred values ​​or dimensions. Where appropriate, some or all features may not be shown to help explain essential features. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a flow diagram of an example of a system operational flow for matching aerial image features with corresponding asset maintenance management levels. [Figure 2A]

[0020] FIG. 1 shows an example of trait classification and corresponding risk profile data. [Figure 2B]

[0021] FIG. 10 illustrates example state characteristics and corresponding example histograms. [Figure 2C]

[0022] FIG. 10 illustrates an example condition profile corresponding to an example asset characteristic type. [Figure 2D]

[0023] 1 is an example graph of error rate when comparing training data with test data when matching aerial image features with corresponding rooftop shapes. [Figure 3]

[0024] FIG. 1 is a block diagram of an example environment for matching topographical features of aerial imagery with corresponding asset maintenance levels. [Figure 4]

[0025] 1 is a flowchart illustrating an example method for classifying condition characteristics of an asset based on aerial image analysis. [Figure 5A]

[0026] FIG. 1 illustrates an example of a structure shape map of a geographic region. [Figure 5B]

[0027] FIG. 5B illustrates an example of a two-dimensional aerial image of the geographic area of ​​FIG. 5A. [Figure 5C]

[0028] 5B shows an example of an overlay map image in which a section of the structure geometry map of FIG. 5A is overlaid on a corresponding section of the aerial image of FIG. 5B. [Figure 6]

[0029] FIG. 1 is a block diagram of an example computing system. [Figure 7]

[0030] FIG. 1 is a block diagram of an example of a distributed computing environment, including a cloud computing environment. [Figure 8A]

[0031] 10A-10C illustrate a series of example screenshots of a user interface for reviewing asset characteristics and maintenance level information. [Figure 8B]

[0031] FIG. 10 is a series of example screenshots of a user interface for reviewing asset characteristics and maintenance management level information. [Figure 8C]

[0031] FIG. 10 is a series of example screenshots of a user interface for reviewing asset characteristics and maintenance management level information. DETAILED DESCRIPTION OF THE INVENTION

[0020]

[0032] The following description of the accompanying drawings is intended to describe various exemplary embodiments of the disclosed subject matter. Although specific features and functions are described in connection with each exemplary embodiment, it will be apparent to those skilled in the art that the disclosed embodiments may be practiced without each of those specific features and functions.

[0021]

[0033] Throughout this specification, the phrase "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic illustrated in connection with one embodiment is included in at least one embodiment of the disclosed subject matter. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Furthermore, embodiments of the disclosed subject matter are intended to cover such modifications and variations.

[0022]

[0034] It should be noted that, as used herein, singular forms (“a,” “an,” “the”) include plural referents unless the context clearly indicates otherwise. That is, unless otherwise specified, as used herein, singular forms (“a,” “an,” “the,” etc.) have the meaning “one or more.” Furthermore, it should be understood that terms such as “left,” “right,” “upper,” “lower,” “front,” “rear,” “side,” “height,” “length,” “width,” “upper,” “lower,” “inner,” “external,” “inside,” “outside,” etc., which may be used herein, merely indicate points of reference and do not necessarily limit embodiments of the present disclosure to any particular orientation or configuration. Also, terms such as “first,” “second,” “third,” etc., merely identify one of the numerous parts, components, steps, operations, functions, and / or points of reference disclosed herein and, similarly, do not necessarily limit embodiments of the present disclosure to any particular configuration or orientation.

[0023]

[0035] Additionally, the terms "approximately," "about," "approximately," "slightly different," and similar terms generally refer to ranges that include a particular value within a 20%, 10%, or preferably 5% difference in certain embodiments, and any value therebetween.

[0024]

[0036] All features described in connection with one embodiment are intended to be applicable to the additional embodiments described below, unless expressly indicated otherwise, or unless the feature or function is incompatible with the additional embodiments. For example, if a given feature or function is expressly described in connection with one embodiment but not expressly stated in connection with an alternative embodiment, it should be understood that the inventors intend that the feature or function can be developed, utilized, or implemented in connection with the alternative embodiment, unless the feature or function is incompatible with the alternative embodiment.

[0025]

[0037] Referring to FIG. 1 , a flow diagram illustrates an example of a system operational flow 100 for matching aerial image features with corresponding asset maintenance management levels. The system acquires 104 an image of an asset location 102b. In some implementations, initiation of at least a portion of the operational flow 100 by the system may be triggered by an asset classification request submitted by a user, such as an insurance company or real estate investment broker, on a dashboard interface screen on a computing device connected to the system via a network. For example, as shown in FIG. 8A , a specific asset may be selected within a map view user interface 800, or alternatively, a neighborhood of assets. The operational flow 100 may be used to analyze the condition of each asset feature at each asset location 102b based on the aerial imagery 102c. In some implementations, the operational flow 100 may assist in estimating, for example, the cost of repair or replacement of each asset feature detected in the aerial imagery. In another implementation, the operational flow 100 may be used to verify that the asset location 102b has been repaired.

[0026]

[0038] Asset location 102b, in some examples, may represent the location of an asset insured by a particular insurance carrier, the location of an asset recently affected by a disaster such as a tornado, hurricane, earthquake, storm surge, or volcanic eruption, the location of an asset in an investment portfolio, or the location of an asset being considered for addition to an investment portfolio. Asset location 102b, in some examples, may be identified by a street address, global positioning system coordinates, or other geographic coordinates.

[0027]

[0039] In some implementations, the operational flow 100 obtains (104) an image of each asset location 102b. The image may include aerial imagery 102c stored by the system or accessed by the system from a remote networked location. The aerial imagery 102c may include, for example, a three-dimensional or two-dimensional image of a geographic location that includes at least one of the asset locations 102b. In one example, the system may query a remote database to obtain the latest aerial imagery 102c that includes an image of at least one asset location 102b based on a street address or geographic coordinates. The remote database, in some implementations, may include Google® Earth imagery from Google, Inc. (Mountain View, California) or a database from a private company such as NTT Spatial Information Corporation (Japan). In other implementations, the remote database may include one or more databases from a public agency such as the Geospatial Information Authority of Japan (GSI), the United States Geological Survey, or the German Federal Office for Mapping and Geodesy. The aerial imagery, in some embodiments, may be provided by a software such as QGIS by the Open Source Geospatial Foundation (OSGeo). The imagery may be collected from one or more remote network locations using an open source geographic information system (GIS). The format of the imagery of the asset locations 102b, in some embodiments, depends on the format received by the various sources available for the aerial imagery 102c. An example of an aerial imagery 510 of multiple asset locations 102b is shown in FIG. 5B. It has been done.

[0028]

[0040] In some embodiments, the system may be configured to query multiple remote database systems to obtain at least two aerial images of a given asset location 102b. The aerial images available in the various databases may vary, for example, in terms of resolution and recency of capture. By collecting more than one image of a particular asset, for example, the system may analyze each image to determine the highest quality image to use in a condition analysis. The condition analysis, in some examples, may include balancing multiple factors, such as clarity, completeness, and recency.

[0029]

[0041] Additionally, in some embodiments, the system may be configured to query multiple remote database systems to obtain both two-dimensional and three-dimensional aerial imagery. Different asset characteristics may be determined based on whether the aerial imagery was captured in two-dimensional or three-dimensional format. Two-dimensional aerial imagery, in some examples, may be used to determine the asset's location coordinates, street name, occupancy type, floor area, presence of skylights, presence of chimneys, roof condition, roof shape, roof covering, roof anchors, roof equipment, and / or pounding. Three-dimensional aerial imagery, when compared to two-dimensional aerial imagery, may be used to determine the location coordinates, street name, building structure type, occupancy type, year built, building height, soft stories, number of stories, roof condition, roof shape, roof covering, roof anchors, roof equipment, cladding, and pounding. In some embodiments, if there is overlap in characteristics identifiable using either two-dimensional or three-dimensional images, machine learning analysis of both images may be combined to increase confidence in the identification of individual characteristics.

[0030]

[0042] In some embodiments, the system obtains (104) a geometric map image 102a for each of the asset locations 102b. The geometric map image 102a includes a block-shaped layout of the existing assets within a city, town, or village, such as a city plan used for urban planning and development. In another embodiment, the geometric map image 102a may include a block-shaped layout map used in providing information to a user of a computer-based navigation system. In one embodiment, the geometric map image 102a may be obtained from the Geospatial Information Authority of Japan or Zenrin Corporation of Japan. An example of a city plan 500 is shown in FIG. 5A (image obtained from the Geospatial Information Authority of Japan, Digital Map 2500 (Spatial Data Framework)).

[0031]

[0043] The shape map image 102a, in some embodiments, is used to locate a particular asset. The shape map image 102a of a geographic region may be overlaid with a corresponding aerial image 102c to, for example, match assets to the image. An example of an image 520 in which the city plan image 500 of FIG. 5A is overlaid with the aerial image 510 of FIG. 5B is shown in FIG. 5C. In some embodiments, the shape map image 102a may not match the aerial image 102c in shape or position. As shown in FIG. 5C, for example, the shape map outline (shown in bold) of location 522 may not be rectangular, but the shape of the rooftops below the outline may be rectangular. In some embodiments, analyzing the multiple aerial images 102c to determine the highest quality image may include identifying one or more aerial images 102c that substantially match the corresponding shape map image 102a. The overlay of the aerial image 102c and the shape map image 102a can also be used to assist in cropping the aerial image 102c to focus the analysis on a particular asset location 102b. For example, referring to FIG. 5C , an asset location 524 is substantially matched to a corresponding city plan image, and the overall outline can be used to assist in cropping the asset location 524. In some examples, an asset location 524 can be identified in the overlaid image 520 if it matches within a predetermined tolerance with an asset in the overlaid image 520 to account for modifications made to the asset 524, such as an asset addition, that may change the overall shape of the asset.

[0032]

[0044] Returning to Figure 1, in some embodiments, the shape map image 102a may be used to correct or compensate for alignment errors or mismatches in the corresponding two-dimensional aerial image 102c. For example, an aerial image showing a standard orthophoto angle may not be directly based on a city plan block. Using the city plan block as a guide, for example, the aerial image may be geometrically corrected to obtain a true orthophoto version of the aerial image.

[0033]

[0045] As shown in FIG. 1 , in some implementations, the system identifies features of each aerial image 102c to classify asset properties (106). Using machine learning for analysis, for example, the system can extract aerial image features of a particular asset location 102b. Groupings of extracted features, such as angles, contours, substantially homogeneous areas, etc., can be used to identify asset features such as roof tops, swimming pools, chimneys, and skylights. Upon extracting image-related features identifiable as asset features (e.g., roof tops), the extracted features can be analyzed to determine one or more asset characteristics of the features (e.g., roof top type). In one example, a swimming pool can be identified as a pool asset feature (e.g., various shapes of contours of a particular size or larger that bound a substantially homogeneous blue area), which can then be characterized as a particular shape (e.g., rectangular, circular, or kidney-shaped), type (e.g., above or below ground), and / or size (e.g., approximate area). The machine learning classifier used in the machine learning analysis, in some embodiments, includes a convolutional neural network (CNN) to preprocess aerial images 102c of specific asset locations 102b and to classify asset features as asset characteristics 110. Using CNN technology, for example, the biological process of visually inspecting and identifying objects or object features is mimicked by a feedforward artificial neural network, where multi-layer artificial perception minimizes preprocessing. CNN is a well-established and popular machine learning method. An example of a CNN processing model is Alexnet. (See "ImageNet Classification with Deep Convolutional Neural Networks," Krizhevksy et al., Advances in neural information processing systems, 2012, the entire contents of which are incorporated herein by reference.) In other embodiments, the machine learning classifier may include a network-in-network (NIN), in which multiple layers of artificial perception results are generated using micro-neural networks with complex structures.The artificial perception results are then stacked and averaged to produce a single global average pooling layer for classification. NIN has shown better performance results than traditional CNN processing. Furthermore, NIN processing is less storage intensive than CNN processing.

[0034]

[0046] In some implementations, asset property profiles 108 are used to classify asset features. Asset property profiles 108 may be developed, for example, by training a machine learning algorithm using aerial images (and in some embodiments, terrestrial images 102d) of known asset properties. Each asset feature may be divided into multiple classes, for example. In an example involving terrestrial image classification, cladding may include stone, brick, stucco, shingles, vertical shingles, horizontal shingles, or metal. The machine learning algorithm may, for example, generate a new aerial image 102c containing a gambrel roof top and a property profile 108 for the gambrel roof top. A confidence percentage may be generated for the match.

[0035]

[0047] Referring to FIG. 2A , in the illustrated example, extracted angular and contour features of an aerial image 204 may be used to classify the roof shape 202 of a particular asset location 102b. In a first example, a gable roof 202b includes a central peak within a generally rectangular contour, as shown in the corresponding aerial image 204b. A gambrel roof 202a includes these characteristics of a gable roof 202b, as shown in the aerial image 204a, but the gambrel roof also includes a "skirt" region surrounding the central gable. The "skirt" region may be identified, in part, based on corner edges extending from the four corners of the gable. A hip roof 202c includes a central peak and two opposing triangular ends extending from either end of the central peak, as shown in the aerial image 204c. The central peak forms the top of two trapezoidal sections. In contrast, a rectangular roof 202d does not have a central peak. Instead, as shown in aerial image 204d, a rectangular roof includes four triangular sections that meet at a central point. Finally, a flat roof 202e is represented by a substantially flat surface with a rectangular outline, as shown in aerial image 204e.

[0036]

[0048] Referring to FIG. 2D, an example error rate graph 240 compares the accuracy of matching training data 242 with matching test data 244 in matching roof top features in aerial images to corresponding roof top types (e.g., shapes). Tests were conducted using the NIN deep learning algorithm to classify roof top shapes in 2D aerial images, including gambrel, gable, hip, square, and flat roof shapes in various colors. Further testing was conducted across different image sizes and mini-batch sizes. In particular, the images were generated using the formula

number

[0037]

[0049] In some embodiments, to improve accuracy, the image may be cropped and / or resized before analysis. For example, the image may be cropped to include the asset of interest or a portion of the asset and its surroundings (e.g., a plot that includes the asset and / or a portion of the neighborhood proximate to where the asset is located). In another example, resizing to a standard image size may help improve accuracy.

[0038]

[0050] In some embodiments, in addition to roof shape, in some examples, feature analysis may be used to determine additional roof features such as roof coverings, roof anchors, roof fixtures, skylights, observation decks, turrets, towers, dormers, and / or chimneys. Furthermore, upon identifying the roof contours, the footprint (e.g., roof size) of the asset location 102b may be calculated based on the scale ratio of the aerial image 102c.

[0039]

[0051] Returning to Figure 1, in some embodiments, rather than analyzing only specific asset locations 102b individually, specific asset locations 102b are each classified based in part on adjacent structures. For example, based on the dimensions of the asset location 102b, which may include lot dimensions, machine learning can be used to identify additional structures at the asset location 102b, such as a deck, swimming pool, shed, gazebo, detached garage, tennis court, fence, retaining wall, dock, playground equipment, or greenhouse, in some embodiments. Additionally, specific asset locations 102b may be classified in part based on their distance (pounding) from adjacent structures.

[0040]

[0052] In some implementations, the terrestrial imagery 102d can be used to classify features that are difficult to recognize using the aerial imagery 102c. The terrestrial imagery 102d can, in some examples, include street view imagery obtained from a street view service or real estate listings. In some examples, street view imagery including a view looking down the street at the asset location 102b can be obtained from Google® Street View by Google, Inc., Bing® Maps Streetside by Microsoft Corporation (Redmond, Washington), or Mapillary by Mapillary AB (Sweden). For example, using the terrestrial imagery, the system can identify features such as building structure type, cladding, building height, number of soft stories, number of floors, location coordinates, street name, slope, elevation, year built, and / or occupancy type. If characteristics identifiable using the terrestrial imagery 102d overlap with characteristics identifiable using the two-dimensional or three-dimensional aerial imagery 102c, in some embodiments, analysis of the terrestrial imagery 102d can be combined with analysis of the aerial imagery to increase confidence in the identification of a particular characteristic or characteristics. For example, house siding features may be more easily detected in the terrestrial image 102d and / or the three-dimensional aerial image 120c than in the two-dimensional aerial image 102c.

[0041]

[0053] In some embodiments, the condition of each asset feature may be classified 112 as a corresponding condition characteristic 116. While a new asset may be in good condition, asset feature condition may deteriorate over time due to normal wear and tear on the asset. Asset features may also be damaged by external forces such as storms and natural disasters. Eventually, the condition of a residential feature may deteriorate to the point that repair and / or replacement may be necessary. As with the asset features described above in block 106, machine learning algorithms may be used to classify the current condition of each detected asset feature. Using machine learning for analysis, for example, the system may extract pixel intensity distributions of previously identified asset features in aerial images of a particular asset location 102b. In some examples, features of newly constructed assets generally have sharp, sharp contrast in machine learning image analysis. On the other hand, features of weathered or damaged assets may have smooth edges, soft contrast, and asymmetric wear in places. The machine learning classifier used in the machine learning condition analysis, in some embodiments, includes machine learning analysis for processing aerial imagery 102c of a particular asset location 102b and for classifying the condition of previously identified asset characteristics 110 as condition characteristics 116. The machine learning analysis, in some examples, may include two-dimensional color histogram analysis or three-dimensional color histogram analysis. In other embodiments, the machine learning analysis may be performed using pattern recognition algorithms (e.g., determining missing fence posts or missing / misaligned roof shingles). In other embodiments, the machine learning classifier includes deep learning analysis, such as CNN or NIN.

[0042]

[0054] In an illustrative example, referring to FIG. 2B, an example of aerial imagery of rooftops for hipped roofs, showing condition characteristics and corresponding example histograms generated by machine learning analysis of pixel intensities, is shown. For example, a set of good-condition aerial images 210a-214a is shown along with corresponding good-condition histograms 210b-214b. In the illustrated embodiment, good-condition histograms 210b-214b show an index of probability of grayscale color distribution, with individual maxima 210c-214c indicating the highest probability of a good-condition roof grayscale color distribution. For example, values ​​can range from 0 (completely black) to 255 (completely white).

[0043]

[0055] Meanwhile, a set of poor condition aerial images 216a-220a is shown along with corresponding poor condition histograms 216b-220b. In the illustrated embodiment, the poor condition histograms 216b-220b represent a measure of the probability of a grayscale color distribution, with each maxima 216c-220c representing the highest probability of a grayscale color distribution for a rooftop in good condition. The maxima 216c-220c corresponding to the poor condition rooftop surfaces 216a-220a are significantly lower than the corresponding maxima 210c-214c for the good condition rooftop surfaces 210a-214a. Furthermore, the distribution widths of the poor condition histograms 216b-220b may be significantly wider than the distribution widths of the good condition histograms 210b-214b. As can be seen from the aerial images of roof tops 216a-220a in poor condition, the color distribution is sparse and washed out, and the roof tops lack sharp edge lines, whereas when viewing aerial images 210a-214a of roof tops in good condition, the roof tops are substantially more uniform in color and have sharp edge lines.

[0044]

[0056] Referring to FIG. 2C , a condition profile 230 of the condition of a rooftop is shown. In some implementations, pixel value distribution histograms, such as histograms 210b-220b, can be combined to create an example condition profile 230 corresponding to the condition of a rooftop. Condition profile 230 can be included, for example, as one of condition profiles 114 of FIG. 1 . Reference bars 232a and 232b on condition profile 230 indicate values ​​corresponding to unaffected hip roof 214a and deteriorated hip roof 216a, respectively. In some implementations, values ​​closer to 1 on the y-axis indicate better roof condition, while values ​​closer to 0 indicate a rooftop in poor condition. In one implementation, reference bar 232 on example condition profile 230 corresponds to a test condition sample dataset containing 2,500 images. In some implementations, reference bars 232 are combined using logistic regression to create roof condition profile graph 236. The accuracy rate of the image histograms is 80% based on several tests.

[0045]

[0057] Although illustrated as one condition profile 230 corresponding to different types of roof tops (e.g., at least hip roof top 216a and gable roof top 234), in other embodiments, individual condition profiles 114 may be created for each property characteristic (e.g., a gable roof-specific profile, a gambrel roof-specific profile, a flat roof-specific profile, a hip roof-specific profile, a square roof-specific profile, etc.). In a further refinement, in certain embodiments, individual condition profiles 114 are created for combinations of property characteristics (e.g., gable shingle roof, gable tile roof, gable metal roof, etc.) to improve accuracy based on pixel density corresponding to the combined characteristics. Many combinations of property characteristic characteristics may be used alone or in combination to create condition profiles 114 designed to accurately identify the condition of the corresponding property characteristics.

[0046]

[0058] Returning to FIG. 1 , in some implementations, asset characteristics 110 and / or condition characteristics 116 may be analyzed to calculate 120 a risk estimate 122 corresponding to the risk of damage based on one or more types of hazards. The types of hazards may include, in some examples, earthquakes, hurricanes, tornadoes, storm surges, straight-line winds, or volcanic eruptions. In some embodiments, the type and estimated severity of the hazard may be based on the particular asset location 102b. For example, assets near active faults may be more susceptible to earthquakes, while coastal assets may be more susceptible to storm surge damage. The risk estimate 122 may, in some examples, include an actuarial percentage indicating the likelihood of damage based on particular asset characteristics. For example, different asset characteristics may be more susceptible to certain types of damage than others. For example, flat roof tops may be more susceptible to damage from excessive snow accumulation, while roof tops with eaves may be more susceptible to high winds.

[0047]

[0059] In some embodiments, one or more disaster risk profiles 118 may be applied based on asset characteristics when calculating damage risk. Vulnerability to disaster damage may vary by location, elevation, slope, roof top shape, roof top condition, cladding type, cladding condition, and / or pounding, in some examples. In particular examples, as shown in FIG. 2A , a graphical risk profile 206 ranks roof top types by vulnerability to strong winds. Vulnerability to wind damage increases from lowest vulnerability (flat roof top 202e) to square roof 202d, hip roof 202c, gable roof 202b, and gambrel roof 202a, with highest vulnerability to wind damage. Vulnerability may be mathematically modeled based on statistical information constructed from past damage results from various types and / or severities of disasters. Combining location-based vulnerability with vulnerability based on individual asset characteristics may more accurately estimate the risk of disaster damage.

[0048]

[0060] Returning to FIG. 1 , risk estimate 122 may, in some embodiments, be used in insurance underwriting. For example, risk estimate 122 may be used to determine whether an asset is adequately insured against potential damage. Additionally, when tracking upcoming natural disasters, insurance companies use risk estimate 122 to estimate insurance benefits based on the storm's expected path and severity. Additionally, risk estimate 122 may be utilized by investors when selecting investment assets for their investment portfolios.

[0049]

[0061] In some embodiments, repair or replacement costs may be calculated 126 to determine a cost estimate 128 for placing one or more asset properties in good condition or “like new.” The cost estimate 128, in some embodiments, may be based in part on one or more replacement cost profiles 120. The replacement cost may vary in some examples based on the type of material (e.g., roof top material, cladding material, etc.), the scale of the work (e.g., estimated roof area, estimated length of fence, etc.), and / or additional asset features (e.g., contributing to the complexity of the job). The replacement profile 120, in another example, may be based in part on location (e.g., different areas have different material and labor costs). In some embodiments, the cost estimate 128 for repairing or replacing an asset property may be output to a dashboard interface screen of a remote computing device of a claimant, such as an insurance carrier or real estate investment broker.

[0050]

[0062] In some embodiments, cost estimates 128 may be used to automatically analyze insurance claims. Cost estimates 128 may also be combined with risk estimates 122 when analyzing the adequacy of insurance coverage for a particular asset location 102b. Cost estimates 128 may also be used by investors in determining the cost of maintaining / upgrading a particular investment asset.

[0051]

[0063] While the operational flow is depicted as a series of computational steps, other implementations may include more or fewer computational steps. For example, aerial imagery 102c may be analyzed after a claim has been paid to verify that the insured property has been repaired and / or to upgrade the property's estimated value (e.g., if the owner upgrades the property rather than directly repairing / replacing the damaged property features).

[0052]

[0064] Additionally, in other implementations, certain computational steps may be performed in a different order. For example, cost estimates 128 may be calculated before risk estimates 122. Other modifications of operational flow 100 are also possible.

[0053]

[0065] 3 is a block diagram of an example environment 300 for matching topographic features in aerial imagery with corresponding asset maintenance management levels. The example environment 300 may be used, for example, to implement the operational flow 100 described with respect to FIG. 1. The example environment 300 includes an asset characterization / condition analysis system 302 in communication with client computing system(s) 306, multiple cost and / or risk data source computing system(s) 308, and multiple geometry map, aerial imagery, and / or terrestrial imagery source computing system(s) 304. The communication connections may be wired or wireless connections over various network types. Generally, the system 302 obtains asset identifier(s) 340 from the client computing system(s) 306, accesses images of the assets identified by the asset identifiers 340 from the image source computing system(s) 304, and analyzes the images to determine asset characteristic(s) 342 and condition characteristic(s) 344. Using the cost and / or risk data obtained from the cost / risk data source computing system(s) 308, the system 302 can determine replacement cost estimate data 350 and / or disaster risk estimate data 352 based on the asset characteristics and asset condition.

[0054]

[0066] In some implementations, a user of a particular client computing system 306 submits a request to the system 302 via a graphical user interface provided by the graphical user interface engine 334. The request may include, for example, identification of at least one asset identifier 340 and one or more asset characteristics 342, and / or a replacement or repair cost estimate, a disaster risk estimate, and a confirmation of repair assessment. The asset identifier 340, in some examples, may include location information (e.g., an address, geolocation coordinates, parcel boundaries, etc.). The location information may match, for example, the asset location information 102b described with respect to FIG. 1. In other embodiments, the asset identifier 340 may be a unique asset identifier (e.g., a number or string of characters) used to look up asset location information in the data store 310. In another embodiment, rather than submitting an asset identifier, a user may submit a unique user identifier used to look up one or more assets associated with the user (e.g., assets insured by an insurance client, assets held by an investment client, etc.).

[0055]

[0067] In one example, referring to FIG. 8A , a user can identify the location of one or more assets by selecting the assets within a graphical map interface 800. Many assets are identified within the map view user interface 800, as indicated by associated pins, such as pin 804. The pins may, in some examples, indicate assets insured by a particular insurance carrier, asset types (e.g., commercial, industrial, residential, etc.), and / or assets within a range of values ​​(e.g., within a range, above a threshold, etc.). The pins may be displayed based on filtering options, for example, provided by the user or otherwise associated with a user profile (e.g., insurance carrier identification). In particular examples, a user can select three assets associated with three callouts 802. As shown, each callout identifies the replacement cost associated with the asset and invites the user to use a details control to drill down for more details about the selected asset.

[0056]

[0068] In some implementations, a user enters asset characteristics associated with each selected asset. In other implementations, the characteristics may be accessed upon asset identification (e.g., selection of a pin identifying the asset, as shown in FIG. 8A). For example, asset characteristics 342 may be retrieved from data store 310, as shown in FIG. 3. Referring to FIG. 8B, an example asset characteristic user interface 820 presents building area characteristics 822 (e.g., building area 822a, number of stories 822b, and building floor area 824c) and structural characteristics 824 (e.g., construction material 824a, building construction class 824b). As described, different or additional characteristics may vary from implementation to implementation. In some embodiments, a user can edit building area characteristics 822 and / or structural characteristics 824 via user interface 820, for example, using edit controls 828a or 828b. For example, in the context of recent development or renovation (e.g., additions), a user may modify previously stored characteristic values. For example, the asset characteristics user interface 820 may be presented to a user upon selection of a details control of one of the callouts 802 shown in the user interface of FIG. 8A.

[0057]

[0069] In some implementations, user interface 820 further presents premium characteristics 826, such as replacement costs. Premium information may relate, for example, to an insurance policy purchased for the property or parameters for insuring the property (e.g., based on upgraded or new characteristics). A user, in some embodiments, can update the premium information using edit controls 828c.

[0058]

[0070] In some implementations, when the system 302 identifies the asset location information, it accesses images of the asset from the image source computing system(s) 304 using the image acquisition engine 336. The images, in some examples, may include a geometric map (e.g., the geometric map 102a described with reference to FIG. 1), a two-dimensional aerial image (e.g., the aerial image 102c described with reference to FIG. 1), a three-dimensional aerial image (e.g., the aerial image 102c described with reference to FIG. 1), and / or a terrestrial image (e.g., the terrestrial image 102d described with reference to FIG. 1). In some embodiments, the images acquired by the image acquisition engine 336 are based on the asset characteristic(s) identified by the user. For example, certain types of images are preferred for analyzing certain types of asset characteristics. For example, house siding is more easily recognized from a terrestrial image or a three-dimensional aerial image than from a two-dimensional aerial image.

[0059]

[0071] In some implementations, the image quality analysis / preprocessing engine 326 analyzes the acquired image(s) to verify that they contain sufficiently clear and detailed images of the asset identified by the asset identifier 340. In some embodiments, the image quality analysis / preprocessing engine 326 crops and / or resizes the acquired image(s). For example, the image quality analysis / preprocessing engine 326 may extract a portion of the acquired image based on the asset (e.g., building size). In some implementations, the image quality analysis / preprocessing engine 326 resizes each acquired image to a standard size. In particular examples, the standard size may be 256x256 pixels per image. Resizing to a standard size may, for example, improve the accuracy of subsequent classification analysis. In some embodiments, the image quality analysis / preprocessing engine 326 may be configured to apply corrections to the acquired image(s). For example, the image quality analysis / pre-processing engine 326 may be configured to adjust the aerial imagery from standard orthophoto angles to true orthophoto versions as described above.

[0060]

[0072] In some implementations, the image quality analysis / preprocessing engine 326 uses shape contours as part of its analysis. For example, the composite image generation engine 318 may overlay an aerial image with a shape map image, as described with respect to FIG. 5C . The image quality analysis / preprocessing engine 326 may use the generated composite image to verify that the aerial image matches a city plan or other asset location guide (e.g., navigation map data, etc.). Additionally, the image quality analysis / preprocessing engine 326 may use the overlaid shape map as a guide in determining whether the aerial image needs to be corrected to a true orthophoto angle and / or how much to crop the image to capture the asset (and, optionally, the asset's immediate surroundings, such as part of the asset's parcel or neighborhood).

[0061]

[0073] In some implementations, if the image quality analysis / pre-processing engine 326 determines that the acquired image is insufficient, the image quality analysis / pre-processing engine 326 may request a replacement image from the image acquisition engine 336. For example, the image acquisition engine 336 may, in some examples, acquire images based on various factors including recency of capture, resolution, cost, and / or applicability to a particular asset characteristic analysis. If the image quality analysis / pre-processing engine 326 determines that a first acquired image is insufficient, for example, the image acquisition engine 336 may determine the next best source for acquiring an image of the asset.

[0062]

[0074] In some implementations, once the images have been approved (and optionally preprocessed and / or corrected) by the image quality analysis / preprocessing engine 326, the feature identification engine 320 extracts features from the asset image(s) that are associated with the identified asset characteristics 342. For example, in the case of a fence, the feature identification engine 320 may identify a perimeter or partial perimeter adjacent to and extending from the asset as indicative of a fence.

[0063]

[0075] In some implementations, once features are identified by feature identification engine 320, asset characteristic classification engine 322 classifies the asset characteristics. For example, the features may be classified using one or more machine learning techniques, as described with respect to computational processing stage 106 of FIG. 1 (classifying the features of each image). Asset characteristic classification engine 322 may generate asset characteristics 342 that represent, for example, a grouping, type, or other characterization of particular asset features. Subsequently, in the particular example of a fence, the extracted features may be analyzed and classified with respect to height and / or material. Once the asset characteristics are classified, the asset characteristic classifications may be stored in data store 310 as asset characteristics 342 (similar to asset characteristics 110 of FIG. 1).

[0064]

[0076] In some implementations, the asset characteristic classification engine 322 uses characteristic profile(s) 360 in classifying asset characteristics. The asset characteristic classification engine 322 may, for example, retrieve a specific characteristic profile 360 ​​from the profile data store 314 for each asset characteristic being analyzed. The characteristic profile may, for example, be similar to the characteristic profile 108 described with respect to FIG. 1.

[0065]

[0077] In some implementations, asset property profile 360 ​​is generated by property classification learning engine 328. Property classification learning engine 328 may generate property profile 360 ​​using a sample dataset learning process such as that generally described with respect to FIG. 2A. In particular examples, property classification learning engine 328 may analyze JPG (a lossy compressed digital image standard developed by the Joint Photographic Experts Group) to recognize various rooftop shapes.

[0066]

[0078] While a particular asset characteristic classification is performed by analysis, in some implementations, one or more characteristic classifications may be obtained from previously stored asset characteristics 342. For example, some basic material and construction information about the asset may be readily available via local and / or remote database system(s), based on tax records, real estate records, etc. In other examples, previously stored asset characteristics 342 may also be obtained from a previous condition assessment performed by system 302 for the asset location.

[0067]

[0079] In some embodiments, once asset characteristic classification engine 322 classifies the asset characteristic(s), characteristic condition classification engine 324 classifies the condition of each asset characteristic. For example, asset characteristics may be classified as described with respect to computational processing step 112 (classifying the condition of each characteristic) of FIG. 1 . Characteristic condition classification engine 324 may generate condition characteristics 344 that indicate, for example, a rating, qualification, or ranking of the condition of a particular characteristic. In some examples, characteristic condition classification engine 324 may qualify a characteristic as a rating such as (a) “good” or “poor,” (b) “like new,” “acceptable,” “needs repair,” or “needs replacement,” or (c) a numerical scale such as a scale of 1 to 5 (5 may correspond to “like new” and 1 may correspond to “needs replacement”). In other embodiments, specific problems may be identified by characteristic condition classification engine 324, such as, for example, in some examples, a fence feature that is partially collapsed or a roof feature that shows signs of structural damage.

[0068]

[0080] In some implementations, the property condition classification engine 324 uses condition profile(s) 362 in classifying the asset property condition. The property condition classification engine 324 may, for example, retrieve a particular condition profile 362 from the profile data store 314 for each asset property being analyzed. The condition profile 362 may, for example, be similar to the condition profile 114 described with respect to FIG. 1.

[0069]

[0081] In some implementations, the state profile 362 is generated by the state classification learning engine 330. The state classification learning engine 330 may generate the state profile 362 using, for example, a sample dataset learning process such as that generally described with respect to Figure 2B. In particular embodiments, the state profile 362 may be generated by identifying peak pixel values ​​of an image in grayscale. Color histogram analysis can be used to classify asset conditions into specific categories using the calculated peak pixel values.

[0070]

[0082] In some embodiments, based on the output of the asset characteristic classification engine 322 and / or the characteristic condition classification engine 324, the risk calculation engine 316 may estimate damage risks based on one or more types of disasters. For example, disaster risk estimate data 352 may be calculated as described with respect to calculation process stage 120 (calculate damage risk) of FIG. 1 . Disaster risk estimate data 350 may, in one example, include a percentage of damage likelihood due to each particular type of risk based on particular asset characteristics and / or asset conditions, as described with respect to risk estimate 122 of FIG. 1 . Additionally, in some embodiments, disaster risk estimate data 350 may include a risk estimate set based on a disaster severity number, a particular path of the disaster, and / or a disaster hazard intensity, in some examples. Disaster severity may correspond to a Richter scale of magnitude of seismic activity or a Saffir-Simpson Hurricane Wind Scale category, in some examples. Disaster path may, in some examples, include a geographic direction, such as northeast, or an impact angle relative to the asset property(ies). The hazard intensity of a disaster may be related to wind speed or precipitation in some examples. Repair costs may be more accurately estimated if, for example, the condition of the roof is known. In one particular example, a roof top in poor condition may make the roof more susceptible to damage requiring replacement. In another particular example, a roof top in poor condition may increase the likelihood of damage to internal assets (e.g., water damage due to cracks in the roof) above a set level of disaster severity.

[0071]

[0083] In some embodiments, the risk calculation engine 316 accesses one or more risk profiles 364 from the data store 314 based on asset characteristics and / or condition characteristics. The risk profiles may be similar to the disaster risk profiles 118 described with respect to FIG. 1, for example. The risk profiles 364 may be obtained from one or more data sources 306, for example, by the cost / risk data acquisition engine 338. In other embodiments, the system 302 may generate the risk profiles 364 (e.g., using risk data obtained from one or more cost / risk data sources 306 and a risk data analysis engine (not shown)).

[0072]

[0084] The risk estimate data 352, in some embodiments, is shared with one or more clients 306. For example, an insurance provider client 306 may use the risk estimate data when conducting insurance underwriting. In another example, a real estate investment broker or firm client 306 may utilize the risk estimate data 352 when selecting investment assets for an investment portfolio.

[0073]

[0085] 8C , an example disaster risk user interface 840 presents risk assessment information based on a risk profile 364 associated with a selected asset (e.g., one of the assets identified by callout 802c in FIG. 8A ). User interface 840 may be accessed, for example, by scrolling down from user interface 820. User interface 840 shows a number of disaster risk factors 842 and associated risk estimate data 844 for the selected asset. As shown, disaster risk factors 842 include a fire factor 842a, a wind factor 842b, a flood factor 842c, an earthquake factor 842d, a tsunami factor 842e, and a landslide factor 842f. In some embodiments, only a subset of the risk estimate data is determined using the risk assessment techniques described herein. For example, flood data 844c may be generated based on governmental flood map assessments rather than using aerial imagery assessments, while fire risk 844a may be determined based at least in part on proximity to neighboring assets derived by image analysis and / or asset structure derived by image analysis. Additionally, wind risk data 842b may be derived by rooftop analysis, as described in further detail herein.

[0074]

[0086] In some implementations, based on the output of asset characteristic classification engine 322 and / or property condition classification engine 324, cost calculation engine 336 calculates repair or replacement costs to remedy the issues identified by property condition classification engine 324 to generate replacement cost estimate data 350. For example, replacement cost estimate data 352 may be calculated as described with respect to calculation process stage 126 (calculate repair or replacement costs) of Figure 1. Replacement cost estimate data 350 may be similar to cost estimate 128 described with respect to Figure 1, for example.

[0075]

[0087] In some implementations, the cost calculation engine 336 is configured, in part, to calculate the cost (e.g., , similar to cost profiles 120 of FIG. 1 . In some embodiments, cost calculation engine 336 accesses one or more cost profiles 366 from data store 314 based on asset characteristics and / or condition characteristics. Cost profiles 366 may, in some embodiments, be obtained from one or more cost data sources 306 by cost / risk data acquisition engine 338. In other embodiments, system 302 may generate cost profiles 366 (e.g., by a cost data analysis engine (not shown) using cost data obtained from one or more cost / risk data sources 306).

[0076]

[0088] Cost estimate data 350, in some embodiments, is shared with one or more clients 306. For example, an insurance carrier client 306 may use the cost estimate data in determining the adequacy of insurance coverage for a particular property. In another example, a real estate investment broker or real estate investment firm client 306 may utilize cost estimate data 350 in determining the cost of maintaining / upgrading a particular investment property.

[0077]

[0089] In some implementations, cost estimate data 350 and / or risk estimate data 352 are provided to a requesting client 306 in real time or near real time in response to receiving an identification of one or more assets. For example, a client 306 accessing a dashboard interface may format a request for risk or cost estimates for the assets identified in the dashboard interface and submit the request to system 302 over a network. In response to the request, system 302 may perform the analysis generally described above and provide a response in the form of a risk or cost analysis to client 306 in real time or near real time. For example, risk analysis information is presented in user interface 840 of FIG. 8C .

[0078]

[0090] In some embodiments, the system may use updated weather data to identify one or more assets in a client's real estate portfolio that have been affected by a disaster. In addition to identifying affected assets, the system may proactively generate and provide repair estimates to clients corresponding to their real estate portfolios shortly after a disaster impacts them. In one example, a weather bureau may update observation data via a website interface within one hour or so of the observation. In this example, the system may provide a repair cost estimate to a client within one to two hours of the disaster occurrence.

[0079]

[0091] In other embodiments, rather than sharing cost estimate data 350 and / or risk estimate data 352 directly with client 306, system 302 may include a report generation engine (not shown) that generates reports regarding the condition, damage, and risk assessment of one or more assets. Additionally, in some embodiments, system 302 may compare current condition characteristics 344 with historical condition characteristics 344 to determine whether the asset owner has made repairs to the asset (e.g., based on an insurance claim payment). Other modifications of system 302 are possible.

[0080]

[0092] 4 is a flowchart illustrating an example method 400 for classifying condition characteristics of an asset based on aerial image analysis. For example, the flowchart may illustrate computational processing steps 104-112 as described with respect to FIG. 1. Method 400 may be performed, for example, by system 302 of FIG. 3.

[0081]

[0093] In some implementations, method 400 begins with accessing (402) a two-dimensional shape map containing the shape of the asset. The shape map may be accessed by image acquisition engine 336 from shape map image source 304, for example, as described with respect to FIG. 3. The two-dimensional shape map may take the form of, for example, any of shape maps 102a described with respect to FIG. 1. In certain examples, the shape map may be in a form similar to shape map 500 described with respect to FIG. 5A.

[0082]

[0094] In some implementations, two-dimensional and / or three-dimensional aerial imagery including images of the asset is accessed 404. The two-dimensional and / or three-dimensional aerial imagery may be accessed by image acquisition engine 336 from aerial image source 304, for example, as described with respect to FIG. 3. The two-dimensional and / or three-dimensional aerial imagery may take the form of, for example, any of aerial imagery 102c described with respect to FIG. 1. In certain examples, the two-dimensional aerial imagery may be in a form similar to aerial imagery 510 described with respect to FIG. 5B.

[0083]

[0095] In some implementations, the aerial image is overlaid with the shape map (406). The synthetic image generation engine 318 of Figure 3 may, for example, overlay the aerial image with the shape map to generate the synthetic image. The synthetic image may, for example, be in a form similar to the synthetic image 520 described with respect to Figure 5C.

[0084]

[0096] If a shape match is not identified (408), in some implementations, method 400 accesses (410) an alternative two-dimensional or three-dimensional aerial image. In some embodiments, image quality analysis / preprocessing engine 326 of FIG. 3 determines whether the composite image shows a sufficient shape match of the asset(s) of interest. If it is determined that the images do not match, image acquisition engine 336 of FIG. 3 may acquire alternative two-dimensional or three-dimensional aerial imagery, for example, from aerial imagery source(s) 304. As described with respect to FIG. 3, for example, image acquisition engine 336 may determine the next best source for acquiring imagery of the asset based on one or more factors. Meanwhile, in other implementations (not shown), method 400 may proceed without matching the shape map to the aerial imagery.

[0085]

[0097] If a shape match is identified (408), in some implementations, a format of the aerial image is prepared (411). In some embodiments, the image is cropped to include the asset of interest or a portion of the asset and its surroundings (e.g., a parcel including the asset and / or a portion of the neighborhood proximate to where the asset is located). Cropping may be based, for example, on a correlation between a shape map and the asset. By way of example, the image may be cropped to include a shape map outline of the asset and a bounding area. In some embodiments, cropping may include cropping to a shape set (e.g., an X x X pixel square, an X x Y rectangle, etc.). In some embodiments, in addition to cropping the image, the image may be resized. For example, based on the resolution of the aerial image, the resolution may be reduced to fit within, for example, an X x X pixel square or an X x Y rectangle. Resizing to a standard image size may contribute to consistent analysis and improved classification accuracy. In another embodiment, the color depth and / or color mapping of the aerial image may be adjusted. For example, the color depth and color mapping may be consistent across the entire analysis image for consistent color histogram analysis of the image. In certain embodiments, color images may be converted to grayscale for grayscale image analysis. Image adjustment may be performed, for example, by the image quality analysis / preprocessing engine 326 described with respect to FIG.

[0086]

[0098] In some embodiments, the orthogonality of the aerial images is assessed (412). For example, the image quality analysis / preprocessing engine 326 may assess the orthogonality of the aerial images. In one embodiment, the image quality analysis / preprocessing engine 326 may determine that orthogonality correction is desirable. In another embodiment, if the images represent a regular orthophoto rather than a true orthophoto, the image quality analysis / preprocessing engine 326 may notify another module that may correct the capture angle of the aerial images. In other embodiments (not shown), orthogonality may be assessed without the use of an overlaid shape map.

[0087]

[0099] Returning to Figure 4, in some implementations, if the aerial imagery is not orthogonal (414), an orthogonal version of the aerial imagery is generated (416). For example, the image quality analysis / preprocessing engine 326 of Figure 3 may correct the normal orthophoto morphology to generate a true orthophoto morphology. The true orthophoto morphology may, for example, be much more consistent with the shape map.

[0088]

[0100] In some implementations, one or more asset characteristics of the asset are classified (418). In some embodiments, the asset is analyzed based on a portion of the aerial image that is substantially surrounded by a shape map of the asset. In other embodiments, the asset analysis encompasses the asset's surroundings (e.g., features proximate to the asset, parcel of the asset, neighboring assets, etc.). The asset characteristic(s) may be classified by asset characteristic classification engine 322, for example, as described with respect to FIG. 3. In particular examples, a rooftop asset characteristic may be classified as one of various rooftop shapes, as shown in FIG. 2A.

[0089]

[0101] In some embodiments, once asset property classifications have been identified (420), the condition of each of the asset properties may be classified (422). The asset property condition(s) may be classified by a property condition classification engine 324, for example, as described with respect to FIG. 3. In particular examples, the classification of a hip roof feature may be classified as one of good condition and poor condition, as shown in FIG. 2B. In some embodiments, the condition classification of each of the asset properties may be output to the claimant's computing device via a dashboard interface screen (424). In some examples, in addition to providing a condition classification for the asset property, the dashboard interface screen may include a replacement cost to replace the asset property, a risk cost estimate for damage from a natural disaster, or confirmation of whether repairs have been made to the asset property.

[0090]

[0102] On the other hand, if a classification match is not identified (420), in some embodiments, an alternative 2D or 3D aerial image is accessed (410), and method 400 resumes registering the alternative aerial image with the shape map (406). Alternatively, in some embodiments, even if a classification match is not identified, the condition may be assessed based on a default profile (not shown) associated with the asset features. For example, if the shape of the rooftop does not match one of the shapes trained into the system, the condition of the rooftop may be further assessed based on a general condition profile trained with various rooftop shapes. In some examples, the condition of the rooftop may be assessed by comparing the current rooftop condition to historical condition characteristics stored from previous condition assessments for the asset.

[0091]

[0103] Although method 400 is shown as a series of steps, other implementations may include more or fewer steps. For example, in some implementations, ground imagery is acquired and used to classify the same and / or different asset characteristics, as described with respect to operational flow 100 of FIG.

[0092]

[0104] Additionally, in other embodiments, certain steps may be performed in a different order. For example, in some embodiments, asset characteristics and condition characteristics may be classified in parallel (418, 422). Other modifications of operational flow 100 are also possible.

[0093]

[0105] Aspects of the present disclosure may be directed to a computing system for classifying the repair condition of asset properties to provide dynamic, real-time asset condition assessments in response to requests received from users, such as insurance carriers or real estate investment brokers, using aerial imagery. The implementations described herein improve upon conventional methods by applying deep learning analytical models to detected asset properties in the acquired imagery to assess whether the condition of the asset properties has deteriorated to the point where repair or replacement is necessary and to determine the asset's risk exposure due to the detected condition of the asset properties. The implementations described herein improve the processing efficiency of the system, reduce the time required to perform condition assessments, and automate the condition assessment process.

[0094]

[0106] Next, a hardware description of an exemplary embodiment of a computing device, mobile computing device, or server will be described with reference to FIG. 6. In FIG. 6, the computing device, mobile computing device, or server includes a CPU 600 that executes the processes described above. Process data and instructions may be stored in memory 602. Furthermore, these process and instructions may be stored on a storage medium disk 604, such as a hard drive (HDD) or a portable storage medium, or may be stored remotely. Furthermore, the claimed technological advance is not limited to the form of computer-readable medium on which the instructions for the inventive process are stored. For example, the instructions may be stored on a CD, DVD, flash memory, RAM, ROM, EPROM, EEPROM, hard disk, or any other information processing device with which the computing device, mobile computing device, or server communicates, such as a server or computer.

[0095]

[0107] Additionally, some of the claimed technological advances may be provided as utility applications, background daemons, or operating system components, or combinations thereof, that run in conjunction with CPU 600 and an operating system (such as Microsoft Windows 6, UNIX, Solaris, LINUX, Apple MAC-OS, and other systems known to those skilled in the art).

[0096]

[0108] CPU 600 may be an Intel (USA) Xenon or Core processor, or an AMD (USA) Opteron processor, or other processor types recognized by those skilled in the art. Alternatively, CPU 600 may be implemented on an FPGA, ASIC, PLD, or using discrete logic circuitry, as recognized by those skilled in the art. Furthermore, CPU 600 may be implemented as multiple processors simultaneously cooperating to execute instructions of the inventive process described above.

[0097]

[0109] The computing device, mobile computing device, or server of Figure 6 further includes a network controller 606, such as an Intel Ethernet PRO network interface card from Intel Corporation (USA), to interface with network 6X. As will be appreciated, network 628 can be a public network such as the Internet, or a private network such as a LAN or WAN network, or a combination thereof, and can further include a PSTN or ISDN subnetwork. Furthermore, network 628 can be wired, such as an Ethernet network, or wireless, such as a cellular network, including EDGE, 3G, and 4G wireless cellular systems. The wireless network can also be Wi-Fi, Bluetooth, or any other known form of wireless communication.

[0098]

[0110] The computing device, mobile computing device, or server further includes a display controller 608, such as an NVIDIA GeForce GTX or Quadro graphics adapter from NVIDIA Corporation (USA), for interfacing with a display 610, such as a Hewlett Packard HPL2445w LCD monitor. A general-purpose I / O interface 612 interfaces with a keyboard and / or mouse 614, as well as a touch screen panel 616, on or separate from the display 610. The general-purpose I / O interface further connects to various peripherals 618, including printers and scanners, such as Hewlett Packard's OfficeJet or DeskJet.

[0099]

[0111] Additionally, a sound controller 620, such as Creative's Sound Blaster X-Fi Titanium, is provided within the computing device, mobile computing device, or server to interface with a speaker / microphone 622 to provide voice and / or music.

[0100]

[0112] The general-purpose storage controller 624 connects to the storage media disk 604 by a communication bus 626 (which may be ISA, EISA, VESA, PCI, etc.) for interconnecting all of the components of the computing device, mobile computing device, or server. A description of the general features and functionality of the display 610, keyboard and / or mouse 614, as well as the display controller 608, storage controller 624, network controller 606, sound controller 620, and general-purpose I / O interface 612 is omitted herein for the sake of brevity, as these features are well known.

[0101]

[0113] One or more processors may be used to implement the various functions and / or algorithms described herein unless otherwise specified. Furthermore, any function and / or algorithm described herein may be executed on one or more virtual processors, on one or more physical computing systems, such as, for example, a computer farm or a cloud drive, unless otherwise specified.

[0102]

[0114] Flowchart diagrams and block diagrams of methods, systems, and computer program products according to embodiments of the present disclosure have been described. These aspects are implemented by computer program instructions that are provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to create a machine, whereby the instructions, executing via the processor of the computer or other programmable data processing apparatus, create means for implementing the function(s) / act(s) specified in the flowchart and / or block diagram block(s).

[0103]

[0115] These computer program instructions may further be stored on a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, thereby creating an article of manufacture that includes instruction means for implementing the functions / acts specified in the flowchart and / or block diagram block(s).

[0104]

[0116] These computer program instructions can then be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable data processing apparatus to generate a computer-implemented process, whereby the instructions executing on the computer or other programmable data processing apparatus can generate a process for implementing the function(s) / act(s) specified in the flowchart and / or block diagram block(s).

[0105]

[0117] Furthermore, the present disclosure is not limited to the particular circuit elements described herein, nor is the disclosure limited to the particular sizing and classification of these elements. For example, one skilled in the art will appreciate that the circuits described herein may be designed based on the sizing and chemistry of the battery or the requirements of the backup load being driven.

[0106]

[0118] The functions and features described herein may also be performed by various distributed components of a system. For example, one or more processors may perform these system functions, where the processors may be distributed across multiple components communicating within a network. Distributed components may include human interface communication devices (e.g., display monitors, smartphones, tablets, personal digital assistants (PDAs)), as well as one or more client and server machines that may share processing, as shown in FIG. 8. The network may be a private network such as a LAN or WAN, or a public network such as the Internet. Input to the system may be received by direct user input, remotely, in real time, or as a batch process. Furthermore, some implementations may be implemented on modules or hardware that are not identical to those described. Accordingly, other implementations are within the scope of what may be claimed.

[0107]

[0119] In some implementations, as illustrated in FIG. 7 , the innovations described herein may interface with a Google Cloud Platform™ cloud computing environment 730 to execute at least a portion of the methods or algorithms detailed above. Processes associated with the methods described herein may be executed by a data center 734 on a computational processor, such as a Google Compute Engine. The data center 734 may also include an application processor, such as a Google App Engine, that may be used to interface with the systems described herein to receive data and output corresponding information. The cloud computing environment 730 may further include one or more databases 738, such as cloud storage and query databases, or other data storage. In some implementations, a cloud storage database 738, such as Google Cloud Storage, may store processed and unprocessed data provided by the systems described herein.

[0108]

[0120] The systems described herein may communicate with a cloud computing environment 730 via a secure gateway 732. In some embodiments, the secure gateway 732 includes a database query interface, such as the Google BigQuery platform.

[0109]

[0121] The cloud computing environment 102 may include a provisioning tool 740 for resource management. The provisioning tool 740 may be connected to computing devices of the data center 734 to facilitate provisioning of computing resources of the data center 734. The provisioning tool 740 may receive requests for computing resources via the secure gateway 732 or the cloud controller 736. The provisioning tool 740 may facilitate connection to particular computing devices of the data center 734.

[0110]

[0122] Network 702 represents one or more networks, such as the Internet, connecting cloud environment 730 to a large number of client devices (e.g., in some examples, mobile phones 710, tablet computers 712, mobile computing devices 714, and desktop computing devices 716). Network 702 may further communicate over wireless networks using various mobile network services 720, such as Wi-Fi, Bluetooth, cellular networks (including EDGE, 3G, and 4G wireless cellular systems), or any other form of wireless communication known in the art. In some embodiments, network 702 is independent of local interfaces and networks associated with client devices, allowing for integration of local interfaces and networks configured to perform the processes described herein.

[0111]

[0123] While specific embodiments have been described, these embodiments are presented by way of example only and do not limit the scope of the present disclosure. Indeed, the novel methods, apparatus, and systems described herein may be embodied in a variety of other forms, and various omissions, substitutions, and changes may be made in the form of the methods, apparatus, and systems described herein without departing from the spirit of the disclosure. The accompanying claims and their equivalents are intended to cover such forms or modifications as are within the scope and spirit of the present disclosure.

Claims

1. 1. A system for automatically assessing asset condition, comprising: a non-transitory computer-readable medium storing a plurality of machine learning classifiers, each machine learning classifier trained to identify at least one asset attribute or at least one asset state; a data store containing asset information for a plurality of assets and a plurality of images of said plurality of assets; processing circuitry; The processing circuitry accessing one or more aerial images of the asset from the data store; identifying a footprint of a structure on the asset within at least a first aerial image of the one or more aerial images; applying a first one or more machine learning classifiers of the plurality of machine learning classifiers to at least one of the one or more aerial images to determine a plurality of asset characteristics of the structure, the plurality of asset characteristics including a particular roof material among a plurality of roof materials and a particular roof shape among a plurality of roof shapes; applying a second one or more machine learning classifiers of the plurality of machine learning classifiers to at least one of the one or more aerial images to determine a roof condition of a roof of the structure, the second one or more machine learning classifiers being trained to identify one or more types of problems with the roof as part of determining the roof condition of the structure; and configured to perform operations including providing, to a remote computing device over a network, at least a portion of at least one of the one or more aerial images of the asset, asset characteristic information corresponding to the plurality of asset characteristics, and status information corresponding to the roof condition, wherein whenever the second one or more machine learning classifiers identify at least one of the one or more types of problem with the roof, the status information includes an identification of one or more problems with the roof. system.

2. The system of claim 1 , wherein the action further comprises comparing the roof condition to a past roof condition.

3. 10. The system of claim 1, wherein the operations further include applying a third one or more machine learning classifiers of the plurality of machine learning classifiers to at least one of the one or more aerial images to determine a plurality of asset attributes of one or more features in a vicinity of the structure, the one or more features comprising one or more of a swimming pool or a tennis court.

4. The system of claim 3 , wherein determining the plurality of asset attributes includes determining at least one of a size or a type of the pool.

5. The system of claim 3 , wherein the operation further comprises determining a distance between the structure and each feature of the at least one feature of the one or more features.

6. The system of claim 1 , wherein the one or more types of problem include structural damage to one or more roof features.

7. The system of claim 6 , wherein the one or more roof features include at least one of a roofing, a skylight, or a chimney.

8. The system of claim 1 , wherein the operations further include obtaining geographic coordinates of the asset, and wherein at least one of the one or more aerial images is accessed using the geographic coordinates.

9. The system of claim 1 , wherein the actions further include determining a numerical condition rating qualifying a relative condition of the roof, and the condition information includes the numerical condition rating.

10. 2. The system of claim 1, wherein providing the at least a portion of at least one of the one or more aerial images of the asset, the asset property information corresponding to the plurality of asset properties, and the status information corresponding to the roof condition comprises presenting a graphical user interface including the at least one portion of the one or more aerial images, the asset property information, and the status information for review by an end user on a display of the remote computing device.

11. The system of claim 1 , wherein at least some of the images are orthophoto images.

12. 1. A method for automatically assessing asset condition, comprising: obtaining geographic coordinates and / or addresses corresponding to the asset; retrieving one or more aerial images of the asset from at least one database; identifying, by a processing circuit, a building footprint on the property within at least a first aerial image of the one or more aerial images; applying, by the processing circuitry, a first one or more machine learning models to at least one of the one or more aerial images to classify a plurality of asset characteristics of the building, the plurality of asset characteristics including a particular roof material among a plurality of roof materials and a particular roof shape among a plurality of roof shapes; applying, by the processing circuitry, second one or more machine learning models to at least one of the one or more aerial images to determine a roof condition of the building roof, the second one or more machine learning models being trained to identify one or more types of problems with the roof as part of determining the building roof condition; providing, in real time in response to obtaining the geographic coordinates and / or the address, to a remote computing device over a network at least a portion of at least one of the one or more aerial images of the asset, asset characteristic information corresponding to the plurality of asset characteristics, and status information corresponding to the roof condition, wherein whenever the second one or more machine learning models identify at least one of the one or more types of problem with the roof, the status information includes identification of one or more problems with the roof; A method comprising:

13. The method of claim 12 , wherein the geographic coordinates and / or the address are provided by a user of the remote computing device in response to a risk analysis request.

14. The method of claim 12 , wherein the at least one database further includes asset information corresponding to a plurality of assets including the asset.

15. The method of claim 12 , wherein retrieving the one or more aerial images comprises assessing orthogonality of the one or more aerial images from a remote location over a network.

16. 13. The method of claim 12, further comprising applying, by the processing circuitry, a third one or more machine learning models to at least one of the one or more aerial images to determine a plurality of asset characteristics of one or more additional structures on the asset.

17. The method of claim 16 , wherein the one or more additional structures include at least one of a swimming pool or a tennis court.

18. 13. The method of claim 12, wherein providing the at least a portion of at least one of the one or more aerial images of the asset, the asset property information corresponding to the plurality of asset properties, and the status information corresponding to the roof condition comprises presenting a graphical user interface including the at least a portion of the at least one of the one or more aerial images, the asset property information, and the status information for review by an end user on a display of the remote computing device.

Images