Remote Real Estate Inspection

Abstract

The system is configured to receive image data, identify a plurality of objects related to the real estate shown in the image data using a first set of one or more machine learning models, determine the number of unique objects shown in the image data, and generate an assessment of the state of the real estate using a second set of one or more machine learning models.

Description

Technical Field

[0001] Priority / Incorporation by Reference This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 363,193, filed Apr. 19, 2022, which is hereby incorporated by reference in its entirety.

Background Art

[0002] An artificial intelligence (AI) system can perform a real estate inspection by autonomously evaluating the condition of a real estate using computer vision and other machine learning techniques. An entity can utilize this type of AI system to provide any of a variety of different types of services. By way of example, without the involvement of a professional claims adjuster, the condition of a real estate can be evaluated by the AI system to provide an estimate of repair costs. In another example, the condition of a real estate can be evaluated by the AI system and the real estate can be appraised without the involvement of a professional appraiser. This type of AI system can also be utilized in other use cases where it is desired to evaluate the condition of a real estate, including but not limited to rental property management, insurance underwriting, and insurance claims processing.

[0003] An entity can release a user-facing application to provide the above types of services. A user can use their mobile device to take images and / or videos of a real estate. The images and videos can be input into the AI system to evaluate the condition of the real estate. However, if the images and videos do not properly capture the object or do not have sufficient quality, the AI system may not be able to evaluate the condition of the real estate.

[0004] The user experience associated with an application is an important factor in attracting and retaining users. Each interaction between the user and the application is a potential friction point that can prevent the user from completing the inspection process and / or using the application in the future. For example, if it is inconvenient or too difficult for the user to take the image data used by the AI system to evaluate the condition of the real estate, the user may decide not to use the application. Therefore, there is a need for a mechanism configured to collect appropriate data for the AI system to evaluate the condition of the real estate without adversely affecting the user experience associated with the application.

Summary of the Invention

[0005] Some exemplary embodiments relate to a method of receiving image data, identifying a plurality of objects related to a real estate shown in the image data using a first set of one or more machine learning models, determining the number of unique objects shown in the image data, and generating an evaluation of the condition of the real estate using a second set of one or more machine learning models.

[0006] Other exemplary embodiments relate to a system having a memory for storing image data, one or more processors that identify a plurality of objects related to a real estate shown in the image data using a first set of one or more machine learning models, determine the number of unique objects shown in the image data, and generate an evaluation of the condition of the real estate using a second set of one or more machine learning models.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

[0008] Exemplary embodiments may be further understood with reference to the following description and the related attached drawings, where like elements are provided with the same reference numerals. Exemplary embodiments introduce a system and method for evaluating the condition of a real estate using artificial intelligence (AI). As will be described in more detail below, computer vision and other types of machine learning techniques can be used to autonomously evaluate the condition of a real estate (e.g., a house, a building, a fence, a land reclamation, etc.).

[0009] Exemplary embodiments are described with respect to an application executed on a user device. However, the reference to the term "user device" is provided for illustrative purposes only. Exemplary embodiments can be used with any electronic component equipped with hardware, software, and / or firmware configured to communicate with a network and collect image and video data, such as a mobile phone, a tablet computer, a smartphone, etc. Therefore, the user device described herein is used to represent any suitable electronic device.

[0010] Furthermore, throughout this description, certain operations may be described as being performed by one or more machine learning models or a series of machine learning models. One of ordinary skill in the art will understand that there are many different types of machine learning models. For example, the exemplary machine learning models described herein may include visual and non-visual algorithms. Additionally, the exemplary machine learning models may include classifiers and / or regression models. One of ordinary skill in the art will understand that a classifier model can generally be used to determine the probability that a particular outcome will occur (e.g., an 80% probability that a part of a house (e.g., a wooden floor) will be replaced rather than repaired). A regression model may provide a numerical value (e.g., 20 labor hours are required for floor repair). Other examples of machine learning models may include multi-task learning models (MTLs) that can perform both classification, regression, and other tasks. The resulting AI system described below may include some or all of the above-described machine learning components or any other type of machine learning model that can be applied to determine the expected results of the AI system. It should be understood that any reference to one or more machine learning models (or a series of machine learning models) may refer to a single machine learning model or a group of machine learning models. Additionally, it should be understood that the machine learning models described as performing different operations may be the same machine learning model or different machine learning models.

[0011] In addition, the exemplary embodiments are described with reference to real estate. The user device may capture images and / or videos of the real estate for the purpose of evaluating the condition of the real estate. However, it should be understood that the exemplary embodiments are not limited to evaluating the condition of any particular type of object associated with the real estate. The exemplary embodiments may be implemented for any tangible object associated with any aspect of the real estate where numerical values or conditions can be evaluated. To provide some non-limiting examples, the exemplary embodiments may be used to evaluate, for example, houses, buildings, rooms, fences, paths, driveways, lawns, shrubs, trees, gardens, crops, sprinkler systems, lighting fixtures, renewable energy devices, and the condition of other objects associated with the real estate.

[0012] In some exemplary embodiments, it can be described that the AI can perform an evaluation by comparing an image of a damaged property with an image of an undamaged property. However, it should be understood that the exemplary embodiments do not require such comparisons. In other exemplary embodiments, the AI may perform an evaluation without directly comparing an image of a damaged property with an image of an undamaged property. That is, the machine learning model described herein can perform a property evaluation of a damaged property regardless of the image of an undamaged property.

[0013] An entity may utilize the AI to evaluate the condition of the real estate and provide any of a variety of different services. As an example, the condition of one or more objects may be evaluated by the AI system to generate an estimated repair cost without the involvement of a professional claims adjuster. In another example, the condition of one or more objects may be evaluated by the AI system to evaluate the real estate without the involvement of a professional appraiser. However, the exemplary embodiments are not limited to the exemplary use cases referenced above. The exemplary techniques described herein may be used independently of each other, in conjunction with currently implemented AI systems, in conjunction with future implementations of AI systems, or independently of other AI systems.

[0014] An AI system can process image data to evaluate the condition of a real estate. Throughout this disclosure, the term "image data" should be understood to refer to data captured by a camera or any other suitable type of image capture device. In some embodiments, the image data may include one or more digital photographs. In another embodiment, the image data may include one or more segments of video data that include a plurality of consecutive frames. One or more segments may be part of a single continuous recording or part of a plurality of different video recordings. Further, the video data may be augmented by individual frames or images taken separately at different resolutions, different angles with respect to the object or point of interest, or different compression algorithms (or without a compression algorithm). In some exemplary embodiments, a machine learning model can identify key frames of a video. For example, the machine learning model may determine that the object is at the center of the frame and the entire object is in the scene. In another embodiment, the AI system can identify the maximum visual distance between frame captures of the same object to maximize the information provided to the machine learning model, such as image variations below, such as reflections / shadows. In another embodiment, the AI system can indicate the point in time that is the optimal viewing point to provide an accurate measurement of the physical dimensions, or the position of the object where occlusion by foreground objects is minimized. The image data may also include data that is not within the human visible range, such as infrared and ultraviolet data.

[0015] The user may collect image data using the camera of their user device. However, if the image and / or video do not properly capture the object or are of insufficient quality, the AI system may not be able to evaluate the condition of the real estate from the image data. In this type of scenario, the user may be requested to provide additional images and / or videos. To ensure a proper user experience, the process of collecting the images and videos required by the AI system to evaluate the condition of the real estate should be an easy task for the user to complete.

[0016] In addition to image data, the AI system can also utilize non-visual information (e.g., non-image information) including audio information, pressure and temperature information, and moisture information that can be collected by the user device 100. The user device 100 can be equipped with additional sensors for detecting events such as moisture or humidity in the ceiling, walls, floor, or floor coverings (such as rugs or carpets). The audio information can include, for example, the sounds of items within home operations (e.g., furnace, air conditioner, sink, toilet, stove, etc.). Alternatively, the audio information can be information about the state of the real estate recorded by the user, and this information may be linked to a particular image or a portion of a video.

[0017] Some of the exemplary mechanisms described herein are configured to reduce friction and improve the user experience associated with the application. For example, in some embodiments, the user device can be configured to provide dynamic feedback to guide the user when collecting image data that appropriately captures the object and has sufficient quality to evaluate the state of the real estate. The dynamic feedback makes the process of recording a video more intuitive and / or easier to use. However, this is only one example of the various types of functions that can be enabled by the exemplary mechanisms introduced herein.

[0018] The AI system can collect and monitor data regarding the quality of data collection, the completion rate of the data collection process, and the user satisfaction of the data collection process across multiple analyses of the real estate. The AI system can be configured to automatically adjust various parameters of the collection process to optimize any of the data selected by the user. Alternatively, the AI system can also propose changes to the collection method to the human administrator of the AI system and the collection system.

[0019] Figure 1 shows an exemplary user device 100 according to various exemplary embodiments described herein. The user device 100 includes a processor 105 for executing AI-based applications. In one embodiment, the AI-based application is a web-based application hosted on a server and accessed via a network (e.g., a wireless access network, a wireless local area network (WLAN), etc.) via a transceiver 115 or some other communication interface. In other embodiments, all of the AI-based applications may be stored and executed locally on the user device 100.

[0020] The reference application executed by the processor 105 is merely exemplary. The functions associated with the application may also be represented as separate built-in components of the user device 100, or as modular components coupled to the user device 100, e.g., integrated circuits with or without firmware. For example, the integrated circuit may include an input circuit for receiving signals, as well as a processing circuit for processing signals and other information. The AI-based application may also be embodied as one application or multiple separate applications. Further, in some user devices, the functions described for the processor 105 are divided among two or more processors. Exemplary embodiments may be implemented in any of these or other configurations of the user device.

[0021] Figure 2 shows an exemplary system 200 according to various exemplary embodiments. System 200 includes a user device 100 that communicates with a server 210 via a network 205. However, the exemplary embodiments are not limited to this type of arrangement. The reference to a single server 210 is provided for illustrative purposes only, and the exemplary embodiments may utilize any suitable number of servers with any suitable number of processors. Further, those skilled in the art will understand that some or all of the functions described herein for server 210 may be performed by one or more processors of a cloud network.

[0022] Server 210 may host a platform associated with an application. The platform may be a set of physical and virtual components configured to execute software to provide any of a variety of different services. The platform may manage stored data, interact with users (e.g., customers, employees, etc.), and perform any of a variety of different operations.

[0023] In one example, user device 100 may store application software, including but not limited to, one or more machine learning models, locally on user device 100. The application may utilize one or more machine learning models or any other suitable type of mechanism to evaluate the condition of a real estate property based on image data collected by user device 100. Next, the data collected and derived by user device 100 may be provided to remote server 210, and optionally, additional operations may be performed. In another example, user device 100 may collect image data and provide it to server 210. Server 210 may utilize one or more machine learning models or any other suitable type of mechanism to evaluate the condition of a real estate property based on the image and / or video of the real estate property.

[0024] The user device 100 further includes a camera 120 for taking videos and a display 125 for displaying the application interface and / or the video in a dynamic overlay. Further details regarding the dynamic overlay are provided below. The user device 100 may be any device having hardware and / or software for implementing the functions described herein. In one embodiment, the user device 100 may be a smartphone having a camera 120 located on a side surface (e.g., the rear) of the user device 100 opposite to the side (e.g., the front) where the display 125 is located. The display 125 may be a touch screen for receiving user input, in addition to displaying images and / or other information via, for example, a web-based application.

[0025] In the embodiment of FIG. 2, it is shown that there can be an interaction between the user device 100 and the server 210. However, it should be understood that information from the user device 100 and / or the server 210 may be distributed to other components via the network 205 or any other network. These other components may be components of the entity operating the server 210 or may be components operated by a third party. Examples of third parties are provided throughout this description and may include, for example, insurance companies, contractors, government agencies, support or relief organizations, etc. That is, the results of the evaluation can be made available for use by any entity permitted to receive the results by the owner of the property and / or the operator of the server 210.

[0026] The examples provided below refer to one or more machine learning models (e.g., classifiers) that perform operations such as, but not limited to, identifying an object shown in image data, identifying a damaged object shown in image data, determining the state of one or more objects shown in image data, determining the dimensions of one or more objects shown in image data, and determining the material of one or more objects shown in image data. Each classifier may be composed of one or more trained models. The AI for classification may be based on the use of one or more of non-linear hierarchical algorithms, neural networks, convolutional neural networks, recurrent neural networks, long short-term memory networks, multi-dimensional convolutional networks, memory networks, transformer networks, fully convolutional networks, gated recurrent networks, gradient boosting techniques, random forest techniques.

[0027] Generally, a machine learning model can be designed to learn progressively as more data is received and processed. Thus, the exemplary applications described herein may periodically send the results to a centralized server to refine the model for future evaluation.

[0028] In some embodiments, a single machine learning model (e.g., a classifier) may be stored locally on user device 100. This may enable an application to produce quick results even when user device 100 does not have an available connection to the Internet (or any other suitable type of data network). This machine learning model may be configured to generate multiple different types of outputs. The use of a single machine learning model trained to perform multiple tasks may be beneficial to user device 100 because it may occupy significantly less storage space compared to multiple machine learning models each specific to a different task. Thus, the machine learning models described herein may be compact enough to execute on user device 100 and may include multi-task learning such that one classifier and / or model may perform multiple tasks. However, the exemplary embodiments are not limited to user device 100 equipped with a single machine learning model. For example, user device 100 may comprise one or more machine learning models dedicated to a single task (e.g., object identification, damage identification, dimension determination, material determination, etc.). Any suitable number of machine learning models may be stored and / or utilized by user device 100.

[0029] Furthermore, for the embodiments discussed below, the evaluation may be performed on the user device, the server, or a combination thereof. If an entity other than the home owner is the intended recipient of the information, the evaluation is more likely to be performed by the server. For example, in a situation where the intended user is concerned about the impact on all or a subset of the structures in the area, the application may execute on a server or a set of servers that execute multiple machine learning models.

[0030] In other embodiments, one or more machine learning models may be stored in a cloud network. The user device 100 may collect image data and upload it to the cloud network for processing by one or more machine learning models. The output of one or more machine learning models may be provided to the user device 100 and / or saved for future use. Some machine learning models may include multi-task learning to enable the execution of multiple tasks by a single machine learning model. In some embodiments, the cloud may be composed of one or more machine learning models each dedicated to a single task (e.g., object identification, damage identification, dimension determination, material determination, etc.). However, the exemplary embodiments are not limited to the examples provided above and may be implemented using any suitable arrangement of devices and machine learning models.

[0031] Figure 3 shows a method 300 for performing a real estate assessment using AI according to various exemplary embodiments. Method 300 provides a general overview of a method that may evaluate the condition of a real estate using image data. Further, various exemplary use cases are described within the context of method 300.

[0032] The image data may include photos and / or videos taken by the user using the camera 120 of the user device 100. The photos and / or videos may be augmented by individual images or frames taken separately at different resolutions, different angles with respect to the object or point of interest, or different compression algorithms (or no compression algorithm). In some embodiments, the image data may further include photos and / or videos taken by devices other than the user device 100. For example, in some use cases, satellite images, images taken by drones, or images taken during an aerial flight may also be utilized to evaluate the condition of the real estate.

[0033] According to some aspects, exemplary embodiments introduce techniques for providing dynamic feedback that guides a user to take photographs and / or videos sufficient for the user to evaluate the condition of a real estate property. In some embodiments, this information may be provided to the user while the user is actively using camera 120 to take photographs and / or videos of the real estate property to be evaluated. During the description of method 300, examples may refer to exemplary techniques for providing instructions and dynamic feedback to the user, which is for guiding the user to collect appropriate image data. A more comprehensive description of an exemplary dynamic feedback mechanism is provided below with respect to method 400 of FIG. 4.

[0034] Throughout this specification, it should be understood that any of the photographs / images / videos may be augmented using, for example, augmented reality (AR) technology, virtual reality (VR) technology, three-dimensional (3D) data such as point clouds, and the like. For example, an application executed on user device 100 may include functionality that enables these technologies to be incorporated into image collection. To provide some specific but non-limiting examples, the application may use AR technology to present to the user a view of the property before damage when collecting a photograph or video. For example, the user may collect an image of the property before damage occurs, which may be used by the AR function to display to the user the appearance of the property before damage. Thereby, the user can compare the current damaged state of the property with the state before damage occurred and collect a photograph / video showing all of the damage. In another example, the AR function may be used to measure the location (e.g., a corner of a room) or the exact coordinates of other objects. Thereby, the application (or AI system) may be able to understand the 3D scenario and the damage.

[0035] Some of the exemplary use cases generally relate to evaluating the condition of a real estate property after the occurrence of an event that may have caused damage to one or more objects. For example, the event can be a storm, flood, fire, termite infestation, construction accident, or any other type of event that may cause damage to one or more objects associated with the real estate property. Accordingly, the evaluation of the real estate property can be used to provide services related to insurance claims, such as estimating initial repair costs, determining whether an in-person inspection is necessary, and determining whether it is possible to live in the home, among others, but is not limited thereto. Other exemplary use cases relate to services such as, but not limited to, underwriting insurance, appraising selling value, managing rental properties, tracking the out-of-hours condition of a real estate property, and identifying improvements that can increase the value of a real estate property. The above examples are not intended to limit the exemplary embodiments in any way. Instead, these examples are intended to provide some context regarding how an exemplary evaluation of a real estate property can be utilized to provide various different types of services.

[0036] In 305, the user takes a photograph and / or video of the real estate property. For example, the user may take a photograph and / or video with the camera 120 of the user device 100. In some embodiments, the photograph and / or video can be evaluated for quality and clarity before being utilized in method 300, as will be described in more detail below with respect to method 400.

[0037] The photograph and / or video may each depict a plurality of objects associated with the real estate property. Throughout this specification, the term object can be used to refer to any tangible thing in the real world that consists of one or more parts. In some embodiments, a part of an object may also be referred to as an object. For example, the term object may be used to refer to an entire building, but may also be used to refer to a window of the building. Accordingly, any example characterizing a particular thing as an object is provided for illustrative purposes only. The exemplary embodiments are not limited to any particular type of object associated with the real estate property.

[0038] To provide some non-limiting examples, within the context of the exterior of a dwelling, relevant objects can include, but are not limited to, exterior walls, windows, doors, screens, roofs, skylights, gutters, fences, shrubs, trees, crops, gardens, yards, parking lots, driveways, walkways, garages, shades, stairs, patios, decks, outdoor furniture, lighting fixtures, electrical appliances, renewable energy devices, and sprinklers. Within the interior or context of a dwelling, objects can include, but are not limited to, interior walls, ceilings, floors, lighting fixtures, windows, screens, blinds, curtains, doors, furniture, appliances, electronic devices, electrical appliances, and exercise equipment. However, the examples provided above are for illustrative purposes only and are not intended to limit the exemplary embodiments in any way.

[0039] In method 300, an example is provided where operations are performed by "one or more machine learning models". As described above, in some embodiments, a single machine learning model can be configured to perform multiple different tasks. In other embodiments, a single machine learning model may be dedicated to a specific task. Thus, the reference to one or more machine learning models can represent any suitable number of machine learning models configured to perform any suitable number of operations.

[0040] In some embodiments, the machine learning model can be agnostic with respect to the architecture, manufacturer, model, and / or style type of the object being evaluated. That is, the user may not need to manually provide any identifying information about the object being evaluated in order to allow the machine learning model to focus its calculations based on the known properties of the object. Instead, the user can simply open the application and begin photographing an image or video of the object without entering initial information regarding the type of object, type of architecture, manufacturer, model, or style. In other embodiments, the type of object being evaluated (e.g., house, window, fence, etc.) may be specified, or some other information may be obtained from the user to determine which machine learning model is to be utilized.

[0041] In addition to the image data, information related to the real estate and / or the customer may also be manually input by the user or obtained from a source remote from the user device 100. This information may be provided before, during, or after the image data is collected. The information may include, but is not limited to, customer identification information, requests for the type of service (e.g., insurance claim, appraisal, insurance underwriting, etc.), indication of the type of object being evaluated, number of unique objects being evaluated, and indication of the parameters or features of the object being evaluated. In some embodiments, the application may request that the user provide additional information or image data related to the real estate and / or the customer based on the analysis of the image data collected at 310. However, the exemplary embodiments may be utilized in a wide variety of different types of use cases, and the image data, real estate information, and / or customer information may be provided by the user in any suitable manner and may include any suitable type of information that can be used to evaluate the condition of the real estate.

[0042] In some examples, each of one or more machine learning models may receive the same input data (e.g., image data collected by the user device 100, image data collected by another source, customer information, real estate information, region-specific information, etc.). In other examples, different machine learning models may receive different input data. For example, a first set of image data may be provided to one machine learning model, a second different set of image data may be provided to another machine learning model, or the output of the first machine learning model may be included as part of the input data provided to the second machine learning model.

[0043] At 310, the objects shown in the image data are identified using AI. To provide a general example, consider a scenario where photos and / or videos show the exterior of a house from multiple locations around the perimeter of the house. The image data may be input into one or more machine learning models configured to identify different types of objects, such as houses, exterior walls, windows, doors, gutters, etc. When identifying an object within the image data, the one or more machine learning models may determine the position of one object relative to another object, a light source, and / or coordinates.

[0044] In some embodiments, multiple machine learning models (e.g., classifiers) may be used, and each machine learning model is trained to identify one or more specific types of objects related to real estate. Each machine learning model may receive all of the available image data, or each machine learning model may receive a subset of the image data determined to be relevant to that respective machine learning model. In other embodiments, a single machine learning model may be used to perform the identification at 310, or as described above, a single machine learning model may perform all of the operations necessary to generate an assessment of the condition of the real estate (e.g., 325).

[0045] At 315, the number of unique objects shown in the image data is determined using AI. Each instance of real estate may be composed of any number of objects. Continuing with the above example, the exterior of the house may include multiple exterior walls, multiple windows, multiple doors, and multiple sections of gutters. Thus, the image data may include multiple photos and / or videos each showing the same object. To ensure that each unique object shown in the image data is described only once, computer vision techniques may be used to count and track each unique object shown in the image data.

[0046] At 320, the damage state is determined for one or more unique objects shown in the image data using AI. For example, the image data may be input into one or more machine learning models configured to determine whether the object is damaged. In some embodiments, the damage state may be determined for each unique object shown in the image data. In other embodiments, only a subset of the unique objects shown in the image data may be evaluated for damage. When determining the damage state, the one or more machine learning models may also determine the extent of damage to the object, the location of the damage to the object, possible repair methodologies including whether the object should be repaired or replaced, the number of labor hours that may be involved in the repair, and the estimated cost of the repair. The estimated total cost of the repair may include costs such as, for example, labor costs, material costs, part costs, scaffolding costs, disposal costs, permit costs, and other costs associated with the repair.

[0047] At 325, an assessment of the condition of the real estate is generated using AI. In some embodiments, the machine learning model may output an assessment of the real estate as a whole. In other embodiments, the assessment of the real estate as a whole may be derived based on the outputs of multiple machine learning models. The content of the assessment may vary depending on the use case, and examples are provided in detail below.

[0048] One or more machine learning models may also be trained to determine the physical real-world dimensions of an object of interest. For example, one or more machine learning models may determine the height and width of a window or door, the height and length of one or more sections of a fence, the dimensions of a house, or the area of a room. The dimensions of the object of interest may be used for purposes such as determining a damage state, performing an assessment at 325, or any other suitable purpose related to the assessment of the state of a real estate property. Additionally, one or more machine learning models may be trained to determine the materials that make up the object of interest. For example, one or more classifiers may determine that a fence is made of polyvinyl chloride (PVC), vinyl, paving material, cinder block, or chain link. In another example, one or more machine learning models may determine that a floor is carpeted, tiled, or made of hardwood. In a further example, one or more machine learning models may determine that an exterior wall is composed of brick, wood, aluminum siding, cedar plank, or vinyl. The material composition of the object of interest may be used for purposes such as determining a damage state, performing an assessment at 325, or any other suitable purpose related to the assessment of the state of a real estate property.

[0049] In some embodiments, multiple machine learning models may be used, where a first set of one or more machine learning models may be trained to perform the identification at 310, a second set of one or more machine learning models may be trained to determine the damage state at 320, and a third set of one or more machine learning models may be trained to perform the assessment at 325. In other embodiments, a single machine learning model may be used to perform multiple tasks, or as described above, a single machine learning model may perform all of the operations to generate an assessment of the state of a real estate property (e.g., 325).

[0050] Before or during method 300, image segmentation may be performed on one or more images or video frames to identify segments of an object or to segment an object. In some exemplary embodiments, image segmentation can be used to identify objects that occlude the view of an object (such as a lamp that occludes the view of a portion of a wall object or a bag that occludes the view of a table). This information can be used in various ways. In one example, the information can be used to request the user to move the occluding object and take a new image. In another example, an application and / or an AI system can remove the occluding object from the image, for example, using AR or VR technology. Segmentation performed on the image data can be used by one or more machine learning models to better identify objects and / or damage that would otherwise be difficult. Thus, in some embodiments, the image data may further include a plurality of sets of image segments, where each set of image segments can be generated from a single image or video. However, exemplary embodiments do not require the use of image segmentation. Any suitable computer vision technology can be utilized to assist the machine learning models in performing their configured tasks on the image data.

[0051] In some embodiments, an application running on user device 100 may perform an assessment of the condition of a real estate. This assessment may then be sent to server 210, or any other suitable remote location for future use by the business entity. For example, the application may display an initial estimated repair cost on user device 100 and then provide data (e.g., image data, information manually entered by the user, estimated repair cost) collected and derived on user device 100 to server 210. Thereafter, any of a variety of different services may be provided by the business entity using the data collected and / or derived on user device 100. In other embodiments, user device 100 may collect image data and then provide it to server 210 where the assessment is performed. The assessment may then be provided to the customer via user device 100 or in any other suitable manner. In any scenario, the data collected and / or derived by user device 100 may be utilized by the business entity to provide any of a variety of different services.

[0052] In one exemplary use case, the application may be used to provide an overall or partial initial estimate for repairing an object damaged after an event. To provide an example, consider a scenario where an event has damaged the exterior of a house within the context of method 300. The user uses camera 120 of user device 100 to take photos and / or videos of the exterior of the house from various points along the perimeter of the home. The image data may be input into one or more machine learning models, and an assessment of the condition of the real estate (e.g., 325) may be provided to user device 100.

[0053] In some embodiments, the assessment may identify the number of damaged objects and provide an estimated repair cost. For example, after a storm, a machine learning model may identify that 2 out of 10 windows are broken and the estimated cost to replace the broken glass. In addition to the image data, the estimate may be based on actions performed by the machine learning model such as determining the dimensions of the glass to be replaced and the estimated number of labor hours to replace the broken glass, but is not limited thereto. In another example, the machine learning model may identify that a section of a fence has been damaged and the estimated cost to replace the damaged section of the fence. In addition to the image data, the estimate may be based on actions performed by the machine learning model such as determining the material of the fence, the dimensions of the section of the fence to be replaced, and the estimated number of labor hours to replace the broken glass, but is not limited thereto.

[0054] To provide another example, within the context of method 300, consider a scenario where an event has caused damage inside a residence. The user uses the camera 120 of the user device 100 to take pictures and / or videos of the inside of the house. The image data may be input into one or more machine learning models, and an assessment of the condition of the real estate (e.g., 325) may be provided to the user device 100.

[0055] In some embodiments, the assessment may identify the number of damaged objects and provide an estimated repair cost. For example, after an event causing water damage, a machine learning model may identify water damage to the inner wall, determine whether and / or how the wall can be repaired, and determine an estimated cost for repairing or replacing the wall. In addition to image data, the estimates may be based on actions performed by the machine learning model such as identifying the material of the wall, determining the dimensions of the wall, and determining the estimated number of labor hours. In another example, the machine learning model may identify fire and / or smoke damage to one or more objects, the estimated cost for replacing damaged objects, and the estimated cost for repairing the damage. In addition to image data, the estimates may be based on actions performed by the machine learning model such as identifying the material of the damaged object, determining the dimensions of the damaged object, and determining the estimated number of labor hours for replacing and / or repairing the damaged object.

[0056] Alternative assessments can be made, including recommendations on whether to file an insurance claim based on an estimated cost value exceeding a threshold cost value, or an analysis of the impact of the claim on future insurance premiums compared to repair costs. For example, if a claim is likely to significantly increase insurance premiums, it may not be economically meaningful for a user to replace a single broken window. Additional assessments may include recommendations on whether the building is habitable in its current state, or whether the damage suffered by the building is severe enough to prevent the life or occupancy of the building prior to repair.

[0057] From the perspective of the entity providing the service, the assessment may indicate whether an on-site inspection is being performed on the real estate. Thereby, if there is an influx of assessments in response to an event, the entity may be able to deploy employees (e.g., inspectors, adjusters, etc.) more efficiently. For example, the entity may be able to quickly identify customers with homes initially assessed as habitable and deploy inspectors as soon as possible.

[0058] In some embodiments, aerial imaging can be used in addition to the images collected by user device 100. For example, satellite images, image data captured by drones, or image data captured during flight over a property that depict an object before and / or after an event can also be provided to one or more machine learning models. This type of imaging can be used to evaluate the condition of a home's roof, crops, land reclamation, equipment, wiring, roads, or any other aspect of a property visible from the air.

[0059] Exemplary machine learning models can also consider the characteristics of the event that caused the damage and determine whether the damage identified in the image data is consistent with the event. If the damage to an object is determined not to be consistent with the event or the cause of the damage, the damage may not be considered in assessment 325. For example, a storm may cause a tree to fall, thereby damaging a portion of a fence. The fence may also have sections where the paint has peeled or rusted. One or more machine learning models may determine that the damage caused by the tree is likely to have been caused by the storm, but the paint / rust damage is likely to have existed prior to the storm. Thus, the initial estimated value of the insurance claim performed during assessment 325 may not consider the cost of repairing damage determined to be inconsistent with the event.

[0060] In another exemplary use case, the application can be used to provide an initial assessment without the involvement of a professional assessor. Compared to the estimation of damages for insurance claims, an accurate assessment may need to consider smaller sized damages and other less visually obvious factors. For example, the inspected real estate may not have been significantly affected by any particular event (e.g., storm, flood, fire, accident, etc.) recently, and thus the image data may not show significant structural damage consistent with natural disasters. However, factors such as rust, paint condition (e.g., discoloration, peeling, flaking, bubbling, etc.) and surface condition, among others, can affect the assessment of the real estate. It has been identified that video data can enable the application to evaluate less obvious factors when identifying smaller sized damages and assessing the condition of the real estate. Video data provides benefits to the use case of performing an initial assessment of the real estate, although exemplary embodiments can utilize any suitable type of image data, including infrared or ultraviolet image data. Additionally, the AI system can use other types of data alone or in combination with the image data to identify damages, such as audio recordings of object movement, oral or text descriptions regarding damages made simultaneously with the image data. Other examples of non-image data (e.g., temperature, moisture, etc.) were also provided above.

[0061] In this type of use case, the damage state determination (e.g., 320) and / or assessment 325 may also include an assessment of minor damage or cosmetic damage. These assessments can be used in situations other than repair, for example, to assist in the assessment of a house to determine the recommended property price, or for use in insurance underwriting. In another example, these assessments can be used for the management of rental property, where, using an exemplary embodiment, the rental unit can be assessed before, during, and / or at the end of a rental contract. In some embodiments, the assessment can be utilized to initiate and / or terminate a smart contract. Thus, the minor damage assessment can optionally be used together with other information about the real estate to determine the overall condition of the real estate.

[0062] In this type of use case, the assessment (e.g., 325) may include the paint condition or surface condition of one or more unique objects. For example, one or more machine learning models may be able to identify the paint condition of a unique object. The paint condition may be output as a score or a pre-set identifier (e.g., fading, flaking, bubbling, scratching, satisfactory, like new, etc.). Similarly, one or more machine learning models may be able to identify the surface condition of a unique object, and the surface condition may be output as a score or a pre-set identifier (e.g., fading, chipping, cracking, scratching, weathering, satisfactory, like new, etc.). Additionally, the assessment may include the rust condition of one or more unique objects. For example, one or more machine learning models may be able to identify the severity of corrosion for each unique object. The rust condition may be output as a score or a pre-set identifier. Further, the application may indicate whether it is possible to treat the rust or whether the object needs to be replaced.

[0063] The evaluation provided by method 300 can be used as part of an end-to-end claims process. For example, in some use cases, an insurance company may present an initial payment to a user based on the evaluation performed at 325. This allows the user to receive compensation from the entity autonomously without a human employee having to confirm or approve the monetary presentation. However, the user may provide additional information later if additional funds are needed. The additional information may be evaluated and the evaluation may be updated. In another use case, the entity may identify a contractor who can perform the repairs and / or corrections of the damages identified in the evaluation.

[0064] The application may generate an inspection report for the real estate that includes insights generated by AI, which includes an assessment of the overall condition of the real estate (excellent, good, fair, poor, etc.). Additionally, or alternatively, the inspection report may include an estimated total cost to repair the real estate to a higher level of condition (e.g., to convert an overall poor condition to good). Further, the report may include a selection of images derived from the image data that show the overall condition of the real estate. Optionally, the inspection report may provide more details regarding the various parts of the real estate that require repair, including the proposed repair work and the cost components of the repair work. The report may include images taken from a video that show the images that the AI has determined most clearly display the identified damages.

[0065] Exemplary embodiments may also be used to track the history of real estate. For example, at a first point in time, the user device 100 may be used to perform a real-time inspection of the real estate. The application may output a signature indicating the state of the real estate at the first point in time, such as an evaluation performed by a machine learning model on image data showing one or more objects. The signature may include, but is not limited to, information such as the type of damage present, the location of the damage, the severity of the damage, the painting condition, the surface condition, the external condition, the internal condition, and the presence and severity of rust. The signature may be stored in a secure database such as a distributed blockchain-based database.

[0066] When a new inspection is performed, the signature may be updated. For example, a real-time inspection of the real estate may be performed using the user device 100 at a second point in time. The signature may be processed by one or more trained models to identify different types of preventive maintenance that can be performed on one or more objects. In addition, the signature may provide a clear history of the real estate that can be used to evaluate the current value of the real estate.

[0067] In some embodiments, the application may determine the value of an undamaged version of one or more objects shown in the image data. This determination may be based on one or more machine learning models, existing pricing grades, a lookup table stored in the user device 100 or the remote server 210, or any other suitable resource. The application may reduce the value derived for one or more undamaged objects based on an evaluation of the state of the real estate to generate an estimated value. For example, factors such as, but not limited to, the geographical location of the real estate, the external condition of the building, the internal condition of the building, the painting condition, the surface condition, and the presence and severity of damage.

[0068] Instead of, or in addition to, reducing the value of an undamaged object, the application may generate an estimated cost for fixing one or more aspects of one or more objects. This may also include an estimate of how fixing one or more objects may improve the estimated valuation of the real estate. To provide one general example, one or more machine learning models may identify that the paint on one or more objects is decorative, the appliances are not energy efficient, there is water damage in multiple interior locations, and the fence surrounding the property has multiple damaged sections. The application may consider these issues identified from the image data and reduce the value derived for the undamaged version of the real estate to generate an estimated value (X). Additionally, the application may estimate the cost of repairing the faded paint (A), the cost of replacing the appliances (B), the cost of repairing the water damage (C), and the cost of repairing the damaged fence (D). The application may further estimate that by repairing the faded paint, the estimated value (X) may increase by an amount (U), by replacing the appliances, the estimated value (X) may increase by an amount (V), by repairing the water damage, the estimated value (X) may increase by an amount (W), and by repairing the damaged fence, the estimated value (X) may increase by an amount (Z). The examples provided above are for illustrative purposes only and are not intended to limit the exemplary embodiments in any way.

[0069] As described above, an exemplary embodiment may enable a user to perform a real-time inspection of a real estate property using the user device 100. The user may collect image data using the camera 120 of the user device 100. The application may include one or more machine learning models for determining which objects were captured within the image data. The one or more machine learning models may be executed on the user device 100 while the user is taking a photo and / or while recording a video. Thus, the application may provide a user interface for identifying what is currently being filmed in the video, and an overlay that is updated to guide the user in tracking their progress and / or collecting sufficient image data to perform an assessment of the real estate property. The dynamic feedback that may be provided to the user will be described in more detail below.

[0070] Figure 4 shows a method 400 for collecting image data to perform a real estate inspection using an AI-based application, according to various exemplary embodiments. Method 400 is described with respect to the user device 100 of FIG. 1, the system 200 of FIG. 2, and the method 300 of FIG. 3.

[0071] The following description of method 400 provides an overview of how an application may process image data, interact with a user, and generate an assessment of the condition of a real estate property.

[0072] At 405, the user device 100 launches an application. For example, the user may select an application icon shown on the display 125 of the user device 100. After launch, the user may interact with the application via the user device 100. To provide a general example of a conventional interaction, the user may be presented with a graphical user interface that provides any of a variety of different interactive features. The user may select one of the features shown on the display 125 via user input entered on the display 125 of the user device 100. In response, the application may provide a new page that includes additional information and / or interactive features. Thus, the user may move within the application by interacting with these features and / or navigating between different application pages.

[0073] At 410, the application receives image data captured by the camera 120 of the user device 100. The application may request that the user capture image data of different objects of a real estate. For example, the user may be prompted to record videos of the exterior of a building, the interior of a building, a fence, a yard, crops, shrubs, equipment, etc. According to some exemplary embodiments, method 400 may be a continuous process in which one or more segments of the video are provided downstream of one or more machine learning models while the user is actively pointing the camera at an object to take a photo or during video recording. Thereby, the application may be able to provide dynamic feedback to guide the user in recording a video of sufficient quality to perform an evaluation of the real estate.

[0074] At 415, the application determines whether the image data meets a predetermined criterion. The predetermined criterion may be based on image quality or video quality. In some embodiments, the predetermined criterion may be based on data collected from other components of the user device 100.

[0075] As an example, the application can identify that one or more images or one or more video segments are blurry and lack sufficient sharpness, that there are areas that feel dazzling, or that the lighting is insufficient. Exemplary embodiments can evaluate any suitable type of quality metric associated with the image or video to determine whether the image or video lacks sufficient clarity. Clarity can be affected by the way the image or video is recorded. For example, when an image is captured or during the recording of one or more segments of a video, if camera 120 moves in a particular manner, the content may be too blurry and it may be difficult to identify the objects captured in the image data. In some embodiments, instead of or in addition to a quality metric, a predetermined criterion may be based on the speed parameter of user device 100, the acceleration parameter of user device 100, and / or any other suitable type of motion-based parameter of user device 100 that exceeds a threshold. This can include applications that collect data from other internal components of user device 100 (such as accelerometers, gyroscopes, motion sensors, etc.), record one or more video segments, and derive parameters associated with the motion of user device 100 while comparing the parameters to a threshold. If the parameter exceeds the threshold, the application may assume that one or more segments of the video were not recorded in a manner that is likely to provide video data that can be used to evaluate the condition of the property and thus is not of sufficient quality.

[0076] In another example, the application may identify that an image or more video segments were recorded from a perspective where the object is too close, too far from, and / or at an inappropriate camera angle. Exemplary embodiments may evaluate any suitable type of quality metric associated with the image or video to determine whether the image data was recorded from an appropriate perspective (e.g., distance, angle, etc.). In some embodiments, instead of or in addition to a quality metric, a predetermined criterion may be based on the distance parameter and / or camera angle parameter between the object and the user device 100 during the recording of one or more segments of the video.

[0077] If the predetermined criterion is not met, method 400 proceeds to 420. At 420, the application may generate a warning indicating to the user that the image data needs to be recorded in a different way. For example, if the application identifies that one or more video segments lack sufficient clarity, the warning may explicitly or implicitly indicate to the user that the camera is moving too fast, and the user should slow down and / or move the camera in a less erratic manner. In another example, if the application identifies that one or more video segments were recorded from an inappropriate distance or angle, the warning may explicitly or implicitly indicate to the user that the camera is too close to the object, too far from the object, or is configured at an inappropriate angle. The warning may be a visual warning provided on the display 125 of the user device 100 and / or an audio warning provided by the audio output device of the user device 100.

[0078] When returning to 415, if the image data does not meet the predetermined criteria, method 400 proceeds to 425. At 425, the application identifies one or more objects shown in the image data. At 430, the application updates the overlay displayed on user device 100. From the user's perspective, display 125 may display an interface that includes the overlay and video data captured by camera 120. As will be described in more detail below, the overlay may be updated to indicate the position of user device 100 relative to the object during recording of the image data, or to indicate the amount of image data collected and / or to be collected for evaluation of the condition of the real estate, or to provide any other type of information that may guide the user in recording video necessary for evaluation of the condition of the real estate.

[0079] As described above, the application may provide dynamic feedback to the user to assist the user in capturing image data of sufficient quality for the user to appropriately capture the object and / or evaluate the condition of the real estate. An example of dynamic feedback is the warning generated at 420. Another example of dynamic feedback is the dynamic overlay referenced at 430.

[0080] The dynamic feedback may indicate the need to move the camera closer to or further away from the area of potential damage based on a damage assessment implemented using a machine learning model. Information related to the need to move the camera closer or further away may be based on information obtained from user device 100 using either a LIDAR sensor or any of the other sensors described above. The application can provide a general indication that the camera should be moved closer or further away, or provide the recommended distance from the area of interest. The application may also recommend or require that additional images be captured from various angles as described above, based on the damage assessment.

[0081] Furthermore, the application may display information indicating the need for a closer image or video of the area of interest. In some embodiments, the display may indicate the area of interest using a bounding box, crosshair arrows, or any other suitable means on an already acquired image or a portion thereof. When the area of interest is being captured, it may be indicated that the user can proceed to capture the image data as normal using visual, auditory, and / or tactile responses. Capturing the image data of the area of interest may include an image, a video, or a combination thereof. The video or image may be captured at a different resolution or using a different compression method than other image data.

[0082] The application may require the user to capture image data of an object from multiple different viewpoints. For example, the application may require the user to capture one or more panoramic photos of the exterior of a house (e.g., 315) in order to enable tracking and counting the number of unique objects. In another example, the application may require the user to capture a video of the exterior of the house while moving around the perimeter of the house to enable counting the number of unique objects (e.g., 315 of method 300).

[0083] In some embodiments, the dynamic feedback may include a graphical display tracking the position of camera 120 relative to the object, and a score indicating the amount of the exterior of the object captured within the image data. The score may be indicated as a percentage or any other suitable quantitative value.

[0084] In some embodiments, while recording a video or panoramic photo, there may be a request from the user to slow down. The warning may further explain that moving too fast may cause the image data to blur. In other embodiments, augmented reality (AR) technology may be used to provide dynamic feedback that is more advanced than two-dimensional graphics. Exemplary embodiments may utilize any suitable graphic or visual component to guide the user in recording a video and / or collecting data to provide the user with dynamic feedback for evaluating the condition of the real estate.

[0085] The application may also obtain data regarding the height of the camera 120 from the user device 100 during video recording. Using the height calculation, the application may increase or decrease the height of the camera 120 to guide the user to capture additional information. As described above, the video may be analyzed to determine the distance of the camera 120 from the object. Alternatively, this distance may be based on information obtained from sensors such as, for example, a light detection and ranging (LIDAR) sensor embedded in the user device 100. Also, information from other types of sensors may be used to determine distances such as ultrasonic, infrared, or LED time-of-flight (ToF). The application may also change the angle of the video to determine whether the application needs to improve its ability to evaluate the condition of the real estate. The angle may be adjusted in the vertical plane and / or the horizontal plane to provide, for example, an image perpendicular to the object, an image at the same height as the midpoint of the height of the object but not perpendicular to the side, or an image from an angle above the object.

[0086] At 435, the application determines whether sufficient image data has been collected to evaluate the condition of the real estate. If more image data is needed to evaluate the condition of the real estate, the method 400 returns to 410, where one or more images or one or more segments of a video are received by the application.

[0087] In some embodiments, the application may prompt the user to obtain additional video or images of a particular object based on conditions identified from the image data. For example, if damage is detected on the exterior wall of a house, the application may request that the user collect image data from the interior part of the house that aligns with the damaged exterior part while the user has it open. In another example, if a damage matching a certain type of event is identified, the application may request that the user capture additional video of other objects that may similarly be damaged by the same type of event.

[0088] If more image data is not required to evaluate the condition of the real estate, method 400 proceeds to 440. At 440, the application generates an evaluation of the condition of the real estate. This may be similar to 325 of method 300.

[0089] In further embodiments, one or more machine learning models may generate a confidence value associated with the evaluation of the real estate. The system may identify an object with a confidence level below a particular level for the evaluation and prompt the user to record additional video of that object. The dynamic display may indicate which objects are currently visible by camera 120 with an appropriate level of confidence. The dynamic display may further indicate which objects have a damage evaluation with a predetermined level of confidence in the images captured earlier in the session. Thereby, the user can distinguish which objects need to be captured to evaluate the condition of the real estate.

[0090] In some embodiments, the application may restrict the way in which a video of the real estate is recorded by the user to ensure that the objects shown in the image data are associated with the same real estate. For example, the application may require the user to take an image or video that shows the entire portion of the object. This acts as a security feature and can ensure that the image data contains multiple objects within the same image or video so that the unique objects shown in the image data can be countered and tracked. In another example, the application may require a continuous video that includes an identifier specific to the object (e.g., an address, a front door, a mailbox, etc.). This can confirm that the video has not been edited in a manner that could change the assessment of the real estate. In another example, when multiple video clips are used, the application may require that each video clip shows the same object. Further, the application can ensure that the objects shown in the first and second video clips are the same object by comparing the color, dimensions, and / or materials of the object in the first video clip with the color, dimensions, and / or materials of the object in the second video clip.

[0091] In some embodiments, the AI system is used to create a floor plan inside a two-dimensional or three-dimensional structure. This can be done by an AI system that analyzes the image or, alternatively, by asking the user to identify the corners of the room in another way. The distance measurement can be done by any of the methods described above. The creation of this 2D or 3D model can be done using visual information obtained from the user device, augmented with aerial, satellite, or drone images. Further, optionally, it may be augmented based on other information such as design drawings, floor plans, and other previously stored information about the real estate.

[0092] The AI system can determine the identifiability of an object using a machine learning model and the other methods described above. Similarly, using technologies such as GPS, triangulation methods from images, and triangulation using an accelerometer, the location of these objects can be determined, including their optional boundaries, which can be recorded and optionally identified with respect to a 2D or 3D model, either inside or outside the real estate.

[0093] The information that can be identified using a machine learning model for these objects includes the identification of the object's components, the object's materials, the object's design, the object's type, and the object's dimensions. Using a machine learning model, it is possible to identify whether the object is damaged, the type of damage (e.g., water damage, cracks, dents, warping, and other classifications of damage related to the object), and to determine the associated repair work or necessary mitigation work.

[0094] The AI system can include a machine learning model that identifies the types of damage that can make a building uninhabitable or dangerous and can provide that information to the user. Additionally, the AI system may include a model that identifies the presence of dangerous objects within the property, such as the likelihood of the presence of a particular type of mold, and can similarly warn the user of those concerns.

[0095] The AI system can be configured to create an overall report of the real estate and related objects that can include some or all of the information identified by the AI system, and can also include evidence related to that aspect of the report, such as still images related to the damage determination, videos related to the determination, audio information related to the determination, and / or other information described above, such as infrared images of the moisture or location of the object. This report can be in any of a single document, an interactive computer report, an augmented reality or virtual reality tour, or any other method for communicating information to either the user or a remote party (such as an insurance company, a repair company, etc.).

[0096] This report may include recommendations for immediate response and recommendations for actions that may occur later. Immediate response items can be determined based on damage that may lead to additional damage if not corrected promptly, or measures that may need to be implemented promptly for safety purposes. The report may also specify the materials required for countermeasures, such as dehumidifiers and fans to reduce moisture, and personal protective equipment in case of toxic mold. The identified actions can be both temporary and permanent. For example, the system can identify that there is a hole in the roof and instruct to place a plastic tarp over the hole until it can be repaired. These actions can be further prioritized and identified based on the prediction of local weather. For example, in case of a storm, a temporary cover for the openings in the structure is prioritized.

[0097] This report can be provided to users such as homeowners or repair technicians for verification and to determine consistency with the evaluation of the AI system. If there is an error in the AI system's determination, the user can identify the discrepancy, and the AI system can request to collect additional information regarding the determination.

[0098] Through the use of AR or other visual means, the AI system can provide real-time evaluation of any of the items discussed above. For example, it can identify lamps, water damage, holes in the roof, damaged fences, and undamaged doors. The AI system can also enable the user to identify in real-time any discrepancies with the AI system's decision and allow for the immediate collection of other relevant information.

[0099] The AI system can require the user to identify damaged structures, objects, components, or regions on the display by surrounding, highlighting, or selecting the components. This can be done, for example, via the use of an AR or VR display. Additionally, the system may require the user to identify areas sensitive to odor, humidity, or airflow that may not be readily apparent to the user device 100. This may refer to the user identifying areas of interest in a 2D or 3D model built for real estate purposes.

[0100] Before information leaving the user device is delivered to other components of the AI system, the AI system can cause the device to delete certain categories of personally identifiable information (such as an individual's face), or information indicating religious or political leanings. This can be done for several reasons, including the desire to avoid bias in insurance claim compensation for improper reasons.

[0101] Regarding the accuracy of the information collected, additional verification can be performed, such as comparison with existing images of the structure, including Google Street View, satellite images, or images collected for a given address. Additionally, information regarding the location from government files such as property databases can be compared with the information collected to confirm that the real estate is the same as that recorded to be at that location. This can be done to avoid cases of simple human error or fraud. In the case of multiple sessions, additional geolocation data can be captured for the various images, videos, and other data collected to confirm that all the data collected is from the same location. Additionally, the images can be compared with images collected by the insurance company at an earlier time, such as at the start of coverage, in relation to earlier claims, or can be taken at the time of structural changes to the real estate.

[0102] The application may also be configured to request image data from the user prior to a predicted event. For example, weather information may be used to predict the occurrence of an event that could cause damage to the user's real estate. The application may request that the user collect image data prior to the event and provide reference image data that can be compared to the image data taken after the event occurs. In some exemplary embodiments, the image data may be obtained prior to the weather event based on satellite, aerial, drone, or ground-based images obtained from a third-party source.

[0103] In another embodiment, the application may autonomously request that image data be obtained by the user after the event occurs. For example, the application may utilize weather information to predict the occurrence of an event that could cause damage to the user's real estate. Additionally, the application may determine whether any policyholder is near the event and / or whether any policyholder owns real estate with characteristics that are vulnerable to the effects of damage that could be caused by the type of event. The application may autonomously send a notification to the user that meets this criteria to collect image data because there is a high likelihood that damage has occurred to the user's real estate.

[0104] In some exemplary embodiments, AI and / or machine learning (ML) techniques can be used to create classifiers and models that can predict potential damage to structures from near-future weather events. Information used by a classifier or model to predict damage can include satellite imagery (including Doppler radar, infrared, visible), predicted wind speed and direction, storm surges, tides, and other weather-related data regarding the approach of hurricanes, typhoons, tropical depressions expected to strike a region over a period of hours to days. These classifiers and models can be trained based on past weather data and information about the structures, such as the damage status of the structures, the type of structures, the materials used in construction, the location of neighboring objects such as trees, rivers, shorelines, and the age of the structures. Next, the classifiers and models can be used to predict damage to structures from impending weather events based on this same type of data (weather data and information about the structures). Similar classifiers and models can be trained based on structures and historical information about local objects and floods resulting from various events affecting the water level in the region to predict potential damage from short-term floods.

[0105] Using these weather-related classifiers and models, it is also possible to determine whether there are steps that can be taken to evaluate existing real estate and structures and reduce potential damage from future weather events. For example, an application can model the likelihood of damage to a structure from future possible weather events (based on past probabilities and trends) based on the current characteristics of the structure, and then evaluate the possible damage based on changes to the structure (such as changes to roofing materials, types of fences, removal of trees, addition of trees and windbreaks, reinforcement of riverbanks, etc.). The cost expected to make the changes can also be calculated. Based on this information, recommendations can be made based on a comparison of the expected reduction in the cost of damage and the expected cost of the changes. Additionally, the information can be provided to the homeowner to enable a series of actions to be determined taking into account any other considerations (such as values not displaced by storm damage, etc.).

[0106] The determination of the impact of a meteorological event can be made not only for one structure, but also for all or a subset of the structures within a region. Based on this information, relevant businesses can make preparations. For example, an insurance company can utilize potential damages for in-house purposes. Construction companies and building material companies can predict the need for specific materials and make the preparations necessary to safely and timely deliver the materials to the area.

[0107] Furthermore, these predictions regarding the impact on a region based on individual structures within the region can be used by government agencies, support or relief organizations, or other agencies to determine the likelihood of the impact of a meteorological event (imminent events or statistical analysis of likely events), and this information can be used to formulate plans for future meteorological disasters. This plan can include a combination of pre-impact evacuation, temporary housing planning, or the provision of repair and reconstruction work after the impact. The area to be evaluated can be of any scale, from a few local structures to villages, towns, cities, postal delivery areas, counties, states, prefectures, or national levels. The number of structures to be evaluated can be less than 10, less than 100, less than 1000, less than 10,000, less than 100,000, or millions of structures (if less). By conducting this evaluation of the impact of a meteorological event based on the actual structures in the region, the accuracy of the plan can be significantly improved.

[0108] Modeling can be based on statistical sampling of typical structures and their characteristics in areas where data for each structure is not available. The structures to be evaluated are not limited to the housing or other structures discussed above and can also include infrastructure such as roads, bridges, railways, dams, power plants, water treatment facilities, warehouses, airports, and harbors. Furthermore, this evaluation can include the evaluation of changes or modifications to structures as described above, but is conducted on a larger scale for multiple structures. This can help any of the businesses mentioned above to determine a proactive and responsive approach to meteorological events such as floods, hurricanes, tornadoes, tropical depressions, and droughts.

[0109] One skilled in the art will understand that the above exemplary embodiments can be implemented in any suitable software or hardware configuration, or a combination thereof. Exemplary hardware platforms for implementing the exemplary embodiments may include, for example, Intel-based platforms with compatible operating systems, Windows OS, Mac platforms, and mobile devices with operating systems such as MAC OS, iOS, and Android. Exemplary embodiments of the above methods may be embodied as software including lines of code stored on a non-transitory computer-readable storage medium, which may be executed on a processor or microprocessor when compiled.

[0110] This application describes various embodiments having different features in various combinations. However, one skilled in the art will understand that any of the features of one embodiment can be combined with the features of other embodiments in any manner that is not specifically excluded or functionally or logically inconsistent with the specified functions of the disclosed embodiments or the operation of the device.

[0111] The use of information that can identify an individual should generally comply with privacy policies and practices recognized as meeting or exceeding industry or government requirements for maintaining user privacy. In particular, information data that can identify an individual needs to be managed and handled to minimize the risk of unintended or unauthorized access or use, and the nature of the approved use should be clearly indicated to the user.

[0112] It will be apparent to those skilled in the art that various modifications can be made to the present disclosure without departing from the spirit or scope of the present disclosure. Accordingly, the present disclosure is intended to cover modifications and variations of the present disclosure when they fall within the scope of the appended claims and their equivalents.

Claims

1. A method comprising: Receiving image data; Using a first set of one or more machine learning models to identify a plurality of objects related to a real estate shown in the image data; Determining the number of unique objects shown in the image data; Using a second set of one or more machine learning models to generate an assessment of the state of the real estate.

2. The method according to claim 1, wherein the first set of one or more machine learning models and the second set of one or more machine learning models are the same set of one or more machine learning models.

3. Generating the assessment of the state of the real estate further comprises: Determining a damage state for at least one unique object.

4. The method according to claim 3, wherein the damage state includes at least one of a location of the damage or a severity of the damage.

5. The method according to claim 3, wherein the damage state includes at least one of an estimated repair cost, a repair method, and an estimated number of labor hours for performing the repair.

6. Generating the assessment of the state of the real estate further comprises: Determining a physical dimension of at least one unique object.

7. Generating the assessment of the state of the real estate further comprises: Determining one or more materials for at least one unique object.

8. The method according to claim 1, wherein the image data includes at least one of a satellite image, an image taken by a drone, or an image taken during an aerial flight of the real estate.

9. Generating feedback to be displayed on a user device, wherein the user device captured at least a portion of the image data, and the feedback is provided to an interface including the feedback and a field of view of a camera of the user device. The method according to claim 1, further comprising.

10. The method according to claim 9, wherein the feedback includes a warning configured to indicate to the user a request to change a distance or an angle between the camera and the real estate.

11. The method according to claim 10, wherein the request to change the distance or the angle is based on the presence of an object, a region of interest for one or more objects, or a damage region for one or more objects.

12. The method according to claim 9, wherein the feedback includes a warning configured to indicate a request to the user during recording of a video that changes the way the user moves the camera.

13. Receiving predicted weather-related data; Determining predicted weather-related damage to the real estate using a third set of one or more machine learning models; The method according to claim 1, further comprising.

14. The method according to claim 1, further comprising constructing a two-dimensional (2D) or three-dimensional (3D) model of the real estate based at least on the image data. The method according to claim 1.

15. The method according to claim 14, wherein the 2D model or 3D model is constructed using augmented reality (AR) or virtual reality (VR) technology.

16. The method according to claim 14, further comprising requesting feedback from a user regarding the 2D model or 3D model, wherein the feedback relates to identifying a region of interest in the 2D model or 3D model. The method according to claim 14.

17. The method according to claim 1, further comprising receiving feedback from a user regarding the evaluation of the state of the real estate. The method according to claim 1.

18. The method according to claim 1, further comprising receiving non-image data related to the real estate, wherein the evaluation of the state of the real estate is generated based on the non-image data. The method according to claim 1.

19. The method according to claim 1, further comprising segmenting the image data to identify one or more of the plurality of objects or an object blocking one or more of the plurality of objects. The method according to claim 1.

20. The method according to claim 1, further comprising verifying the accuracy of the image data based on an image received from a third-party source. The method according to claim 1.

Images