House state provision device and method

JPWO2023167017A5Pending Publication Date: 2024-11-11
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024504613
Authority / Receiving Office
JP · JP
Patent Type
Applications
Priority Date
2023-02-17
Filing Date
2023-02-17
Publication Date
2024-11-11

AI Technical Summary

Technical Problem

Existing disaster situation understanding systems face challenges in accurately extracting and matching feature points from 3D aerial images before and after a disaster, especially due to changes like new construction or reconstruction, making it difficult to assess damage effectively.

Method used

A house status providing device and method that acquires aerial images, matches them with stored maps, extracts external shape information, and uses a learning model to classify house status into easily understandable categories, such as completely or partially destroyed, and superimposes this information onto the images for clear display.

Benefits of technology

Enables quick and accurate recognition of house status post-disaster, allowing for effective damage assessment and association of results with individual houses in aerial images, improving the understanding of disaster impact and facilitating relief efforts.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

Provided are a house state provision device and method which are capable of satisfactorily recognizing the state of individual houses in an area in which a disaster occurred, and associating the recognition results with individual houses in an aerial image. A processor (200) of this house state provision device acquires an aerial image (IM) capturing a disaster-stricken area, and on the basis of the aerial image (IM) and a map (MP) stored in a database (220), matches the map (MP) to the aerial image (IM). From the matching process results and region information for a house included in the map (MP), external appearance information for the house in the aerial image (IM) is acquired, and a house image (H) showing the house is extracted from the aerial image (IM) on the basis of the external appearance information for the house. The damage status is recognized respectively for the houses on the basis of the extracted house images (H), and from the recognition results, the houses are classified into one of a plurality of classes, namely safe, half-destroyed, and totally destroyed. The classified class is associated with the respective houses in the aerial image (IM). The information indicating the classification results is synthesized in the aerial image (IM), and is displayed to an image display device.
Need to check novelty before this filing date? Find Prior Art

Description

House status providing device and method

[0001] The present invention relates to a house status providing device and method, and more particularly to a technology for recognizing the status of individual houses shown in aerial images taken when a disaster occurs and providing the recognition results in an easy-to-understand manner.

[0002] A technology has been proposed for determining the status of individual houses (structures) shown in aerial photographs taken at the time of a disaster, such as partial or complete destruction, based on the aerial photographs taken at the time of a disaster (Patent Document 1).

[0003] The disaster situation assessment system described in Patent Document 1 uses a 3D camera mounted on an aircraft to capture three-dimensional ground images of a target area when a disaster occurs, and acquires imaging condition information including the three-dimensional position coordinates of the aircraft when capturing the three-dimensional ground images. Then, feature points are extracted from the three-dimensional ground images, and the three-dimensional coordinates of the extracted feature points are acquired based on the three-dimensional ground images and the imaging condition information.

[0004] On the other hand, the disaster situation assessment system stores three-dimensional ground images taken before a disaster occurs (in normal times) and the three-dimensional coordinates of feature points extracted from those three-dimensional ground images in a normal time storage device, and performs damage analysis by three-dimensionally comparing the three-dimensional ground images taken before and after a disaster, the three-dimensional coordinates of feature points extracted at the time of the disaster, and the three-dimensional coordinates of feature points during normal times, and detecting feature points in areas that have displaced after the disaster occurs.

[0005] In addition, the disaster situation assessment system has a database that stores area information and related information (place names, road names, building names, addresses, map codes, etc.) of structures existing within the target area, and identifies damaged structures by comparing the three-dimensional coordinates of characteristic points of the displaced area with the area information stored in the database, and obtains related information of the damaged structures from the database.

[0006] Then, related information about the damaged structures is superimposed on a map or three-dimensional ground image of the target area, and the damage status (partially destroyed / completely destroyed markers) is also superimposed.

[0007] International Publication No. 2013 / 051300

[0008] The disaster situation assessment system described in Patent Document 1 performs damage analysis by three-dimensionally comparing three-dimensional ground images taken before and after a disaster, the three-dimensional coordinates of feature points extracted at the time of the disaster with the three-dimensional coordinates of feature points stored in a normal storage device, and detecting feature points of areas displaced after the disaster. However, it is generally difficult to accurately extract feature points from captured images, and it is particularly not easy to accurately extract multiple feature points for each building captured in a three-dimensional ground image taken from the air. Furthermore, it is even more difficult to determine the correspondence between feature points extracted from three-dimensional ground images taken at the time of the disaster and feature points extracted from three-dimensional ground images taken in normal times and stored in a normal storage device, making it difficult to perform damage analysis effectively.

[0009] In addition, three-dimensional ground images taken during normal times may differ from the three-dimensional ground images taken immediately before a disaster occurs due to new construction, reconstruction, extensions, renovations, etc. of houses. On the other hand, there is the problem that it is cumbersome to constantly update the three-dimensional ground images and three-dimensional coordinates of feature points stored in a normal time storage device to the latest data.

[0010] The present invention has been made in consideration of these circumstances, and aims to provide a house status providing device and method that can accurately recognize the status of individual houses in areas where a disaster has occurred and associate the recognition results with individual houses in aerial images.

[0011] In order to achieve the above-mentioned object, the invention of a first aspect is a house status providing device equipped with a processor, wherein the processor performs the following processes: a process of acquiring an aerial image of a target area where a plurality of houses exist; a matching process of matching the map to the aerial image based on the aerial image and a map stored in memory; a process of acquiring outline information of the houses on the aerial image from the result of the matching process and area information of the houses contained in the map; a process of extracting house images showing the houses from the aerial image based on the acquired outline information of the houses; a classification process of recognizing the status of each of the plurality of houses based on the extracted house images and classifying the houses into one of a plurality of first classes based on the recognition results; and a process of associating the classified first classes with the houses on the aerial image.

[0012] According to the first aspect of the present invention, a map is matched to an aerial image, and outline information of houses on the aerial image is obtained from the result of this matching process and area information of houses included in the map. Then, house images showing houses are extracted from the aerial image based on the obtained outline information of the houses, and the status of each house in the disaster-hit area can be recognized accurately and quickly based on the extracted house images, and the recognition results can be associated with each house in the aerial image.

[0013] In the second aspect of the house status providing device of the present invention, in the first aspect, it is preferable that the memory stores three-dimensional area information of the house included in the map, and the processor estimates the position and attitude of the camera that took the aerial image by matching processing, performs perspective projection conversion of the three-dimensional area information of the house based on the estimated position and attitude of the camera, and obtains external shape information of the house on the aerial image.

[0014] In the house status providing device according to the third aspect of the present invention, in the first or second aspect, the processor preferably causes the display device to superimpose first information indicating a first class associated with the house on the aerial image, thereby making it possible to clearly display the status of each house on the aerial image.

[0015] In the fourth aspect of the present invention, in the third aspect, the house status providing device is preferably such that the first information is a frame line surrounding the house, and the color or line type of the frame line differs depending on the first class into which the house is classified.

[0016] In the house status providing device of the fifth aspect of the present invention, in any of the first to fourth aspects, it is preferable that the multiple first classes are multiple first classes including completely destroyed and partially destroyed, corresponding to the damage status of the house due to the disaster.

[0017] In a sixth aspect of the house status providing device of the present invention, in any of the first to fifth aspects, the memory stores a learning model that outputs a classification result indicating a first class when a house image is input, and the classification process of the processor preferably uses the learning model stored in the memory, inputs the house image into the learning model, and obtains the classification result estimated by the learning model.

[0018] In a seventh aspect of the present invention, in the sixth aspect of the house status providing device, the learning models stored in the memory preferably include a plurality of learning models corresponding to the types of disasters, and the processor preferably selects a learning model corresponding to the type of disaster from the plurality of learning models and obtains a classification result using the selected learning model. Since the condition of the house differs depending on the type of disaster, selecting and applying a learning model corresponding to the type of disaster makes it possible to better grasp the condition of the house and appropriately classify it.

[0019] In the house status providing device of the eighth aspect of the present invention, in any of the first to seventh aspects, it is preferable that the memory stores attribute information related to the houses included in the map, and the processor obtains the attribute information related to the houses from the memory and associates it with the houses on the aerial image.

[0020] In the ninth aspect of the present invention, in the eighth aspect of the house status providing device, it is preferable that the processor superimposes first information and attribute information indicating a first class associated with a house on an aerial image on the aerial image and displays them on the display device.

[0021] In the tenth aspect of the present invention, in the eighth aspect, the attribute information related to the house includes the age of the house or the type of the house, and the processor preferably classifies the age of the house or the type of the house into one of a plurality of second classes based on the age of the house or the type of the house, associates the classified second class with the house on the aerial image, and displays second information indicating the second class superimposed on the aerial image on the display device.

[0022] The invention of an eleventh aspect is a house status providing method executed by a house status providing device equipped with a processor, comprising the steps of: the processor acquiring an aerial image of a target area in which a plurality of houses exist; the processor performing a matching process to match the aerial image to the map based on the aerial image and a map stored in a memory; the processor acquiring outline information of the houses on the aerial image from the result of the matching process and area information of the houses included in the map; the processor extracting house images showing the houses from the aerial image based on the acquired outline information of the houses; the processor recognizing the status of each of the plurality of houses based on the extracted house images and classifying the houses into one of a plurality of first classes based on the recognition result; and the processor associating the classified first class with the house on the aerial image.

[0023] In the 12th aspect of the present invention, in the 11th aspect, the method for providing house status is preferably such that the memory stores three-dimensional area information of the house included in the map, and the processor estimates the position and attitude of the camera that took the aerial image by a matching process, and performs perspective projection conversion of the three-dimensional area information of the house based on the estimated position and attitude of the camera, thereby obtaining external shape information of the house on the aerial image.

[0024] In the house status providing method according to the thirteenth aspect of the present invention, in the eleventh or twelfth aspect, it is preferable that the processor superimposes first information indicating a first class associated with the house on the aerial image on the display device and displays it.

[0025] In the house status providing method according to the fourteenth aspect of the present invention, in the thirteenth aspect, it is preferable that the first information is a frame line surrounding the house, and the color or line type of the frame line differs depending on the first class into which the house is classified.

[0026] In the house status providing method according to the 15th aspect of the present invention, in any of the 11th to 14th aspects, it is preferable that the multiple first classes are multiple first classes including completely destroyed and partially destroyed, corresponding to the damage status of the house due to the disaster.

[0027] According to the present invention, the condition of each house in a disaster-hit area can be accurately recognized, and the recognition results can be associated with each house in an aerial image.

[0028] FIG. 1 is a schematic diagram showing an example of the configuration of a system 10 including a house status providing device according to the present invention. FIG. 2 is a block diagram showing an embodiment of the hardware configuration of the house status providing device shown in FIG. 1. FIG. 3 is a functional block diagram showing an embodiment of the house status providing device according to the present invention. FIG. 4 is a diagram showing an aerial image IM and a map MP corresponding to the aerial image IM. FIG. 5 is a diagram used to explain the position and orientation of a camera. FIG. 6 is a diagram showing an example of the relationship between a three-dimensional spatial coordinate system having three axes corresponding to the three-dimensional coordinates (x', y', z') obtained by the coordinate transformation of Equation 1 and the image coordinate system of the camera's image sensor. FIG. 7 is a diagram showing an example of a composite image in which the map is converted into image coordinates using a camera matrix using sensor data as parameter values, and the positions of houses and roads are superimposed on the aerial image IM. FIG. 8 is a diagram showing an example of a composite image in which the optimal camera matrix is ​​automatically calculated by a matching processing unit, and the map MP is converted into image coordinates using the optimal camera matrix, and the positions of houses and roads are superimposed on the aerial image IM. FIG. 9 is a diagram showing a flow in which a house outer shape information acquisition unit acquires house outer shape information. Fig. 10 is a diagram showing an example of an aerial image onto which first information indicating the classification result of a house has been synthesized. Fig. 11 is a block diagram showing another embodiment of a classification processing unit. Fig. 12 is a diagram showing a list of attribute information related to houses. Fig. 13 is a flowchart showing an embodiment of a house status providing method according to the present invention.

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the house status providing device and method according to the present invention will now be described with reference to the accompanying drawings.

[0030] [System Including House Status Providing Device] FIG. 1 is a schematic diagram showing an example of the configuration of a system 10 including a house status providing device according to the present invention.

[0031] This system 10 is a system for quickly grasping and providing information on the status of houses when various disasters occur, and is installed in, for example, local governments.

[0032] The system 10 shown in FIG. 1 is configured by connecting an aerial photography drone 12, a remote controller 16, a house status providing device 20, and a terminal device 24 via a network 22.

[0033] The drone 12 is an unmanned aerial vehicle that is remotely controlled using a remote controller 16 .

[0034] When a disaster occurs, an investigator investigating the damage situation, or a drone operator commissioned by a local government, operates the drone 12 using a remote controller 16 and takes aerial photographs of a target area (disaster area) where multiple houses exist using a camera 14 mounted on the drone 12.

[0035] The camera 14 is mounted on the drone 12 via a gimbal head 13. The camera 14 or the drone 12 has a GPS (Global Positioning System) receiver, a barometric pressure sensor, a direction sensor, a gyro sensor, etc., and acquires information indicating the position (latitude, longitude, altitude) and attitude (azimuth angle and depression angle indicating the shooting direction) of the camera 14 during aerial photography.

[0036] Images captured using the camera 14 (hereinafter referred to as "aerial images IM") can be stored in an internal storage built into the camera 14 and / or in a storage device such as a memory card removably attached to the camera 14. The aerial images IM can be transferred to the remote controller 16 via wireless communication or to the house status providing device 20. Information on the position and attitude of the camera 14 when capturing the aerial image can be recorded in the header of the image file in which the aerial images IM are recorded.

[0037] The house status providing device 20 is configured using a computer. The computer applied to the house status providing device 20 may be a server, a personal computer, or a workstation.

[0038] The home status providing device 20 may perform data communication with the drone 12, the remote controller 16, and the terminal device 24 via a network 22. The network 22 may be a local area network or a wide area network.

[0039] The house status providing device 20 can acquire the aerial images IM from the drone 12 or the camera 14 via the network 22 or via the remote controller 16. The house status providing device 20 can also acquire the aerial images IM from the memory card of the camera 14 or the like without going via the network 22. This is because it is conceivable that the network in some areas may be down due to a disaster.

[0040] The house status providing device 20 also acquires the necessary map from an internal memory (database) or an external database that provides maps. In this case, the map corresponds to the aerial image IM and preferably includes the area captured by the aerial image IM. Details of the house status providing device 20 and the map will be described later.

[0041] The terminal device 24 is, for example, a smartphone or a tablet terminal carried by a local government official, a fire department staff member, etc. The terminal device 24 may also have the processing function of the house status providing device 20.

[0042] The display device 230 displays the aerial image IM and various pieces of information superimposed on the aerial image IM. The terminal device 24 includes a display 24A and can perform the same display as the display device 230.

[0043] [Hardware Configuration of House Status Providing Apparatus] FIG. 2 is a block diagram showing an embodiment of the hardware configuration of the house status providing apparatus shown in FIG.

[0044] The house status providing device 20 shown in FIG. 2 includes a processor 200 , a memory 210 , a database 220 , a display device 230 , an input / output interface 240 , and an operation unit 250 .

[0045] The processor 200 is composed of a CPU (Central Processing Unit) and the like, and controls all parts of the house status providing device 20, and also performs a matching process of matching a map with the aerial image IM, a process of acquiring outer shape information of the house on the aerial image IM from the result of the matching process and area information of the house included in the map, a process of extracting house images showing the house from the aerial image IM based on the outer shape information of the house, and a classification process of recognizing the status of each of a plurality of houses based on the house images and classifying the houses into one of a plurality of classes (first class) based on the recognition results. Details of the various processes of this processor 200 will be described later.

[0046] The memory 210 includes flash memory, read-only memory (ROM), random access memory (RAM), a hard disk drive, etc. The flash memory, ROM, or hard disk drive is a non-volatile memory that stores various programs including an operating system. The RAM functions as a working area for processing by the processor 200 and temporarily stores programs stored in the flash memory, etc. Note that the processor 200 may have a portion of the memory 210 (RAM) built in.

[0047] The memory 210 also functions as an image storage unit that stores the aerial images IM, and can store and manage the aerial images IM.

[0048] The database 220 is a part that manages maps of areas photographed from the air, and in this example, in addition to the maps, it also manages house attribute information related to the houses on the maps.

[0049] Details of the information managed by the database 220 will be described later. The database 220 may be configured inside the house status providing device 20 or may be an external database connected via communication, such as a database of the Geospatial Information Authority of Japan that manages basic map information or an OpenStreetMap database.

[0050] The display device 230 displays the aerial image IM in response to an instruction from the processor 200, and also displays first information indicating a first class (information classified according to the condition of the house) associated with the house on the aerial image IM, superimposed on the aerial image IM. The display device 230 is also used as part of a GUI (Graphical User Interface) when various information is received from the operation unit 250.

[0051] The display device 230 may be included in the house status providing device 20, or may be provided separately from the house status providing device 20 as shown in FIG.

[0052] The input / output interface 240 includes a connection unit that can be connected to an external device, a communication unit that can be connected to a network, etc. As the connection unit that can be connected to an external device, a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), etc. can be applied. The processor 200 can acquire aerial images IM via the input / output interface 240, or output necessary information in response to a request from the outside (for example, the external terminal device 24).

[0053] The operation unit 250 includes a pointing device such as a mouse, a keyboard, and the like, and functions as part of a GUI that accepts input of various information and instructions by user operation.

[0054] [Embodiment of House Status Providing Device] FIG. 3 is a functional block diagram showing an embodiment of a house status providing device according to the present invention.

[0055] FIG. 3 is a functional block diagram mainly showing the functions of the processor 200 of the house status providing device 20 shown in FIG.

[0056] In FIG. 3, the processor 200 functions as a matching processing unit 202 , a house shape information acquisition unit 203 , a house image extraction unit 204 , a house condition classification processing unit 205 , an association processing unit 206 , and a synthesis processing unit 207 .

[0057] The image acquisition unit 201 acquires aerial images IM of the disaster area taken by the camera 14 of the drone 12 from the drone 12 via the network 22 or from the memory card of the camera 14.

[0058] The aerial image IM acquired by the image acquisition unit 201 is output to the matching processing unit 202 , the house image extraction unit 204 , and the synthesis processing unit 207 .

[0059] The matching processing unit 202 performs processing to match the aerial image IM with the map MP based on the aerial image IM and the map MP stored in memory (database 220 in this example).

[0060] Here, the matching processing unit 202 reads information indicating the position and attitude of the camera 14 attached to the aerial image IM from the header section of the image file of the aerial image IM, predicts the area (block) photographed from the aerial based on the read information indicating the position and attitude of the camera 14, and obtains a map MP of the area corresponding to the aerial image IM from the database 220.

[0061] FIG. 4 is a diagram showing an aerial image IM and a map MP corresponding to the aerial image IM.

[0062] The black circles on the map MP shown in FIG. 4 indicate characteristic points such as corners of the ground perimeter of a house, and have three-dimensional information indicating latitude, longitude, and altitude.

[0063] The matching processing unit 202 identifies positions on the aerial image IM that correspond to each of a plurality of specific points indicated by black circles on the map MP, based on the aerial image IM and the map MP.

[0064] In map MP, each house is assigned a house ID (Identification) as identification information to identify the house, and three-dimensional information of multiple specific points on the ground perimeter of the house is recorded, linked to the house ID.

[0065] <Principle of matching process> In order to identify a specific point on the map MP and its corresponding position on the aerial image IM, it is necessary to determine the correspondence between the three-dimensional spatial coordinates in the world coordinate system (map coordinate system) and the two-dimensional image coordinates in the local coordinate system (camera coordinate system).

[0066] <<About the camera matrix>> The problem of finding the correspondence between three-dimensional spatial coordinates and two-dimensional image coordinates can be solved by finding the camera matrix as a transformation matrix for perspective projection transformation using the following formula based on the camera model: Image coordinates (u, v) = camera matrix * three-dimensional coordinates (x, y, z)

[0067] The camera matrix can be expressed as the product of an intrinsic parameter matrix and an extrinsic parameter matrix. The extrinsic parameter matrix is ​​a matrix that converts three-dimensional coordinates (world coordinates) into camera coordinates. The extrinsic parameter matrix is ​​a matrix determined by the position and orientation of the camera during aerial photography, and includes translational parameters and rotational parameters.

[0068] The internal parameter matrix is ​​a matrix that converts camera coordinates into image coordinates, and is determined by the specifications of the camera 14, such as the focal length of the camera, the sensor size and aberration (distortion) of the image sensor, etc.

[0069] By converting from the three-dimensional coordinates (x, y, z) to the camera coordinates using an extrinsic parameter matrix, and then converting from the camera coordinates to the image coordinates (u, v) using an intrinsic parameter matrix, the three-dimensional coordinates (x, y, z) can be associated (converted) with the image coordinates (u, v).

[0070] The intrinsic parameter matrix can be specified in advance, whereas the extrinsic parameter matrix depends on the position and orientation of the camera and therefore needs to be set for each aerial image IM.

[0071] The camera matrix can be calculated if there are six or more corresponding points between three-dimensional coordinates on the map MP in the real three-dimensional space and image coordinates on the aerial image IM. In Figure 4, one feature point on the map MP and the corresponding point on the aerial image IM that corresponds to this feature point are indicated by arrows.

[0072] These multiple corresponding points can be specified by a human, or automatically searched for corresponding points, which are local image features, using SIFT (Scale-Invariant Feature Transform) or the like. However, the matching processing unit 202 in this example acquires the camera matrix without searching for corresponding points and performs matching processing.

[0073] <<Issues When Sensor Data is Used for the Extrinsic Parameter Matrix>> It is conceivable to calculate the extrinsic parameter matrix using sensor data (sensor values) obtained from various sensors, such as a GPS receiver, a barometric pressure sensor, a direction sensor, and a gyro sensor, mounted on the drone 12, as information indicating the position and attitude of the camera 14. However, a camera matrix actually calculated using sensor data has a problem in that it is not possible to correctly match (align) the three-dimensional positions of houses, etc. on the map MP with the corresponding houses, etc. on the aerial image IM due to the influence of errors in the sensor data.

[0074] The matching processing unit 202 in this example automatically searches for the parameter values ​​of the camera matrix based on the sensor data taken during aerial photography, and determines the optimal parameter values, i.e., the camera matrix that can accurately correspond (align) the three-dimensional position on the map MP with the position on the aerial image IM.

[0075] In the process of searching for the parameter values ​​of the camera matrix, the matching processing unit 202 assigns parameter values ​​based on the sensor data values, converts the map MP into image coordinates using the camera matrix of those parameter values, evaluates the degree of correspondence between the conversion result and the position on the aerial image IM, selects the parameter values ​​that result in the highest evaluation score, and determines the camera matrix.

[0076] In the process of evaluating the degree of match, the matching processor 202 extracts line segments such as the perimeter of houses and roads from both the result of converting the map MP into image coordinates and the aerial image IM, and calculates an evaluation value that quantitatively evaluates the degree of match between the line segments. A line segment is specified by the coordinates of two points (start and end points). The "degree of match" here may refer to the degree of match, including tolerances, for at least one, and preferably multiple, of the distance between the line segments, the difference in length of the line segments, and the difference in inclination angle of the line segments.

[0077] The matching processing unit 202 generates a map MP1 aligned with the aerial image IM by perspective projection converting the three-dimensional information of the map MP onto the aerial image IM using the camera matrix Mc determined by automatic parameter value search using line segment matching.

[0078] <<Explanation of Perspective Projection Transformation Using Camera Matrix>> Here, we will explain in detail the calculation method for converting the three-dimensional coordinates (x, y, z) of a specific point, such as the ground perimeter of a house, included in the map MP into coordinates when projected onto the image sensor of the camera 14, i.e., image coordinates (u, v).

[0079] At a specific point (x, y, z), such as the ground perimeter of a house, x and y are latitude and longitude converted into UTM coordinates, which are a Cartesian coordinate system, and z is altitude. If height information is available for a building such as a house, it is desirable to use that height information to calculate the position of the roof on the image. For houses without height information, the height of the roof may be calculated assuming a height of, for example, 6 m.

[0080] The three-dimensional position of the camera 14 during aerial photography is defined as (xc, yc, zc), where xc and yc are the latitude and longitude of the camera 14 converted into UTM coordinates, and zc is the altitude.

[0081] The attitude (shooting direction) of the camera 14 during aerial photography is specified by the azimuth angle θh, tilt angle θt, and roll angle θr. The azimuth angle θh is the angle from north, with north as the reference point. The tilt angle θt is the camera angle (depression angle) toward the ground. The roll angle θr is the inclination from the horizontal.

[0082] FIG. 5 is a diagram used to explain the position and orientation of the camera.

[0083] In the UTM coordinate system, the x-axis is defined as east and the y-axis is defined as north. In Fig. 5, the position of the camera 14 is defined as Pc(xc, yc, zc). Arrow A indicates the shooting direction specified by the attitude of the camera 14.

[0084] The equation for converting the coordinates of a feature point (x, y, z) such as the ground perimeter of a house into the origin of the projection center (i.e., the camera position at the time of shooting) is expressed by the following [Equation 1].

[0085]

[0086] Furthermore, the rotation matrices Mh, Mt, and Mr are defined by the following [Equation 2], [Equation 3], and [Equation 4].

[0087]

[0088]

[0089]

[0090] The coordinates of feature points such as the ground perimeter of the house, with the projection center as the origin, are converted into camera coordinates using the following equation.

[0091]

[0092] The origin of the camera coordinate system is the center of projection, the X axis is the horizontal direction of the image sensor, the Y axis is the vertical direction of the image sensor, and the Z axis is the depth direction.

[0093] FIG. 6 is a diagram illustrating an example of the relationship between a three-dimensional spatial coordinate system having three axes corresponding to the three-dimensional coordinates (x', y', z') obtained by the coordinate transformation of Equation 1 and the image coordinate system of the image sensor 140 of the camera 14.

[0094] The feature points (in meters) of the camera coordinates obtained by the above formula (5) are converted into coordinates (in pixels) on the aerial image IM by the following formula.

[0095]

[0096] In Equation 6, f is the focal length, and p is the pixel pitch. The pixel pitch p is the distance between pixels of the image sensor 140 and is usually the same in both the vertical and horizontal directions. Uc and Vc are the image center coordinates (in pixels).

[0097] <<Example of Searching for Optimal Parameter Values>> An example of a specific procedure for calculating the camera matrix Mc will be described.

[0098] [Step 1] The matching processing unit 202 acquires the position and orientation of the camera 14 at the time of image capture from sensor data. The position (xc_0, yc_0, zc_0) and orientation (θh_0, θt_0, θr_0) acquired from the sensor data are used as reference values ​​for searching for parameter values.

[0099] [Step 2] The matching processing unit 202 sets a search range and a search step size for each of the parameter values ​​of the position and orientation of the camera 14. For example, the matching processing unit 202 sets the search range for the x coordinate of the position of the camera 14 to a range of ±10 m from a reference value, with the step size set to 1 m. That is, the search range for the x coordinate of the position of the camera 14 is set to "xc_0-10<xc<xc_0+10", and the search step size is set to 1 (unit: meter). xc-10, which indicates the lower limit of the search range, is an example of a search lower limit value, and xc+10, which indicates the upper limit of the search range, is an example of a search upper limit value.

[0100] A search range and step size are also set for each of the parameters, namely, the y and z coordinates of the position of the camera 14 and the attitude (θh, θt, θr). For example, for the azimuth angle θh, the search range is set to a range of ±45° from the reference value indicated by the sensor data, and the parameter value is changed in 1° step sizes. Different search ranges and step sizes can be set for each parameter.

[0101] [Step 3] The matching processing unit 202 determines a combination of parameter values ​​by adjusting the step size within each search range for the position and orientation parameters of the camera 14. Then, using the determined combination of parameter values ​​(xc, yc, zc), (θh, θt, θr), the positions (latitude, longitude, altitude) of houses and roads included in the map MP are converted into coordinates on the two-dimensional aerial image IM.

[0102] [Step 4] The matching processing unit 202 evaluates the matching between the map MP1 and the aerial image IM by the line segment matching described above.

[0103] [Step 5] The matching processing unit 202 performs steps 3 and 4 above, varying the parameter values ​​of the three-dimensional position and shooting direction of the camera 14 using all step sizes within the search range of each parameter, and adopts the parameter values ​​of the position and orientation of the camera 14 that produce the best evaluation score for the line segment matching as the correct position and orientation of the camera 14. In this way, the optimal camera matrix is ​​automatically calculated for each aerial image IM, and a converted map MP1 that is accurately aligned with each aerial image IM is obtained.

[0104] Overview of Automatic Alignment Using Line Segment Matching Figure 7 shows an example of a composite image in which a map is converted into image coordinates using a camera matrix that uses sensor data as parameter values, and the positions of houses and roads are superimposed on an aerial image IM.

[0105] 7, each of the polygons PG superimposed on the aerial image IM represents the perimeter of a house on the map transformed using a camera matrix that uses sensor data as parameter values, and the line RL superimposed on the aerial image IM represents a road on the map transformed using the same camera matrix.

[0106] 7, the polygon PG and the line RL are significantly deviated from the positions of the houses and roads in the aerial image IM. This is because a camera matrix using sensor data (sensor values) as parameters for the position and attitude of the camera 14 has errors in the sensor data, making it impossible to correctly match houses and the like on the map with houses and the like on the aerial image IM.

[0107] FIG. 8 shows an example of a composite image in which the optimal camera matrix is ​​automatically calculated by the matching processing unit, the map MP is converted into image coordinates using the optimal camera matrix, and the positions of houses and roads are superimposed on the aerial image IM.

[0108] The matching processing unit 202 in this example can accurately determine the camera matrix Mc as described above, and by using this camera matrix Mc to perform perspective projection conversion of the three-dimensional information of the map MP onto the aerial image IM, it is possible to accurately match the map (the map indicated by the thick line) to the aerial image IM as shown in Figure 8.

[0109] Furthermore, by correctly matching the map MP with the aerial image IM, the house ID on the map MP can be associated with the house on the aerial image IM, and attribute information of the house (such as the house's address, age, and type) can be obtained from the house ID associated with each house on the aerial image IM.

[0110] In Figure 3, the house shape information acquisition unit 203 acquires two-dimensional shape information of the house on the aerial image IM from the result of the matching process by the matching processing unit 202 (e.g., the camera matrix Mc) and the area information of the house included in the map MP.

[0111] Here, the area information of the house is three-dimensional information that indicates the three-dimensional area of ​​the house in real space.

[0112] The process of matching the map MP to the aerial image IM by the matching processing unit 202 is not limited to the automatic alignment by line segment matching described above, but may be a process of matching the map MP to the map MP based on known corresponding point detection. For example, correspondence points between a plurality of feature points that serve as landmarks in the aerial image IM and feature points of corresponding landmarks on the map MP are detected, and parameters of a geometric transformation (projective transformation, affine transformation, etc.) are determined so that the corresponding points match, and the map MP is geometrically transformed, thereby matching the map MP to the aerial image IM.

[0113] FIG. 9 is a diagram showing a flow in which the house outer shape information acquisition unit acquires the house outer shape information.

[0114] In this example, as described above, the camera matrix Mc is used to perform perspective projection transformation on the map MP having three-dimensional information, thereby correctly matching the map MI onto the aerial image IM.

[0115] Now, a case will be described in which area information (three-dimensional outer shape information) of one house on the map MP shown in FIG. 9A is correctly matched on the aerial image IM.

[0116] An example of the three-dimensional area information of this house is shown in the following table.

[0117]

[0118] As shown in Table 1, the three-dimensional area information of a house includes the three-dimensional positions (latitude, longitude, and altitude) of four points on the ground perimeter of the house and information on a uniform building height. In this example, the area information of the house includes three-dimensional information on the vertices of a polygonal prism (rectangular parallelepiped).

[0119] The three-dimensional information of the house represented by this rectangular parallelepiped can be accurately matched onto the aerial image IM by perspective projection transformation using the camera matrix Mc (Figure 9(B)). The four points on the ground side of the polygon shown in Figure 9(B) are the projection positions of the four points on the ground perimeter shown in [Table 1], and the four points on the roof side are the projection positions of four points at a uniform building height (= 6.0 m) relative to the four points on the ground side.

[0120] Next, the perimeter information of the house on the map MP after the matching process is extracted (FIG. 9C). The perimeter information of the house is the outer shape information of the house obtained by connecting the six outermost points of the eight points shown in FIG. 9B.

[0121] The contour information acquisition unit 203 generates a mask image in which the inside of the extracted perimeter information (contour information) of the house is white (transparent) and the outside is black (opaque) ( ​​FIG. 9(D) ). This mask image is used to extract (cut out) the house image from the aerial image IM.

[0122] The contour information acquisition unit 203 acquires contour information about each house shown in the aerial image IM and generates a mask image for cutting out. Note that the contour information may be generated as an area that is one size larger than the adjacent house, as long as it does not overlap significantly with the adjacent house.

[0123] An aerial image IM is input to the house image extraction unit 204 shown in Figure 3, and the house image extraction unit 204 extracts (cuts out) a house image H from the aerial image IM based on the exterior shape information of the house (mask image corresponding to the exterior shape information) acquired by the exterior shape information acquisition unit 203.

[0124] The house image extraction unit 204 extracts house images H showing each house shown in the aerial image IM, but even if a house has been lost or submerged in a disaster, for example, it is possible to extract an image of the area corresponding to the house before the disaster occurred.

[0125] The house image H extracted by the house image extraction unit 204 is output to a house condition classification processing unit 205 and an association processing unit 206 .

[0126] The classification processing unit 205 recognizes the state of the house after the disaster occurs based on the house image H, and performs classification processing to classify the house into one of a plurality of classes (first class) based on the recognition result. The classification processing unit 205 can classify the state of the house (disaster state) using a damage determination AI (artificial intelligence) that inputs the house image.

[0127] The memory 210 (Figure 2) stores a damage determination AI model (learning model) that outputs a classification result indicating the first class when a house image H is input, and the classification processing unit 205 uses the learning model stored in the memory 210, inputs the house image H into the learning model, and obtains the classification result estimated by the learning model.

[0128] The classification processing unit 205 has, for example, three first classes: intact, partially destroyed, and completely destroyed, and classifies the house into one of the three classes of intact, partially destroyed, and completely destroyed based on the input house image, and outputs the classification result to the association processing unit 206.

[0129] The association processing unit 206 performs processing to associate the classification result (one of the multiple first classes) obtained by the classification processing unit 205 with the house (house image) on the aerial photograph image IM.

[0130] Since houses on the aerial image IM can be associated with houses on the map MP through a matching process, the association processing unit 206 can obtain a house ID corresponding to the house image H, and based on this house ID, can associate the classification results with the house information (map information, attribute information) managed in the database 220.

[0131] The composition processing unit 207 performs composition processing to compose (superimpose) the house classification results (first information indicating the first class) associated with each house image H onto the aerial image IM. The first information in this example is a frame line surrounding the house on the aerial image IM, and the color of the frame line varies depending on the class classification assigned by the classification processing unit 205. For example, a green frame is assigned to an intact house, a yellow frame to a partially destroyed house, and a red frame to a completely destroyed house as the first information. Furthermore, the composition processing unit 207 can determine the position and size of the frame line to be composed to surround the house based on the outer shape information acquired by the outer shape information acquisition unit 203.

[0132] The aerial image IMs onto which the first information indicating the classification result of the house has been synthesized by the synthesis processing unit 207 is output to the display device 230 (FIGS. 1 and 2) and displayed on the display device 230.

[0133] FIG. 10 is a diagram showing an example of an aerial image onto which first information indicating the classification result of houses is combined.

[0134] In the aerial image IMs shown in Fig. 10, three types of frame lines are synthesized to surround houses. In Fig. 10, thin solid frame lines indicate safe buildings, dotted frame lines indicate partially destroyed buildings, and thick solid frame lines indicate completely destroyed buildings. In Fig. 10, the three types of frame lines indicating safe, partially destroyed, and completely destroyed buildings are distinguished by line type, but the visibility of the classification results can be further improved by distinguishing them by color, such as green frame, yellow frame, and red frame.

[0135] In addition, in this example, the classification results for each house are displayed using the color of the frame line surrounding the house, but this is not limited to this. Characters indicating whether the house is safe, partially destroyed, or completely destroyed may be synthesized as the first information, and the color of the characters may be color-coded to green, yellow, or red depending on whether the house is safe, partially destroyed, or completely destroyed.

[0136] Furthermore, to prevent the display of the first information indicating the classification result from becoming too complicated, the first information indicating the classification result may not be displayed for intact houses. Furthermore, when a desired class is selected by the operation unit 250, the processor 200 may superimpose only the first information corresponding to the selected class on the aerial image IM so that the user can check only houses classified into a desired class (for example, completely destroyed).

[0137] By using the house status providing device 20 of this embodiment when a disaster occurs, the secretariat of a local government's disaster response headquarters can quickly grasp the damage status of houses in the affected area and quickly formulate plans for disaster relief activities.

[0138] Furthermore, local government officials, fire department personnel, and the like who possess the terminal device 24 can use the terminal device 24 to check the aerial images IMs on which the classification results of the damage status of houses are superimposed.

[0139] FIG. 11 is a block diagram showing another embodiment of the classification processing unit.

[0140] The classification processing unit 2050 shown in Figure 11 is another embodiment of the house condition classification processing unit 205 shown in Figure 3, and uses one of four damage determination AIs (learning models) 2052, 2054, 2056, and 2058 to obtain a classification result that classifies the house condition.

[0141] The four damage determination AIs 2052, 2054, 2056, and 2058 are learning models corresponding to the types of disasters, with damage determination AI 2052 being an earthquake determination AI that determines the status of houses damaged by earthquakes, damage determination AI 2054 being a flood determination AI that determines the status of houses damaged by floods, damage determination AI 2056 being a fire determination AI that determines the status of houses damaged by fires, and damage determination AI 2058 being a typhoon / tornado determination AI that determines the status of houses damaged by typhoons and tornadoes.

[0142] Since the damage situation of houses varies depending on the type of disaster, it is preferable to select the damage determination AI (learning model) to be used depending on the type of disaster that has occurred.

[0143] The classification processing unit 2050 selects a damage determination AI corresponding to the type of disaster from multiple (four) damage determination AIs 2052, 2054, 2056, and 2058, and obtains a classification result using the selected damage determination AI.

[0144] The selection of the damage determination AI can be done in two ways: by receiving a user instruction to select the damage determination AI from the operation unit 250, or by inputting an aerial image into the AI ​​that determines the type of disaster to estimate the type of damage and automatically select it.

[0145] The types of damage determination AI are not limited to those corresponding to the four types of disasters described above. Two or more types of damage determination AI may be used in combination to classify the damage status of a house.

[0146] FIG. 12 is a table showing a list of attribute information related to houses.

[0147] The attribute information related to the house can be, for example, information managed in a ledger (memory 210 or database 220, etc.) owned by a local government.

[0148] In the example shown in FIG. 12, the address, type of house, age of the house, damage status of the house (classification result), etc. are managed in association with the house ID.

[0149] The type of house is the type of building structure, such as wooden, steel-framed reinforced concrete, reinforced concrete, etc. In the classification column, the classification results obtained by classifying the damage status of houses at the time of the current disaster are registered in association with the house (house ID).

[0150] As described above, by correctly matching the map MP with the aerial image IM, it is possible to associate the house (house ID) on the map MP with the house on the aerial image IM, and therefore it is possible to associate the house on the aerial image IM with the attribute information registered in association with the house ID.

[0151] The house status providing device 20 classifies the age of the house or the type of the house into one of multiple classes (second classes) based on the attribute information of the house (age of the house or type of house), associates the classified second class with the house on the aerial image, and can superimpose second information indicating the second class on the aerial image IM and display it on a display device 230, etc.

[0152] When second information indicating the age of the house or a second class indicating the type of house is superimposed on a house on an aerial image IM, the house status providing device 20 can accept an instruction to switch the information to be displayed and superimpose the second information instead of the first information indicating the classification result of the damage status of the house.

[0153] The plurality of second classes based on the age of the house may be, for example, four classes: less than 10 years, 10 to 30 years, 30 to 50 years, and 50 years or more, and the second information indicating the second class superimposed on the aerial image IM may be a color frame colored according to the four classes.Furthermore, the plurality of second classes based on the type of house may be, for example, four classes: wood, reinforced concrete, steel-reinforced concrete, and other structure, and the second information indicating the second class superimposed on the aerial image IM may be a color frame colored according to the four classes.

[0154] By displaying the second class indicating the age of the house or the type of house in this way, it is possible to easily understand what types of houses are resistant to disasters.

[0155] [Embodiment of House Status Providing Method] FIG. 13 is a flowchart showing an embodiment of a house status providing method according to the present invention.

[0156] The house status providing method shown in FIG. 13 is a method performed by the house status providing device of the embodiment shown in FIG.

[0157] In FIG. 13, the processor 200 acquires an aerial image IM of the disaster area taken by the camera 14 mounted on the drone 12 (step S10).

[0158] Next, the processor 200 matches the aerial image IM with the map MP (step S12). The map MP contains three-dimensional information of the disaster area photographed from the air, and is read from the database 220. In this example, the position and orientation of the camera 14 at the time the aerial image IM was captured are searched for with high accuracy, and the camera matrix Mc determined by the search is used to perform perspective projection transformation of the three-dimensional information of the map MP onto the aerial image IM, thereby generating a map MP1 matched to the aerial image IM.

[0159] Next, the processor 200 acquires the outer shape of the house on the aerial image IM from the matching result and the house area information included in the map MP (step S14). The house area information is three-dimensional information that indicates the three-dimensional area of ​​the house in real space, and in this example, it includes the three-dimensional positions (latitude, longitude, altitude) of multiple points on the ground perimeter of the house and information on a uniform building height. This house area information (three-dimensional information) is matched to the house on the aerial image IM by perspective projection transformation using the camera matrix Mc, and the outer shape of the house on the aerial image IM is acquired.

[0160] The processor 200 extracts (cuts out) a house image H showing the house from the aerial image IM based on the external shape of the house (step S16).

[0161] The processor 200 recognizes the status of the house based on the extracted house image H, and classifies the house into one of a plurality of first classes based on the recognition result (step S18). The processor 200 uses a damage determination AI (learning model) and inputs the house image H into the learning model to obtain a classification result estimated by the learning model. The plurality of first classes may include, for example, three classes: intact, partially destroyed, and completely destroyed, and the damage determination AI outputs one of the three classes: intact, partially destroyed, and completely destroyed, as a classification result for the status (disaster status) of the house.

[0162] Next, the processor 200 associates the classified class (one of the multiple first classes) with the house on the aerial image IM (step S20). Because the map MP is aligned with the aerial image IM, information (house ID) of the house on the map MP corresponding to the house image H can be obtained, and based on this house ID, the information (map information, attribute information) of the house managed in the database 220 can be associated with the classification result.

[0163] The processor 200 superimposes information (first information) indicating the classification result of the houses on the aerial image IM and displays it on the display device 230 (step S22). The first information is a frame line surrounding the house on the aerial image IM, and the color of the frame line varies depending on the class classification. For example, a green frame is assigned to an intact house, a yellow frame to a partially destroyed house, and a red frame to a completely destroyed house.

[0164] Next, the processor 200 determines whether or not classification of all the houses in the aerial image IM has been completed (step S24). If the processor 200 determines that classification of all the houses has not been completed (in the case of "No"), the processor 200 transitions to step S14. As a result, the processor 200 obtains the outer shape information of the other houses in the aerial image IM, and repeats the processes from step S16 to step S24.

[0165] On the other hand, when the processor 200 determines that classification of all houses has been completed (if "Yes"), it terminates this process, but can display on the display device 230 aerial images IMs of all houses on the aerial image IM with color frames superimposed to indicate the damage status of those houses.

[0166] These aerial images IMs allow for a quick understanding of the damage to houses in the affected areas, enabling the rapid formulation of plans for disaster relief activities.

[0167] [Others] In this embodiment, the hardware structure of a processing unit that executes various processes, such as a CPU (Central Processing Unit), is the following various processors: The various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processes.

[0168] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include: a first configuration, as typified by client or server computers, in which a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0169] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0170] Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications are possible without departing from the spirit of the present invention.

[0171] DESCRIPTION OF SYMBOLS 12...Drone 13...Gimbal head 14...Camera 16...Remote controller 20...House status providing device 22...Network 24...Terminal device 200...Processor 201...Image acquisition unit 202...Matching processing unit 203...External shape information acquisition unit 204...House image extraction unit 205, 2050...Classification processing unit 206...Association processing unit 207...Synthesis processing unit 210...Memory 220...Database 230...Display device 240...Input / output interface 250...Operation unit 2052, 2054, 2056, 2058...Damage determination AI S10 to S24...Steps

Claims

1. A home status providing device including a processor, The processor, acquiring an aerial image of a target area in which a plurality of houses are present; a matching process for matching the aerial image with the map based on the aerial image and a map stored in a memory; A process of acquiring outer shape information of the house on the aerial image from a result of the matching process and area information of the house included in the map; A process of extracting a house image showing the house from the aerial image based on the acquired exterior information of the house; A classification process of recognizing the state of each of the plurality of houses based on the extracted house image and classifying the house into one of a plurality of first classes based on a recognition result; and associating the classified first class with the house in the aerial image. Home status information providing device.

2. The memory stores three-dimensional area information of the house included in the map, The processor, The position and orientation of the camera that captured the aerial image are estimated by the matching process; performing perspective projection transformation on three-dimensional area information of the house based on the estimated position and orientation of the camera, and acquiring external shape information of the house on the aerial image; The house status providing device according to claim 1.

3. The processor causes a display device to display first information indicating the first class associated with the house on the aerial image in a superimposed manner on the aerial image. The house status providing device according to claim 1.

4. The first information is information about a frame line surrounding the house, and a color or a line type of the frame line differs depending on the first class. The house status providing device according to claim 3.

5. The plurality of first classes are a plurality of first classes including completely destroyed and partially destroyed corresponding to the damage state of the house due to the disaster, The house status providing device according to claim 1.

6. the memory stores a learning model that outputs a classification result indicating the first class when the house image is input; The classification process of the processor uses the learning model stored in the memory, inputs the house image into the learning model, and obtains a classification result estimated by the learning model. The house status providing device according to any one of claims 1 to 5.

7. the learning model stored in the memory includes a plurality of learning models corresponding to types of disasters; The processor selects a learning model corresponding to a type of disaster from the plurality of learning models, and obtains the classification result using the selected learning model. The house status providing device according to claim 6.

8. The memory stores attribute information related to houses included in the map, The processor retrieves attribute information related to the house from the memory and associates the attribute information with the house in the aerial image. The house status providing device according to claim 1.

9. The processor causes a display device to display the first information indicating the first class associated with the house on the aerial image and the attribute information superimposed on the aerial image. The house status providing device according to claim 8.

10. The attribute information related to the house includes the age of the house or the type of the house, The processor, Classifying the age or type of the house into one of a plurality of second classes based on the age or type of the house; Associating the second class with the house on the aerial image; displaying second information indicating the second class on a display device by superimposing the second information on the aerial image; The house status providing device according to claim 8.

11. A home status providing method executed by a home status providing device having a processor, comprising: acquiring an aerial image of a target area having a plurality of houses; A step of performing a matching process by the processor to match the aerial image with the map based on the aerial image and a map stored in a memory; The processor acquires outer shape information of the house on the aerial image from a result of the matching process and area information of the house included in the map; The processor extracts a house image showing the house from the aerial image based on the acquired exterior information of the house; The processor recognizes the state of each of the plurality of houses based on the extracted house image, and classifies the house into one of a plurality of first classes based on a recognition result; The processor associates the classified first class with a house on the aerial image; Methods for providing home status, including:

12. The memory stores three-dimensional area information of the house included in the map, The processor estimates the position and orientation of the camera that captured the aerial image by the matching process, performing perspective projection transformation on three-dimensional area information of the house based on the estimated position and orientation of the camera, and acquiring external shape information of the house on the aerial image; The house status providing method according to claim 11.

13. The processor causes a display device to display first information indicating the first class associated with the house on the aerial image in a superimposed manner on the aerial image. The house status providing method according to claim 11.

14. The first information is information about a frame line surrounding the house, and a color or a line type of the frame line differs depending on the first class. The house status providing method according to claim 13.

15. The plurality of first classes are a plurality of first classes including completely destroyed and partially destroyed corresponding to the damage state of the house due to the disaster, The house status providing method according to any one of claims 11 to 14.