Information processing device, abnormality determination result display method, and program

Abstract

Provided are an information processing device, an abnormality determination result display method, and a program that make it easy to understand, for example, the scale of a disaster and the status of damage to features such as houses in a target area without the need for complex operations. One or more processors execute: processing for acquiring an aerial image; processing for acquiring information relating to a feature on a map from map information; processing for aligning the aerial image and the feature on the map; processing for cutting out the region of the feature within the aerial image to generate a feature image; processing for determining a degree of abnormality of the feature within the feature image on the basis of the feature image; processing for acquiring attribute of the feature within the feature image; processing for narrowing down the feature to be displayed on the basis of at least one of the attribute and the degree of abnormality of each feature; and processing for displaying information relating to the narrowed-down feature on the display device.

Description

Information processing device, abnormality determination result display method and program

[0001] The present disclosure relates to an information processing device, an abnormality determination result display method, and a program, and in particular to an information processing technology that supports understanding of damage conditions of houses and the like based on aerial images.

[0002] Patent Literature 1 proposes a house status providing device and method that can accurately recognize the status of individual houses in a disaster-stricken area and associate the recognition results with individual houses in an aerial image. The processor of the house status providing device described in Patent Literature 1 performs the following processes: acquiring an aerial image of a target area where multiple houses exist, matching the map to the aerial image based on the aerial image and a map stored in memory, acquiring outline information of the houses in the aerial image from the matching result and area information of the houses included in the map, extracting house images showing the houses from the aerial image based on the acquired outline information of the houses, recognizing the status of each of the multiple houses based on the extracted house images and classifying the houses into one of multiple first classes based on the recognition result, and associating the classified first classes with the houses in the aerial image. This processor recognizes the damage status of each house based on the extracted house images, classifies the house into one of multiple classes (sustained, partially destroyed, or completely destroyed) based on the recognition results, associates the classified class with the house in the aerial image, and then superimposes information indicating the classification result on the aerial image and displays it on an image display device.

[0003] Patent Document 2 proposes an image analysis system that uses aerial images (equivalent to aerial photographs) to determine the damage status of houses with high accuracy. The destroyed house extraction unit in the image analysis system described in Patent Document 2 determines the damage level of houses from the aerial images. Based on the locations of houses determined to be destroyed, users to be notified are selected, and a notification is sent to the selected users inquiring about the situation. Auxiliary information received from the users in response to the notification is further used to determine the damage level.

[0004] Patent Document 3 proposes a determination result estimation device, a determination result estimation method, and a program for estimating the determination result of a secondary investigation without conducting a secondary investigation on-site in order to improve the efficiency of residential damage certification investigations. The processor of the determination result estimation device described in Patent Document 2 acquires information about the house, acquires the determination result of a primary investigation related to the house, extracts characteristic information about the house from the house information, estimates the determination result of a secondary investigation related to the house that is different from the primary investigation based on the determination result of the primary investigation and the characteristic information about the house, and presents the estimated determination result of the secondary investigation.

[0005] Patent Document 4 proposes a method for estimating earthquake damage that improves the accuracy of estimating damage to structures during earthquake disasters by using information other than seismometer information. In the method described in Patent Document 4, businesses that install infrastructure such as underground pipes in cities install multiple seismometers and acquire images taken from the sky using satellites or other devices. When an earthquake disaster occurs, collapsed buildings, liquefaction, ground deformation, etc. are confirmed based on the images, and a damage estimate is made based on the confirmation results, taking into account the information obtained from the seismometers. The estimated results are also distributed to contracted customers via the Internet and can be used effectively for disaster recovery, etc.

[0006] International Publication No. 2023 / 167017 Japanese Patent Application Laid-Open No. 2023-1469 International Publication No. 2022 / 059604 Japanese Patent Application Laid-Open No. 2003-287573 Japanese Patent Application Laid-Open No. 2011-113237

[0007] When a disaster such as an earthquake occurs, it is necessary to quickly grasp the damage status of houses and other structures in the target area. One possible technology to support damage status investigations is to use a drone to take aerial photographs of the target area, extract the areas of individual houses from the obtained aerial images, perform image analysis, and build a system to determine the damage status of each house.

[0008] However, simply linking the results of damage assessments obtained from images taken after a disaster (for example, assessments of total destruction, partial destruction, or less than partial destruction) to geographic space and displaying them makes it difficult for humans to understand the scale and extent of a disaster.

[0009] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device, an abnormality determination result display method, and a program that make it easy to grasp the damage status and scale of disasters of buildings and other features in a target area without requiring complicated operations.

[0010] An information processing device according to a first aspect of the present disclosure is an information processing device having one or more processors, and the one or more processors perform the following processes: acquiring an aerial image; acquiring information about features on the map from map information; aligning the aerial image with the features on the map; generating a feature image by cutting out an area of ​​the feature in the aerial image that corresponds to the feature on the map from the aerial image; determining the degree of abnormality of the feature in the feature image based on the feature image; acquiring attributes of the feature in the feature image; narrowing down the features to be displayed based on at least one of the feature attributes and the degree of abnormality; and displaying information about the narrowed down features to be displayed on a display device.

[0011] According to the first aspect, the features to be displayed are narrowed down from the perspective of at least one of the features' attributes and abnormality level, and information about features that meet the conditions is displayed, making it easy to understand the abnormal conditions of features such as houses in the target area, such as the damage situation and scale of the disaster, without requiring complex operations.

[0012] The information processing device of the second aspect may be configured such that, in the information processing device of the first aspect, the alignment process includes a process of calculating camera parameters representing the position and attitude of the camera at the point where the aerial image was taken from one or more aerial images, and a process of overlaying features on the map on the aerial image using the camera parameters.

[0013] An information processing device according to a third aspect may be configured in the information processing device according to the first or second aspect, wherein the features include houses.

[0014] An information processing device according to a fourth aspect may be configured such that, in the information processing device according to any one of the first to third aspects, the attributes of the feature include at least one attribute of building type, building material, roof type, building structure, floor level, floor area of ​​the first floor, construction date, address, and geographic coordinates.

[0015] An information processing device according to a fifth aspect may be configured such that, in an information processing device according to any one of the first to fourth aspects, the narrowing-down process is performed using a logical expression including at least one condition related to the attributes of the features and the degree of abnormality.

[0016] An information processing device according to a sixth aspect may be configured in the information processing device according to the fifth aspect such that one or more processors receive input of a logical expression and execute a narrowing-down process using the specified logical expression.

[0017] The information processing device according to a seventh aspect may be configured in the information processing device according to the sixth aspect, further comprising an input device used for inputting logical expressions, and a display device.

[0018] An information processing device according to an eighth aspect may be configured such that, in the information processing device according to any one of the first to seventh aspects, the information about the feature to be displayed includes an image of the feature.

[0019] An information processing device according to a ninth aspect may be configured in the information processing device according to the eighth aspect, wherein the information about the feature to be displayed includes attributes of the feature.

[0020] An information processing device according to a tenth aspect may be configured such that, in an information processing device according to any one of the first to ninth aspects, the display process includes a process of displaying feature images of the feature to be displayed in order of the degree of abnormality.

[0021] An information processing device according to an eleventh aspect may be configured such that, in an information processing device according to any one of the first to tenth aspects, the display process includes a process of changing the display color of the border line of the feature image for the feature to be displayed depending on the degree of abnormality.

[0022] An information processing device according to a twelfth aspect may be configured such that, in an information processing device according to any one of the first to eleventh aspects, the display process includes a process of displaying a map including the location of the feature to be displayed and a point mark indicating the position of the feature on the map.

[0023] An information processing device according to a thirteenth aspect may be configured in the information processing device according to the twelfth aspect, such that the information about the feature to be displayed is displayed in association with a point mark.

[0024] An information processing device according to a fourteenth aspect may be configured such that, in an information processing device according to any one of the first to thirteenth aspects, the process of determining the degree of abnormality is performed using a trained model that has been trained by machine learning to receive an input of a feature image and output the degree of abnormality of the feature image.

[0025] An information processing device according to a fifteenth aspect is the information processing device according to the fourteenth aspect, wherein the trained model may be an anomaly detection model generated by unsupervised learning.

[0026] An information processing device according to a 16th aspect may be configured such that, in an information processing device according to any one of the first to fifteenth aspects, one or more processors acquire attributes of features from a database containing at least some data from the fixed asset tax register and the real estate register.

[0027] An information processing device according to a seventeenth aspect is the information processing device according to any one of the first to sixteenth aspects, wherein the aerial images are images of a disaster-stricken area.

[0028] The method for displaying abnormality determination results relating to the 18th aspect includes the steps of one or more processors performing the following steps: acquiring an aerial image; acquiring information about features on the map from map information; aligning the aerial image with the features on the map; generating a feature image by cutting out from the aerial image an area of ​​the feature in the aerial image that corresponds to the feature on the map; determining the degree of abnormality of the feature in the feature image based on the feature image; acquiring attributes of the feature in the feature image; narrowing down the features to be displayed based on at least one of the feature attributes and the degree of abnormality; and displaying information about the narrowed down features to be displayed on a display device.

[0029] The one or more processors may automatically execute the processing of each step according to a program, or may accept instructions input from a user for some or all of the steps and execute the processing in accordance with the received instructions.

[0030] The method for displaying an abnormality determination result according to the eighteenth aspect may be configured to include the same specific aspects as the information processing device according to any one of the second to seventeenth aspects.

[0031] The program of the 19th aspect enables a computer to perform the following functions: acquire aerial images; acquire information about features on a map from map information; align the aerial images with the features on the map; generate a feature image by cutting out from the aerial images areas of features in the aerial images that correspond to the features on the map; determine the degree of abnormality of the features in the feature image based on the feature image; acquire attributes of the features in the feature image; narrow down the features to be displayed based on at least one of the feature attributes and the degree of abnormality; and display information about the narrowed down features to be displayed on a display device.

[0032] The program according to the nineteenth aspect may be configured to include the same specific aspects as the information processing device according to any one of the second to seventeenth aspects.

[0033] The present disclosure also includes a non-transitory, tangible computer-readable storage medium on which the program according to the nineteenth aspect is stored.

[0034] According to the present disclosure, it becomes easy to understand the damage status and scale of disasters of land features such as houses in a target area without requiring complicated operations.

[0035] FIG. 1 is a schematic diagram illustrating an example of the configuration of a photographed image processing system according to an embodiment. FIG. 2 is a block diagram illustrating an example of the electrical configuration of a drone equipped with a camera. FIG. 3 is a block diagram illustrating an example of the hardware configuration of an information processing device according to an embodiment. FIG. 4 is an example of an aerial image including a house matched with a map. FIG. 5 is an example of a display when the logical expression for the condition for the features to be displayed is "display all." FIG. 6 is an example of a display when the features to be displayed are narrowed down based on a condition combining the roof type and the anomaly level. FIG. 7 is an example of a display when the houses to be displayed are narrowed down based on the anomaly level, and only houses with very severe damage are displayed. FIG. 8 is an example of a display when the houses to be displayed are narrowed down based on the anomaly level, and only houses with an average level of damage are displayed. FIG. 9 is an example of a display when the display is limited to wooden houses with old earthquake-resistant structures. FIG. 10 is an example of a display when the administrative district of a specific address is specified and the display is limited to wooden houses with old earthquake-resistant structures. FIG. 11 is a functional block diagram showing the functional configuration of an information processing device that operates as an abnormality determination result display device.

[0036] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In this specification, the same components are designated by the same reference numerals, and redundant explanations will be omitted where appropriate.

[0037] [Configuration example of a photographed image processing system] As an application example of the present disclosure, an example of a system will be described in which damage status of houses and other structures in an image is determined by image analysis from an image taken using a camera mounted on a drone, and various information including the determination results is provided.

[0038] 1 is a schematic diagram showing an example of the configuration of a photographed image processing system 10 according to an embodiment. The photographed image processing system 10 includes a drone 12 for aerial photography, a camera 14 mounted on the drone 12, a remote controller 16, and an information processing device 20. The drone 12 is an unmanned aerial vehicle that is remotely controlled using the remote controller 16. The drone 12 may have an autopilot function that flies according to a program. The drone 12 is an example of an air vehicle.

[0039] The camera 14 is mounted on the drone 12 via a gimbal head 13. The camera 14 includes an optical system, an image sensor, and a signal processing circuit (not shown). The optical system includes one or more lenses such as a focus lens. The image sensor may be, for example, a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor.

[0040] The camera 14 generates digital image data of the photographed subject by processing signals obtained from the image sensor using a signal processing circuit. The digital image data generated by the camera 14 can be used as a photographed image. Images photographed using the camera 14 (photographed images) can be stored in an internal storage built into the drone 12 and / or in a storage device such as a memory card removably attached to the drone 12. Images photographed using the camera 14 can also be transferred to the remote controller 16, the information processing device 20, and other terminal devices 24 using wireless communication.

[0041] The remote controller 16 is a transmitter that controls the operation of the camera 14 and the drone 12 via wireless communication. The wireless communication may be in the form of a wireless local area network (LAN), a communication format using radio waves in the 2.4 GHz or 5.7 GHz band, or a format using a mobile communication network. The communication format for the control signals for operating the drone 12 and the communication format for transferring images captured by the camera 14 may be different or may be the same.

[0042] The remote controller 16 includes left and right sticks for controlling the flight operation of the drone 12, a lever for operating the gimbal head 13, a shooting button for instructing the camera 14 to take a picture, and a shooting mode button for switching between video shooting and still image shooting. By employing a touch panel display for the display 16A, the shooting button and other operation buttons can be realized by the touch panel display.

[0043] The live video captured by the camera 14 can be displayed on the display 16A of the remote controller 16. The remote controller 16 can also grasp the status of the drone 12, such as its flight position and speed, in real time based on data from various sensors provided on the drone 12. Flight information indicating the status of the drone can be displayed on the display 16A.

[0044] 1 is an example of an image captured using the camera 14. In this embodiment, at least one still image is captured from the air, and the captured image is processed in the information processing device 20.

[0045] The information processing device 20 is configured using a computer. The computer applied to the information processing device 20 may be a server, a personal computer, or a workstation.

[0046] The information processing device 20 can perform data communication with the remote controller 16 and the terminal device 24 via a network 22. The network 22 may be a local area network or a wide area network. The information processing device 20 acquires various information from the drone 12 and the camera 14. The information processing device 20 can also acquire map data of the imaging target range from a geographic information system (not shown) via the network 22. The map data may be acquired in advance before imaging, or may be acquired after imaging.

[0047] The terminal device 24 may be a mobile information terminal such as a smartphone or a tablet terminal. The terminal device 24 includes a display 24A. The terminal device 24 may have the functionality of the remote controller 16. The terminal device 24 may also have the processing functionality of the information processing device 20.

[0048] [Configuration Example of Drone with Camera] Figure 2 is a block diagram that schematically illustrates an example of the electrical configuration of a drone 12 equipped with a camera 14. The drone 12 includes a Global Navigation Satellite System (GNSS) receiver 30, a barometric pressure sensor 32, a direction sensor 34, an inertial measurement unit (IMU) 36, and a motor 38. The IMU 36 includes a gyro sensor 362, an acceleration sensor 364, and a temperature sensor 366, and detects translational and rotational motion in three orthogonal axes. The motor 38 is a power source that rotates a rotor (not shown), and the drone 12 includes multiple motors 38 that drive multiple rotors.

[0049] The GNSS receiver 30 acquires position information including the latitude and longitude of the drone 12. The barometric pressure sensor 32 detects the barometric pressure in the drone 12. The drone 12 can acquire its altitude based on the barometric pressure detected using the barometric pressure sensor 32. Note that the term "acquire" includes the concept of generating information by data processing such as calculation. The latitude, longitude, and altitude of the drone 12 constitute position information of the drone 12 and the camera 14.

[0050] The orientation sensor 34 may be, for example, a geomagnetic sensor, and may detect the azimuth angle at which the lens of the camera 14 is facing.

[0051] The gyro sensor 362 detects a roll angle representing the angle of rotation about the roll axis, a pitch angle representing the angle of rotation about the pitch axis, and a yaw angle representing the angle of rotation about the yaw axis. The drone 12 acquires attitude information of the drone 12 based on the rotation angles acquired using the gyro sensor 362. Note that some or all of the sensors, such as the GNSS receiver 30, the barometric pressure sensor 32, the orientation sensor 34, and the IMU 36, may be located on the camera 14 side.

[0052] The drone 12 includes a processor 40, a storage device 42, and a communication interface 44. The storage device 42 may be a memory, an internal storage device, an external storage device, or a combination thereof. The processor 40 serves as a flight controller and performs various calculations necessary for flight control of the drone 12 based on sensor data obtained from various sensors.

[0053] The communication interface 44 is a communication unit that performs wireless communication with the remote controller 16, etc. The communication interface 44 may also include a communication terminal that supports wired communication. Furthermore, the drone 12 includes a battery and a battery charging terminal (not shown).

[0054] 3 is a block diagram showing an example of the hardware configuration of the information processing device 20. The information processing device 20 includes a processor 202, a computer-readable medium 204 which is a non-transitory tangible entity, a communication interface 206, and an input / output interface 208.

[0055] The processor 202 includes a central processing unit (CPU) and may include a graphics processing unit (GPU). The processor 202 is connected to a computer-readable medium 204, a communication interface 206, and an input / output interface 208 via a bus 210.

[0056] The computer-readable medium 204 includes a memory 212 serving as a primary storage device and a storage 214 serving as a secondary storage device. The computer-readable medium 204 may be, for example, a semiconductor memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these. The computer-readable medium 204 stores various programs, data, and the like, including an image processing program and a display control program.

[0057] The information processing device 20 may include an input device 222 and a display device 224. The input device 222 and the display device 224 are connected to the bus 210 via the input / output interface 208. The input device 222 is configured by, for example, a keyboard, a mouse, a multi-touch panel, or other pointing device, or a voice input device, or an appropriate combination of these.

[0058] The display device 224 is configured by, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination of these.

[0059] [Overview of Processing Functions of Information Processing Device 20] The information processing device 20 functions as an image processing device that processes images captured by the camera 14. The processor 202 of the information processing device 20 acquires images captured by the camera 14 and information related to the approximate position (approximate shooting position) and approximate attitude of the camera 14 when the images were captured. The information related to the approximate position and approximate attitude of the camera 14 may be position information and attitude information indicated by sensor data obtained by sensors such as the GNSS receiver 30, barometric pressure sensor 32, orientation sensor 34, and IMU 36 mounted on the drone 12.

[0060] The position information and attitude information indicated by the sensor data of the drone 12 includes GNSS positioning errors, errors caused by the sensor, and the like, and indicates the approximate camera position (shooting position) and camera attitude (shooting attitude). The position information and attitude information at the time of shooting indicated by the sensor data of the drone 12 are referred to as camera position information and camera attitude information at the time of shooting. The camera information including the camera position information and camera attitude information at the time of shooting may be recorded as metadata such as tag information attached to the image file of the captured image, or may be recorded as a separate file linked to the image file.

[0061] The processor 202 calculates camera parameters that represent the position and orientation of the camera at the point where each image was captured, based on one or more images captured by the camera 14 and the camera information for each image. The processor 202 may calculate the camera parameters based on, for example, information about ground control points (GCPs) captured in the image. The processor 202 may also use map information to calculate the camera parameters based on the correspondence between features in the image and features on the map. The map information includes information about the geographical position and shape of the features.

[0062] The processor 202 may use the technology described in Patent Document 1 to match the map and the image and calculate the camera parameters. The camera parameters correspond to transformation parameters that convert geospatial coordinates into image coordinates. Once the parameters that convert geospatial coordinates into image coordinates are determined, the image coordinates can be converted back to geospatial coordinates by inverse transformation. Therefore, calculating the transformation parameters that convert geospatial coordinates into image coordinates is understood to include the concept of calculating the transformation parameters that convert image coordinates into geospatial coordinates. The transformation matrix applied to the coordinate conversion between image coordinates and geospatial coordinates is called the camera matrix, and is expressed as the product of the intrinsic parameter matrix and the extrinsic parameter matrix. Once the intrinsic parameter matrix is ​​set, it can be used repeatedly as long as the focal length of the imaging optical system is not changed. Therefore, the transformation parameters that actually matter are the elements of the extrinsic parameter matrix.

[0063] In the map information referenced by the information processing device 20, for example, each house is assigned a house ID (Identification) as an identification code for identifying the house, and location data indicating the respective positions of multiple specific points constituting the perimeter of the house is recorded, linked to the house ID. The location data for each specific point may be three-dimensional data of latitude, longitude, and altitude. In the case of Japan, map data including such geographic coordinate data can be obtained, for example, from basic map information provided by the Geospatial Information Authority of Japan. Alternatively, such map data can be obtained from the OpenStreetMap database.

[0064] The coordinate system of the geographic coordinates may be a geographic coordinate system expressed by latitude and longitude, or may be a coordinate system to which a map projection method such as the Universal Transverse Mercator (UTM) coordinate system is applied. When the map data is expressed as latitude and longitude data, the processor 202 preferably converts the map data including the latitude and longitude into Cartesian coordinate data. The processor 202 can obtain three-dimensional geospatial coordinates (X, Y, Z) from the map information.

[0065] The processor 202 compares features in the aerial image with features on the map using map information and camera parameters about the geographic space including the area photographed by the camera 14. Features on the map refer to features on the map information shown in the map information. Here, an example will be described in which a house is used as the target feature.

[0066] The processor 202 acquires features on the map from the map information and performs a process of superimposing the features on the map using the camera parameters. The processor 202 converts the geographic coordinates of the features on the map into coordinates within the image through coordinate transformation using the camera parameters, superimposes the features on the map on the photographed image, and associates the features on the image with the features on the map.

[0067] This correspondence allows the houses in the image to be identified as houses on the map, and the processor 202 extracts the areas of the identified houses in the image and cuts out (individual) house images for each house. A house image is an image in which the main subject is a house. A house image is an example of a feature image.

[0068] An example of an aerial image IM1 is shown in Figure 4. Aerial image IM1 shows multiple houses HS, and the areas of the houses in the image that are matched with the houses on the map are surrounded by a rectangular frame FR. Processor 202 cuts out the areas of each house HS from aerial image IM1 as house images. Aerial image IM1 is an example of a first aerial image in the present disclosure.

[0069] The processor 202 assigns information indicating the attributes of the house to the extracted house image. The term "assign" includes the concept of linking (associating). The attributes of the house include, for example, the building type, building materials, roof type, building structure, construction date, floor number, and floor area of ​​the first floor.

[0070] Building types include, for example, residential, apartment complex, inn, and store. Building materials include, for example, wood construction and steel-reinforced concrete construction. Roof types include, for example, tiled roofing, slate roofing, and galvanized steel roofing. Building structure levels include, for example, single-story and two-story buildings.

[0071] The attributes of such a building may be defined, for example, in accordance with the contents prescribed in Article 113, Paragraph 1 of the Real Estate Registration Regulations and the contents prescribed in the Real Estate Registration Procedures Handling Guidelines.

[0072] The processor 202 may acquire the attributes of the target house from a database such as a real estate register or a fixed asset tax ledger. Note that the data of the real estate register or the fixed asset tax ledger may be included in the map information.

[0073] By similarly extracting house images from each of a plurality of aerial images taken of the disaster area, a large number of house images in the target area are obtained.

[0074] The processor 202 performs image analysis on each house image extracted from the aerial image and determines the degree of abnormality for each house. Artificial intelligence (AI) can be used for the process of determining the degree of abnormality. For example, the processor 202 estimates (infers) the degree of abnormality using an anomaly detection model that has been trained in advance through machine learning.

[0075] The processor 202 displays information including the determination result of the degree of abnormality for the house on the display device 224. The processor 202 narrows down the houses to be displayed based on the attributes and the degree of abnormality of the houses, and displays information about the houses that meet the desired conditions on the display device 224.

[0076] As a means for narrowing down the display targets, for example, a logical expression that specifies a condition can be used. The logical expression can specify desired conditions using logical operators (e.g., AND and OR) for multiple condition elements such as feature attributes, anomaly levels, addresses, and geographic coordinates. Note that the geographic coordinates may be expressed in latitude and longitude.

[0077] The input device 222 is used to input a logical expression. The logical expression may be specified by key input from the input device 222 in a command-like format, or may be specified using a GUI (Graphical User Interface) method that supports the creation of a logical expression by allowing the user to select desired elements from menus or icons of pre-prepared condition items and logical operator selection candidates. By using the logical expression, the user can freely narrow down the images to be observed.

[0078] [Display Example 1: Example with Mixed Building Types] Figure 5 shows a display example when the logical expression of the condition for the features to be displayed is "display all." In this case, all house images acquired from the aerial imagery are targeted for display, regardless of the attributes and abnormality levels of each feature, and are displayed on the display device 224. There are various possible methods (display formats) for displaying multiple house images. For example, as shown in Figure 5, the feature images may be displayed in order of abnormality level based on the abnormality level value determined by AI from each feature image.

[0079] The degree of abnormality is determined by AI as a value ranging from 0% to 100% according to the degree of abnormality in the house. Note that the expression of the degree of abnormality is not limited to percentage, and may be expressed as a number between "0" and "1."

[0080] In the example of Fig. 5, the multiple house images are displayed in an order in which the degree of abnormality is relatively smaller toward the left side of Fig. 5 and relatively larger toward the right side of Fig. 5. In order to inform the user that the images are arranged in this order of abnormality, it is also preferable to display the word "normal" at the left end and the word "abnormal" at the right end of a double-headed arrow parallel to the arrangement direction of the multiple house images (the horizontal direction in Fig. 5).

[0081] Instead of or in combination with the text displays of "normal" and "abnormal," text displays such as "low abnormality level" and "high abnormality level" may be used (see Figure 6), or the value of the abnormality level may be displayed (see Figures 7 and 8).

[0082] Furthermore, in order to visually indicate whether or not there is an abnormality (damage) in each house image, it is also preferable to configure the display color of the frame line FR of the house image to be different depending on the value of the abnormality level. For example, the display color of the frame line FR of a house image whose abnormality level exceeds a threshold may be "red," and the display color of the frame line FR of a house image whose abnormality level is less than the threshold may be "green."

[0083] In the case of display example 1 shown in Figure 5, images of houses with various attributes such as building type (e.g., residential, warehouse, garage, etc.), building material (e.g., wooden, steel-reinforced concrete, steel frame, etc.), roof type (e.g., tiled roof, slate roof, galvanized steel roof, etc.), floor number, and age of construction are mixed together. As a result, the durability of each house differs, making it difficult to accurately grasp the scale of damage in the target area.

[0084] [Display Example 2: Example of selection by roof type] Figure 6 shows an example of a display when the features to be displayed are narrowed down by a condition combining the type of roof and the abnormality level. Figure 6 shows an example of a display when the abnormality level exceeds 50% and the roof type is selected as "tiled roof." In this case, the logical expression for narrowing down is "(abnormality level > 50%) & (roof type = "tiled roof")."

[0085] The processor 202 selects house images that satisfy the logical formula from a plurality of house images extracted from the aerial photograph, and displays the selected images in order of abnormality level (damage level). As a result, the house images that satisfy the conditions of the logical formula are sorted and displayed in order of the abnormality level (damage level) value on the display device 224. If there are many house images to be displayed that satisfy the logical formula and it is not possible to display all the images on one screen, the images may be displayed by scrolling or splitting into pages.

[0086] The processor 202 may also display the specified logical expression and the number of houses that satisfy the conditions of the logical expression.

[0087] [Display Example 3: Example of displaying only houses with severe damage] Figure 7 shows an example of displaying only houses with extremely severe damage. Figure 7 shows an example of display when the conditions specified are that the abnormality level exceeds 90% and the roof type is "tiled roof." In this case, the logical expression for narrowing down the results is "(abnormality level > 90%) & (roof type = "tiled roof")."

[0088] 7 shows an example in which house images to be displayed that satisfy the logical formula are displayed in the display area on the right side of the screen, but this is to make it easier to understand in comparison with Fig. 6 that only images with a high abnormality level exceeding 90% are to be displayed. The display form is not limited to that shown in Fig. 7, and house images to be displayed that satisfy the logical formula may also be displayed in the display area in the center of the screen.

[0089] [Display Example 4: Example of displaying houses with average damage] Figure 8 shows an example of a display when narrowing down the display to houses with an average level of damage. Figure 8 shows an example of a display when houses with an abnormality level greater than 40% and less than 60% are selected. In this case, the logical expression for narrowing down the display is "(40% < abnormality level) & (abnormality level < 60%)".

[0090] As shown in Fig. 8, images of houses that satisfy the specified conditions are displayed in order of abnormality level. For convenience of illustration, images of houses with "tiled roof" roofs are displayed in Fig. 8, but house images that satisfy the conditions of the logical formula may be displayed regardless of the roof type. Furthermore, the display is not limited to the form in which selected house images are displayed in order of abnormality level as shown in Fig. 8, and it is also possible to display house images that satisfy the conditions of the logical formula in a matrix (two-dimensional array).

[0091] [Display Example 5: Example of selecting houses with old earthquake-resistant construction] Figure 9 is an example of a display when narrowing down the display to wooden houses with old earthquake-resistant construction. In Japan, the Building Standards Act was revised in 1981, and the earthquake resistance standards for buildings before and after this revision are different. Buildings that received confirmation applications by May 31, 1981 are buildings that comply with the "old earthquake resistance standards," while buildings that received confirmation applications after June 1, 1981 are buildings that comply with the "new earthquake resistance standards."

[0092] 9 shows an example in which the houses to be displayed are narrowed down to those with an abnormality level (damage level) by specifying building information (attribute information) such as building structure as "wooden," construction year before 1981, and abnormality level above 50%. In this case, the logical formula for narrowing down the houses is "(structure = wood) & (construction year < 1981) & (abnormality level > 50%)."

[0093] According to the display example 5 shown in FIG. 9, it is easy to grasp the damage status of buildings that satisfy the conditions specified by the logical expressions.

[0094] [Display Example 6: Example of selecting houses in the administrative district of a specified address] Figure 10 shows an example of a display when the administrative district of a specific address is specified and the display is narrowed down to wooden houses with old earthquake-resistant structures. Figure 10 shows an example of a display where the specific address is specified as "1-chome, B-cho, A-city," the building structure is set to "wooden," and the year of construction is set to 1981 or earlier, narrowing down the houses to be displayed and displaying them in order of abnormality level (damage level). In this case, the logical formula for narrowing down is "(address = "1-chome, B-cho, A-city) & (structure = wooden) & (year of construction < 1981) & (abnormality level > 50%)."

[0095] As shown in FIG. 10 , a map of the administrative district containing the specified address and information on houses that satisfy the conditions of the logical expression are displayed. For example, the left side of the screen is a map display area, and the right side of the screen is a house information display area. The map display area displays a map containing the administrative district of the specified address, and the area of ​​the target administrative district is displayed in a visually differentiated manner. The differentiated display may be, for example, a highlight display, a thick border display, a color-coded display, or a combination of these. Point marks indicating the locations of houses that satisfy the conditions of the logical expression are displayed on the map shown in FIG. 10. Each point mark is associated with house information, and for example, when a user selects a point mark, the information on the corresponding house is displayed in a differentiated manner.

[0096] The house information display area displays images and attribute information of houses that satisfy the conditions of the logical formula. In the house information display area, information on houses that satisfy the conditions of the logical formula is displayed in order of abnormality level.

[0097] 5 to 10, the user can specify desired conditions from the input device 222. The processor 202 can accept, from the user via the input device 222, specification of conditions relating to the attributes and abnormality levels of houses to be observed, and can display information narrowed down to houses that satisfy the accepted conditions.

[0098] [Method for Displaying Abnormality Determination Results Executed by Information Processing Device 20] The processor 202 of the information processing device 20 that realizes the displays exemplified in FIGS. 5 to 10 executes the following steps 1 to 8.

[0099] [Step 1] The processor 202 acquires an aerial image of the target area.

[0100] [Step 2] The processor 202 obtains information about features on the map from the map database.

[0101] [Step 3] The processor 202 uses the camera parameters to align the aerial image with the features on the map.

[0102] [Step 4] The processor 202 cuts out from the aerial image the area of ​​the feature in the aerial image that corresponds to the feature on the map, and generates a feature image.

[0103] [Step 5] The processor 202 determines the degree of abnormality of the features in the feature image based on the extracted feature image.

[0104] [Step 6] The processor 202 obtains attributes of the features in the feature image from a feature attribute database such as a fixed asset tax register.

[0105] [Step 7] The processor 202 narrows down the features to be displayed based on at least one condition of the feature attributes and the abnormality level.

[0106] [Step 8] The processor 202 causes the display device 224 to display information such as feature images and attributes relating to the selected features to be displayed.

[0107] This makes it possible to narrow down the features to be observed from a large number of feature images and display information about features that meet the conditions.

[0108] [Functional Configuration of Information Processing Device 20] Figure 11 is a functional block diagram showing the functional configuration of the information processing device 20 operating as an anomaly determination result display device. The information processing device 20 includes an image acquisition unit 230, a camera information acquisition unit 232, a map information acquisition unit 235, a feature acquisition unit 236, an attribute acquisition unit 238, a camera parameter calculation unit 240, an alignment unit 242, a feature image clipping unit 244, an attribute assignment unit 246, an anomaly degree determination unit 270, a display narrowing unit 274, a display image generation unit 282, and a display control unit 284. Each of these units can be realized by computer hardware and software. Software is synonymous with a program.

[0109] The image acquisition unit 230 acquires aerial images. The aerial images are images captured from the air by the camera 14. The camera information acquisition unit 232 acquires camera information including camera position information and camera attitude information at the time the aerial images were captured. The camera information acquisition unit 232 includes a camera position information acquisition unit 233 that acquires camera position information, and a camera attitude information acquisition unit 234 that acquires camera attitude information.

[0110] The map information acquisition unit 235 includes a feature acquisition unit 236 and an attribute acquisition unit 238. The feature acquisition unit 236 acquires necessary map information related to features from a map database 260. The map database 260 may include some or all of the GIS data, for example, basic map information, location reference information, national land numerical information, map data attached to land registry documents, and city, ward, town, and village boundary data. The term "map data" includes the concept of GIS data.

[0111] The information processing device 20 may include a map database 260. In this case, the information processing device 20 includes a map data storage unit that stores the map database 260. The map data storage unit may be a storage area of ​​the computer-readable medium 204 in the information processing device 20, or may be a storage area of ​​an external storage device separate from the information processing device 20.

[0112] The attribute acquisition unit 238 acquires information indicating attributes of the target feature from the feature attribute database 262. The feature attribute database 262 may include, for example, some or all of the data in a real estate registry and a fixed asset tax register. The feature attribute database 262 is an example of a "database" in this disclosure. The feature attribute database 262 may be linked to the map database 260. Furthermore, the feature attribute database 262 may be included in the map database 260.

[0113] The information processing device 20 may include a feature attribute database 262. In this case, the information processing device 20 includes a feature attribute data storage unit that stores the feature attribute database 262. The feature attribute data storage unit may be a storage area of ​​the computer-readable medium 204 in the information processing device 20, or may be a storage area of ​​an external storage device separate from the information processing device 20.

[0114] The camera parameter calculation unit 240 calculates the camera parameters of the point where the aerial image was taken, based on the aerial image acquired via the image acquisition unit 230 and the camera information acquired via the camera information acquisition unit 232. The calculated camera parameters are applied to the feature matching process in the positioning unit 242.

[0115] The positioning unit 242 uses the aerial image acquired via the image acquisition unit 230, the feature information acquired via the feature acquisition unit 236, and the camera parameters to perform a process of overlaying the aerial image with the features on the map, and matches the features shown in the aerial image with the features on the map. The positioning unit 242 may include a geocoding processing function that matches geospatial coordinates with information such as addresses.

[0116] The feature image cutting unit 244 cuts out a feature image from the area of ​​the target feature in the aerial image based on the matching result by the alignment unit 242. In this embodiment, the target feature is a house, and an image of the house is cut out from the aerial image.

[0117] The attribute assigning unit 246 assigns attributes to the feature images extracted by the feature image extracting unit 244. The attribute assigning unit 246 associates information indicating the attributes of the target feature, acquired via the attribute acquiring unit 238, with the feature image. In this way, an attributed image, which is a feature image to which the feature attributes have been assigned, is generated. The generated attributed image is stored in a data storage unit (not shown). The data storage unit may be a storage area of ​​the computer-readable medium 204 (see FIG. 3 ).

[0118] The abnormality degree determination unit 270 determines the degree of abnormality of a feature in a feature image based on the feature image extracted from the aerial image. The degree of abnormality may also be referred to as the "degree of damage" or "degree of damage." The abnormality degree determination unit 270 may be configured to determine the degree of abnormality using an AI model 272. The AI ​​model 272 is a trained model that has been trained by machine learning so as to receive an input of a feature image and output an abnormality degree. Note that the AI ​​model 272 is essentially a program.

[0119] The AI ​​model 272 may be, for example, an anomaly detection model generated by unsupervised learning. The unsupervised learning may be good learning.

[0120] The AI ​​model 272 may output a numerical value indicating the degree of anomaly in the form of a continuous value. The value of the degree of anomaly may be normalized. For example, the AI ​​model 272 may output a numerical value ranging from "0" to "1" depending on the degree of anomaly of the feature depicted in the input feature image. The degree of anomaly output from the AI ​​model 272 may also be converted into a percentage format ranging from 0% to 100%.

[0121] If the abnormality level value output from the AI ​​model 272 exceeds a threshold, it may be determined to be "abnormal," and if the abnormality level value is below the threshold, it may be determined to be "normal."

[0122] The AI ​​model 272 is not limited to an anomaly detection model, but may also be a classification model trained to estimate an anomaly rank (class) indicating the degree of damage to a house. Such a classification model may be generated, for example, by performing supervised learning using, as training data, images associated with labels indicating the correct anomaly level.

[0123] The display narrowing unit 274 performs processing to narrow down the display targets of the determination results by the abnormality degree determination unit 270. The display narrowing unit 274 narrows down the display targets of the features based on at least one of the feature attributes and the abnormality degree.

[0124] The display narrowing unit 274 accepts input of a logical expression that is a narrowing condition via the input device 222, and narrows down the features to be displayed according to the specified logical expression. The features to be displayed may also be referred to as features to be observed. The user can specify the conditions for the features to be observed via the input device 222.

[0125] The logical expression that is the condition for narrowing down may be pre-programmed, and the display narrowing unit 274 may automatically narrow down the features to be displayed in accordance with the program.

[0126] The processing result of the display narrowing down unit 274 is sent to the display image generating unit 282 .

[0127] The display image generation unit 282 performs processing to generate an image to be displayed on the display device 224. The display image generation unit 282 can generate various display images such as those illustrated in Figures 5 to 10. The display image generation unit 282 can also apply camera parameters to convert three-dimensional geographic coordinate data obtained from map information into coordinates within the image, and from the conversion result, generate a composite image for display in which map information aligned with the captured image is superimposed on the captured image. Furthermore, the display image generation unit 282 can generate various images, such as a display image displaying an investigation route for a damage assessment investigation based on the abnormality level determination results for each house, and a display image displaying an investigation schedule.

[0128] The display control unit 284 generates data for display on the display device 224. The display image generated by the display image generation unit 282 is displayed on the display device 224 via the display control unit 284.

[0129] [Hardware configuration of each processing unit] The hardware structure of processing units that execute various processes, such as the image acquisition unit 230, camera information acquisition unit 232, camera position information acquisition unit 233, camera attitude information acquisition unit 234, map information acquisition unit 235, feature acquisition unit 236, attribute acquisition unit 238, camera parameter calculation unit 240, alignment unit 242, feature image cropping unit 244, attribute assignment unit 246, abnormality degree determination unit 270, display narrowing down unit 274, display image generation unit 282, and display control unit 284 of the information processing device 20, is, for example, various processors as shown below.

[0130] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units, GPUs, which are processors specialized for image processing, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration designed specifically for executing specific processes.

[0131] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0132] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0133] [Regarding the program that operates the computer] A program that causes a computer to realize some or all of the processing functions of the information processing device 20 can be recorded on a computer-readable medium, such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible, non-transitory information storage medium, and the program can be provided through this information storage medium.

[0134] In addition, instead of providing the program by storing it on such a tangible, non-transitory computer-readable medium, it is also possible to provide the program signal as a download service using a telecommunications line such as the Internet.

[0135] Furthermore, some or all of the processing functions of the information processing device 20 may be realized by cloud computing, and may also be provided as a SaaS (Software as a Service) service.

[0136] [Regarding sharing of processing among multiple information processing devices] The processing functions of the information processing device 20 may be realized by multiple information processing devices. In addition, some or all of the processing functions of the information processing device 20 may be implemented in the remote controller 16 and / or the terminal device 24.

[0137] Advantages of this embodiment The information processing device 20 according to this embodiment has the following advantages.

[0138] [1] For feature images extracted from aerial images, the display items are narrowed down based on the attributes and abnormality level of the features, and information about features that meet the conditions is displayed. This makes it easy to understand the damage status and scale of the disaster of features such as houses in the target area without requiring complex operations.

[0139] [2] The information processing device 20 makes it possible to comprehensively assess information (non-image data) indicating the attributes of features such as buildings before the disaster and the analysis results of aerial images taken after the disaster, thereby enabling the creation of an efficient on-site investigation plan.

[0140] [3] The information processing device 20 can accurately and quickly grasp the damage status of houses and other structures in a disaster-hit area, which can contribute to the prompt issuance of disaster certificates.

[0141] [Variation 1] In the above embodiment, an example of processing a still image as a captured image has been described, but the camera 14 may also capture a video, and the information processing device 20 may extract some frames from the captured video and perform similar processing.

[0142] [Variation 2] In the above embodiment, an example has been described in which features on a map obtained from map information in the alignment unit 242 are projected onto a photographed image and superimposed on the photographed image, but a process may also be executed in which features on an image are projected onto a map and superimposed on the map image.

[0143] [Variation 3] In the above embodiment, an example is given of processing an image captured by the camera 14 mounted on the drone 12, but the scope of application of the present disclosure is not limited to this example. For example, an image captured by a camera installed at a high location overlooking the ground, such as on the roof of a building or on a steel tower, may be processed.

[0144] [Variation 4] The technology disclosed herein is not limited to the task of determining damage to houses, but can also be applied to information processing devices that perform tasks of determining abnormalities in various geographical features, such as investigating the current status of social infrastructure such as roads and railways.

[0145] [Others] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technical idea of ​​the present disclosure.

[0146] 10 Photographed image processing system 12 Drone 13 Gimbal head 14 Camera 16 Remote controller 16A Display 20 Information processing device 22 Network 24 Terminal device 24A Display 30 GNSS receiver 32 Barometric pressure sensor 34 Orientation sensor 36 Inertial measurement unit 38 Motor 40 Processor 42 Storage device 44 Communication interface 202 Processor 204 Computer readable medium 206 Communication interface 208 Input / output interface 210 Bus 212 Memory 214 Storage 222 Input device 224 Display device 230 Image acquisition unit 232 Camera information acquisition unit 233 Camera position information acquisition unit 234 Camera attitude information acquisition unit 235 Map information acquisition unit 236 Feature acquisition unit 238 Attribute acquisition unit 240 Camera parameter calculation unit 242 Positioning unit 244 Feature image extraction unit 246 Attribute assignment unit 260 Map database 262 Feature attribute database 270 Abnormality degree determination unit 272 AI model 274 Display narrowing unit 282 Display image generation unit 284 Display control unit 362 Gyro sensor 364 Acceleration sensor 366 Temperature sensor FR Frame line HS House IM Image IM1 Aerial image

Claims

1. An information processing device having one or more processors, wherein the one or more processors perform the following processes: acquiring an aerial image; acquiring information about features on a map from map information; aligning the aerial image with the features on the map; generating a feature image by cutting out from the aerial image an area of ​​the feature in the aerial image that corresponds to the feature on the map; determining the degree of abnormality of the feature in the feature image based on the feature image; acquiring attributes of the feature in the feature image; narrowing down the features to be displayed based on at least one of the feature attributes and the degree of abnormality; and displaying information about the narrowed down features to be displayed on a display device.

2. The information processing device of claim 1, wherein the alignment process includes: a process of calculating camera parameters representing the position and attitude of the camera at the point where the aerial image was taken from one or more of the aerial images; and a process of overlaying features on the map onto the aerial image using the camera parameters.

3. The information processing device according to claim 1, wherein the features include houses.

4. The information processing device according to claim 1, wherein the attributes of the features include at least one attribute of building type, building material, roof type, building structure, floor, ground floor area, construction date, address, and geographic coordinates.

5. The information processing device according to claim 1, wherein the narrowing down process is performed using a logical expression including at least one condition from among a condition related to an attribute of the feature and a condition related to the degree of anomaly.

6. The information processing device according to claim 5, wherein the one or more processors accept input of the logical expression and execute the narrowing process using the specified logical expression.

7. The information processing device according to claim 6, comprising: an input device used for inputting the logical formula; and the display device.

8. The information processing device according to claim 1, wherein the information about the feature to be displayed includes an image of the feature.

9. The information processing device according to claim 8, wherein the information about the feature to be displayed includes attributes of the feature.

10. The information processing device according to claim 1, wherein the display process includes a process of displaying the feature images of the feature to be displayed in order of the degree of abnormality.

11. The information processing device according to claim 1, wherein the display process includes a process of changing the display color of the border of the feature image for the feature to be displayed depending on the degree of abnormality.

12. An information processing device according to claim 1, wherein the display process includes a process of displaying a map including the location of the feature to be displayed and a point mark indicating the position of the feature on the map.

13. The information processing device according to claim 12, wherein the information about the feature to be displayed is displayed in correspondence with the point mark.

14. The information processing device according to claim 1, wherein the process of determining the degree of anomaly is performed using a trained model that has been trained by machine learning to receive the input of the feature image and output the degree of anomaly of the feature image.

15. The information processing device according to claim 14, wherein the trained model is an anomaly detection model generated by unsupervised learning.

16. The information processing device according to claim 1, wherein the one or more processors acquire attributes of the features from a database containing at least some data from a fixed asset tax register and a real estate register.

17. The information processing device according to claim 1, wherein the aerial images are images of a disaster-stricken area.

18. A method for displaying an abnormality determination result, in which one or more processors perform the following steps: acquiring an aerial image; acquiring information about features on the map from map information; aligning the aerial image with the features on the map; generating a feature image by cutting out from the aerial image an area of ​​the feature in the aerial image that corresponds to the feature on the map; determining the degree of abnormality of the feature in the feature image based on the feature image; acquiring attributes of the feature in the feature image; narrowing down the features to be displayed based on at least one of the attributes of the feature and the degree of abnormality; and displaying information about the narrowed down features to be displayed on a display device.

19. A program that causes a computer to perform the following functions: acquire an aerial image; acquire information about features on a map from map information; align the aerial image with the features on the map; generate a feature image by cutting out from the aerial image an area of ​​the feature in the aerial image that corresponds to the feature on the map; determine the degree of abnormality of the feature in the feature image based on the feature image; acquire attributes of the feature in the feature image; narrow down the features to be displayed based on at least one of the feature attributes and the degree of abnormality; and display information about the narrowed down features to be displayed on a display device.

20. A non-transitory computer-readable recording medium on which the program according to claim 19 is recorded.

Images