Information processing device, information processing method, program, and information processing system

Abstract

Provided are an information processing device, an information processing method, a program, and an information processing system with which it is possible to ascertain a change with respect to existing map information. This information processing device: acquires a map of a target region; acquires a captured image in which the target region is imaged from the sky; acquires first information in which map information representing the position of each building in the map is added with respect to the captured image, in which the map and the position are aligned; acquires second information in which a building is detected from the captured image; and acquires association information (MR) in which the first information and the second information are associated, the association information including at least either new building information (FN) representing a new building included in the captured image but not included in the map or reconstruction building information (FR) representing a reconstructed building for which the shape in the map and the shape in the captured image are different.

Description

Information processing device, information processing method, program, and information processing system

[0001] The present invention relates to an information processing device, an information processing method, a program, and an information processing system.

[0002] A technique is known for accurately overlaying map information onto an aerial image using aerial images, map information, and sensor data, which may include the camera position and the camera attitude when the aerial image is taken.

[0003] For example, by aligning a photographed image obtained by using a camera mounted on a drone with a map and associating the photographed image with map information contained in the map, correspondence between each building contained in the photographed image and its address can be realized. Note that the correspondence between the photographed image and map information can be referred to as matching the photographed image with map information.

[0004] In the event of a disaster, the damage status of each building can be confirmed by applying AI-based assessment to photographed images associated with map information, which assesses the damage status of each building. An example of a building is a house where people live.

[0005] Patent Document 1 describes a map data storage method that aligns map graphic data representing a map of a predetermined area with map image data obtained by photographing the area represented by the map graphic data, specifies an area for which difference extraction is to be performed, extracts difference data representing the difference between the two, and stores the extracted difference data. The map data storage method described in this document updates the map graphic data for buildings determined to be demolished, replaced, or newly constructed. The difference data is superimposed on the map graphic data and displayed on a display.

[0006] Patent document 2 describes an information processing device that extracts individual building images from an overhead image, identifies unphotographed buildings that are not shown in the overhead image among buildings included in map information that associates buildings with addresses, and requests a terminal among multiple terminals that is located near the unphotographed building to photograph the unphotographed building.

[0007] JP 2002-297641 A International Publication No. 2022 / 070956

[0008] However, information about new buildings that were constructed after the map was created is not included in the map, and new buildings cannot be detected from the map. Furthermore, when a building is demolished and the site is left vacant, the vacant lot is detected based on the information present in the map, even though no building is present in the captured image. Furthermore, it is difficult to accurately detect the shape of buildings that have been expanded or remodeled.

[0009] In other words, it is difficult to reflect actual changes that have occurred since the map was created in the map information associated with the captured image, and as a result, it can be difficult to accurately associate the captured image with the map information.

[0010] In the method described in Patent Document 1, the map graphic data and the map image data are aligned, but the document does not describe or suggest how to associate the map graphic data with the map image data, and the method described in the document makes it difficult to use map graphic data, such as information indicating the location of buildings in the map image data.

[0011] The device described in Patent Document 2 identifies the addresses of buildings included in the aerial image based on address information that associates the building with an address for all buildings present on the map. However, address information is missing for buildings newly constructed after the map was created, and it is possible that buildings included in the aerial image may have addresses whose addresses cannot be identified.

[0012] The present invention has been made in view of the above circumstances, and has as its object to provide an information processing device, an information processing method, a program, and an information processing system that are capable of grasping changes to existing maps.

[0013] An information processing device according to a first aspect of the present disclosure includes one or more processors and one or more memories that store instructions to be executed by the one or more processors, and the one or more processors acquire a map of a target area, acquire photographed images of the target area from above, acquire first information in which map information representing the position of each building on the map is attached to the photographed images that have been aligned with the map, acquire second information in which buildings are detected from the photographed images, and acquire correspondence information in which the first information and the second information are associated, and the correspondence information includes at least one of new building information representing a new building that is not included on the map but is included in the photographed images, and renovated building information representing a renovated building whose shape on the map differs from that in the photographed images.

[0014] According to the information processing device according to the first aspect of the present disclosure, first information in which map information is attached to a photographed image of a target area is associated with second information in which buildings are detected from the photographed image, and association information including at least one of new buildings and renovated buildings is obtained, thereby making it possible to grasp changes to an existing map.

[0015] The location of the building may be determined by the address of the building or the latitude and longitude of the center of the building.

[0016] The association information may include missing images that exist on the map but do not exist in the captured image.

[0017] An information processing device according to a second aspect is the information processing device of the first aspect, wherein the one or more processors may display association information superimposed on the captured image.

[0018] The display mode for superimposing the association information on the captured image may be a display mode on a display device, or may be printing.

[0019] In the information processing device of the third aspect, in the information processing device of the second aspect, one or more processors may display new building information and renovated building information separately when displaying a captured image on which correspondence information is superimposed.

[0020] The new building information and the renovated building information may be surrounded by a frame. The new building information and the renovated building information may be distinguished from each other by the color and type of frame.

[0021] In the information processing device of the fourth aspect, in an information processing device of any one of the first to third aspects, one or more processors may display, as correspondence information, an image cut out from a captured image, a type, and an address for each of a new building and a renovated building.

[0022] The correspondence information in this embodiment may be in the form of a list.

[0023] In an information processing device according to a fifth aspect, in an information processing device according to any one of the first to fourth aspects, one or more processors may acquire first information in which the address of a building is associated with the captured image as map information.

[0024] An information processing device according to a sixth aspect is the information processing device of the fifth aspect, wherein one or more processors are existing buildings included in a map and in a captured image, and may obtain the address of the new building based on the latitude and longitude of existing buildings around the new building.

[0025] An information processing device according to a seventh aspect is an information processing device according to any one of the first to sixth aspects, wherein one or more processors may detect buildings from the captured images by applying a trained learning model that has learned the correspondence between the captured images and the buildings contained in the captured images.

[0026] In an information processing device according to an eighth aspect, in an information processing device according to any one of the first to seventh aspects, one or more processors may align the captured image with the map based on sensor data including the position and attitude of the imaging device applied when photographing the target area.

[0027] In an information processing device according to a ninth aspect, in an information processing device according to any one of the first to eighth aspects, one or more processors may assign renovated building information representing a renovated building to a building whose shape on the map differs from that in the captured image.

[0028] An information processing method according to a tenth aspect of the present disclosure is an information processing method in which an information processing device to which a computer is applied acquires a map of a target area, acquires photographed images of the target area from above, acquires first information in which map information indicating the position of each building on the map is attached to the photographed images whose position is aligned with the map, acquires second information in which buildings are detected from the photographed images, and acquires correspondence information in which the first information and the second information are associated, and the correspondence information includes at least one of new building information indicating a new building that is not included on the map but is included in the photographed images, and renovated building information indicating a renovated building whose shape on the map differs from that in the photographed images.

[0029] In the information processing method according to the tenth aspect, it is possible to appropriately combine the same items as those specified in the second to ninth aspects. In this case, the components that perform the processing or function specified in the information processing device can be understood as components of the information processing method that perform the corresponding processing or function.

[0030] A program according to an eleventh aspect of the present disclosure is a program that enables a computer to perform the following functions: acquire a map of a target area; acquire an image of the target area photographed from above; acquire first information in which map information indicating the position of each building on the map is attached to the photographed image whose position is aligned with the map; acquire second information in which buildings are detected from the photographed image; and acquire correspondence information in which the first information and the second information are associated and which includes at least one of new building information indicating a new building that is not included on the map but is included in the photographed image, and renovated building information indicating a renovated building whose shape on the map differs from that in the photographed image.

[0031] In the program according to the eleventh aspect, it is possible to appropriately combine the same items as those specified in the second to ninth aspects. In this case, the components that perform the processes or functions specified in the information processing device can be understood as components of the program that perform the corresponding processes or functions.

[0032] An information processing system according to a twelfth aspect of the present disclosure is an information processing system including an information processing device electrically connected to a network so as to be able to communicate freely with the network, wherein the information processing device includes one or more processors and one or more memories for storing instructions to be executed by the one or more processors, and the one or more processors acquire a map of a target area, acquire photographed images of the target area taken from above, acquire first information in which map information representing the position of each building on the map is attached to the photographed images that have been aligned with the map, acquire second information in which buildings are detected from the photographed images, and acquire correspondence information in which the first information and the second information are associated, and the correspondence information includes at least one of new building information representing a new building that is not included on the map but is included in the photographed images, and renovated building information representing a renovated building whose shape on the map differs from that in the photographed images.

[0033] In the information processing system according to the twelfth aspect, it is possible to appropriately combine the same items as those specified in the second to ninth aspects. In this case, the components that perform the processes or functions specified in the information processing device can be understood as the components of the information processing system that perform the corresponding processes or functions.

[0034] The information processing system according to the twelfth aspect may include a remote controller electrically connected to a network so as to be able to communicate freely with the network, an aircraft remotely controlled using the remote controller, an imaging device remotely controlled using the remote controller, and a terminal device electrically connected to the network so as to be able to communicate freely with the network.

[0035] According to the present invention, first information in which map information is attached to photographed images of a target area is associated with second information in which buildings are detected from the photographed images, and associated information including at least one of new buildings and renovated buildings is obtained, thereby making it possible to grasp changes to an existing map.

[0036] FIG. 1 is an explanatory diagram of a problem in an information processing method of a comparative example. FIG. 2 is a schematic diagram of an information processing method according to an embodiment. FIG. 3 is a schematic diagram showing a specific example of a processing result. FIG. 4 is a schematic diagram showing another example of a processing result. FIG. 5 is an explanatory diagram of an address assignment process for a newly built house. FIG. 6 is a diagram showing mathematical formulas applied to calculating latitude and longitude. FIG. 7 is a configuration diagram showing a schematic configuration of an information processing system according to an embodiment. FIG. 8 is a functional block diagram showing the electrical configuration of a drone equipped with a camera. FIG. 9 is a functional block diagram showing the electrical configuration of the information processing device shown in FIG. 7. FIG. 10 is a block diagram showing the hardware configuration of the information processing device shown in FIG. 7. FIG. 11 is a flowchart showing the procedure of an information processing method according to an embodiment.

[0037] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, identical components are designated by the same reference numerals, and duplicate explanations will be omitted where appropriate. Furthermore, when multiple components are listed as examples, it can be interpreted as including at least one of the multiple components.

[0038] [Issues of Information Processing Methods Related to Conventional Technology] Below, information processing for associating aerial images of a target area taken from above with map information included in a map of the target area will be described using an example of a damage survey conducted by a local government such as a city, town, or village. The term "above" does not limit the distance from the ground, and may refer to the flight path of a drone or helicopter, or the orbit of an artificial satellite, etc.

[0039] In the case of Japan, the map of the target area may be obtained from the basic map information provided by the Geospatial Information Authority of Japan, or from the OpenStreetMap database, etc. The map information includes the addresses of the houses shown on the map. The map information may also include the latitude and longitude of the houses shown on the map. The addresses of the houses shown on the map are visualized as frames representing the outlines of the houses with addresses. Note that the houses described in the embodiments are an example of buildings.

[0040] FIG. 1 is an explanatory diagram of a problem with the information processing method of the comparative example. The figure shows a schematic diagram of a registration process that aligns the scale and position of a map MP with an aerial image IM, and a result of a correspondence process CR that associates map information with the aerial image IM. By applying sensor data including the position and attitude of the camera used to capture the aerial image IM, map information can be accurately overlaid on the aerial image IM. The camera described in the embodiment is an example of an image capture device.

[0041] 1, a frame FG using a dotted line represents the ground perimeter of a house shown on the map MP, and a frame FR using a solid line represents the roof perimeter of a house shown on the map MP.

[0042] 1, a frame FG representing the perimeter of the ground surface of the house shown on the map MP or a frame FR representing the perimeter of the roof of the house is superimposed on the aerial image IM. Note that it is sufficient that at least one of the frame FG representing the perimeter of the ground surface of the house or the frame FR representing the perimeter of the roof of the house is superimposed on the correspondence processing result CR.

[0043] For example, in a damage survey of a disaster or the like, a damage assessment process is performed for each house included in the correspondence process result CR shown in Figure 1, making it possible to grasp the damage situation for each house. The damage assessment process is performed using a damage assessment AI. The damage assessment AI uses a trained learning model that has learned the correspondence between photographed images of houses and the damage situation.

[0044] The matching process result CR includes a frame FG or a frame FR representing a lost house VH that exists on the map MP but not on the aerial image IM. On the other hand, the matching process result CR does not include a frame FG or a frame FR representing a newly built house NH that does not exist on the map MP but exists on the aerial image IM.

[0045] In a damage assessment process that uses damage assessment AI or the like, it is difficult to deal with newly constructed houses NH that have changed since the map was created, and it is difficult to accurately grasp the damage situation of the newly constructed houses NH. Furthermore, in the case of a renovated house RH, due to differences in shape between the frame FG or frame FR representing the renovated house RH and the renovated house RH in the aerial image IM, it is difficult to accurately grasp the damage situation of the renovated house RH in a damage assessment process that uses damage assessment AI or the like. Below, an information processing method that solves the above problems will be described.

[0046] [Information Processing Method According to the Embodiment] Fig. 2 is a schematic diagram of an information processing method according to the embodiment. In the information processing method according to the embodiment, a process of aligning a map MP with an aerial image IM, a process of associating map information with the aerial image IM, and house detection with respect to the aerial image IM are combined. This realizes information processing for extracting a newly built house NH shown in Fig. 1. Furthermore, information processing for distinguishing a renovated house RH shown in Fig. 1 from an existing house is realized. Map information is associated with the houses extracted in the aerial image IM, and an address is assigned to each house. A separate address is assigned to a newly built house that does not have map information and cannot be associated. Details of assigning addresses to newly built houses will be described later.

[0047] That is, in the information processing method according to the embodiment, the association processing result CR shown in Fig. 1 is derived, and house detection is performed on the aerial image IM to derive the house detection result DR. Fig. 2 shows an example in which a rectangular frame FH representing the outline of each house detected on the aerial image IM is superimposed on the house detection result DR.

[0048] House detection AI can be applied to detect houses in the aerial image IM. AI is an abbreviation for Artificial Intelligence. The house detection AI applies a trained learning model that has learned the correspondence between the aerial image IM and one or more houses included in the aerial image IM.

[0049] Furthermore, in the information processing method according to the embodiment, a process of associating the association process result CR with the house detection result DR is performed. The process result MR obtained by combining the association process result CR with the house detection result DR is illustrated in FIG. 3 .

[0050] In the information processing method of the embodiment, segmentation to detect house areas is applied to house detection in the aerial image IM, and a determination of renovated houses can be made based on the results of a comparison between the shape of the house projected onto the aerial image IM from the map MP and the shape of the house area detected from the aerial image IM.

[0051] 2 is an example of first information to which map information indicating the position of each building on a map is added, and the house detection result DR shown in the same figure is an example of second information in which buildings are detected from the captured image.

[0052] [Specific Example of Processing Result] Fig. 3 is a schematic diagram showing a specific example of the processing result. In the figure, the processing result MR is shown in which rectangular frames surrounding the houses, each distinguishing between different types of houses, are superimposed on the multiple houses detected from the aerial image IM shown in Fig. 2. The processing result MR shown in Fig. 3 may be displayed on a display device.

[0053] In the processing result MR shown in Figure 3, each of the multiple houses is classified into existing houses, new houses, renovated houses, and lost houses. Existing houses are houses included in the aerial image IM and the map MP, and the shape of the area of ​​the house in the aerial image IM matches the shape of the area of ​​the house in the map MP. "Match" here does not mean a perfect match, but may include partial mismatches that occur depending on the shooting conditions of the aerial image IM, etc.

[0054] In the processing result MR shown in Figure 3, each of the multiple houses is surrounded by a frame FE representing an existing house, a frame FN representing a new house, a frame FR representing a renovated house, or a frame FL representing a lost house. Note that the symbol FE attached to the frame is omitted for some of the existing houses.

[0055] 3, the frame FN representing a newly constructed house is an example of new building information representing a new building. The frame FR representing a renovated house is an example of renovated building information representing a renovated building. The processing result MR shown in the figure is an example of association information including at least one of new building information representing a new building and renovated building information representing a renovated building.

[0056] 4 is a schematic diagram showing another example of the processing result. The figure shows the processing result MRV, which is output as a list of images, types, and addresses for each house. Examples of output here include display on a display device and printing.

[0057] In the image column of the processing result MRV according to another example shown in Fig. 4, an image of each house cut out from the aerial image IM is posted. In the image column of the missing house, an area of ​​the missing image when the aerial image IM was taken, such as an empty lot, may be posted, or a blank column may be applied.

[0058] The type field is filled with one of the following: existing, new construction, renovation, and lost. The address field is filled with an address assigned based on the association with map information for each house. Note that the block numbers and street numbers representing the street numbers shown in the list shown in Figure 4 are arbitrary numbers. The address field shown in Figure 4 does not include prefecture names, city / town / village names, or town names, but may include city / town / village names, town names, etc.

[0059] Although FIG. 4 illustrates a processing result MRV that includes a lost house, the processing result MRV may not include the lost house, or processing such as overlaying a hatched area on the lost house may be performed.

[0060] [Specific example of address assignment process for newly constructed houses] Fig. 5 is an explanatory diagram of the address assignment process for newly constructed houses. Fig. 6 is a diagram showing mathematical formulas applied to calculate latitude and longitude. The upper part of Fig. 5 shows coordinate values ​​of multiple houses included in the aerial image IM shown in Fig. 2. The lower part of Fig. 5 shows the latitudes and longitudes of the multiple houses shown in the upper part of Fig. 5.

[0061] Of the houses included in the aerial image IM, existing houses for which map information exists have their latitude and longitude specified. On the other hand, the latitude and longitude of a newly constructed house for which no corresponding map information exists are unknown, but the latitude and longitude of the newly constructed house can be calculated based on the latitude and longitude of surrounding existing houses. The surrounding existing houses here refer to existing houses that are within a specified distance from the newly constructed house of interest, such as existing houses adjacent to the newly constructed house of interest.

[0062] The latitude and longitude of the house are the latitude and longitude of the center of the house. The center of the house may be the geometric center of the planar shape of the house in the aerial image IM. The planar shape of the house may be the shape of the house projected onto the ground in the aerial image IM.

[0063] A two-dimensional Cartesian coordinate system is set for the aerial image IM, with the coordinate values ​​of the first house H1 being (x1, y1), the coordinate values ​​of the second house H2 being (x2, y2), and the coordinate values ​​of the third house H3 being (x3, y3).

[0064] The latitude of the first house H1 is lat1, and the longitude of the first house H1 is lon1. Furthermore, the latitude of the second house H2 is lat2, and the longitude of the second house H2 is lon2. Furthermore, the latitude of the third house H3 is lat3, and the longitude of the third house H3 is lon3. The latitude of the second house H2, lat2, and the longitude of the second house H2, lon2, are expressed using the formula shown in FIG. 6.

[0065] The address of the newly constructed house of interest is obtained from the latitude and longitude of the newly constructed house of interest calculated using the formula shown in Figure 6. For example, the distance to the surrounding existing houses and the direction relative to the surrounding existing houses are determined from the latitude and longitude of the newly constructed house of interest and the latitude and longitude of the surrounding existing houses. Using the addresses of the surrounding existing houses as a reference, the address of the newly constructed house of interest can be estimated based on the distance to the surrounding existing houses and the direction relative to the surrounding existing houses. Note that the existing houses described in the embodiments are an example of existing buildings.

[0066] 7 is a diagram showing a schematic configuration of an information processing system according to an embodiment. The information processing system 10 shown in the figure includes a drone 12, a camera 14, a remote controller 16, and an information processing device 20.

[0067] The remote controller 16 and the information processing device 20 are electrically connected to be able to communicate with each other via a network 22. The information processing device 20 is also electrically connected to be able to communicate with a terminal device 24 via the network 22. The network 22 may be accessible from a plurality of different terminal devices 24.

[0068] The drone 12 is an aerial vehicle for aerial photography, and is an unmanned aerial vehicle that is remotely controlled using a remote controller 16. The drone 12 may have an autopilot function that flies according to a program.

[0069] The camera 14 is mounted on the drone 12 via a gimbal head 13. The camera 14 photographs the target area from above. The camera 14 includes an optical system, an image sensor, and a signal processing circuit. The optical system, image sensor, and signal processing circuit are not shown in the drawings.

[0070] The optical system includes one or more lenses such as a focus lens. The image sensor may be a CCD image sensor or a CMOS image sensor. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal-Oxide Semiconductor.

[0071] The camera 14 processes the signal output from the image sensor using a signal processing circuit to generate digital image data of the subject. The digital image data generated using the camera 14 can be used as a captured image. The captured image generated using the camera 14 may be stored in at least one of an internal storage device built into the drone 12 and a storage device such as a memory card removably attached to the drone 12. The captured image generated using the camera 14 may also be transferred to at least one of the remote controller 16, the information processing device 20, and the terminal device 24 via wireless communication. The aerial image IM shown in FIG. 7 is an example of a captured image generated using the camera 14.

[0072] The remote controller 16 is a control device that controls the operation of the camera 14 and the drone 12 by wireless communication. The wireless communication format may be a wireless LAN format, a communication format using radio waves in the 2.4 GHz band or 5.7 GHz band, or a format using a mobile communication network. The communication format for control signals used to control the drone 12 may be different from or the same as the communication format used to transfer captured images generated using the camera 14.

[0073] The remote controller 16 includes left and right sticks used to fly the drone 12, a lever used to operate the gimbal head 13, a shooting button used to take pictures with the camera 14, and a shooting mode button for switching between video shooting and still image shooting. A touch panel display may be used for the display 16A provided on the remote controller 16, and operation buttons such as the shooting button may be implemented using the touch panel display.

[0074] The live video generated by the camera 14 can be displayed on the display 16A. The remote controller 16 can grasp the status of the drone 12, such as the flight position and flight speed, in real time based on output signals from various sensors equipped on the drone 12. The remote controller 16 can display the status of the drone 12 on the display 16A.

[0075] A computer is applied to the information processing device 20. The computer may be a personal computer, a server, a workstation, or a virtual machine.

[0076] The information processing device 20 can acquire a map of the target area from a map information system via the network 22. The map includes map information including addresses, latitudes, and longitudes of buildings and the like to be included on the map. When an image of the target area is captured using the camera 14, the information processing device 20 may acquire the map before, during, or after the image is captured. The map information system is designated by the symbol MIS and is illustrated in FIG. 9.

[0077] The terminal device 24 may be a mobile information terminal such as a smartphone, a tablet terminal, or a portable computer. The terminal device 24 includes a display 24A. The terminal device 24 may have the functions of the remote controller 16. The terminal device 24 may also have the functions of the information processing device 20.

[0078] 8 is a functional block diagram showing the electrical configuration of a drone equipped with a camera. The drone 12 is equipped with a GPS receiver 30. The GPS receiver 30 acquires location information including the latitude and longitude of the drone 12. Note that GPS is an abbreviation for Global Positioning System.

[0079] The drone 12 includes various sensors, such as a barometric pressure sensor 32, a direction sensor 34, and a gyro sensor 36. The barometric pressure sensor 32 detects the barometric pressure in the drone 12. The drone 12 may acquire its own altitude based on the barometric pressure detected using the barometric pressure sensor 32. The term "acquire" in this specification may include the concept of information being generated by applying data processing such as calculation. The latitude, longitude, and altitude of the drone 12 constitute position information of the drone 12 and the camera 14.

[0080] A geomagnetic sensor may be used as the orientation sensor 34. The orientation sensor 34 detects the direction in which the lens of the camera 14 is facing.

[0081] The gyro sensor 36 detects a roll angle representing the angle of rotation about the roll axis, a pitch angle representing the angle of rotation about the pitch axis, and a yaw angle representing the angle of rotation about the yaw axis. The drone 12 acquires attitude information representing the attitude of the drone 12 based on the roll angle, pitch angle, and yaw angle detected using the gyro sensor 36. Note that various sensors such as the air pressure sensor 32, the orientation sensor 34, and the gyro sensor 36 may be provided in the camera 14.

[0082] The drone 12 includes a processor 40, a storage device 42, and a communication interface 44. The processor 40 functions as a flight controller that performs calculations necessary for flight control of the drone 12 based on sensor data obtained from various sensors. The storage device 42 may be a memory, an internal storage device, an external storage device, or the like. The storage device 42 may also be configured by appropriately combining memories and the like.

[0083] The communication interface 44 functions as a communication device that performs wireless communication with the remote controller 16 shown in Fig. 7. The communication interface 44 may be provided with a communication terminal that supports wired communication.

[0084] The drone 12 includes a battery and a battery charging terminal, which are not shown in the drawings.

[0085] [Electrical Configuration of Information Processing Device] Figure 9 is a functional block diagram showing the electrical configuration of the information processing device shown in Figure 7. The information processing device 20 includes a map acquisition unit 100. The map acquisition unit 100 acquires a map MP of a target area from a map information system MIS. The map acquisition unit 100 may acquire a wide-area map including the map MP of the target area. For example, if the target area is defined in units of chome, the map acquisition unit 100 may acquire a map including one or more chomes that are adjacent to the chome of the target area but different from the target area.

[0086] The information processing device 20 includes a map information acquisition unit 101. The map information acquisition unit 101 acquires map information included in a map MP acquired using the map acquisition unit 100. The map information acquisition unit 101 may be configured integrally with the map acquisition unit 100. That is, the map acquisition unit 100 may acquire the map MP and acquire map information from the acquired map MP.

[0087] The information processing device 20 includes a sensor data acquisition unit 102. The sensor data acquisition unit 102 acquires sensor data from the remote controller 16. For example, the information processing device 20 may transmit a signal to the remote controller 16 requesting acquisition of sensor data, and acquire the sensor data transmitted from the remote controller 16 in response to the request. The information processing device 20 may also acquire sensor data from the drone 12.

[0088] The information processing device 20 includes a captured image acquisition unit 104. The captured image acquisition unit 104 acquires, from the captured image storage system IMS, aerial images IM of the target area that have been captured and generated using the camera 14 and stored in the captured image storage system IMS. The information processing device 20 may acquire the aerial images IM of the target area from the camera 14.

[0089] The information processing device 20 includes an alignment processing unit 106. The alignment processing unit 106 references sensor data acquired using the sensor data acquisition unit 102 and performs alignment processing between the map MP acquired using the map acquisition unit 100 and the aerial image IM acquired using the photographed image acquisition unit 104. When aligning the positions of the map MP and the aerial image IM, the alignment processing unit 106 performs scale alignment processing to match the scales of the map MP and the aerial image IM.

[0090] The information processing device 20 includes a house detection unit 108. The house detection unit 108 detects houses from captured images acquired using the captured image acquisition unit 104. A trained learning model that has learned the correspondence between aerial images IM and houses can be applied to the house detection unit 108. The trained learning model can be referred to as a house detection AI.

[0091] The information processing device 20 includes an association processing unit 110. The association processing unit 110 associates map information of houses posted on the map MP with houses detected from the aerial image IM. For example, the association processing unit 110 assigns an address to each house included in the aerial image IM.

[0092] The information processing device 20 includes a display control unit 112. The display control unit 112 generates a display signal representing the captured image associated with the map information, and transmits the display signal to a display device. An example of the display device is a display 24A provided in the terminal device 24.

[0093] [Hardware Configuration of Information Processing Device] Fig. 10 is a block diagram showing the hardware configuration of the information processing device shown in Fig. 7. The information processing device 20 includes one or more processors 132 and one or more computer-readable media 134. The information processing device 20 includes a communication interface 136 and an input / output interface 368.

[0094] The processor 132 executes various programs stored in the memory 140 of the computer-readable medium 134 to realize various functions of the information processing device 20. The processor 132 includes a CPU (Central Processing Unit). The processor 132 may also include a GPU (Graphics Processing Unit). The processor 132 is connected to the computer-readable medium 134, the communication interface 136, and the input / output interface 138 via a bus 142.

[0095] The computer-readable medium 134 includes a memory 140 serving as a main storage device and a storage 144 serving as an auxiliary storage device. The computer-readable medium 134 may be a semiconductor memory, a hard disk drive, a solid-state drive, or the like. The computer-readable medium 134 may be any combination of multiple devices.

[0096] A hard disk drive may be referred to as an HDD, which is an abbreviation of the English term Hard Disk Drive, and a solid state drive may be referred to as an SSD, which is an abbreviation of the English term Solid State Drive.

[0097] The memory 140 of the computer-readable medium 134 stores a map acquisition program 150, a map information acquisition program 151, a sensor data acquisition program 152, a captured image acquisition program 154, an alignment program 156, a house detection program 158, a correspondence program 160, and a display control program 162.

[0098] The map acquisition program 150 is applied to the map acquisition unit 100 shown in Fig. 9 to realize the map acquisition function. The map information acquisition program 151 is applied to the map information acquisition unit 101 to realize the map information acquisition function.

[0099] The sensor data acquisition program 152 is applied to the sensor data acquisition unit 102 to realize a sensor data acquisition function. The captured image acquisition program 154 is applied to the captured image acquisition unit 104 to realize a captured image acquisition function.

[0100] The alignment program 156 is applied to the alignment processing unit 106 to realize a function of aligning the map MP and the aerial image IM. The alignment program 156 may also realize a scale adjustment function of adjusting the scales of the map MP and the aerial image IM. The memory 140 may be provided with a scale adjustment program separate from the alignment program 156.

[0101] The house detection program 158 is applied to the house detection unit 108 to realize a house detection function. The house detection program 158 can be configured as a trained learning model that has learned the correspondence between the aerial images IM and houses.

[0102] The association program 160 is applied to the association processing unit 110, and realizes an association function of associating the map information of houses posted on the map MP with houses detected from the aerial image IM and assigning an address to each house.

[0103] The display control program 162 is applied to the display control unit 112 to realize a display control function. That is, the display control program 162 realizes a display signal generation function for generating a display signal and a display signal output function for transmitting the display signal to a display device.

[0104] The various programs stored in the computer-readable medium 134 include one or more instructions. The computer-readable medium 134 stores various data, various parameters, etc. The term "program" is synonymous with the term "software."

[0105] The hardware structure of the processor 132 is various processors as shown below. The various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various functional units, a GPU, which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0106] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple functional units may also be configured with a single processor. Examples of multiple functional units configured with a single processor include: a first configuration, as typified by a client or server computer, in which a single processor is configured with a combination of one or more CPUs and software, and this processor operates as multiple functional units; and a second configuration, as typified by a SoC (System on Chip), in which a processor is used to realize the functions of an entire system including multiple functional units on a single IC (Integrated Circuit) chip. In this way, the various functional units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0107] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0108] The memory 140 stores instructions to be executed by the processor 132. The memory 140 includes a RAM (Random Access Memory) and a ROM (Read Only Memory), both not shown. The processor 132 uses the RAM as a working area, executes software using various programs and parameters, including the map acquisition program 150, stored in the ROM, and also executes various processes of the information processing device 20 by using the parameters stored in the ROM, etc.

[0109] [Procedure of Information Processing Method] Fig. 11 is a flowchart showing the procedure of the information processing method according to the embodiment. Each step of the information processing method shown in Fig. 11 is executed by using various processing units provided in an information processing device 20 to which a computer is applied.

[0110] In the map acquisition step S10, the map acquisition unit 100 shown in Fig. 9 acquires a map MP. After the map acquisition step S10, the process proceeds to a map information acquisition step S12. In the map information acquisition step S12, the map information acquisition unit 101 acquires map information from the map MP acquired in the map acquisition step S10. After the map information acquisition step S12, the process proceeds to a sensor data acquisition step S14.

[0111] In the sensor data acquisition step S14, the sensor data acquisition unit 102 acquires sensor data from the remote controller 16. After the sensor data acquisition step S14, the process proceeds to an aerial image acquisition step S16.

[0112] In the aerial image acquisition step S16, the image acquisition unit 104 acquires an aerial image IM from the image storage system IMS. After the aerial image acquisition step S16, the process proceeds to a positioning step S18 and a house detection step S20.

[0113] In the registration process S18, the registration processing unit 106 performs registration processing between the map MP and the aerial image IM. In the registration process S18, after performing processing to match the scale of the map MP and the aerial image IM, processing to match the position of the map MP and the aerial image IM may be performed. After the registration process S18, the process proceeds to the association processing process S22.

[0114] In the house detection step S20, the house detection unit 108 detects houses from the aerial image IM. Also, in the house detection step S20, it is determined whether the house detected from the aerial image IM is an existing house or a house to be renovated.

[0115] Specifically, in the house detection step S20, segmentation is performed on the aerial image IM to detect house regions. Next, a first house shape obtained by projecting houses on the map MP onto the aerial image IM is compared with a second house shape of the house region extracted from the aerial image IM.

[0116] A house where the first house shape and the second house shape match is determined to be an existing house and is assigned existing house information indicating that it is an existing house, while a house where the first house shape and the second house shape differ is determined to be a remodeled house and is assigned remodeled house information indicating that it is a remodeled house.

[0117] After the house detection step S20, the process proceeds to the association processing step S22. The house detection step S20 may be executed after the aerial image IM is acquired in the aerial image acquisition step S16, and may be executed in parallel with the alignment step S18, for example. The house detection step S20 may be executed before or after the alignment step S18.

[0118] In the association processing step S22, the association processing unit 110 associates the map information acquired in the map information acquisition step S12 with the houses detected in the house detection step S20. For example, in the association processing step S22, addresses are assigned to existing houses and renovated houses.

[0119] In the association process S22, a newly constructed house is detected and an address is assigned to the newly constructed house. Furthermore, in the association process S22, new house information indicating that the newly constructed house is a newly constructed house is assigned to the newly constructed house. After the association process S22, the process proceeds to the association result display process S24.

[0120] In the association result display step S24, the display control unit 112 generates a display signal representing the processing result of the association processing step S22, transmits the generated display signal to a display device, and causes the display device to display the association processing result. After the association result display step S24, the process proceeds to an end determination step S26.

[0121] In the termination determination step S26, the information processing device 20 determines whether a specified termination condition is satisfied. If it is determined in the termination determination step S26 that the specified termination condition is not satisfied, the determination is No. If the determination is No, the process returns to the map acquisition step S10, and each step from the map acquisition step S10 to the termination determination step S26 is repeatedly executed until the determination is Yes in the termination determination step S26.

[0122] On the other hand, if it is determined that the specified termination condition is satisfied in the termination determination step S26, a Yes determination is made. If the Yes determination is made, a specified termination process is executed, and the procedure of the information processing device is terminated.

[0123] [Operational Effects of the Embodiment] The information processing device and information processing method according to the embodiment can achieve the following operational effects.

[0124] [1] Map information is acquired from a map MP of the target area. Houses are detected from an aerial image IM of the target area. The map of the target area and the aerial image of the target area are associated with each other, and information on newly constructed houses and renovated houses in the aerial image IM is acquired. This makes it possible to grasp changes in the target area since the map MP was created.

[0125] [2] The latitude of the new house is calculated based on the latitude of the existing house. The longitude of the new house is calculated based on the longitude of the existing house. Based on the calculated latitude and longitude of the new house, the address of the existing house is referenced and the address of the new house is obtained. This allows the new house to be identified in the aerial image IM, and an address is assigned to the new house.

[0126] [3] Segmentation is applied to detect house regions during house detection. A first house shape representing the shape of a house on the map MP projected onto the aerial image IM is compared with a second house shape representing the shape of the house region detected from the aerial image IM. A house whose first house shape differs from the second house shape is determined to be a renovated house. This allows the renovated house in the aerial image IM to be identified.

[0127] [4] Existing houses, new houses, and renovated houses are distinguished from each other in the aerial image IM and displayed on the display device. As a result, existing houses, new houses, and renovated houses are visualized in the aerial image IM.

[0128] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other within the scope that does not deviate from the spirit of the present invention.

[0129] 10 Information processing system 12 Drone 13 Gimbal head 14 Camera 16 Remote controller 16A Display 20 Information processing device 22 Network 24 Terminal device 24A Display 30 GPS receiver 32 Barometric pressure sensor 34 Orientation sensor 36 Gyro sensor 38 Motor 40 Processor 42 Storage device 44 Communication interface 100 Map acquisition unit 101 Map information acquisition unit 102 Sensor data acquisition unit 104 Captured image acquisition unit 106 Alignment processing unit 108 House detection unit 110 Correspondence processing unit 112 Display control unit 132 Processor 134 Computer readable medium 136 Communication interface 138 Input / output interface 140 Memory 142 Bus 144 Storage 150 Map acquisition program 151 Map information acquisition program 152 Sensor data acquisition program 154 Photographed image acquisition program 156 Alignment program 158 House detection program 160 Correspondence program 162 Display control program CR Correspondence processing result DR House detection result FE Frame representing existing house FL Frame representing lost house FN Frame representing newly built house FR Frame representing renovated house H1 First house H2 Second house H3 Third house IM Aerial image MP Map MR Processing result MRV Processing result NH Newly built house S10 to S26 Each step of information processing method

Claims

1. An information processing device comprising: one or more processors; and one or more memories in which instructions to be executed by the one or more processors are stored, wherein the one or more processors: acquire a map of a target area; acquire photographed images of the target area taken from the air; acquire first information in which map information indicating the position of each building on the map is added to the photographed images which have been aligned with the map; acquire second information in which buildings are detected from the photographed images; and acquire correspondence information in which the first information and the second information are associated with each other, the correspondence information including at least one of new building information indicating a new building which is not included on the map but is included in the photographed images, and renovated building information indicating a renovated building whose shape on the map differs from that in the photographed images.

2. The information processing device according to claim 1, wherein the one or more processors display the correspondence information by superimposing it on the captured image.

3. An information processing device as described in claim 2, wherein the one or more processors, when displaying the captured image on which the correspondence information is superimposed, display the new building information and the remodeled building information in a distinguishable manner.

4. An information processing device as described in claim 1, wherein the one or more processors display, as the association information, an image cut out from the captured image, a type, and an address for each of the new building and the renovated building.

5. The information processing device according to claim 1, wherein the one or more processors acquire the first information in which the address of the building is associated with the captured image as the map information.

6. The information processing device according to claim 5, wherein the one or more processors obtain the address of the new building based on the latitude and longitude of existing buildings in the vicinity of the new building, the existing buildings being included in the map and the captured image.

7. An information processing device as described in claim 1, wherein the one or more processors detect the buildings from the captured images by applying a trained learning model that has learned the correspondence between the captured images and the buildings contained in the captured images.

8. An information processing device as described in claim 1, wherein the one or more processors align the captured image with the map based on sensor data including the position and attitude of an imaging device applied when photographing the target area.

9. The information processing device according to claim 1, wherein the one or more processors assign remodeled building information representing a remodeled building to a building whose shape on the map differs from that in the captured image.

10. An information processing method in which an information processing device to which a computer is applied acquires a map of a target area, acquires images of the target area taken from the air, acquires first information in which map information indicating the position of each building on the map is added to the captured images whose position is aligned with the map, acquires second information in which buildings are detected from the captured images, and acquires association information in which the first information and the second information are associated with each other, the association information including at least one of new building information indicating a new building that is not included on the map but is included in the captured images, and renovated building information indicating a renovated building whose shape on the map differs from that in the captured images.

11. A program that enables a computer to perform the following functions: acquire a map of a target area; acquire images of the target area taken from the air; acquire first information in which map information indicating the position of each building on the map is added to the captured images whose position is aligned with the map; acquire second information in which buildings are detected from the captured images; and acquire association information in which the first information and the second information are associated with each other, and which includes at least one of new building information indicating a new building that is not included on the map but is included in the captured images, and renovated building information indicating a renovated building whose shape on the map differs from that in the captured images.

12. A non-transitory computer-readable recording medium having the program according to claim 11 recorded thereon.

13. An information processing system comprising an information processing device electrically connected to a network so as to be able to communicate freely with the network, the information processing device comprising: one or more processors; and one or more memories in which instructions to be executed by the one or more processors are stored, the one or more processors acquiring a map of a target area, acquiring photographed images of the target area taken from the air, acquiring first information in which map information representing the position of each building on the map is added to the photographed images which have been aligned with the map, acquiring second information in which buildings are detected from the photographed images, and acquiring association information in which the first information and the second information are associated with each other, the association information including at least one of new building information representing a new building not included on the map but included in the photographed images, and renovated building information representing a renovated building whose shape on the map differs from that in the photographed images.

Images