Information processing device, information processing method, program, and information collection system

Abstract

Provided are an information processing device, an information processing method, a program, and an information collection system which enable quick understanding of the damaged state of buildings not included in a bird's eye view picture shot by a fixed camera. A processor included in this information processing device performs: an acquisition process for acquiring a bird's eye view picture shot by a fixed camera; an extraction process for individually extracting pictures of buildings from the bird's eye view picture, an identification process for identifying an unshot building which is a building not included in the bird's eye view picture, among buildings included in the extracted pictures extracted in the extraction process and in map information in which buildings are associated with addresses; and a shooting request process for transmitting request information for requesting shooting of the unshot building to a terminal detected, from location information, to be present near the unshot building among a plurality of terminals having shooting functions and functions of transmitting the location information.

Description

[Technical field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, a program, and an information collecting system. [Background technology]

[0002] When a large-scale disaster occurs, local governments need to quickly grasp the extent of the damage in order to minimize damage in the affected area and to support the lives of residents. In recent years, damage has been grasped based on aerial images taken by aircraft, drones, or fixed cameras installed on the roofs of tall buildings.

[0003] Aircraft or drones take time to arrive at the scene of a disaster, so they cannot quickly grasp the extent of the damage. In addition, when taking pictures using aircraft or drones, it is difficult to grasp the detailed damage situation of each building because the distance to the building is far. On the other hand, when taking pictures using fixed cameras installed on the roofs of tall buildings, it is possible to grasp the detailed damage situation of each building because the distance to the building is relatively close. However, since it is not possible to capture the blind spots of tall buildings, the extent of the damage situation that can be grasped is limited.

[0004] A related technology is known in which, within a monitored area where multiple surveillance cameras are installed, a range a certain distance from the location of a disaster such as a fire is determined as a requested photography area, and a person in the vicinity of the requested photography area is asked to photograph the requested photography area (see Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 116485 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 aims to grasp the situation in a relatively rough area, i.e., the area where photography is requested, and does not aim to grasp the damage situation for each building. When grasping the damage situation during a disaster, it is necessary to obtain the damage situation for each building.

[0007] In particular, when grasping the damage situation in a disaster-affected area using a fixed camera installed on the roof of a tall building, it is necessary to grasp the damage situation of buildings that are not visible in the overhead image because they are in the blind spots of other buildings, for example.

[0008] The technology disclosed herein aims to provide an information processing device, an information processing method, a program, and an information collection system that enable the quick understanding of the damage situation of buildings that are not visible in overhead images taken by a fixed camera. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the information processing device of the present disclosure includes a processor and a memory connected to or built into the processor, and the processor executes an acquisition process for acquiring an overhead image captured by a fixed camera, an extraction process for individually extracting images of buildings from the overhead image, an identification process for identifying unphotographed buildings that are not shown in the overhead image from among the extracted images extracted by the extraction process and buildings included in map information in which buildings and addresses are associated with each other, and an image capture request process for transmitting request information to request an image of the unphotographed building to be captured by a terminal that is detected to be located near the unphotographed building based on location information from among a plurality of terminals having an image capture function and a location information transmission function.

[0010] In the extraction process, the processor preferably converts coordinates representing the perimeter of the building at each address included in the map information into coordinates on the overhead image and extracts an image of the building based on the converted coordinates, and in the identification process determines whether a structure appearing in the extracted image extracted by the extraction process is a building, and if the structure is not a building, identifies the building at the address from which the extracted image was extracted as an unphotographed building.

[0011] In the identification process, it is preferable that the processor determines whether or not a structure shown in the extracted image is a building based on a trained model trained using training data indicating whether or not the structure is a building.

[0012] It is also preferable that the map information includes a polygon image representing the outline of a building, and in the identification process, the processor determines whether a structure shown in the extracted image is a building based on the similarity between the outline of the structure shown in the extracted image and the polygon image corresponding to the extracted image.

[0013] In the extraction process, the processor extracts an image containing an object having the characteristics of a building from the overhead image as an extracted image, and in the identification process, it preferably associates the extracted image with an address included in the map information and identifies a building at an address that has not been associated as an unphotographed building.

[0014] The request information preferably includes a map showing the addresses of the buildings not yet photographed.

[0015] The request information preferably includes a sample image that represents an example of the image to be captured.

[0016] The information processing method disclosed herein includes an acquisition process for acquiring an overhead image captured by a fixed camera; an extraction process for individually extracting images of buildings from the overhead image; an identification process for identifying unphotographed buildings that are not shown in the overhead image from among the extracted images extracted by the extraction process and buildings included in map information in which buildings are associated with addresses; and an image capture request process for transmitting request information to capture an image of the unphotographed building to a terminal that is detected as being near the unphotographed building based on location information from among a plurality of terminals having an image capture function and a location information transmission function.

[0017] The program disclosed herein causes a computer to execute an acquisition process for acquiring an overhead image captured by a fixed camera; an extraction process for individually extracting images of buildings from the overhead image; an identification process for identifying unphotographed buildings that are not shown in the overhead image from among the extracted images extracted by the extraction process and buildings included in map information in which buildings are associated with addresses; and an image capture request process for sending request information to a terminal, among multiple terminals having an image capture function and a location information transmission function, that is detected to be located near an unphotographed building based on location information, to capture the unphotographed building.

[0018] The information collection system disclosed herein is an information collection system including a fixed camera that captures overhead images, an information processing device having a processor, and a plurality of terminals having an imaging function and a function of transmitting location information, in which the processor executes an acquisition process for acquiring the overhead image captured by the fixed camera, an extraction process for individually extracting images of buildings from the overhead image, an identification process for identifying unphotographed buildings that are not shown in the overhead image among the buildings included in the map information based on the extracted images extracted by the extraction process and map information in which buildings are associated with addresses, and an image capture request process for transmitting request information to request an image of the unphotographed building to be captured by a terminal that is detected as being near an unphotographed building based on the location information among the plurality of terminals having an imaging function and a function of transmitting location information. Effect of the Invention

[0019] According to the technology disclosed herein, it is possible to provide an information processing device, an information processing method, a program, and an information collection system that enable the quick understanding of the damage situation of buildings that are not captured in overhead images taken by a fixed camera. [Brief description of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information collecting system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a configuration of an information processing server. [Diagram 3] FIG. 2 is a block diagram showing an example of a configuration of a mobile terminal. [Figure 4] 2 is a block diagram showing an example of a functional configuration of an information processing server; FIG. [Diagram 5] FIG. 13 is a diagram showing an example of an overhead image. [Figure 6] FIG. 2 is a diagram illustrating an example of map information. [Figure 7] 11 is a flowchart illustrating an example of an extraction process. [Figure 8] FIG. 11 is a diagram showing an example of an extracted image extracted by the extraction process. [Figure 9] FIG. 11 is a diagram showing another example of an extracted image extracted by the extraction process. [Figure 10] 11 is a flowchart illustrating an example of a specification process. [Figure 11] 11 is a block diagram showing an example of a configuration of a determination unit. FIG. [Figure 12] FIG. 13 is a diagram illustrating a learning phase. [Figure 13] FIG. 11 is a diagram showing an example of unphotographed building information. [Figure 14] 11 is a flowchart illustrating an example of a photographing request process. [Figure 15] FIG. 11 is a diagram showing an example of request information displayed on a display unit of the mobile terminal. [Figure 16] FIG. 11 is a diagram showing an example of map information according to the second embodiment. [Figure 17] FIG. 11 is a block diagram showing an example of the configuration of a determination unit according to the second embodiment. [Figure 18] FIG. 13 is a diagram illustrating an example of determination using a similarity calculation unit. [Figure 19] FIG. 11 is a diagram illustrating another example of determination using the similarity calculation unit. [Figure 20] 13 is a flowchart illustrating an example of an extraction process according to the third embodiment. [Figure 21] FIG. 1 is a diagram for explaining a sliding window method. [Figure 22] 13 is a flowchart illustrating a specification process according to the third embodiment. [Diagram 23]13A to 13C are diagrams for explaining a specific process according to the third embodiment; [Figure 24] FIG. 13 is a diagram illustrating an example of coordinate transformation of peripheral coordinates. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] [First embodiment] 1 is a diagram showing a configuration example of an information collection system according to a first embodiment of the present disclosure. The information collection system 2 includes an information processing server 10, a fixed camera 20, and a plurality of mobile terminals 30. The information collection system 2 collects images of individual buildings (hereinafter referred to as building images) so that local governments can grasp the damage situation in urban areas when a disaster occurs, for example. The buildings include various types of buildings such as detached houses, apartment buildings, stores, and office buildings.

[0022] The information processing server 10, the fixed camera 20, and the multiple mobile terminals 30 are connected to each other so as to be able to communicate with each other via a network N. The network N is, for example, a WAN (Wide Area Network) such as the Internet or a public communication network. When using a WAN, it is preferable to build a VPN (Virtual Private Network) or use a communication protocol with a high security level such as HTTPS (Hypertext Transfer Protocol Secure) in consideration of information security.

[0023] The information processing server 10 is installed, for example, in a disaster countermeasure headquarters (government office, government office, etc.) of a local government such as a prefecture, a city, a town, a village, etc. The information processing server 10 is an example of an "information processing device" according to the technology of the present disclosure.

[0024] A client terminal 12 is also connected to the network N in a wired or wireless manner. The client terminal 12 is, for example, a desktop personal computer provided to staff at the disaster countermeasure headquarters, and has a display 13 and an input device 14. Various screens are displayed on the display 13. The input device 14 is a keyboard, a mouse, a touch panel, a microphone, etc. The client terminal 12 is operated, for example, by staff at the disaster countermeasure headquarters. Note that, although only one client terminal 12 is illustrated in FIG. 1, the number of client terminals 12 may be two or more.

[0025] Fixed camera 20 is installed on the rooftop of a tall building 21 or the like so as to be able to overlook urban area 22. Fixed camera 20 is a digital camera, and takes still images based on shooting instructions from client terminal 12. An overhead image of urban area 22 taken by fixed camera 20 is transmitted to information processing server 10 via network N. The overhead image is an image that includes the inside of the municipality.

[0026] Although the fixed camera 20 is fixed, the azimuth and elevation angle of the image capturing may be changeable. The fixed camera 20 may also be configured so that the angle of view α is changeable.

[0027] The mobile terminal 30 is a mobile electronic terminal such as a smartphone or tablet terminal having a photographing function and a location information transmission function. The mobile terminal 30 is connected to a network N via a base station 31. The mobile terminal 30 transmits images photographed by the photographing function and location information acquired by the location information acquisition function to the information processing server 10 via the base station 31 and the network N. The mobile terminal 30 acquires location information by receiving radio waves transmitted from a GPS (Global Positioning System) satellite 32, for example. The mobile terminal 30 is an example of a "terminal" according to the technology of the present disclosure.

[0028] The mobile terminal 30 is owned, for example, by a resident living in the urban area 22. Among the multiple mobile terminals 30, request information for requesting an image of the unphotographed building, which is a building not captured in the overhead image, is transmitted from the information processing server 10 to the mobile terminal 30 that is present near the unphotographed building. For example, the unphotographed building is a building that is present in the blind spot 23 of a relatively tall building as viewed from the fixed camera 20, among the buildings present in the urban area 22.

[0029] The information processing server 10 identifies a mobile terminal 30 that is located near an unphotographed building based on the location information transmitted from each of the multiple mobile terminals 30, and transmits request information to the identified mobile terminal 30. The owner of the mobile terminal 30 photographs the unphotographed building in accordance with the request information, and transmits the photographed image to the information processing server 10.

[0030] 2 shows an example of the configuration of the information processing server 10. The information processing server 10 is configured by a computer having a CPU (Central Processing Unit) 101, a memory 102, a storage 103, and a communication unit 104, which are interconnected via a bus line 105. The CPU 101 is an example of a "processor" according to the technology of the present disclosure.

[0031] The storage 103 is a hard disk drive built into a computer constituting the information processing server 10, or connected via a cable or a network. The storage 103 may be a disk array in which a plurality of hard disk drives are connected in series. The storage 103 stores control programs such as an operating system (OS), various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0032] The memory 102 is a work memory such as a RAM (Random Access Memory) for the CPU 101 to execute processing. The memory 102 is connected to or built into the CPU 101. The CPU 101 loads an operating program 106 stored in the storage 103 into the memory 102 and executes processing in accordance with the operating program 106. In this way, the CPU 101 comprehensively controls the operation of each part of the computer. The communication unit 104 controls the transmission of various information to and from external devices such as the client terminal 12, the fixed camera 20, and the mobile terminal 30 via the network N.

[0033] 3 shows an example of the configuration of the mobile terminal 30. The mobile terminal 30 is configured to include a general computer configuration. The mobile terminal 30 includes, for example, a CPU 301, a memory 302, an input unit 303, a display unit 304, a camera unit 305, a position information acquisition unit 306, a speaker unit 307, a communication unit 308, and a bus line 309.

[0034] The CPU 301 reads out and processes programs and data stored in the memory 302 to implement various functions of the mobile terminal 30. The memory 302 stores an OS, application programs, image data, various types of data, and the like.

[0035] The input unit 303 is an input device such as a touch panel that accepts input by touch operation. The input unit 303 is used to input each operation signal to the mobile terminal 30. The input unit 303 may include a microphone or the like and may accept voice input via the microphone or the like.

[0036] The display unit 304 is a display device such as a display, and displays the processing results by the mobile terminal 30. The input unit 303 and the display unit 304 may be a display input device having an input function and a display function, such as a touch panel display. The display input device is an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescent) display with a built-in touch panel sensor. The camera unit 305 is an imaging device including an image sensor and an image processing device.

[0037] The location information acquisition unit 306 generates location information by receiving a positioning signal included in the radio waves transmitted from the GPS satellite 32, and outputs the generated location information. The speaker unit 307 is an audio circuit that generates an audio signal, and an audio output device that converts the audio signal into audio and outputs it.

[0038] The communication unit 308 is a communication interface for connecting the mobile terminal 30 to the network N via the base station 31. The communication unit 308 is a wireless communication device that connects the mobile terminal 30 to the network N via a mobile communication network such as LTE (Long Term Evolution), and includes an antenna, a wireless transmission / reception circuit, a communication control circuit, etc. The communication unit 308 communicates with the information processing server 10, etc.

[0039] 4 shows an example of the functional configuration of the information processing server 10. When the operating program 106 is started, the CPU 101 of the computer constituting the information processing server 10 functions as an RW (Read / Write) control unit 40, an extraction unit 41, a specification unit 42, and an image capture request unit 43 in cooperation with the memory 102 and the like.

[0040] The RW control unit 40 controls the storage of various data in the storage 103 and the reading of various data from the storage 103. For example, the RW control unit 40 performs an acquisition process to acquire an overhead image PO captured by the fixed camera 20, and stores the acquired overhead image PO in the storage 103. The fixed camera 20 captures an image of the city area 22 in response to a shooting instruction SI transmitted from the client terminal 12, and transmits the captured overhead image PO to the information processing server 10. Fig. 5 shows an example of the overhead image PO.

[0041] Storage 103 prestores map information MI corresponding to the overhead image PO captured by fixed camera 20. Fig. 6 shows an example of map information MI. Map information MI includes map M and address information AI. Map M is, for example, a two-dimensional residential map corresponding to an area of ​​urban area 22 captured by fixed camera 20. Address information AI is information that associates all buildings present in map M with their addresses.

[0042] The RW control unit 40 reads the overhead image PO from the storage 103, and outputs the read overhead image PO to the extraction unit 41. The RW control unit 40 also reads the map information MI from the storage 103, and outputs the read map information MI to the extraction unit 41.

[0043] The extraction unit 41 performs an extraction process to individually extract images of buildings corresponding to each address in the overhead image PO from the overhead image PO based on the map information MI. The extraction unit 41 outputs the extracted extraction image PE to the specification unit 42. The extraction image PE is stored in the storage 103 via the RW control unit 40. The extraction image PE is provided with address information of the building.

[0044] The identification unit 42 performs an identification process to identify unphotographed buildings, which are buildings not captured in the overhead image PO, among buildings included in the map information MI, based on the extracted image PE extracted by the extraction unit 41. The identification unit 42 also generates information (hereinafter referred to as unphotographed building information) NS indicating the identified unphotographed buildings, and outputs the generated unphotographed building information NS to the photography requesting unit 43. For example, when the extracted image PE includes an outline of a building, the identification unit 42 outputs the extracted image PE as a building image P1 to the storage 103 via the RW control unit 40. When the extracted image PE does not include an outline of a building, the identification unit 42 identifies the building at the address from which the extracted image PE is extracted as an unphotographed building.

[0045] The photography requesting unit 43 performs photography request processing to identify a mobile terminal 30 that is located near an unphotographed building based on the location information LI transmitted from each of the multiple mobile terminals 30, and transmit request information RI to the identified mobile terminal 30. The vicinity of an unphotographed building is, for example, within a certain radius (for example, a radius of 100 meters) centered on the location of the unphotographed building. The photography requesting unit 43 generates screen data to be displayed on the display unit 304 of the mobile terminal 30 as the request information RI based on, for example, the unphotographed building information NS. The request information RI includes the address of the unphotographed building, etc. An application program for displaying the request information RI on the display unit 304 is pre-installed in the mobile terminal 30, for example.

[0046] When there are unphotographed buildings at multiple locations in the overhead image PO, the photography requesting unit 43 transmits the request information RI to the mobile terminals 30 located near each of the unphotographed buildings. The photography requesting unit 43 may transmit the request information RI to multiple mobile terminals 30 for one unphotographed building.

[0047] The storage 103 stores mobile terminal information TI for individually communicating with each mobile terminal 30. The mobile terminal information TI includes a telephone number, an email address, etc. The photography requesting unit 43 refers to the mobile terminal information TI and transmits request information RI to the target mobile terminal 30 by SMS (Short Message Service), email, etc.

[0048] When the owner of the mobile terminal 30 located near the unphotographed building performs a predetermined operation, the request information RI is displayed on the display unit 304. For example, when a link described in a message transmitted from the photography requesting unit 43 is pressed, the request information RI is displayed on the display unit 304. The owner of the mobile terminal 30 can photograph the unphotographed building according to the request information RI. When the unphotographed building is photographed by the mobile terminal 30, an image (i.e., a building image) P2 of the unphotographed building is transmitted to the information processing server 10. When the information processing server 10 receives the building image P2 from the mobile terminal 30, it stores the received building image P2 in the storage 103 via the RW control unit 40.

[0049] Fig. 7 illustrates an example of extraction processing by the extraction unit 41. As shown in Fig. 7, first, the extraction unit 41 acquires an overhead image PO from the storage 103 via the RW control unit 40 (step S10). Next, the extraction unit 41 acquires map information MI from the storage 103 via the RW control unit 40 (step S11).

[0050] Next, the extraction unit 41 selects one address included in the map information MI (step S12). Next, the extraction unit 41 identifies the coordinates of the periphery of the building present at the selected address on the map M (hereinafter, referred to as the periphery coordinates) (step S13). The periphery coordinates are, for example, latitude and longitude information that indicates the position of the periphery of the building on the map M. Next, the extraction unit 41 converts the periphery coordinates on the map M into coordinates on the overhead image PO (step S14). Then, the extraction unit 41 extracts an image from the overhead image PO based on the converted coordinates (step S15). At this time, for example, the extraction unit 41 sets a rectangular area on the overhead image PO so as to include the converted periphery coordinates, and extracts an image from the set rectangular area.

[0051] Thereafter, the extraction unit 41 determines whether or not all addresses included in the map information MI have been selected (step S16). If the extraction unit 41 determines that all addresses have not been selected (step S16: NO), the process returns to step S12 and another address is selected. The extraction unit 41 repeatedly executes the processes from step S12 to step S16 until it is determined in step S16 that all addresses have been selected. If the extraction unit 41 determines that all addresses have been selected (step S16: YES), the process ends.

[0052] Through the above process, the extraction unit 41 extracts images of buildings corresponding to all addresses included in the map information MI from the overhead image PO.

[0053] Fig. 8 shows an example of an extracted image PE extracted by the extraction process. As shown in Fig. 8, the perimeter coordinates of one building on the map M are converted into coordinates on the overhead image PO. On the overhead image PO, an image is extracted from an area including the building represented by the coordinate-converted perimeter coordinates. In this example, the extracted image PE shows the outline of the building present at the selected address. Here, the outline of the building means, for example, a line representing the perimeter of the building.

[0054] Fig. 9 shows another example of an extracted image PE extracted by the extraction process. Fig. 9 shows an extracted image PE extracted from an address different from that of Fig. 8. In this example, the outline of a building present at the address selected by the extraction unit 41 is obscured by another building present in the foreground, and is not captured in the extracted image PE (i.e., is not captured in the overhead image PO). In this way, of the buildings present at the address selected by the extraction unit 41, buildings not captured in the overhead image PO are identified by the identification unit 42 as unphotographed buildings.

[0055] In addition, a building located at an address selected by the extraction unit 41 may not appear in the overhead image PO and may be identified as an unphotographed building not only when it is blocked by other buildings but also when it is blocked by obstacles such as smoke generated in the event of a fire.

[0056] Fig. 10 illustrates an example of the identification process by the identification unit 42. As shown in Fig. 10, first, the identification unit 42 acquires one of the multiple extraction images PE extracted by the extraction unit 41 (step S20). Next, the identification unit 42 determines whether or not a structure in the acquired extraction image PE is a building (step S21). For example, the identification unit 42 determines whether or not a structure in the extraction image PE is a building based on whether or not the outline of a building is included in the extraction image PE.

[0057] If the identification unit 42 determines that the structure in the extracted image PE is a building (step S21: YES), it stores the extracted image PE in the storage 103 as a building image P1 (step S22). On the other hand, if the identification unit 42 determines that the structure in the extracted image PE is not a building (step S21: NO), it identifies the building at the address from which the extracted image PE was extracted as an unphotographed building (step S23).

[0058] Then, the identification unit 42 judges whether or not all of the extracted images PE extracted by the extraction unit 41 have been acquired (step S24). If the identification unit 42 judges that all of the extracted images PE have not been acquired (step S24: NO), the identification unit 42 returns the process to step S20 and acquires another extracted image PE. The identification unit 42 repeatedly executes the processes from step S20 to step S24 until it is determined in step S24 that all of the extracted images PE have been acquired. If the identification unit 42 judges that all of the extracted images PE have been acquired (step S24: YES), the identification unit 42 generates unphotographed building information NS and outputs the generated unphotographed building information NS to the photography requesting unit 43 (step S25). This ends the identification process.

[0059] 11 shows an example of the configuration of the identification unit 42. The identification unit 42 has a structure determination model 42A that determines whether or not a structure shown in the extracted image PE is a building. The structure determination model 42A is a machine learning model constructed by a method such as a neural network, a support vector machine, or boosting. The structure determination model 42A is stored in the storage 103, and is read out from the storage 103 by the RW control unit 40 and output to the identification unit 42. The structure determination model 42A is an example of a "trained model" according to the technology of the present disclosure.

[0060] The identification unit 42 inputs the extracted image PE to the structure determination model 42A. The structure determination model 42A determines whether or not the extracted image PE includes an outline of a building (i.e., whether or not the structure is a building), and outputs the determination result JR. The identification unit 42 generates unphotographed building information NS indicating the address of the unphotographed building based on the determination result JR output from the structure determination model 42A. The unphotographed building information NS includes address information attached to the extracted image PE that has been determined not to include an outline of a building (i.e., the structure is not a building).

[0061] 12 illustrates a learning phase in which the structure determination model 42A is trained by machine learning. The structure determination model 42A learns using training data TD. The training data TD includes training images PL and correct answer data CA. The training images PL include images that include building contours and images that do not include building contours. The correct answer data CA is data that represents the correct answer as to whether or not the training images PL include building contours.

[0062] In the learning phase, the learning image PL is input to the structure determination model 42A. The structure determination model 42A outputs a determination result JR for the learning image PL. A loss calculation is performed using a loss function based on the determination result JR and the correct answer data CA. Then, various coefficients of the structure determination model 42A are updated according to the result of the loss calculation, and the structure determination model 42A is updated according to the update setting.

[0063] In the learning phase of the structure determination model 42A, a series of processes are repeatedly performed, including input of the learning image PL to the structure determination model 42A, output of the determination result JR from the structure determination model 42A, loss calculation, update setting, and updating of the structure determination model 42A. The repetition of this series of processes ends when the discrimination accuracy reaches a predetermined set level. The structure determination model 42A whose discrimination accuracy has thus reached the set level is stored in the storage 103 and used by the identification unit 42.

[0064] The learning image PL is, for example, an image of individual buildings extracted from the overhead image PO. The correct answer data CA is, for example, a result of a person visually determining whether or not the learning image PL contains the outlines of buildings. The learning image PL may be an image extracted from an overhead image of an urban area different from the urban area 22 captured by the fixed camera 20. The learning image PL may also be an image extracted from an overhead image of the same urban area captured under various conditions, such as different dates, times, weather, etc.

[0065] 13 shows an example of unphotographed building information NS. For example, the unphotographed building information NS is information that represents an area 50 in which an unphotographed building exists on a map M. For example, a plurality of areas 50 exists on the map M.

[0066] Fig. 14 illustrates an example of a photography request process by the photography requesting unit 43. As shown in Fig. 14, the photography requesting unit 43 acquires unphotographed building information NS from the identifying unit 42 (step S30). After acquiring the unphotographed building information NS, the photography requesting unit 43 acquires position information from each of the multiple mobile terminals 30 present on the map M (step S31). Then, the photography requesting unit 43 detects the mobile terminals 30 present near the unphotographed buildings based on the unphotographed building information NS and the position information of the mobile terminals 30 (step S32).

[0067] Next, the photography requesting unit 43 generates request information RI based on the unphotographed building information NS (step S33). Specifically, the photography requesting unit 43 generates individual request information RI for each detected mobile terminal 30. Then, the photography requesting unit 43 refers to the mobile terminal information TI stored in the storage 103, and transmits the request information RI to the detected mobile terminal 30 by SMS, email, or the like (step S34). This ends the photography request process.

[0068] 15 shows an example of the request information RI displayed on the display unit 304 of the mobile terminal 30. The request information RI is displayed on the display unit 304 by the owner of the mobile terminal 30 that has received the request information RI operating the input unit 303.

[0069] For example, the request information RI includes a message 60 encouraging a user to take a picture of an unphotographed building, an address 61 of the unphotographed building, a map 62 showing the location of the unphotographed building, and a sample image 63 showing an example of an image to be taken.

[0070] For example, an arrow 64 indicating the position of an unphotographed building is displayed on the map 62. A current location mark 65 indicating the current location of the mobile terminal 30 is also displayed on the map 62. The owner of the mobile terminal 30 can ascertain the location of the unphotographed building based on the address 61 and the map 62. Note that the map 62 may be updated from time to time from map data on the Internet in accordance with the movement of the mobile terminal 30.

[0071] It is preferable that the sample image 63 shows the entire building. It is also preferable to display a plurality of sample images 63 on the display unit 304. It is preferable that the sample images 63 include all four walls of the building. It is also preferable to display notes 66 for taking pictures together with the sample images 63 on the display unit 304.

[0072] Furthermore, the display unit 304 displays an execute button 67 for starting shooting. By pressing the execute button 67, the owner of the mobile terminal 30 can start the camera unit 305 (see FIG. 3) and start shooting the unphotographed building. The building image P2 of the unphotographed building photographed by the mobile terminal 30 is transmitted to the information processing server 10 and stored in the storage 103.

[0073] The information collection system 2 configured as above starts the information collection process, for example, immediately after the occurrence of a disaster, when a local government official operates the client terminal 12. In this case, the overhead image PO is an image including the disaster occurrence area. The local government official can check the collected building images P1 and P2 on the client terminal 12.

[0074] According to the technology of the present disclosure, building images P1 of buildings captured in the overhead image PO are collected from the overhead image PO, and building images P2 of unphotographed buildings not captured in the overhead image PO are collected by capturing images with a mobile terminal 30 located near the unphotographed buildings. Therefore, according to the technology of the present disclosure, the damage situation of buildings not captured in the overhead image PO can be quickly grasped.

[0075] The information collection system 2 may acquire unphotographed building information NS by executing the information collection process during normal times when no disaster has occurred. In this case, since the unphotographed building information NS is acquired in advance before a disaster occurs, a request to photograph unphotographed buildings can be made promptly after a disaster occurs. It is also preferable that the information collection system 2 periodically executes the information collection process during normal times when no disaster has occurred, thereby updating the unphotographed building information NS to the latest information as needed. This makes it possible to quickly respond to changes in the unphotographed building information NS due to the construction or demolition of buildings, etc.

[0076] [Second embodiment] Next, a second embodiment will be described. In the first embodiment, the map M included in the map information MI is a two-dimensional planar map, but in the second embodiment, the map M is a three-dimensional relief map in which buildings are represented by polygons.

[0077] 16 shows an example of map information MI according to the second embodiment. In a map M included in the map information MI, buildings represented by polygon images are recorded in association with addresses. As in the first embodiment, address information AI is information that associates buildings with their addresses for all buildings present in the map M.

[0078] In this embodiment, the configurations of the extraction unit 41 and the photography requesting unit 43 are similar to those in the first embodiment, but the configuration of the identification unit 42 is different from that in the first embodiment.

[0079] 17 shows an example of the configuration of the identification unit 42 according to the second embodiment. In this embodiment, the identification unit 42 performs the determination using a similarity calculation unit 42B instead of a structure determination model 42A. An extraction image PE and a polygon image PG corresponding to the extraction image PE are input to the similarity calculation unit 42B. In the map M, the polygon image PG is an image of a building corresponding to the address from which the extraction image PE is extracted.

[0080] The similarity calculation unit 42B calculates the similarity between the outline of the structure in the extraction image PE and the polygon image PG. The identification unit 42 judges whether the outline of a building is included in the extraction image PE (i.e., whether the structure is a building) based on the similarity calculated by the similarity calculation unit 42B. Then, the identification unit 42 generates unphotographed building information NS indicating the address of the unphotographed building based on the judgment result JR.

[0081] The similarity calculation unit 42B calculates the similarity by, for example, a pattern matching method. Note that the similarity calculation unit 42B may calculate the similarity by a method using machine learning such as a neural network.

[0082] Fig. 18 shows an example of a determination using the similarity calculation unit 42B. Since the outline of a building is captured in the extraction image PE shown in Fig. 18, the similarity between the extraction image PE and the polygon image PG calculated by the similarity calculation unit 42B is equal to or greater than a threshold value. When the similarity is equal to or greater than a threshold value in this manner, the determination unit 42 determines that the extraction image PE includes the outline of a building. In this case, the extraction image PE is stored in the storage 103 as a building image P1.

[0083] Fig. 19 shows another example of the determination using the similarity calculation unit 42B. Since the outline of the building is not completely captured in the extraction image PE shown in Fig. 19, the similarity between the extraction image PE and the polygon image PG calculated by the similarity calculation unit 42B is less than the threshold value. When the similarity is less than the threshold value in this way, the determination unit 42 determines that the extraction image PE does not include the outline of the building. In this case, the building at the address from which the extraction image PE is extracted is determined as an unphotographed building and is included in the unphotographed building information NS.

[0084] [Third embodiment] Next, a third embodiment will be described. In the first embodiment, the extraction unit 41 extracts an image of a building corresponding to each address in the overhead image PO as an extracted image PE based on the map information MI. In contrast, in the third embodiment, the extraction unit 41 extracts an image including an object having a characteristic of a building from the overhead image PO as an extracted image PE without using the map information MI. In this embodiment, the map information MI is used by the identification unit 42.

[0085] FIG. 20 illustrates an example of extraction processing by the extraction unit 41 according to the third embodiment. As shown in FIG. 20, first, the extraction unit 41 acquires an overhead image PO from the storage 103 (step S40). Next, the extraction unit 41 searches for an object having the characteristics of a building from the overhead image PO by a sliding window method (step S41). Then, the extraction unit 41 extracts the object having the characteristics of a building as an extracted image PE (step S42). This ends the extraction processing.

[0086] 21 illustrates the sliding window method. The extraction unit 41 sets a window W in the overhead image PO, and detects an object having the characteristics of a building based on the feature amount of the structure included in the window W while changing the position of the window W. The extraction unit 41 searches for the object while changing the size and shape of the window W in addition to the position of the window W.

[0087] The extraction unit 41 is configured by, for example, a convolutional neural network. The search using the sliding window method may be performed based on a feature map obtained by converting the overhead image PO by a convolutional layer.

[0088] FIG. 22 explains the identification process by the identification unit 42 according to the third embodiment. As shown in FIG. 22, first, the identification unit 42 acquires map information MI from the storage 103 (step S50). Next, the identification unit 42 acquires an extraction image PE extracted by the extraction unit 41 (step S51). The identification unit 42 associates the acquired extraction image PE with an address included in the map information MI (step S52). Here, the identification unit 42 associates all extraction images PE acquired from the extraction unit 41 with addresses included in the map information MI.

[0089] Then, the identifying unit 42 identifies, among the addresses of buildings included in the map information MI, buildings with addresses that have not been associated with the extracted image PE as unphotographed buildings (step S53). This ends the identifying process.

[0090] Fig. 23 is a schematic diagram of the identification process according to the third embodiment. As shown in Fig. 23, the identification process associates an extracted image PE extracted from an overhead image PO as an image including an object having a characteristic of a building with an address included in map information MI. After all extracted images PE extracted from the overhead image PO are associated with an address, a building with an unassociated address is identified as an unphotographed building.

[0091] [Conversion of outer perimeter coordinates] Next, the coordinate conversion of the perimeter coordinates of the building executed by the extraction process will be described. Fig. 24 shows an example of the coordinate conversion of the perimeter coordinates. In this example, the map M is a two-dimensional planar map.

[0092] First, four corresponding points S1 to S4 are set on the map M and the overhead image PO. The four points S1 to S4 may be set manually or by image processing, etc. Next, a projective transformation matrix is ​​calculated based on the coordinates of the four corresponding points S1 to S4 on the map M and the coordinates on the overhead image PO.

[0093] The map MP is a map obtained by projectively transforming the map M using a projective transformation matrix. The perimeter coordinates of the buildings on the map MP correspond to the perimeter coordinates on the overhead image PO. The extraction unit 41 may use the projective transformation matrix to perform coordinate transformation of the perimeter coordinates of the buildings.

[0094] Furthermore, if the map M is a three-dimensional relief map, it is possible to generate a map similar to the projectively transformed map MP by changing parameters such as the line of sight and angle of view of the relief map and determining the parameters that match most closely with the overhead image PO.

[0095] In addition, if it is possible to obtain information on latitude, longitude, and altitude for six points in the overhead image PO, it is possible to convert the overhead image PO into a planar image. Based on this converted overhead image PO and the two-dimensional planar map M, the perimeter coordinates of the buildings in the overhead image PO may be obtained.

[0096] In each of the above embodiments, the following various processors can be used as the hardware structure of the processing unit that executes various processes, such as the RW control unit 40, the extraction unit 41, the identification unit 42, and the photography request unit 43. As described above, the various processors include the CPU 101, which is a general-purpose processor that executes software (operation program 106) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.

[0097] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0098] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as represented by computers such as client and server. Second, there is a form in which a processor is used to realize the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as represented by a system on chip (SoC). In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0099] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0100] The technology of the present disclosure can be appropriately combined with the above-mentioned various embodiments and / or various modified examples. In addition, the present invention is not limited to the above-mentioned embodiments, and various configurations can be adopted without departing from the gist of the present invention. Furthermore, in addition to the program, the present invention also covers a computer-readable storage medium that non-temporarily stores the program.

[0101] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0102] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0103] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0104] 2. Information Collection System 10. Information Processing Server 12 Client terminal 13. Display 14 Input Devices 20 Fixed Camera 21 Tall Building 22 Urban Area 23 blind spot 30 Mobile Devices 31 Base station 32 GPS satellites 40 RW control section 41 Extraction part 42 Specific part 42A Structure Judgment Model 42B Similarity calculation part 43 Photoshoot Request Department 50 Area 60 Messages 61 Address of unphotographed building 62 Map 63 Sample Images 64 Arrow 65 Current location mark 66 Notes 67 Run button 101 CPU 102 Memory 103 Storage 104 Communications Department 105 Bus Line 106 Operating Program 301 CPU 302 Memory 303 Input section 304 Display section 305 Camera Department 306 Location information acquisition unit 307 Speaker section 308 Communications Department 309 Bus Line α Angle of View AI address information CA Answer Data JR judgment results LI location information M Map MI Map Information MP Map N Network NS Unphotographed building information P1, P2 Building images PE Extracted Image PG Polygon Image PL learning images PO overhead view image RI Request Information SI Shooting Instructions TD Teacher Data TI Mobile Device Information W Window

Claims

1. A processor; A memory connected to or embedded in the processor; Equipped with The processor, Obtain overhead images taken by a fixed camera, An image estimated to be a building is individually extracted as an extracted image from the overhead image; Based on the extracted image, a non-photographed building that is not captured in the overhead image is identified from among buildings included in map information in which buildings and addresses are associated with each other; transmitting request information to a terminal, among a plurality of terminals having an image capturing function and a position information transmitting function, which is detected to be in the vicinity of the unphotographed building based on the position information, to request the unphotographed building to be photographed; Information processing device.

2. The extracted image is an image obtained by converting coordinates representing the perimeter of a building at each address included in the map information into coordinates on the overhead image, and extracting an area corresponding to the converted coordinates from the overhead image, The processor determines whether a structure shown in the extracted image is a building, and if the structure is not a building, identifies a building at an address from which the extracted image is extracted as the unphotographed building. The information processing device according to claim 1 .

3. The processor, determining whether or not the structure shown in the extracted image is a building based on a trained model trained using training data indicating whether or not the structure is a building; The information processing device according to claim 2 .

4. The map information includes a polygon image representing an outline of a building, The processor, determining whether the structure is a building based on a similarity between an outline of the structure and the polygon image corresponding to the extracted image; The information processing device according to claim 2 .

5. The extracted image is an image including an object having a characteristic of a building extracted from the overhead image, The processor associates the extracted image with an address included in the map information, and identifies a building at an address that has not been associated as the unphotographed building. The information processing device according to claim 1 .

6. The request information includes a map showing the address of the unphotographed building. The information processing device according to claim 1 .

7. The request information includes a sample image representing an example of an image to be captured. The information processing device according to any one of claims 1 to 6.

8. An acquisition process for acquiring an overhead image captured by a fixed camera; an extraction process of individually extracting an image estimated to be a building from the overhead image as an extracted image; a process of identifying an unphotographed building that is not captured in the overhead image, among buildings included in map information in which buildings and addresses are associated with each other based on the extracted image; an image capturing request process for transmitting request information to a terminal, among a plurality of terminals having an image capturing function and a position information transmitting function, that is detected to be in the vicinity of the unphotographed building based on the position information, to capture the unphotographed building; An information processing method comprising:

9. An acquisition process for acquiring an overhead image captured by a fixed camera; an extraction process of individually extracting an image estimated to be a building from the overhead image as an extracted image; a process of identifying an unphotographed building that is not captured in the overhead image, among buildings included in map information in which buildings and addresses are associated with each other based on the extracted image; an image capturing request process for transmitting request information to a terminal, among a plurality of terminals having an image capturing function and a position information transmitting function, that is detected to be in the vicinity of the unphotographed building based on the position information, to capture the unphotographed building; A program that causes a computer to execute the following.

10. An information collection system including a fixed camera for capturing an overhead image, an information processing device having a processor, and a plurality of terminals having a capturing function and a transmitting function of position information, The processor, An acquisition process for acquiring an overhead image captured by a fixed camera; an extraction process of individually extracting an image estimated to be a building from the overhead image as an extracted image; a process of identifying an unphotographed building that is not captured in the overhead image, among buildings included in map information in which buildings and addresses are associated with each other based on the extracted image; an image capturing request process for transmitting request information to a terminal, among a plurality of terminals having an image capturing function and a position information transmitting function, that is detected to be in the vicinity of the unphotographed building based on the position information, to capture the unphotographed building; Execute Information gathering system.

Images