Image processing device, image processing method, and program

Abstract

The present invention provides an image processing device comprising: a disaster information acquiring unit that acquires disaster information indicating a type of disaster; a weighting unit that sets a weight for each reference object included in a plurality of reference images having position information, on the basis of the type of disaster indicated by the disaster information; a comparing unit that compares an input image with the plurality of reference images, taking into account the weight of each reference object; and a position identifying unit that identifies the imaging location of the input image on the basis of the comparison results.

Description

Image processing device, image processing method, and recording medium

[0001] The present invention relates to an image processing device, an image processing method, and a program.

[0002] Techniques related to the present invention are disclosed in Patent Documents 1 to 3 and Non-Patent Documents 1 to 5.

[0003] Patent Literature 1 discloses a technology for estimating the location at the time of image capture based on the results of comparing captured image with images registered in a database. This technology changes the priority of comparison with images registered in the database based on the date, time, and weather conditions.

[0004] Patent Document 2 and Non-Patent Documents 1 to 5 disclose techniques for estimating the range of activities of users who have accounts on social media such as SNSs (social networking services).

[0005] Patent Document 3 discloses a technique for identifying the location of a user when text data or the like is posted on a social media such as an SNS.

[0006] Japanese Patent Application Laid-Open No. 2011-113245 International Publication No. 2021 / 028988 Japanese Patent Application Laid-Open No. 2018-010378

[0007] Keisuke Ikeda, Kazufumi Kojima, and Masahiro Tani, "A study on residential area estimation methods focusing on the geographical proximity of friend groups," IEICE Technical Report, Vol. 119, No. 317, pp. 37-42, AI2019-36, November 2019. Dan Xu, Peng Cui, Wenwu Zhu, and Shiqiang Yang, "Graph-based residence location inference for social media users," IEEE Computer Society, IEEE MultiMedia, Volume 21, Issue 4, pp. 76-83, October 2014. Lars Backstrom, Eric Sun, and Cameron Marlow, "Find me if you can: Improving geographical prediction with social and spatial proximity," Proceedings of the 19th international conference on World Wide Web, 2010, pp. 61-70. Liu Zhi and Yan Huang, "Closeness and structure of friends help to estimate user locations," International Conference on Database Systems for Advanced Applications, Springer, pp. 33-48. Keisuke Ikeda, Kazufumi Kojima, Masahiro Tani, "A method for estimating the activity area of ​​social media users using kernel density estimation," IEICE Technical Report, Vol. 120, No. 379, pp. 18-23, AI2020-42, February 2021.

[0008] Techniques for identifying the location where an image was taken by image analysis have been widely studied. As a result of studying such techniques, the present inventor has newly discovered the following problem.

[0009] In a technology for identifying the location where an image was taken using image analysis, the location of the image is identified by detecting one or more landmarks present in each location within the image. However, when a disaster such as a natural disaster or a man-made disaster occurs, some of the landmarks present in each location may be hidden (e.g., hidden by flood or smoke), or their appearance may change or disappear due to damage, etc. If such situations are not taken into consideration, the accuracy of identifying the location of an image taken in a disaster-stricken area will be poor.

[0010] Although Patent Document 1 relates to a technology for estimating the location where an image was taken by image analysis, it does not describe or suggest the above-mentioned problem or a means for solving the problem.

[0011] Patent Document 2 and Non-Patent Documents 1 to 5 relate to technology for estimating the range of activity of users who have accounts on social media such as SNS, but do not describe or suggest the above-mentioned problem or a means for solving the problem.

[0012] Patent Document 3 relates to a technology for identifying the location of a user when posting text data or the like on a social media such as an SNS, and does not describe or suggest the above-mentioned problem or a means for solving the problem.

[0013] In view of the above-mentioned problems, one example of the object of the present invention is to provide an image processing device, an image processing method, and a program that solve the problem of accurately identifying the location where an image taken at a disaster site was taken.

[0014] According to one aspect of the present invention, there is provided an image processing device having: a disaster information acquisition means for acquiring disaster information indicating a type of disaster; a weighting means for setting a weight for each of reference objects included in a plurality of reference images with location information based on the type of disaster indicated in the disaster information; a matching means for matching an input image with the plurality of reference images taking into account the weight for each of the reference objects; and a location identification means for identifying a location where the input image was taken based on the matching result.

[0015] According to one aspect of the present invention, an image processing method is provided in which one or more computers acquire disaster information indicating a type of disaster, set weights for each reference object included in a plurality of reference images with location information based on the type of disaster indicated in the disaster information, compare an input image with the plurality of reference images taking into account the weights for each reference object, and identify the location where the input image was taken based on the comparison result.

[0016] According to one aspect of the present invention, there is provided a program that causes a computer to function as: disaster information acquisition means for acquiring disaster information indicating the type of disaster; weighting means for setting a weight for each reference object included in a plurality of reference images with location information based on the type of disaster indicated in the disaster information; matching means for matching an input image with a plurality of the reference images taking into account the weight for each of the reference objects; and location identification means for identifying the location where the input image was taken based on the matching result.

[0017] According to one aspect of the present invention, an image processing device, an image processing method, and a program are realized that solve the problem of accurately estimating the shooting location of an image taken in a disaster area.

[0018] The above-mentioned objects, as well as other objects, features, and advantages, will become more apparent from the following description of the preferred embodiments and the accompanying drawings.

[0019] FIG. 1 is a diagram showing an example of a functional block diagram of an image processing device. FIG. 2 is a diagram showing an example of a hardware configuration of an image processing device. FIG. 3 is a diagram showing another example of a functional block diagram of an image processing device. FIG. 4 is a diagram showing an example of information processed by the image processing device. FIG. 5 is a flowchart showing an example of a processing flow of the image processing device. FIG. 6 is a diagram showing another example of a functional block diagram of an image processing device. FIG. 7 is a diagram for explaining a process in which the image processing device estimates a disaster area. FIG. 8 is a flowchart showing an example of a processing flow of the image processing device.

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0021] 1 is a functional block diagram showing an overview of an image processing device 10 according to a first embodiment. The image processing device 10 includes a disaster information acquisition unit 11, a weighting unit 12, a matching unit 13, and a position identification unit 18.

[0022] The disaster information acquisition unit 11 acquires disaster information indicating the type of disaster. The weighting unit 12 sets weights for each reference object included in multiple reference images with location information based on the type of disaster indicated in the disaster information. The matching unit 13 matches the input image with the multiple reference images, taking into account the weights for each reference object. The location identification unit 18 identifies the shooting location of the input image based on the matching result.

[0023] The image processing device 10 having such a configuration solves the problem of accurately identifying the shooting location of an image taken in a disaster area.

[0024] Second Embodiment Overview An image processing device 10 according to a second embodiment is a more specific version of the image processing device 10 according to the first embodiment.

[0025] The inventors have studied techniques for identifying the location where an image was taken by image analysis and have found that "a wide variety of objects are included among one or more landmarks present at each location" and that "the landmarks that may be hidden, damaged, or destroyed vary depending on the type of disaster that has occurred." Based on this knowledge, the inventors have invented the image processing device 10 of this embodiment.

[0026] First, the image processing device of this embodiment compares an input image with multiple reference images with pre-registered location information, and identifies the shooting location of the input image based on the comparison result. The reference images include reference objects, which are one or more landmarks present at each location. The image processing device identifies reference images whose similarity with the input image satisfies a predetermined condition, and identifies the location indicated by the location information associated with the reference image as the shooting location of the input image.

[0027] The image processing device then adjusts the weights of the multiple landmarks (reference objects) used in matching the input image with the reference image, depending on the type of disaster that has occurred. That is, when a first disaster occurs, the image processing device matches the input image with the reference image by relatively decreasing the weights of landmarks (reference objects) that are likely to be hidden, damaged, or disappeared when the first disaster occurs, and relatively increasing the weights of landmarks (reference objects) that are less likely to be hidden, damaged, or disappeared when the second disaster occurs. Furthermore, when a second disaster occurs, the image processing device matches the input image with the reference image by relatively decreasing the weights of landmarks (reference objects) that are likely to be hidden, damaged, or disappeared when the second disaster occurs, and relatively increasing the weights of landmarks (reference objects) that are less likely to be hidden, damaged, or disappeared when the second disaster occurs.

[0028] In this way, the image processing device of this embodiment adjusts the weights of landmarks (reference objects) depending on the type of disaster that has occurred, and compares the input image with multiple reference images (calculates similarities) taking the set weights into consideration. As a result, it becomes possible to accurately identify the shooting location of an image taken at a disaster site. The configuration of the image processing device 10 will be described in more detail below.

[0029] "Hardware Configuration" Next, an example of the hardware configuration of the image processing device 10 will be described. Each functional unit of the image processing device 10 is realized by any combination of hardware and software, centered around a CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk that stores the programs (this can store programs that are pre-loaded when the device is shipped, as well as programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface. Those skilled in the art will understand that there are many variations in the implementation methods and devices.

[0030] FIG. 2 is a block diagram illustrating an example of the hardware configuration of an image processing device 10. As shown in FIG. 2, the image processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the image processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0031] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to and from each other. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of input devices include a keyboard, a mouse, a microphone, physical buttons, a touch panel, etc. Examples of output devices include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0032] "Functional Configuration" Next, the functional configuration of the image processing device 10 of the second embodiment will be described in detail. FIG. 3 shows an example of a functional block diagram of the image processing device 10. As shown in the figure, the image processing device 10 has a disaster information acquisition unit 11, a weighting unit 12, a matching unit 13, an input image acquisition unit 14, a storage unit 15, and a position identification unit 18. Note that the image processing device 10 does not necessarily have to have the storage unit 15. In this case, an external device configured to be able to communicate with the image processing device 10 has the storage unit 15.

[0033] The disaster information acquisition unit 11 acquires disaster information indicating the type of disaster.

[0034] There are various ways to define "type of disaster." Disasters include at least one of natural disasters and man-made disasters. For example, types of disasters may be classified into heavy rain, torrential rain, typhoons, earthquakes, tsunamis, floods, storm surges, heavy snowfall, tornadoes, volcanic eruptions, landslides, subsidence, inundation, fires, forest fires, etc. As another example, types of disasters may be classified into wind disasters, water disasters, fire disasters, earth disasters, etc.

[0035] There are various means for acquiring disaster information. For example, a user (such as an operator) may input a disaster type. Then, the disaster information acquisition unit 11 may acquire disaster information indicating the type of disaster specified by the user. Alternatively, the disaster information acquisition unit 11 may acquire disaster warnings or advisories publicly announced by the Japan Meteorological Agency, local governments, etc. via a server, etc., as disaster information. Alternatively, the disaster information acquisition unit 11 may analyze images or text posted on social media such as SNS to detect the occurrence and type of disaster. Then, the disaster information acquisition unit 11 may generate disaster information indicating the type of disaster detected.

[0036] The weighting unit 12 sets a weight for each of the reference objects included in the plurality of reference images with position information, based on the type of disaster indicated in the disaster information.

[0037] Before describing the process of setting the weights, a description of the reference object, the reference image, and related matters will be given.

[0038] A "reference object" is one or more landmarks present at each location. What is selected as a reference object is a design matter, but a wide variety of objects may be selected as reference objects. The selected multiple reference objects may include objects of different types, heights, weights, materials, and elevations. For example, man-made objects such as buildings, houses, schools, hospitals, government offices, other buildings, traffic lights, roads, sidewalks, road markings, curbs, guardrails, utility poles, and steel towers, as well as natural objects such as trees, waterfalls, cliffs, and mountains may be selected as reference objects.

[0039] The "reference image" includes one or more reference objects. A plurality of reference images are stored in advance in the storage unit 15. It is preferable that a large number of reference images taken at various locations are stored in the storage unit 15.

[0040] As shown in FIG. 4, the related information of each reference image is stored in the storage unit 15 in association with the identification information (reference image identification information) of each reference image.

[0041] The "related information" includes location information and reference object information.

[0042] "Location information" indicates the location where each reference image was taken. The location information indicates the location using, for example, latitude and longitude, but may also indicate the location using other information.

[0043] "Reference object information" is information about one or more reference objects included in each reference image. The weighting unit 12 determines the weight of each reference object based on the reference object information. In other words, the reference object information is information used when determining the weight of each reference object, and includes information necessary for determining the weight.

[0044] For example, the reference object information may be information in which the type of disaster and a weight are linked for each reference object. Examples of such reference object information include "first disaster: weight 1," "second disaster: weight 0.3," etc. Here, the weights are set between 0 and 1, but this is not limiting. Note that when multiple reference objects are included in one reference image, the multiple reference objects are distinguished from one another using information indicating their positions within the image, etc. Then, the above-described reference object information is registered, linked to each reference object.

[0045] As another example, the reference object information may include material information for determining the weight. The material information is useful for determining the possibility of an object being hidden, damaged, or destroyed in the event of a disaster, and may indicate, for example, at least one of the type of the reference object (a building, a house, a school, a hospital, a government office, another building, a traffic light, a road, a sidewalk, a road marking, a curb, a guardrail, a utility pole, a steel tower, a tree, a waterfall, a cliff, a mountain, etc.), height, weight, material, elevation of the location, earthquake resistance, degree of damage, age, etc.

[0046] Next, a process for setting a weight for each reference object will be described. As described above, the weighting unit 12 sets a weight for each reference object according to the type of disaster indicated in the disaster information, based on the reference object information registered in advance.

[0047] If the reference object information is information that associates a disaster type with a weight for each reference object, the weighting unit 12 determines the weight of each reference object in accordance with the information. For example, if the reference object information for a first reference object is defined as "first disaster: weight 1," "second disaster: weight 0.3," etc., and the disaster information acquired by the disaster information acquisition unit 11 indicates a "second disaster," the weighting unit 12 determines the weight of the first reference object to be "0.3."

[0048] In addition, when the reference object information includes material information for determining a weight, the weighting unit 12 determines a weight for each reference object based on a weight calculation method prepared in advance for each type of disaster. The weight calculation method is an arithmetic expression or calculation algorithm that receives various material information indicated in the reference object information as input and outputs a weight. When the weighting unit 12 identifies a weight calculation method corresponding to the type of disaster indicated in the disaster information acquired by the disaster information acquisition unit 11, the weighting unit 12 inputs the material information of each reference object into the identified weight calculation method and calculates the weight of each reference object.

[0049] The details of the calculation formula and calculation algorithm for calculating the weight are design matters, but are designed to satisfy certain conditions. The conditions include at least one of the following: If the type of disaster indicated in the disaster information is a first disaster, a higher weight is set for the reference object that is taller; If the type of disaster indicated in the disaster information is a second disaster, a higher weight is set for the reference object that is located at a higher altitude; If the type of disaster indicated in the disaster information is a third disaster, a higher weight is set for the reference object that has a lower risk of damage; If the type of disaster indicated in the disaster information is a fourth disaster, a higher weight is set for the reference object that is heavier.

[0050] "First disasters" include heavy rain, torrential rain, tsunami, storm surge, heavy snowfall, inundation, flooding, and submersion. When such disasters occur, low-height reference objects may be hidden by water or snow. Therefore, the higher the height of a reference object, the higher the weight is set.

[0051] "Secondary disasters" include heavy rain, torrential rain, tsunami, high tide, inundation, flooding, and submergence. When such disasters occur, reference objects located at low elevations may be hidden by water. Therefore, the higher the elevation of a reference object located at, the greater the weight is set.

[0052] "Third disasters" include typhoons, earthquakes, tsunamis, floods, storm surges, heavy snowfall, tornadoes, volcanic eruptions, and landslides. When such disasters occur, there is a possibility that parts of the reference object will be damaged or destroyed (completely destroyed). Therefore, the lower the risk of damage to a reference object, the higher the weight is set. The risk of damage can be calculated using a predetermined calculation algorithm based on seismic performance, materials, damage level, age of the building, etc.

[0053] The "fourth disaster" is heavy rain, torrential rain, tsunami, flood, high tide, etc. When such a disaster occurs, there is a possibility that light reference objects will be washed away by water. Therefore, the heavier the reference object, the greater the weight is set.

[0054] Returning to FIG. 3 , the input image acquisition unit 14 acquires an input image. The input image is an image of a target for which a shooting location is to be identified. The input image acquisition unit 14 may acquire an image input by a user as the input image, or may acquire images stored in a database in a predetermined order as the input images. Furthermore, the input image acquisition unit 14 may acquire an image posted on social media such as an SNS as the input image, or may acquire the input image by other means.

[0055] The matching unit 13 matches the input image with multiple reference images, taking into account the weight of each reference object. Then, the position identification unit 18 identifies the shooting location of the input image based on the matching result by the matching unit 13. Specifically, the matching unit 13 calculates the similarity between the input image and each of the multiple reference images. Then, the position identification unit 18 identifies, as the shooting location of the input image, the location indicated by the position information associated with the reference image whose similarity satisfies a predetermined condition (e.g., the reference image with the highest similarity). The matching unit 13 takes into account the weight of each reference object in calculating the similarity. The greater the weight of a reference object, the greater its contribution to the calculated similarity.

[0056] Here, an example of a matching process that takes into account the weight of the reference object will be described.

[0057] First Example In this example, the matching unit 13 calculates the similarity between the input image and the reference image using a feature point matching technique.

[0058] First, the matching unit 13 performs feature point matching between the input image and the reference image using well-known techniques. Then, the matching unit 13 calculates the number of feature points that match in the matching with the input image for each reference object included in the reference image (hereinafter, "matching number"). For example, if the first reference image includes a first reference object and a second reference object, the matching unit 13 calculates the matching number for each reference object, such as "matching number for the first reference object: 126" and "matching number for the second reference object: 118." Note that the area in which each reference object exists within the reference image may be specified in advance and stored in the storage unit 15 as, for example, reference object information.

[0059] Next, the matching unit 13 calculates a score for each reference object by correcting the number of matches for each reference object included in the reference image based on the weight of each reference object. The correction method varies, but the larger the weight, the higher the score. For example, the product of the number of matches and the weight may be used as the score for each reference object. In the above example, if the weight of the first reference object is 1 and the weight of the second reference object is 0.5, the matching unit 13 calculates a score for each reference object, such as "score of first reference object: 126 (= 126 × 1)" and "score of second reference object: 59 (= 118 × 0.5)."

[0060] The matching unit 13 then calculates the similarity between the reference image and the input image based on the score for each reference object included in the reference image. The calculation of the similarity based on the score for each reference object included in the reference image can be done in various ways. For example, the sum of the scores for each reference object included in the reference image may be calculated as the similarity. In the above example, the sum of the score for the first reference object (126) and the score for the second reference object (59) is calculated as the similarity between the input image and the first reference image, which is 185.

[0061] "Second Example" In this example, the matching unit 13 calculates the similarity between the input image and the reference image using a feature point matching technique. The matching unit 13 then adjusts the criterion for determining whether there is a match (the similarity between feature points) for each feature point based on the weight. Specifically, the matching unit 13 lowers the criterion for determining whether there is a match for feature points extracted from reference objects with larger weights (i.e., makes it easier to determine whether there is a match), and raises the criterion for determining whether there is a match for feature points extracted from reference objects with smaller weights (i.e., makes it harder to determine whether there is a match).

[0062] The matching unit 13 then calculates the number of feature points that match in the matching with the input image performed using this method as the similarity between the input image and the reference image. That is, if the number of feature points that match between the input image and the first reference image in this matching is 118, the matching unit 13 determines 118 as the similarity between the input image and the first reference image.

[0063] Next, an example of the processing flow of the image processing device 10 will be described using the flowchart in Figure 5. Note that the purpose here is to explain the processing flow. Since the details of each process have been described above, the description here will be omitted as appropriate.

[0064] First, the image processing device 10 acquires disaster information indicating the type of disaster (S10). Next, the image processing device 10 sets weights for each of the reference objects included in the multiple reference images with location information, based on the type of disaster indicated in the disaster information (S11).

[0065] Thereafter, when the image processing device 10 acquires an input image (S12), it compares the input image acquired in S12 with a plurality of reference images, taking into consideration the weight of each reference object set in S11 (S13).

[0066] Then, based on the comparison result, the image processing device 10 identifies the shooting location of the input image acquired in S12 (S14). Specifically, the image processing device 10 identifies a reference image whose similarity to the input image acquired in S12 satisfies a predetermined condition (e.g., a reference image with the highest similarity). The image processing device 10 then identifies the location indicated by the location information associated with the identified reference image as the shooting location of the input image acquired in S12.

[0067] "Effects" The image processing device 10 of this embodiment identifies the shooting location of an input image based on the result of matching the input image with multiple reference images with pre-registered location information. The image processing device 10 then adjusts the weights of each of the multiple landmarks (reference objects) used in the matching between the input image and the reference image, depending on the type of disaster that has occurred. That is, when a first disaster occurs, the image processing device 10 matches the input image with the reference image by reducing the weights of landmarks (reference objects) that are likely to be hidden, damaged, or destroyed in the event of the first disaster, and increasing the weights of landmarks (reference objects) that are unlikely to be hidden, damaged, or destroyed in the event of the second disaster. When a second disaster occurs, the image processing device 10 matches the input image with the reference image by reducing the weights of landmarks (reference objects) that are likely to be hidden, damaged, or destroyed in the event of the second disaster, and increasing the weights of landmarks (reference objects) that are unlikely to be hidden, damaged, or destroyed in the event of the second disaster.

[0068] In this way, the image processing device 10 of this embodiment can adjust the weights of landmarks (reference objects) depending on the type of disaster that has occurred, and can compare the input image with multiple reference images (calculate the similarity) taking the set weights into consideration. As a result, it becomes possible to accurately identify the shooting location of an image taken at a disaster site.

[0069] The image processing device 10 of this embodiment has a function of determining whether an area included in an input image (an area shown in the input image) has been affected by a disaster, and estimating the affected area based on the result of the determination and the result of identifying the location where the input image was taken, as described in the first and second embodiments. This will be described in detail below.

[0070] 6 shows an example of a functional block diagram of the image processing device 10 according to this embodiment. The image processing device 10 includes a disaster information acquisition unit 11, a weighting unit 12, a matching unit 13, an input image acquisition unit 14, a storage unit 15, a determination unit 16, an estimation unit 17, and a position identification unit 18.

[0071] The determination unit 16 determines whether or not an area included in the input image has been affected by disaster by analyzing the input image or by receiving a user input.

[0072] First, a process for determining whether an area has been affected by a disaster by analyzing an input image will be described. In this example, the image processing device 10 stores in advance the appearance characteristics of the affected area. If the determination unit 16 can detect the characteristics from the input image, it determines that the area included in the input image has been affected by the disaster. If the determination unit 16 cannot detect the characteristics from the input image, it determines that the area included in the input image has not been affected by the disaster. Note that the image processing device 10 may store the appearance characteristics of the affected area for each type of disaster. The determination unit 16 may then detect, from the input image, characteristics corresponding to the type of disaster indicated in the disaster information acquired by the disaster information acquisition unit 11. This eliminates the need for unnecessary processing, such as detecting characteristics of an area affected by a second disaster, when a first disaster has occurred.

[0073] Next, a process for determining whether an area has been affected by a disaster based on a user input will be described. In this example, the image processing device 10 displays an input image on a display. Next, the image processing device 10 receives an input from the user as to whether an area included in the displayed input image has been affected by a disaster. The image processing device 10 then determines whether the area included in the input image has been affected by a disaster based on the response input by the user.

[0074] The estimation unit 17 estimates the disaster area based on the result of identifying the shooting location of each of the plurality of input images and the result of determining whether or not the area included in each of the plurality of input images has been affected by disaster.

[0075] For example, the estimation unit 17 determines that a predetermined area (e.g., an area within a radius R) centered on the shooting location of the input image determined to be affected by the disaster has been affected. Then, the estimation unit 17 estimates, as the affected area, an area occupied by one or more areas determined to be affected by the disaster.

[0076] An example will be described with reference to FIG. 7. Area B shown in FIG. １ is a predetermined area centered on the location where the first input image was taken, and area B ２ is a predetermined area centered on the location where the second input image was taken, and area B ３is a predetermined area centered on the location where the third input image was taken, and area B ４ is a predetermined area centered on the location where the fourth input image was taken. The first to fourth input images are all input images in which the area included in the image has been determined to be affected by the disaster. The estimation unit 17 estimates area B １ ~B ４ The area occupied by is estimated as the affected area.

[0077] Next, an example of the processing flow of the image processing device 10 will be described using the flowchart in Fig. 8. Note that the purpose here is to explain the processing flow. Since the details of each process have been described above, the description here will be omitted as appropriate.

[0078] First, the image processing device 10 acquires disaster information indicating the type of disaster (S20). Next, the image processing device 10 sets weights for each of the reference objects included in the multiple reference images with location information, based on the type of disaster indicated in the disaster information (S21).

[0079] After that, when the image processing device 10 acquires an input image (S22), it compares the acquired input image with multiple reference images (S23) while taking into consideration the weight of each reference object set in S21. Then, the image processing device 10 identifies the shooting location of the input image acquired in S22 based on the comparison result (S24).

[0080] Furthermore, when the image processing device 10 acquires input images (S22), it determines whether the area included in each of the multiple input images has been affected by disaster (S25).

[0081] The processes of S23 and S24 and the process of S25 may be performed in parallel as shown in FIG. 8, or one may be performed after the other.

[0082] Thereafter, the image processing device 10 estimates the disaster area based on the result of identifying the shooting location of each of the plurality of input images in S24 and the result of determining whether the area included in each of the plurality of input images in S25 is disaster-affected (S26). For example, the image processing device 10 determines that a predetermined area (e.g., an area within a radius R) centered on the shooting location of the input image determined to be disaster-affected is disaster-affected. Then, the image processing device 10 estimates the area occupied by one or more areas determined to be disaster-affected as the disaster-affected area.

[0083] Other configurations of the image processing device 10 of this embodiment are similar to those of the image processing device 10 of the first and second embodiments.

[0084] The image processing device 10 of this embodiment achieves the same effects as the image processing device 10 of the first and second embodiments. Furthermore, the image processing device 10 of this embodiment can estimate a disaster area.

[0085] Even if it is possible to roughly identify the area where a disaster has occurred, it is not easy to specifically identify the area that has actually been affected. The image processing device 10 of this embodiment can accurately estimate the affected area based on multiple input images. Note that, in order to improve estimation accuracy, it is preferable to process a larger number of input images. Therefore, it is preferable for the image processing device 10 of this embodiment to acquire images posted on social media such as SNS as input images. In this way, it becomes possible to acquire and process images taken by a wide variety of people in a wide variety of locations as input images.

[0086] Fourth Embodiment In order to improve the accuracy of identifying the shooting location of an input image, it is preferable to store a large number of reference images taken in various locations in the storage unit 15. However, the greater the number of reference images, the greater the processing load on the computer required to match them with the input image. Therefore, the image processing device 10 of this embodiment has a function of narrowing down the targets to be matched with the input image from among the large number of reference images registered in advance. This will be described in detail below.

[0087] The disaster information acquisition unit 11 acquires disaster information that indicates the disaster occurrence area in addition to the type of disaster.

[0088] The disaster information roughly indicates the disaster-hit area. For example, the disaster-hit area indicated in the disaster information is a predetermined area in relatively large units such as "Kanto region," "Tokyo," or "XX ward of Tokyo." The information is more detailed than the disaster area described in the third embodiment.

[0089] The matching unit 13 narrows down the reference images to be matched with the input image from among the multiple reference images based on the disaster area indicated in the disaster information and the location information of each of the multiple reference images. For example, the matching unit 13 selects reference images whose location indicated by the location information is included in the disaster area indicated in the disaster information as the target for matching with the input image. Note that the matching unit 13 may also select reference images whose location indicated by the location information is included in an area obtained by enlarging or reducing the disaster area indicated in the disaster information using any method as the target for matching with the input image.

[0090] Other configurations of the image processing device 10 of this embodiment are similar to those of the image processing device 10 of the first to third embodiments.

[0091] The image processing device 10 of this embodiment achieves the same effects as the image processing device 10 of the first to third embodiments. Furthermore, in order to accurately identify the shooting location of an input image, it is preferable that a large number of reference images taken in various locations be stored in the storage unit 15. However, the greater the number of reference images, the greater the processing burden on the computer. The image processing device 10 of this embodiment can narrow down the reference images to be compared with the input image based on the disaster-occurring area indicated in the disaster information. As a result, the processing burden on the computer can be reduced while maintaining a high level of accuracy in identifying the shooting location of the input image.

[0092] Fifth Embodiment An image processing device 10 of this embodiment acquires, as input images, images posted and made public on social media such as SNS. Then, from the many input images acquired in this way, it has a function of narrowing down targets to be compared with a reference image for estimating the disaster area (the process described in the third embodiment). This will be described in detail below.

[0093] The disaster information acquisition unit 11 acquires disaster information that indicates the disaster occurrence area in addition to the type of disaster.

[0094] The disaster information roughly indicates the disaster-hit area. For example, the disaster-hit area indicated in the disaster information is a predetermined area in relatively large units such as "Kanto region," "Tokyo," or "XX ward of Tokyo." The information is more detailed than the disaster area described in the third embodiment.

[0095] The input image acquisition unit 14 acquires, as an input image, a posted image that has been posted to a social media such as a social networking service (SNS) and made public. Social media is a medium through which anyone can transmit information via the Internet, and examples of such media include social networking services, video sharing sites, and messaging apps.

[0096] The matching unit 13 narrows down the posted images to be matched with the reference image from among the multiple posted images based on the disaster-hit area indicated in the disaster information and the activity area of ​​the poster of each of the multiple posted images. For example, the matching unit 13 targets posted images posted by posters whose activity area at least partially overlaps the disaster-hit area indicated in the disaster information to be matched with the reference image. Alternatively, the matching unit 13 may further target posted images posted by posters whose activity area does not overlap the disaster-hit area indicated in the disaster information but whose distance (e.g., shortest distance) between the activity area and the disaster-hit area is less than a threshold to be matched with the reference image.

[0097] The activity area of ​​the poster can be estimated using any known technology, such as the technologies disclosed in Patent Document 2 and Non-Patent Documents 1 to 5.

[0098] Other configurations of the image processing device 10 of this embodiment are similar to those of the image processing device 10 of the first to fourth embodiments.

[0099] The image processing device 10 of this embodiment achieves the same effects as the image processing device 10 of the first to fourth embodiments. Furthermore, the image processing device 10 of this embodiment acquires publicly available images posted on social media such as SNS as input images, and estimates the disaster area based on the input images. By processing a larger number of posted images, the disaster area can be estimated more accurately. However, because there are a huge number of posted images, processing all of them places a heavy processing burden on the computer. The image processing device 10 of this embodiment can narrow down the posted images (input images) to be compared with the reference image based on the disaster-affected area indicated in the disaster information and the poster's activity area. As a result, the processing burden on the computer can be reduced while maintaining high accuracy in identifying the shooting location of the input image.

[0100] Sixth Embodiment An image processing device 10 of this embodiment acquires, as input images, images posted and made public on social media such as SNS. The image processing device 10 has a function of narrowing down, from among a large number of input images, targets to be compared with a reference image for estimating a disaster area (the process described in the third embodiment) using a method different from that of the fifth embodiment. This will be described in detail below.

[0101] The disaster information acquisition unit 11 acquires disaster information that indicates the timing of the occurrence of the disaster in addition to the type of disaster. The disaster information may further indicate the area where the disaster occurred.

[0102] The disaster occurrence timing is the timing when a disaster occurs (or is detected), and is indicated by a date and time.

[0103] The input image acquisition unit 14 acquires, as an input image, a posted image that has been posted to a social media such as a social networking service (SNS) and made public. Social media is a medium through which anyone can transmit information via the Internet, and examples of such media include social networking services, video sharing sites, and messaging apps.

[0104] The matching unit 13 narrows down the posted images to be matched with the reference image from among the multiple posted images based on the timing of the disaster occurrence indicated in the disaster information and the posting timing of each of the multiple posted images. For example, the matching unit 13 targets the posted images posted after the timing of the disaster occurrence to be matched with the reference image. Alternatively, the matching unit 13 may target the posted images posted after any timing after the timing of the disaster occurrence to be matched with the reference image.

[0105] The image processing device 10 of this embodiment may further have the "function of narrowing down the target of matching with the reference image from the input image" described in the fifth embodiment. That is, the matching unit 13 may narrow down the target of matching with the reference image from the input image by performing both the narrowing down based on the timing of the disaster occurrence described above and the narrowing down based on the disaster occurrence area described in the fifth embodiment.

[0106] Other configurations of the image processing device 10 of this embodiment are similar to those of the image processing device 10 of the first to fifth embodiments.

[0107] The image processing device 10 of this embodiment achieves the same effects as the image processing devices 10 of the first to fifth embodiments. Furthermore, the image processing device 10 of this embodiment acquires, as input images, images posted and made public on social media such as SNS, and estimates the disaster area based on the input images. By processing a larger number of posted images, the disaster area can be estimated more accurately. However, because there are a huge number of posted images, processing all of them would impose a heavy processing burden on the computer. The image processing device 10 of this embodiment can narrow down the posted images (input images) to be compared with the reference image based on the timing of the disaster occurrence indicated in the disaster information and the posting timing of the posted images. As a result, the processing burden on the computer can be reduced while maintaining high accuracy in identifying the shooting location of the input images.

[0108] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations may be adopted. The configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, various modifications may be made to the configurations of the above-described embodiments without departing from the spirit of the invention. Furthermore, the configurations and processes disclosed in the above-described embodiments and modified examples may be combined with each other.

[0109] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps executed in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content. Furthermore, the above-described embodiments can be combined as long as the content is not contradictory.

[0110] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. 1. An image processing device comprising: disaster information acquisition means for acquiring disaster information indicating a type of disaster; weighting means for setting a weight for each reference object included in a plurality of reference images with position information based on the type of disaster indicated in the disaster information; matching means for matching an input image with the plurality of reference images taking into account the weight of each reference object; and position identification means for identifying a location where the input image was taken based on the matching result. 2. The image processing device described in 1, further comprising: determination means for determining whether an area included in the input image has been damaged by analyzing the input image or by receiving user input; and estimation means for estimating a damaged area based on the result of identifying the shooting location of each of the plurality of input images and the result of determining whether the area included in each of the plurality of input images has been damaged. 3. The image processing device described in 1 or 2, wherein the weighting means sets a greater weight for the reference object as it becomes taller when the type of disaster indicated in the disaster information is a first disaster. 4. 5. The image processing device according to 1 or 2, wherein the weighting means assigns a greater weight to the reference object where the location is at a higher altitude when the type of disaster indicated in the disaster information is a second disaster. 6. The image processing device according to 1 or 2, wherein the weighting means assigns a greater weight to the reference object where the risk of damage is lower when the type of disaster indicated in the disaster information is a third disaster. 7. The image processing device according to 1 or 2, wherein the disaster information further indicates a disaster-occurred area, and the matching means narrows down the reference images to be matched with the input image from among the plurality of reference images based on the disaster-occurred area indicated in the disaster information and the location information of each of the plurality of reference images. 8. The image processing device according to 1 or 2, wherein the disaster information further indicates a disaster-occurred area, and the input image is a posted image posted on social media, and the matching means narrows down the posted images to be matched with the reference image from among the plurality of posted images based on the disaster-occurred area indicated in the disaster information and an activity area of ​​the poster of each of the plurality of posted images.8. The image processing device according to 1 or 2, wherein the disaster information further indicates a timing of the occurrence of the disaster, the input image is a posted image posted on social media, and the matching means narrows down the posted images to be matched with the reference image from among the multiple posted images based on the timing of the disaster occurrence indicated in the disaster information and the posting timing of each of the multiple posted images. 9. An image processing method, wherein one or more computers: acquire disaster information indicating a type of disaster; set a weight for each reference object included in multiple reference images with location information based on the type of disaster indicated in the disaster information; match the input image with the multiple reference images taking into account the weight for each reference object; and identify the shooting location of the input image based on the matching result. A program that causes a computer to function as: disaster information acquisition means that acquires disaster information indicating the type of disaster; weighting means that sets a weight for each reference object included in multiple reference images with location information based on the type of disaster indicated in the disaster information; matching means that matches an input image with multiple reference images taking into account the weight for each reference object; and location identification means that identifies the location where the input image was taken based on the matching result.

[0111] This application claims priority based on Japanese Patent Application No. 2023-040575, filed March 15, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0112] REFERENCE SIGNS LIST 10 Image processing device 11 Disaster information acquisition unit 12 Weighting unit 13 Matching unit 14 Input image acquisition unit 15 Storage unit 16 Determination unit 17 Estimation unit 18 Position identification unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. disaster information acquisition means for acquiring disaster information indicating the type of disaster; a weighting means for setting a weight for each of the reference objects included in the plurality of reference images with position information based on the type of disaster indicated in the disaster information; a matching means for matching an input image with a plurality of the reference images, taking into consideration a weight of each of the reference objects; a location specifying means for specifying a location where the input image was taken based on the result of the comparison; An image processing device having:

2. a determination means for determining whether an area included in the input image is affected by a disaster by analyzing the input image or by receiving a user input; an estimation means for estimating a disaster-stricken area based on a result of identifying a photographing location of each of the plurality of input images and a result of determining whether or not an area included in each of the plurality of input images is damaged by disaster; The image processing device according to claim 1 , further comprising:

3. 3. The image processing device according to claim 1, wherein the weighting means sets a larger weight for the reference object as the height of the reference object increases when the type of disaster indicated by the disaster information is a first disaster.

4. 3. The image processing device according to claim 1, wherein the weighting means sets a larger weight for the reference object located at a higher altitude when the type of disaster indicated by the disaster information is a second disaster.

5. 3. The image processing device according to claim 1, wherein the weighting means sets a larger weight for the reference object having a lower risk of damage when the type of disaster indicated by the disaster information is a third disaster.

6. The disaster information further indicates a disaster occurrence area, 3. The image processing device according to claim 1, wherein the comparison means narrows down the reference images to be compared with the input image from among the plurality of reference images based on the disaster area indicated in the disaster information and the location information of each of the plurality of reference images.

7. The disaster information further indicates a disaster occurrence area, the input image is a posted image posted on a social media site, The image processing device described in claim 1 or 2, wherein the comparison means narrows down the posted images to be compared with the reference image from among the multiple posted images based on the disaster area indicated in the disaster information and the activity area of ​​the poster of each of the multiple posted images.

8. The disaster information further indicates a timing of the occurrence of the disaster, the input image is a posted image posted on a social media site, The image processing device described in claim 1 or 2, wherein the comparison means narrows down the posted images to be compared with the reference image from among the multiple posted images based on the timing of the disaster occurrence indicated in the disaster information and the posting timing of each of the multiple posted images.

9. One or more computers Obtain disaster information indicating the type of disaster, setting a weight for each of the reference objects included in a plurality of reference images with position information based on the type of disaster indicated by the disaster information; matching an input image with a plurality of the reference images, taking into account a weight of each of the reference objects; Identifying the location where the input image was taken based on the matching result. Image processing methods.

10. Computer, disaster information acquisition means for acquiring disaster information indicating the type of disaster; a weighting means for setting a weight for each of reference objects included in a plurality of reference images with position information based on the type of disaster indicated in the disaster information; a matching means for matching an input image with a plurality of the reference images, taking into consideration a weight of each of the reference objects; a location specifying means for specifying a location where the input image was taken based on the result of the comparison; A program that functions as a