Information processing device, information processing method, and program

Abstract

Provided are, for example, an information processing device with which it is possible to effectively search for a photographed location of a search image the photographed location of which is unknown. An information processing device according to the present disclosure comprises: a reception unit that receives a search image the photographed location of which is unknown; and a database that stores a plurality of ground images, each including location information, and a plurality of aerial images, each including location information. Further, the information processing device according to the present disclosure comprises: a collation unit that collates the search image with each of the plurality of ground images and the plurality of aerial images; and a display unit that displays information about one or more candidate locations for the photographed location on the basis of the result of the collation by the collation unit.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] Some images, such as photographs, uploaded to networks such as the Internet have GPS (Global Positioning System) information (location information) embedded in them. By using a specific application, it is possible to determine when and where such images were taken. Meanwhile, many images are uploaded to blogs, social networking sites, and the like, without knowing the location information, such as the location where they were taken.

[0003] Patent Literature 1 discloses a method for searching for an image from a desired location and date and time from images captured by multiple cameras. In this way, by comparing ground images (images of the subject captured from the side) of a subject such as a building with each other, it is possible to determine the location information of an image whose capture location is unknown.

[0004] JP 2016-115082 A

[0005] However, there is a problem that it is not efficient to compare an image with a large number of ground images in order to determine the location information of an image whose shooting location is unknown, and the accuracy of the comparison results is not high. Therefore, there is a demand for efficiently determining the location information of images whose shooting location is unknown. The technology described in Patent Document 1 does not anticipate such a problem, and does not disclose any method for solving this problem.

[0006] The present disclosure has been made to solve such problems, and its purpose is to provide an information processing device, an information processing method, and a program that can effectively search for the shooting location of a search image whose shooting location is unknown.

[0007] An information processing device according to one aspect of the present disclosure includes: a reception unit that receives a search image whose shooting location is unknown; a database that stores a plurality of ground images, each of which includes location information, and a plurality of aerial images, each of which includes location information; a matching unit that matches the search image with each of the plurality of ground images and the plurality of aerial images; and a display unit that displays information regarding one or more candidate locations for the shooting location based on the matching result by the matching unit.

[0008] An information processing method according to one aspect of the present disclosure includes the steps of: accepting a search image whose shooting location is unknown; storing a plurality of ground images, each of which includes location information, and a plurality of aerial images, each of which includes location information; comparing the search image with each of the plurality of ground images and the plurality of aerial images; and displaying information regarding one or more candidate locations for the shooting location based on the results of the comparison.

[0009] A program in one aspect of the present disclosure causes a computer to perform the following processes: accepting a search image whose shooting location is unknown; storing a plurality of ground images, each of which includes location information, and a plurality of aerial images, each of which includes location information; matching the search image with each of the plurality of ground images and the plurality of aerial images; and displaying information regarding one or more candidate locations for the shooting location based on the matching results of the matching process.

[0010] According to the present disclosure, it is possible to provide an information processing device, an information processing method, and a program that can effectively search for the shooting location of a search image whose shooting location is unknown.

[0011] 1. A block diagram showing an example of a configuration of an information processing device according to the present disclosure. 1. A diagram conceptually showing a method of matching a search image with a ground image and an aerial image. 2. A diagram showing an example of a matching result obtained by the matching method shown in FIG. 2. 3. A flowchart showing an example of an image search process executed by the information processing device shown in FIG. 1. 4. A flowchart showing an example of an image registration process executed by the information processing device shown in FIG. 1. 5. A block diagram showing an example of a configuration of an information processing device according to the present disclosure. 6. A diagram conceptually showing a first matching between a search image and an aerial image. 7. A diagram conceptually showing a second matching between a search image and a candidate ground image. 8. A flowchart showing an example of an image search process executed by the information processing device shown in FIG.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the present embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are assigned the same reference numerals, and duplicate explanations are omitted as necessary.

[0013] Each drawing referenced in the embodiments is merely an example for describing one or more embodiments. Each drawing is not related to only one particular embodiment, but may also be related to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0014] <Background to the Disclosure> Photographs taken by users are posted or published on social media and blogs, but in many cases, location information is not attached to such images to prevent the location where the photograph was taken from being identified. Here, in methods of utilizing information on social media, blogs, and the Internet, such as the following methods, it is important to identify the user's base of activity to some extent.

[0015] The first possible use is for local governments to use images uploaded to the Internet showing the damage caused by natural disasters such as typhoons and earthquakes, or man-made disasters such as fires in factories and buildings. By knowing where the images of damaged buildings and structures were taken, local governments can more accurately grasp the extent of the damage caused by the disaster and, if necessary, provide timely evacuation guidance to victims.

[0016] The second application method is to identify the user's range of activity from the posted images, which can be used for retail marketing or to support the development of targeted advertising. In this case, too, it is expected that the range of a user's activities can be understood from multiple images posted by a user with certain characteristics, making it possible to narrow down the target marketing based on those characteristics.

[0017] The third use case is to identify the range of a person's movements from images that show that person, thereby supporting criminal investigations and the search for children, etc. For example, providing such information to law enforcement agencies in various countries could lead to the early resolution of crimes and searches.

[0018] In order to fully ensure these utilization methods, it is necessary to efficiently search for the location where a search image was taken, even if the location is unknown, by using not only ground images but also aerial images, as shown in the following embodiment.

[0019] In this specification, the terms "ground image" and "aerial image" are used to refer to the following images: A ground image is an image such as a photograph taken from the ground in a city, such as an urban area, or from inside or on the roof of a building in the city. In other words, a ground image is a landscape image taken near the ground from an upward or generally lateral direction. Thus, a ground image is an image including a ground view of a certain location. For example, a ground image is taken from a pedestrian or from inside a vehicle. On the other hand, an aerial image is an image such as a photograph taken from an artificial satellite or an aircraft located in the sky. In other words, an aerial image is a landscape image taken from a generally straight down direction from the sky (also called an aerial image), and includes satellite images and aerial images. Thus, an aerial image is taken from a drone, an airplane, an artificial satellite, or the like. Note that each aerial image is an image of a square section with a predetermined side length, identified by the longitude and latitude of the center point, and one or more candidate locations for the ground image are displayed within the aerial image.

[0020] (First Embodiment) <Configuration of Information Processing Device> The configuration of an information processing device according to this embodiment will be described. Note that this information processing device is associated with matching of images such as photographs, but the installation location of the information processing device may be on the cloud, at a user's location, or the like, as long as it can perform the processing described below.

[0021] 1 is a block diagram showing an example of the configuration of an information processing device according to the present disclosure. As shown in FIG. 1, the information processing device 1 of this example includes a reception unit 10, a database 20, a comparison unit 30, a display unit 40, and a registration unit 50. The database 20 also includes a ground image database 21 and an aerial image database 22.

[0022] The reception unit 10 is configured to receive a search image (an image to be searched) whose shooting location is unknown from a user of the information processing device 1. A search image is an image for which the user wants to search for the location where it was taken. In this example, the user specifies a ground image whose shooting location is desired to be known, and the reception unit 10 receives the ground image as the search image.

[0023] The ground image database 21 stores a plurality of ground images, each including location information. The aerial image database 22 stores a plurality of aerial images, each including location information. Here, location information is, for example, information indicating GPS information or longitude and latitude. Each ground image or aerial image may be assigned GPS information corresponding to the image, or the longitude and latitude, etc. may be assigned in text format. As will be described later, by matching each of the plurality of ground images and aerial images with the search image, location information of the location where the search image was taken can be obtained according to the matching result.

[0024] The administrator (or user) of the information processing device 1 may purchase image data of aerial images or ground images from, for example, the Geospatial Information Authority of Japan or a map vendor, or may prepare the image data himself / herself, and register the images in each of the databases 21 and 22 via a registration unit 50, which will be described later. As another registration method, registration of images by a user using the registration unit 50 will be described later.

[0025] The matching unit 30 is configured to match the search image received by the receiving unit 10 with each of a plurality of ground images stored in the ground image database 21 and a plurality of aerial images stored in the aerial image database 22. In this example, the matching unit 30 is characterized by matching the search image with both the ground image and the aerial image. Note that, when matching the search image with the ground image, the degree of matching between the subjects captured in each image is calculated. Furthermore, when matching the search image with the aerial image, the degree of matching between the subjects captured in each image is calculated.

[0026] In this case, the matching unit 30 may sequentially match the search image against all images stored in the ground image database 21 and the aerial image database 22. Furthermore, the matching unit 30 may, for example, use an area displayed on the display unit 40 (described later) as a search range and match the search image against ground images and aerial images included within the search range. Alternatively, the matching unit 30 may limit the images to be matched based on the user's category setting or the contents of the text description. Furthermore, for example, when multiple users form a group, the matching unit 30 may limit the images to be matched based on the attributes of the group.

[0027] Figure 2 is a conceptual diagram showing a method for matching a search image with a ground image and an aerial image. For example, as shown in Figure 2, the search image shows two buildings on the left side of the image with trees growing in front of the buildings. It also shows that a low building stands on the right side of the image. Furthermore, it shows that a pedestrian crossing is painted in the lower center of the image.

[0028] Based on these features, the matching unit 30 performs matching with each of a plurality of ground images as shown in the upper right of Fig. 2 and a plurality of aerial images as shown in the lower right of Fig. 2. As a matching result, the matching unit 30 is configured to calculate a matching score for the target ground image or aerial image against the search image.

[0029] The matching unit 30 is configured to sort the matching results of the plurality of ground images and the plurality of aerial images based on the matching scores, and then outputs the sorted matching results to the display unit 40.

[0030] If the receiving unit 10 is configured to receive text data together with the search image, the matching unit 30 may match the received text data with the ground image (which may include not only the image but also the text data information) stored in the ground image database 21. By using the text data, it is possible to shorten the time required to obtain the matching result.

[0031] The display unit 40 is configured to display information about one or more candidate locations for the shooting location based on the collation result by the collation unit 30. The information about the one or more candidate locations includes at least one of the location information (GPS information) of each of the one or more candidate locations, an image taken at the candidate location, or a point on an aerial image that includes the candidate location.

[0032] In this example, the display unit 40 is configured to display the matching results output from the matching unit 30. The display unit 40 may display a predetermined number of ground images or aerial images (i.e., up to a predetermined rank) in order of matching score. Alternatively, the matching unit 30 may determine whether the matching score is equal to or greater than a predetermined threshold, and cut off the matching results, and the display unit 40 may display only the matching results whose matching scores are equal to or greater than the predetermined threshold among the sorted matching results.

[0033] Fig. 3 is a diagram showing an example of a matching result obtained by the matching method shown in Fig. 2. In this example, a search image is displayed in the upper left, and three ground images are displayed below it in order of matching score. In the example shown in Fig. 3, the ground image in the upper left of the three ground images obtained as a matching result has been selected by the user, and an aerial image including the location where the ground image was taken is displayed in the aerial image on the right.

[0034] The matching result in Figure 3 may be configured such that the user first selects the aerial image shown on the right from multiple aerial images sorted by matching score, and one or more candidate locations for the shooting location included in the selected aerial image are listed on the left.

[0035] The receiving unit 10 may also receive user information of the user performing the search along with the search image. In this case, the matching unit 30 may perform matching of the search image taking into consideration the user information, and the display unit 40 may be configured to not display some or all of the images taken at each of one or more candidate locations based on the received user information. For example, a user belonging to a certain group may set the ground image registered by the user using the registration unit 50 (described later) to be displayed only to users in the group, and set it so that only the location information is displayed to other users.

[0036] The registration unit 50 is configured to register, in response to a user's operation, location information of a predetermined location and an image taken at the predetermined location in association with each other. In this way, the information processing device 1 is configured to allow a user to register an image taken by himself or herself along with the location information. The registration unit 50 may also be configured to register, in association with each other, location information of a predetermined location and an image taken at the predetermined location based on an image uploaded to a network. In this way, not only can the administrator of the information processing device 1 purchase and register ground images and aerial images to store in the database 20, but new ground images and aerial images may also be registered during actual operation of the information processing device 1.

[0037] For example, multiple users may be registered in the information processing device 1, and the registration unit 50 may accept registration of images including location information from each of these multiple users. Furthermore, the registration unit 50 may accept registration of multiple images taken at the same location and in the same shooting direction. In this case, it is sufficient that at least one of the multiple images is linked to (includes) location information.

[0038] Furthermore, when registering an image including location information in response to a user's operation, the registration unit 50 may register detailed image information from the user received via the reception unit 10, as well as image classification, viewing authority, search authority, and the like set by the information processing device 1. Examples of detailed image information include building names such as apartment complex names and supermarket names, and road names. Examples of image classification include building exteriors and interiors, trees, and utility poles. Viewing authority and search authority indicate whether each image can be viewed or searched, and may be set on a user-by-user basis. For example, the information processing device 1 may set an image registered by a user (person in charge) of a certain local government, such as a prefecture or city, town, or village, so that users of other local governments have search authority but not viewing authority. In this case, the display unit 40 displays only the location information of the shooting location as a comparison result.

[0039] <Operation of Information Processing Device> Next, the operation of the information processing device 1 according to this embodiment will be described. Here, an image search process by a user of the information processing device 1 will be described. Fig. 4 is a flowchart showing an example of the image search process executed by the information processing device 1 shown in Fig. 1. This image search process is executed as appropriate in response to an operation by the user of the information processing device 1.

[0040] In the image search process, the receiving unit 10 receives a search image whose shooting location is unknown from a user (step S11). The matching unit 30 then matches the received search image with each of a plurality of ground images stored in the ground image database 21 and a plurality of aerial images stored in the aerial image database 22 (step S12).

[0041] At this time, the matching unit 30 calculates the matching scores of each ground image and each aerial image against the search image (step S13). After completing the matching for all images, the matching unit 30 sorts the matching results according to the matching scores (step S14) and outputs the sorted matching results to the display unit 40.

[0042] The display unit 40 displays the sorted matching results as a list of search results (step S15), and the information processing device 1 ends the image search process. As described above, the display unit 40 may display a predetermined number of ground images or aerial images, or may display one or more ground images or aerial images whose matching scores are equal to or greater than a predetermined threshold.

[0043] Next, a description will be given of an image registration process performed by a user of the information processing device 1. Fig. 5 is a flowchart showing an example of the image registration process executed by the information processing device 1 shown in Fig. 1. This image registration process is executed as appropriate in response to an operation by a user who wishes to register an image in the information processing device 1, for example.

[0044] In the image registration process, the accepting unit 10 acquires an image including location information such as GPS information (step S21). For example, the accepting unit 10 acquires the image by accepting an image including location information in response to a user operation.

[0045] At this time, the receiving unit 10 may acquire text information related to the image in response to a user operation (step S22). The text information may be detailed information about the image, such as the type and name of the building, the name of the road, and so on.

[0046] Then, the registration unit 50 registers the image in association with the position information and text information (step S23), and the information processing device 1 ends this image registration process.

[0047] As described above, the information processing device 1 according to the present embodiment is configured to include a reception unit 10 that receives a search image whose shooting location is unknown, a ground image database 21 that stores a plurality of ground images, each of which includes location information, and an aerial image database 22 that stores a plurality of aerial images, each of which includes location information, a matching unit 30 that matches the search image with each of the plurality of ground images and the plurality of aerial images, and a display unit 40 that displays information regarding one or more candidate locations based on the matching results by the matching unit 30. By configuring the information processing device 1 according to the present embodiment in this manner, it is possible to efficiently search for the shooting location of an image whose shooting location is unknown. Therefore, according to the information processing device 1 according to the present embodiment, such image matching can be used for disaster relief, marketing support, investigation support, etc.

[0048] The information processing device 1 according to the present embodiment may further include a registration unit that registers location information of a predetermined location in association with an image captured at the predetermined location in response to a user operation or based on an image uploaded to the network. By configuring the information processing device 1 according to the present embodiment in this manner, it is possible to prepare many ground images and aerial images for a search image.

[0049] The information processing method according to the present embodiment is configured to include the steps of: receiving a search image whose shooting location is unknown; storing a plurality of ground images each including location information and a plurality of aerial images each including location information; comparing the search image with each of the plurality of ground images and the plurality of aerial images; and displaying information about one or more candidate locations for the shooting location based on the comparison result in the comparison step. By configuring the information processing method according to the present embodiment in this way, it is possible to obtain the same effects as those of the information processing device 1.

[0050] (Embodiment 2) <Configuration of Information Processing Device> The configuration of an information processing device according to embodiment 2 will be described. FIG. 6 is a block diagram showing an example of the configuration of an information processing device according to the present disclosure. As shown in FIG. 6, the information processing device 2 according to this embodiment includes a reception unit 10, a database 20, a matching unit 30′, and a display unit 40. The database 20 includes a ground image database 21 and an aerial image database 22. The matching unit 30′ includes a first matching unit 31 and a second matching unit 32. Note that the configurations of the reception unit 10, the database 20, and the display unit 40 in the information processing device 2 have the same functions as the corresponding components of the information processing device 1 according to embodiment 1, and therefore detailed descriptions thereof will be omitted as appropriate, except for functions unique to this embodiment.

[0051] In the information processing device 2 of this example, in order to efficiently perform searches over a wide area, the matching unit 30' includes a first matching unit 31 and a second matching unit 32, and is characterized in that it performs matching with the search image in two stages.

[0052] The first matching unit 31 is configured to match the search image received by the receiving unit 10 with each of the multiple aerial images stored in the aerial image database 22. Here, similar to the information processing device 1 according to the first embodiment, each aerial image stored in the aerial image database 22 is an image of a square section with a side of a predetermined length specified by the longitude and latitude of the center point.

[0053] FIG. 7 is a conceptual diagram illustrating the first matching between a search image and an aerial image. The first matching unit 31 matches the search image (left side of FIG. 7 ) with each aerial image in a square section (right side of FIG. 7 ), calculates a matching score for each aerial image, and sorts the matching results of the aerial images based on the matching score. At this time, the first matching unit 31 prunes a predetermined number of aerial images starting from the aerial images with the highest matching scores. Alternatively, the first matching unit 31 selects one or more aerial images with a matching score equal to or greater than a predetermined threshold. In this way, matching by the first matching unit 31 selects aerial images with a high matching score (high matching rate). Here, it is assumed that the first matching unit 31 selects, for example, five aerial images, each of which includes longitude and latitude information of a center point.

[0054] The second matching unit 32 is configured to match the search image received by the receiving unit 10 with each of the plurality of ground images stored in the ground image database 21 based on the matching result of the first matching unit 31. The second matching unit 32 is configured, for example, to match the search image with each of one or more ground images included in each of one or more aerial images for which the matching score by the first matching unit 31 is equal to or greater than a predetermined threshold. Furthermore, the second matching unit 32 may be configured, for example, to match the search image with each of one or more ground images included in each of a predetermined number of aerial images for which the matching score by the first matching unit 31 is among the highest.

[0055] FIG. 8 is a diagram conceptually showing the second matching between the search image (left side of FIG. 8) and the ground image (left side of FIG. 8) that has become a candidate through matching by the first matching unit 31.

[0056] Specifically, the second matching unit 32 is configured to extract a ground image included in each of the one or more aerial images that are the matching results of the first matching unit 31, and match each of the extracted ground images with the search image. In this way, by performing a two-stage matching, i.e., matching with aerial images that have a fast search speed to narrow down the range, and then matching with ground images included in a predetermined number of top aerial images, it is possible to more efficiently search for ground images whose shooting locations are unknown.

[0057] Here, the first collation unit 31 may include information on the shooting direction for each aerial image as a collation result. In this case, the second collation unit 32 may collate the aerial image with the ground image, taking into consideration the information on the shooting direction as a collation result of the first collation unit 31.

[0058] The display unit 40 may display a predetermined number of aerial images including points that are candidate shooting locations based on the collation result of the first collation unit 31. This allows the user to check candidate shooting locations based on the aerial images. Furthermore, the user can also eliminate candidates that are not considered to be shooting locations, as necessary.

[0059] <Operation of Information Processing Device> Next, the operation of the information processing device 2 according to this embodiment will be described. Fig. 9 is a flowchart showing an example of image search processing executed by the information processing device 2 shown in Fig. 6. This image search processing is executed as appropriate in response to an operation by a user of the information processing device 2.

[0060] In the image search process, the reception unit 10 receives a search image whose shooting location is unknown from a user (step S31). Then, the first comparison unit 31 compares the received search image with each of a plurality of aerial images stored in the aerial image database 22 (step S32).

[0061] At this time, the first matching unit 31 calculates a matching score for each aerial image against the search image (step S33). Then, the first matching unit 31 selects a predetermined number of aerial images with the highest matching scores and prunes the aerial images below them.

[0062] Next, the second matching unit 32 extracts ground images that are included in a predetermined number of aerial images with the highest matching scores (step S34), and matches the search image with the extracted ground images (step S35). At this time, the second matching unit 32 calculates the matching score of each ground image against the search image (step S36).

[0063] Next, the second matching unit 32 sorts the matching results of the ground images according to the matching scores (step S37), and outputs the sorted matching results to the display unit 40. The display unit 40 displays the sorted matching results as a list of search results (step S38), and the information processing device 2 ends this image search process.

[0064] As described above, the information processing device 2 according to this embodiment includes a matching unit 30′ including a first matching unit 31 and a second matching unit 32. The first matching unit 31 is configured to match the search image received by the reception unit 10 with each of the plurality of aerial images stored in the aerial image database 22, and the second matching unit 32 is configured to match the search image received by the reception unit 10 with each of the plurality of ground images stored in the ground image database 21 based on the matching result of the first matching unit 31. By configuring the information processing device 2 according to this embodiment in this way, in addition to the effects of the information processing device 1 according to the first embodiment, it is possible to more efficiently search for ground images whose shooting locations are unknown.

[0065] (Third Embodiment) <Configuration of Information Processing Device> The configuration of an information processing device according to the third embodiment will be described. FIG. 10 is a block diagram showing an example of the configuration of an information processing device according to the present disclosure. As shown in FIG. 10, the information processing device 3 according to the present embodiment includes a reception unit 11, a database 20′, a matching unit 30, a display unit 40, and a narrowing-down unit 60. The database 20′ includes a ground image database 21, an aerial image database 22, and a map database 23. Note that the configurations of the matching unit 30 and the display unit 40 in the information processing device 3 have the same functions as the corresponding components of the information processing device 1 according to the first embodiment, and therefore detailed descriptions thereof will be omitted as appropriate, except for functions unique to this embodiment.

[0066] The reception unit 11 is configured to receive, from the user of the information processing device 3, a search image (an image to be searched) whose shooting location is unknown, as well as terrain information as a search target. Here, terrain information refers to terrain such as a coastline or a lakeside. For example, if the user can read the terrain information from the content of a post on a social media site or blog where the search image whose shooting location is unknown is posted, the user can simply input the name of the terrain that will become the terrain information to the reception unit 11.

[0067] The receiving unit 11 may be configured to further receive information about the country or region to be searched. The information processing device 3 may acquire map data that is publicly available, for example, at a URL of the Geospatial Information Authority of Japan, without going through the map database 23.

[0068] The map database 23 of the database 20′ is configured to store map data of a country or region (Japan or the Kanto region in this example) that is the search target of the user of the information processing device 3. When the information processing device 3 acquires map data that has been made public on a network, the map database 23 only needs to store the acquired map data.

[0069] The narrowing-down unit 60 is configured to, when the receiving unit 11 receives the topographical information, narrow down the map data stored in advance in the map database 23 to one or more sections that match the received topographical information before the matching unit 30 performs matching. Here, each of the one or more sections on the map data may correspond to one aerial image. Therefore, the multiple sections narrowed down by the narrowing-down unit 60 include multiple square aerial images with sides of a predetermined length.

[0070] The reception unit 11 may further receive a region to be narrowed down. In this case, the narrowing down unit 60 may use map data of the region to be narrowed down as a target for narrowing down. When the reception unit 11 receives the region to be narrowed down, the information processing device 3 may acquire map data of the region to be narrowed down via a network such as the Internet.

[0071] 11 is a diagram showing an example of map data in which the sections to be searched have been narrowed down by the narrowing-down unit 60. For example, when the receiving unit 11 receives the topographical information "lakeside" and the narrowing-down target area "northern Kanto," the narrowing-down unit 60 is configured to narrow down the map data of northern Kanto, which is the search target, to one or more sections that match lakeside, as shown in FIG.

[0072] In this example, the matching unit 30 is configured to match the search image with the ground images and aerial images contained within one or more areas narrowed down as described above, from among the plurality of ground images stored in the ground image database 21 and the plurality of aerial images stored in the aerial image database 22.

[0073] <Operation of Information Processing Device> Next, the operation of the information processing device 3 according to this embodiment will be described. Fig. 12 is a flowchart showing an example of image search processing executed by the information processing device 3 shown in Fig. 10. This image search processing is executed as appropriate in response to an operation by a user of the information processing device 3.

[0074] In the image search process, the reception unit 11 receives from the user a search image whose shooting location is unknown and terrain information of the search target (step S41). Then, the narrowing-down unit 60 narrows down the search target area to one or more sections that match the terrain information from the map data stored in the map database 23 (step S42). Note that when the reception unit 11 receives the area to be narrowed down, the narrowing-down unit 60 acquires map data of the area to be narrowed down from the map data stored in the map database 23, and narrows down the search target area to one or more sections that match the terrain information from the acquired map data.

[0075] Next, the matching unit 30 matches the received search image with one or more ground images and one or more aerial images included in one or more sections narrowed down by the narrowing unit 60 (step S43).

[0076] At this time, the matching unit 30 calculates the matching score of each ground image and each aerial image against the search image (step S44). After the matching unit 30 has finished matching all the images, it sorts the matching results according to the matching score (step S45) and outputs the sorted matching results to the display unit 40.

[0077] The display unit 40 displays the sorted matching results as a list of search results (step S46), and the information processing device 3 ends this image search process. As described in the first embodiment, the display unit 40 may display a predetermined number of ground images or aerial images, or may display one or more ground images or aerial images whose matching scores are equal to or greater than a predetermined threshold.

[0078] Note that, as in the second embodiment, the information processing device 3 may be provided with a matching unit 30' including a first matching unit 31 and a second matching unit 32, instead of the matching unit 30, and may first match the search image with the aerial image. In this case as well, the second matching unit 32 may extract ground images included in a predetermined number of aerial images having the highest matching scores, and match each of the extracted ground images with the search image.

[0079] As described above, the information processing device 3 according to this embodiment includes a receiving unit 11 that receives topographical information as a search target along with a search image; a database 20' including a map database 23 that stores map data of the country or region to be searched; and a narrowing-down unit 60 that narrows down the map data of the country or region to be searched to one or more sections that match the topographical information. The matching unit 30 is configured to match the search image with the narrowed-down one or more ground images and one or more aerial images. By configuring the information processing device 3 according to this embodiment in this manner, in addition to the effects of the information processing device 1 according to the first embodiment and the information processing device 2 according to the second embodiment, narrowing down the topographical information as a search target allows for faster search of ground images whose shooting locations are unknown. Furthermore, the search processing load on the information processing device 3 can be reduced.

[0080] (Fourth Embodiment) <Configuration of Information Processing Device> The configuration of an information processing device according to the fourth embodiment will be described. Fig. 13 is a block diagram showing an example of the configuration of an information processing device according to the present disclosure. As shown in Fig. 13, an information processing device 4 according to this embodiment includes a reception unit 10, a database 20, a matching unit 33, a display unit 40, and a distance estimation unit 70. The database 20 also includes a ground image database 21 and an aerial image database 22.

[0081] The matching unit 33 has the same function as the matching unit 30 of the information processing device 1 according to the first embodiment. That is, the matching unit 33 is configured to match the search image received by the receiving unit 10 with each of the plurality of ground images stored in the ground image database 21 and the plurality of aerial images stored in the aerial image database 22.

[0082] The matching unit 33 is also configured to calculate a matching score for the target ground image (or aerial image) against the search image. The matching unit 33 is configured to sort the matching results of the multiple ground images (and multiple aerial images) based on the matching score.

[0083] In addition to the above functions, the matching unit 33 is configured to identify a predetermined ground image with the highest matching score based on the matching score, and identify a subject that appears in the predetermined ground image. The highest matching score is considered to indicate that the subject that appears in the predetermined ground image also appears in the search image. This allows the matching unit 33 to acquire location information of the identified subject. Note that possible subjects include buildings with distinctive exteriors such as department stores, factories such as waste disposal plants and their chimneys, and radio towers such as Tokyo Tower and Skytree.

[0084] The distance estimation unit 70 is configured to estimate the distance between the candidate position of the shooting location and the subject based on information about the subject appearing in the search image, the shooting location of a predetermined ground image, and information about the subject appearing in the predetermined ground image. Here, the information about the subject includes the name of the subject, the size of the subject, the shooting direction or angle, etc. The distance estimation unit 70 may estimate the distance between the candidate position of the shooting location and the subject based on, for example, the size of the subject appearing in the search image and the size of the subject appearing in the predetermined ground image. Note that the distance estimation by the distance estimation unit 70 may be performed using a well-known conventional method.

[0085] The display unit 40 is configured to display the position of the subject and a circle with a radius of the estimated distance centered on the subject position for an aerial image including the shooting position of a predetermined ground image. Fig. 14 is a diagram showing an example of an aerial image displayed on the display unit 40 of the information processing device 4 shown in Fig. 13. For example, the display unit 40 displays a point of the subject and a circle with an estimated distance R (m) from the subject on the aerial image. In this case, the point displayed on the circle is a candidate position for the shooting location.

[0086] Although not shown in the figure, the display unit 40 may display not only points on a circle as candidate locations for the shooting location, but also points included within concentric circles of radii R-A (m) and R+A (m), for example, using a length A (m) that is sufficiently smaller than the radius R (m).

[0087] <Operation of Information Processing Device> Next, the operation of the information processing device 4 according to this embodiment will be described. Fig. 15 is a flowchart showing an example of image search processing executed by the information processing device 4 shown in Fig. 13. This image search processing is executed as appropriate in response to an operation by a user of the information processing device 4.

[0088] In the image search process, the receiving unit 10 receives a search image from a user whose shooting location is unknown (step S51). The matching unit 33 then matches the received search image with each of the plurality of ground images stored in the ground image database 21 (step S52). In this example, for the purpose of distance estimation in the subsequent stage, the matching unit 33 may omit matching with the plurality of aerial images stored in the aerial image database 22. However, as in the information processing device 1 according to the first embodiment, the matching unit 33 may also match with a plurality of ground images and a plurality of aerial images.

[0089] At this time, the matching unit 33 calculates the matching score of each ground image against the search image (step S53). Then, after the matching unit 33 has finished matching all the images, it sorts the matching results according to the matching score (step S54).

[0090] Next, the matching unit 33 identifies a predetermined ground image with the highest matching score based on the sorted matching results (step S55), and further identifies a subject appearing in the predetermined ground image (step S56). The matching unit 33 outputs information about the identified subject to the display unit 40.

[0091] Next, the distance estimation unit 70 estimates the distance between the subject and the candidate location of the photographing location based on information about the subject appearing in the search image, the photographing location of the predetermined ground image, and information about the subject appearing in the predetermined ground image (step S57). The distance estimation unit 70 outputs this estimated distance information to the display unit 40.

[0092] The display unit 40 displays an aerial image including the subject based on the subject information. Then, for the aerial image including the subject and the shooting position of the predetermined ground image, the display unit 40 displays the position of the subject (a point indicating the subject) and a circle with a radius equal to the estimated distance R (m) centered on the subject position (step S58). With the above processing, the information processing device 4 ends this image retrieval processing.

[0093] As described above, the information processing device 4 according to the present embodiment is configured to include a matching unit 33 that calculates matching scores for each of a plurality of ground images and a plurality of aerial images for a search image, identifies a predetermined ground image with the highest matching score based on the matching scores, and identifies a subject appearing in the predetermined ground image, a distance estimation unit 70 that estimates the distance between a candidate position of the shooting location and the subject based on information about the subject appearing in the search image, the shooting location of the predetermined ground image, and information about the subject appearing in the predetermined ground image, and a display unit 40 that displays the position of the subject and a circle with a radius estimated from the position of the subject and a radius of the estimated distance for an aerial image including the shooting location of the predetermined ground image. By configuring the information processing device 4 according to the present embodiment in this way, it is possible to display a circle (or concentric circles) estimated as the shooting location, and to identify a candidate position of the shooting location on the circle.

[0094] In the above-described embodiments, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited to this. The present disclosure can also be realized by having a processor in a computer execute a computer program to perform the processes of the information processing devices 1 to 4 described in the above-described embodiments.

[0095] Finally, the hardware configuration of the information processing devices 1 to 4 will be described. FIG. 16 is a block diagram showing an example of the hardware configuration of the information processing devices 1 to 4 of the present disclosure. As shown in FIG. 16, the information processing devices 1 to 4 include a network interface 100, a processor 200, and a memory 300. The network interface 100 may be used to communicate with not only the above-mentioned networks but also network nodes. The network interface 100 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series. IEEE stands for Institute of Electrical and Electronics Engineers.

[0096] The processor 200 reads and executes software (computer programs) from the memory 300 to perform various processes of the information processing devices 1 to 4 described above. The processor 200 may be, for example, a microprocessor, an MPU (Micro-Processing Unit), or a CPU (Central Processing Unit). The processor 200 may include multiple processors.

[0097] The memory 300 is configured by a combination of volatile memory and non-volatile memory. The memory 300 may include storage located remotely from the processor 200. In this case, the processor 200 may access the memory 300 via an I / O (Input / Output) interface (not shown).

[0098] 16, the memory 300 is used to store a group of software modules. The processor 200 reads and executes these software modules from the memory 300, thereby performing various processes of the information processing devices 1 to 4 described in the above embodiments.

[0099] As explained using Figure 16, each of the processors possessed by the information processing devices 1 to 4 in the above-mentioned embodiments executes one or more programs including a group of instructions for causing a computer to perform the algorithm explained using the drawings.

[0100] Some or all of the processes performed by the information processing devices 1 to 4 described above can be implemented as a computer program. Such programs can be stored on various types of non-transitory computer-readable media and supplied to a computer. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The programs may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0101] Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified as appropriate within the scope of the invention.

[0102] Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to the following. (Supplementary Note 1) An information processing device comprising: a reception unit that receives a search image whose shooting location is unknown; a database that stores a plurality of ground images, each including location information, and a plurality of aerial images, each including location information; a matching unit that matches the search image with each of the plurality of ground images and the plurality of aerial images; and a display unit that displays information about one or more candidate locations for the shooting location based on the matching result by the matching unit. (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the information about the one or more candidate locations includes at least one of location information for each of the one or more candidate locations, an image taken at the candidate location, or a point on the aerial image that includes the candidate location. (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the reception unit receives user information of a user performing a search along with the search image, and the display unit does not display images taken at each of the one or more candidate locations based on the user information. (Supplementary Note 4) The information processing device according to Supplementary Note 1, wherein the matching unit calculates a matching score for each of the plurality of ground images and the plurality of aerial images with respect to the search image and sorts the matching results based on the matching score, and the display unit displays, from the sorted matching results, a matching result whose matching score is equal to or greater than a predetermined threshold. (Supplementary Note 5) The information processing device according to Supplementary Note 1, wherein the database includes a ground image database that stores the plurality of ground images and an aerial image database that stores the plurality of aerial images, and the matching unit includes: a first matching unit that matches the search image with each of the plurality of aerial images stored in the aerial image database, and a second matching unit that matches the search image with each of the plurality of ground images stored in the ground image database based on the matching result of the first matching unit.(Supplementary Note 6) The information processing device of Supplementary Note 5, wherein each of the plurality of aerial images is an image of a square section with one side of a predetermined length specified by the longitude and latitude of a center point, the first matching unit calculates a matching score of each of the plurality of aerial images against the search image, and the second matching unit matches the search image with each of one or more ground images included in each of the one or more aerial images for which the matching score by the first matching unit is equal to or greater than a predetermined threshold. (Supplementary Note 7) The information processing device of Supplementary Note 5, wherein each of the plurality of aerial images is an image of a square section with one side of a predetermined length specified by the longitude and latitude of a center point, the first matching unit calculates a matching score of each of the plurality of aerial images against the search image, and the second matching unit matches the search image with each of one or more ground images included in each of a top predetermined number of aerial images for which the matching score by the first matching unit is equal to or greater than a predetermined threshold. (Supplementary Note 8) The information processing device according to any one of Supplementary Notes 1 to 7, further comprising a registration unit that registers location information of a predetermined location in association with an image taken at the predetermined location in response to a user operation or based on an image uploaded to a network. (Supplementary Note 9) The information processing device according to any one of Supplementary Notes 1 to 7, wherein the reception unit receives topographical information as a search target along with the search image, the database includes a ground image database that stores the plurality of ground images, an aerial image database that stores the plurality of aerial images, and a map database that stores map data of a country or region that is the search target, the information processing device further comprises a narrowing-down unit that narrows down the map data of the country or region that is the search target to one or more sections that match the topographical information, and the matching unit matches the search image with each of the plurality of ground images and the plurality of aerial images within the one or more narrowed-down sections.(Supplementary Note 10) The information processing device described in any one of Supplementary Notes 1 to 7, wherein the matching unit calculates a matching score for each of the plurality of ground images against the search image, identifies a predetermined ground image with the highest matching score based on the matching score, and identifies the subject that appears in the predetermined ground image; the information processing device further includes a distance estimation unit that estimates the distance between a candidate position of the shooting location and the subject based on information about the subject that appears in the search image, the shooting location of the predetermined ground image, and information about the subject that appears in the predetermined ground image; and the display unit displays the position of the subject and a circle with the estimated distance as its radius, centered on the position of the subject, for an aerial image including the shooting location of the predetermined ground image. (Supplementary Note 11) An information processing method comprising the steps of: accepting a search image whose shooting location is unknown; storing a plurality of ground images, each including location information, and a plurality of aerial images, each including location information; matching the search image with each of the plurality of ground images and the plurality of aerial images; and displaying information about one or more candidate locations for the shooting location based on the matching result of the matching step. (Supplementary Note 12) A program that causes a computer to execute the steps of: accepting a search image whose shooting location is unknown; storing a plurality of ground images, each including location information, and a plurality of aerial images, each including location information; matching the search image with each of the plurality of ground images and the plurality of aerial images; and displaying information about one or more candidate locations for the shooting location based on the matching result of the matching process.

[0103] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 10 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 11 and 12 in the same dependency relationship as Supplementary Notes 2 to 10. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0104] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0105] This application claims priority based on Japanese Patent Application No. 2024-130123, filed on August 6, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0106] 1, 2, 3, 4 Information processing device 10, 11 Reception unit 20, 20' Database 21 Ground image database 22 Aerial image database 23 Map database 30, 30', 33 Matching unit 31 First matching unit 32 Second matching unit 40 Display unit 50 Registration unit 60 Narrowing unit 70 Distance estimation unit 100 Network interface 200 Processor 300 Memory

Claims

1. An information processing device comprising: a reception unit that receives a search image whose shooting location is unknown; a database that stores a plurality of ground images, each of which includes location information, and a plurality of aerial images, each of which includes location information; a matching unit that matches the search image with each of the plurality of ground images and the plurality of aerial images; and a display unit that displays information regarding one or more candidate locations for the shooting location based on the matching results by the matching unit.

2. The information processing device according to claim 1, wherein the information relating to the one or more candidate locations includes at least one of location information for each of the one or more candidate locations, an image taken at the candidate location, or a point on an aerial image including the candidate location.

3. The information processing device described in claim 2, wherein the reception unit receives user information of the user performing the search along with the search image, and the display unit does not display images taken at each of the one or more candidate locations based on the user information.

4. The information processing device of claim 1, wherein the matching unit calculates a matching score for each of the plurality of ground images and the plurality of aerial images against the search image, and sorts the matching results based on the matching score; and the display unit displays, from the sorted matching results, those whose matching score is equal to or greater than a predetermined threshold.

5. The information processing device of claim 1, wherein the database includes a ground image database that stores the plurality of ground images and an aerial image database that stores the plurality of aerial images, and the matching unit includes: a first matching unit that matches the search image with each of the plurality of aerial images stored in the aerial image database; and a second matching unit that matches the search image with each of the plurality of ground images stored in the ground image database based on the matching result of the first matching unit.

6. The information processing device of claim 5, wherein each of the plurality of aerial images is an image of a square section with a side of a predetermined length specified by the longitude and latitude of a center point, the first matching unit calculates a matching score of each of the plurality of aerial images against the search image, and the second matching unit matches the search image with each of one or more ground images included in each of the one or more aerial images for which the matching score calculated by the first matching unit is equal to or greater than a predetermined threshold.

7. The information processing device of claim 5, wherein each of the plurality of aerial images is an image of a square section with a side of a predetermined length specified by the longitude and latitude of a center point, the first matching unit calculates a matching score of each of the plurality of aerial images against the search image, and the second matching unit matches the search image with each of one or more ground images whose matching scores by the first matching unit are included in each of a predetermined number of top aerial images.

8. An information processing device according to any one of claims 1 to 7, further comprising a registration unit that registers location information of a specified location in association with an image taken at the specified location in response to a user operation or based on an image uploaded to a network.

9. An information processing device as described in any one of claims 1 to 7, wherein the reception unit receives topographical information as the search target along with the search image, the database includes a ground image database storing the plurality of ground images, an aerial image database storing the plurality of aerial images, and a map database storing map data of the country or region to be searched, the information processing device further includes a narrowing-down unit that narrows down the map data of the country or region to be searched to one or more areas that match the topographical information, and the matching unit matches the search image with each of the plurality of ground images and the plurality of aerial images within the narrowed-down one or more areas.

10. An information processing device as described in any one of claims 1 to 7, wherein the matching unit calculates a matching score for each of the plurality of ground images against the search image, identifies a predetermined ground image with the highest matching score based on the matching score, and identifies a subject that appears in the predetermined ground image; the information processing device further comprises a distance estimation unit that estimates a distance between a candidate position of the shooting location and the subject based on information about the subject that appears in the search image, the shooting location of the predetermined ground image, and information about the subject that appears in the predetermined ground image; and the display unit displays the position of the subject and a circle with the estimated distance as its radius, centered on the position of the subject, for an aerial image including the shooting location of the predetermined ground image.

11. An information processing method comprising: a step of accepting a search image whose shooting location is unknown; a step of storing a plurality of ground images, each including location information, and a plurality of aerial images, each including location information; a step of comparing the search image with each of the plurality of ground images and the plurality of aerial images; and a step of displaying information regarding one or more candidate locations for the shooting location based on the comparison result in the comparing step.

12. A program that causes a computer to perform the following processes: accepting a search image whose shooting location is unknown; storing a plurality of ground images, each of which includes location information, and a plurality of aerial images, each of which includes location information; matching the search image with each of the plurality of ground images and the plurality of aerial images; and displaying information regarding one or more candidate locations for the shooting location based on the matching results of the matching process.

Images