Image matching device, image matching model training device, photographing location estimate device, and program

Abstract

To improve matching accuracy between ground images and aerial images.SOLUTION: Ground image feature extraction means 11 of an image matching device 1 generates a single ground image AA from N ground images A1, ..., AN captured from different viewpoints at the same location, and uses a ground NN 13 to compute a ground feature vector AV from the ground image AA. Aerial image feature extraction means 12 uses an aerial NN 14 to compute an aerial feature vector BV from an aerial image B. Matching determination means 20 calculates a similarity score S based on the ground feature vector AV and the aerial feature vector BV, and determines whether or not the ground images A1, ..., AN match the aerial image B based on the similarity score S.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for matching two images, and in particular to an image matching device that matches between ground images and aerial images, an image matching model training device that trains a model used when matching between ground images and aerial images, a shooting location estimation device that estimates the shooting location of a ground image, and a program. [Background technology]

[0002] Conventionally, a technology has been proposed that estimates the location of subjects, scenery, etc. contained in a ground image (hereinafter referred to as the "location where the ground image was taken") by matching an image taken on the ground (hereinafter referred to as the "ground image") with a satellite image, aerial photograph, drone-taken image, etc. taken from the sky (hereinafter referred to as the "aerial image") (see, for example, Non-Patent Document 1 and Patent Documents 1 and 2).

[0003] Non-Patent Document 1 describes a technology called ground-to-aerial cross-view matching (matching between ground images and aerial images). This technology uses two convolutional neural networks (CNNs) to extract features from the ground images and aerial images to be compared.

[0004] One of the two CNNs acquires ground images and their corresponding directional information, and calculates combined features from a mixed signal containing the ground images and directional information. The other CNN acquires aerial images and their corresponding directional information, and calculates combined features from a mixed signal containing the aerial images and directional information. Based on these combined features, it is then determined whether the ground images match the aerial images.

[0005] Furthermore, the technology of Patent Document 1 uses augmented images in which specific object regions have been removed when generating a training dataset containing ground-to-aerial cross-view matching images for training CNN.

[0006] The technology in Patent Document 2 generates a feature map using an attention method and a feature map not using an attention method from a ground image and an aerial image, respectively, and calculates a similarity score between the ground image and the aerial image based on these feature maps. This technology then determines whether the ground image and the aerial image match based on the similarity score. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2023-537373 [Patent Document 2] Special Publication No. 2023-538053 [Non-patent literature]

[0008] [Non-Patent Document 1] Liu Liu and Hongdong Li, “Lending orientation to neural networks for cross-view geo-localization”, Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019 Summary of the Invention [Problem to be solved by the invention]

[0009] Incidentally, social networking sites (SNSs) have traditionally been used as a place for users to interact with each other on the Internet, and images taken with smartphones, for example, are routinely posted to social media, etc. However, due to the specifications of SNSs and crime prevention measures, location information is often not attached to posted images, making it difficult to identify the location where the posted images were taken.

[0010] For example, if a posted image is related to a disaster or incident, identifying the location where the image was taken can help quickly and accurately grasp the local situation at the time of the disaster or incident. It can also help determine the authenticity of content posted on social media, etc.

[0011] Generally, when taking photos with a smartphone, rather than taking a 360-degree panoramic image, multiple images with a limited viewing angle are taken at once, and these multiple images with a narrow viewing angle are posted on social media, etc.

[0012] The ground-to-aerial cross-view matching systems described in Non-Patent Document 1 and Patent Documents 1 and 2 determine whether a ground image matches an aerial image by matching one ground image with an aerial image. Therefore, these systems have the problem of being unable to adequately accommodate the current usage patterns of SNS, in which multiple images (ground images) with narrow viewing angles are posted. In other words, these systems have the problem of being unable to accommodate two or more ground images.

[0013] Here, when determining whether a ground image and an aerial image match, it is expected that the matching accuracy will improve if two or more ground images taken at the same location can be used.

[0014] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide an image matching device, an image matching model training device, a shooting location estimation device, and a program that can improve the matching accuracy between ground images and aerial images. [Means for solving the problem]

[0015] In order to solve the above problem, the image matching device of claim 1 is an image matching device that determines matching between ground images and aerial images, and comprises: a ground image feature extraction means that extracts features of a plurality of ground images based on a plurality of ground images taken at the same location from different viewpoints and determines a ground feature vector AV; an aerial image feature extraction means that extracts features of a single aerial image based on the single aerial image and determines an aerial feature vector BV; and a matching determination means that determines the matching based on the ground feature vector AV determined by the ground image feature extraction means and the aerial feature vector BV determined by the aerial image feature extraction means and generates a determination result, wherein the ground image feature extraction means comprises a single image generation means that generates a single ground image based on the plurality of ground images, and a ground NN (neural network) that determines the ground feature vector AV from the single ground image generated by the single image generation means, and the aerial image feature extraction means comprises an aerial NN that determines the aerial feature vector BV from the single aerial image.

[0016] Furthermore, the image matching device of claim 2 is an image matching device that determines matching between ground images and aerial images, and comprises: a ground image feature extraction means that extracts features of a plurality of ground images based on a plurality of ground images taken at the same location from different viewpoints and determines a ground feature vector AV; an aerial image feature extraction means that extracts features of one aerial image based on the single aerial image and determines an aerial feature vector BV; and a matching determination means that determines the matching based on the ground feature vector AV determined by the ground image feature extraction means and the aerial feature vector BV determined by the aerial image feature extraction means and generates a determination result, wherein the ground image feature extraction means comprises a ground NN (neural network) that determines a ground feature vector from each of the plurality of ground images, and a combination means that generates the ground feature vector AV by combining the ground feature vectors for each of the plurality of ground images determined by the ground NN, and the aerial image feature extraction means comprises an aerial NN that determines the aerial feature vector BV from the single aerial image.

[0017] Furthermore, the image matching model training device of claim 3 is an image matching model training device that trains a model for generating parameters of a ground NN (neural network) and parameters of an aerial NN to be used in the image matching device of claim 1, and is characterized in that it comprises: a single image generation means that generates a single ground image based on a plurality of ground images taken from different viewpoints at the same location; a ground NN that obtains a ground feature vector AV' that indicates the characteristics of the single ground image generated by the single image generation means; an aerial NN that obtains an aerial feature vector BV' that indicates the characteristics of a single aerial image; and an update means that calculates a loss based on the ground feature vector AV' obtained by the ground NN and the aerial feature vector BV' obtained by the aerial NN, and updates the parameters of the ground NN and the aerial NN based on the loss.

[0018] The image matching model training device of claim 4 is an image matching model training device that trains a model for generating parameters for a ground NN (neural network) and parameters for an aerial photography NN to be used in the image matching device of claim 2, and is characterized by comprising: a ground NN that obtains a ground feature vector that indicates the characteristics of each of a plurality of ground images taken at the same location from different viewpoints; a synthesis means that generates a ground feature vector AV' by synthesizing the ground feature vectors for each of the plurality of ground images obtained by the ground NN; an aerial photography NN that obtains an aerial photography feature vector BV' that indicates the characteristics of a single aerial image; and an update means that calculates a loss based on the ground feature vector AV' generated by the synthesis means and the aerial photography feature vector BV' obtained by the aerial photography NN, and updates the parameters of the ground NN and the aerial photography NN based on the loss.

[0019] Furthermore, a photography location estimation device according to claim 5 is a photography location estimation device that determines a match between a ground image and an aerial image and estimates the photography location of the ground image based on the determination result, and includes the image matching device according to claim 1 or 2, and photography location estimation means that receives input of a plurality of ground images taken at the same location from different viewpoints, identifies an aerial image that matches the plurality of ground images based on the determination result generated by the image matching device, and estimates position information of the aerial image as the photography location of the ground image, The method is characterized in that the method reads out each of the plurality of sets of data from a location DB (database) consisting of the plurality of sets of data, with the location information of the aerial image as one set, obtains the judgment result for the plurality of ground images and the one aerial image in the data set from the image matching device for each of the plurality of sets of data, identifies an aerial image that matches the plurality of ground images based on the judgment result, and extracts the location information corresponding to the identified aerial image included in the data set, thereby estimating the location information as the shooting location.

[0020] Furthermore, a program according to claim 6 is characterized in that it causes a computer to function as the image matching device according to claim 1 or 2.

[0021] A seventh aspect of the present invention provides a program for causing a computer to function as the image matching model training device according to the third or fourth aspect of the present invention. [Effects of the Invention]

[0022] As described above, according to the present invention, it is possible to improve the matching accuracy between ground images and aerial images. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a block diagram showing an example of the configuration of an image matching device according to an embodiment of the present invention. [Figure 2] 10 is a flowchart illustrating an example of processing performed by the image matching device according to the embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a ground image feature extraction means. [Figure 4] 10 is a flowchart showing an example of processing by a first ground image feature extraction means. [Figure 5] 10 is a flowchart showing an example of processing by a second ground image feature extraction means. [Figure 6] 10 is a flowchart showing an example of processing by a third ground image feature extraction means. [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of a matching determination means. [Figure 8] 10 is a flowchart showing an example of processing by a matching determination means. [Figure 9] FIG. 1 is a block diagram showing an example of the configuration of a first image matching model training device according to an embodiment of the present invention. [Figure 10] 1 is a flowchart showing an example of processing performed by the first image matching model training device according to an embodiment of the present invention. [Figure 11]FIG. 2 is a block diagram showing an example of the configuration of a second image matching model training device according to an embodiment of the present invention. [Figure 12] 10 is a flowchart showing an example of processing performed by the second image matching model training device according to an embodiment of the present invention. [Figure 13] FIG. 10 is a block diagram showing an example of the configuration of a third image matching model training device according to an embodiment of the present invention. [Figure 14] 10 is a flowchart showing an example of processing performed by the third image matching model training device according to an embodiment of the present invention. [Figure 15] 1 is a schematic diagram illustrating an example of the overall configuration of a filming location estimating system including a filming location estimating device according to an embodiment of the present invention. [Figure 16] FIG. 2 is a diagram illustrating an example of the data configuration of a position DB. [Figure 17] 1 is a block diagram showing an example of the configuration of a filming location estimation device according to an embodiment of the present invention. [Figure 18] 10 is a flowchart showing an example of processing by a photography location estimation means. DETAILED DESCRIPTION OF THE INVENTION

[0024] The present invention aims to improve the accuracy of matching between a ground image and an aerial image by using multiple ground images taken at the same location from different viewpoints when determining whether the ground image matches an aerial image.

[0025] In the first example described below, a single ground image is generated from multiple ground images, a ground feature vector AV is obtained from the single ground image using a ground NN (Neural Network), and an aerial feature vector BV is obtained from the aerial image using an aerial NN, and matching is determined based on the ground feature vector AV and the aerial feature vector BV.

[0026] In addition, in the second example, a ground feature vector is obtained from each of multiple ground images using a ground NN, and the multiple ground feature vectors are combined to obtain a single ground feature vector AV, and an aerial photography NN is used to obtain an aerial photography feature vector BV from the aerial photography image, and matching is determined based on the ground feature vector AV and the aerial photography feature vector BV.

[0027] In addition, in the third example, a ground feature vector AV is obtained from multiple ground images using a ground NN, and an aerial feature vector BV is obtained from an aerial image using an aerial NN, and matching is determined based on the ground feature vector AV and the aerial feature vector BV.

[0028] [Image Matching Device] First, an image matching device according to an embodiment of the present invention will be described. Fig. 1 is a block diagram showing an example of the configuration of an image matching device according to an embodiment of the present invention, and Fig. 2 is a flowchart showing an example of processing performed by the image matching device according to the embodiment of the present invention.

[0029] This image matching device 1 includes feature extraction means 10 having ground image feature extraction means 11 and aerial image feature extraction means 12, and matching determination means 20. The ground image feature extraction means 11 and the aerial image feature extraction means 12 are implemented by one or more trained models such as neural networks (NNs). As the trained model, for example, CNN or a Vision Transformer encoder is used. In the example of FIG. 1, the ground image feature extraction means 11 is implemented with a ground NN 13, and the aerial image feature extraction means 12 is implemented with an aerial NN 14.

[0030] The image matching device 1 receives N (N is an integer of 2 or more) ground images A1,...,AN taken at the same location from different viewpoints and one aerial image B, and performs matching between the ground images A1,...,AN and the aerial image B. N indicates the number of ground images.

[0031] Then, the image matching device 1 generates and outputs a determination result indicating whether or not the ground images A1, . . . , AN match the aerial image B.

[0032] It should be noted that the image matching device 1 is still functional even when N=1. In this case, the image matching device 1 inputs one ground image A and one aerial image B, matches the two images, and generates and outputs a determination result indicating whether the two images match. Hereinafter, ground image A refers to any one of ground images A1,...,AN, or a collective term for these images.

[0033] The ground image A is an image taken from, for example, a pedestrian or a vehicle, and the location where the image was taken is unknown. The ground image A may be a panoramic image (an image with a 360° angle of view) or an image with a limited angle of view (an image with a angle of view less than 360°).

[0034] Aerial image B is an image including a planar view of a location photographed from, for example, a drone, an airplane, an artificial satellite, etc., and the photographing location is assumed to be known.

[0035] "Terrestrial image A and aerial image B match" means that the location where terrestrial image A was taken (the location (taking location) where the subject, scenery, etc. included in terrestrial image A exists) is included in aerial image B.

[0036] The ground image feature extraction means 11 of the feature extraction means 10 inputs N ground images A1,...,AN taken from different viewpoints at the same location, and the aerial image feature extraction means 12 of the feature extraction means 10 inputs one aerial image B (step S201).

[0037] The ground image feature extraction means 11 extracts features of the ground images A1,...,AN based on the ground images A1,...,AN by using the ground NN 13, and obtains a ground feature vector AV (step S202). Then, the ground image feature extraction means 11 outputs the ground feature vector AV to the matching determination means 20. The ground image feature extraction means 11 equipped with the ground NN 13 will be described in detail later.

[0038] The aerial image feature extraction means 12 extracts features of the aerial image B based on the aerial image B using the aerial NN 14, and obtains an aerial feature vector BV (step S203). Then, the aerial image feature extraction means 12 outputs the aerial feature vector BV to the matching determination means 20.

[0039] Specifically, the aerial photography NN14 uses the aerial image B as input data and the aerial feature vector BV as output data, and calculates the aerial photography NN14 using pre-set parameters to obtain the aerial photography feature vector BV from the aerial image B.

[0040] The matching determination means 20 inputs the ground feature vector AV from the ground image feature extraction means 11 and inputs the aerial feature vector BV from the aerial image feature extraction means 12. Then, the matching determination means 20 calculates a similarity score S indicating the degree of similarity based on the ground feature vector AV and the aerial feature vector BV, and determines whether there is a match between the ground image A1,...,AN and the aerial image B (step S204).

[0041] The matching determination means 20 generates a determination result indicating whether or not the ground images A1, . . . , AN match the aerial image B, and outputs the determination result (step S205).

[0042] (Ground image feature extraction means 11) Next, the processing of step S202 shown in FIG. 2 by the ground image feature extraction means 11 shown in FIG. 1 will be described in detail.

[0043] Figure 3 is a block diagram showing an example of the configuration of the ground image feature extraction means 11, where Figure 3(1) shows an example of the configuration of the first ground image feature extraction means 11, Figure 3(2) shows an example of the configuration of the second ground image feature extraction means 11, and Figure 3(3) shows an example of the configuration of the third ground image feature extraction means 11.

[0044] (Ground image feature extraction method 11 / 1st example) First, the first ground image feature extraction means 11 shown in Fig. 3(1) will be described. Fig. 4 is a flowchart showing an example of processing by the first ground image feature extraction means 11. This first ground image feature extraction means 11 includes a single image generation means 15 and a ground NN 13-1.

[0045] The single image generation means 15 receives N ground images A1,...,AN taken at the same location from different viewpoints (step S401), and generates a single ground image AA from the ground images A1,...,AN (step S402).The single image generation means 15 then outputs the ground image AA to the ground NN 13-1.

[0046] For example, the single image generation means 15 performs image mosaicing on the ground images A1,...,AN to combine (combine) the ground images A1,...,AN to generate a single ground image AA. Image mosaicing is a technique for seamlessly combining multiple images taken at the same location to generate a single large image. For example, image mosaicing is performed using an existing projective transformation matrix or a neural network.

[0047] Also, for example, the single image generating means 15 generates one ground image AA by arranging the ground images A1, . . . , AN through a concatenation process on the ground images A1, . . . , AN.

[0048] The ground NN 13-1 receives the ground image AA from the single image generation means 15. The ground NN 13-1 then uses the ground image AA as input data and the ground feature vector AV as output data, and performs calculations on the ground NN 13-1 using preset parameters to determine the ground feature vector AV from the ground image AA (step S403). The ground NN 13-1 outputs the ground feature vector AV to the matching determination means 20 (step S404).

[0049] (Ground image feature extraction method 11 / 2nd example) Next, the second ground image feature extraction means 11 shown in Fig. 3(2) will be described. Fig. 5 is a flowchart showing an example of processing by the second ground image feature extraction means 11. This second ground image feature extraction means 11 includes a ground NN 13-2, a memory 16, and a synthesis means 17.

[0050] The ground NN 13-2 receives input of N ground images A1,...,AN taken at the same location from different viewpoints (step S501). Then, the ground NN 13-2 uses the ground images A1,...,AN as input data and the ground feature vectors A1V,...,ANV as output data, and performs calculations of the ground NN 13-2 using preset parameters to obtain the ground feature vectors A1V,...,ANV from the ground images A1,...,AN (step S502). The ground NN 13-2 stores the ground feature vectors A1V,...,ANV in memory 16.

[0051] That is, the ground NN 13-2 determines a ground feature vector A1V from the ground image A1, . . . , and determines a ground feature vector ANV from the ground image AN.

[0052] The combining means 17 reads out the ground feature vectors A1V,...,ANV from the memory 16 and combines the ground feature vectors A1V,...,ANV to obtain a single ground feature vector AV (step S503). Then, the combining means 17 outputs the ground feature vector AV to the matching determination means 20 (step S504). The process of combining the ground feature vectors A1V,...,ANV to obtain a single ground feature vector AV is known, so a description thereof will be omitted here.

[0053] (Ground image feature extraction method 11 / 3rd example) Next, the third ground image feature extraction means 11 shown in Fig. 3(3) will be described. Fig. 6 is a flowchart showing an example of processing by the third ground image feature extraction means 11. This third ground image feature extraction means 11 includes a ground NN 13-3.

[0054] The ground NN 13-3 receives as input N ground images A1,...,AN taken at the same location from different viewpoints (step S601).The ground NN 13-3 then uses the ground images A1,...,AN as input data and the ground feature vector AV as output data, and performs calculations within the ground NN 13-3 using preset parameters to determine the ground feature vector AV from the ground images A1,...,AN (step S602).The ground NN 13-3 outputs the ground feature vector AV to the matching determination means 20 (step S603).

[0055] The ground NN 13-3 is composed of a CNN corresponding to the number of images in multiple channels. The ground NN 13-3 takes N-channel ground images A1,...,AN as input data, overlaps them in the input layer, and then performs NN calculations to obtain a single ground feature vector AV as output data.

[0056] (Matching determination means 20) Next, the processing of steps S204 and S205 shown in FIG. 2 by the matching determination means 20 shown in FIG. 1 will be described in detail.

[0057] 7 is a block diagram showing an example of the configuration of the match determination means 20, and FIG. 8 is a flowchart showing an example of the processing of the match determination means 20. This match determination means 20 includes a score calculation means 21 and a determination means 22.

[0058] The score calculation means 21 receives the ground feature vector AV from the ground image feature extraction means 11 and also receives the aerial photograph feature vector BV from the aerial photograph image feature extraction means 12 (step S801).

[0059] The score calculation means 21 calculates a similarity score S (similarity) between the ground images A1,...,AN and the aerial image B based on the ground feature vector AV and the aerial feature vector BV (step S802). The score calculation means 21 outputs the similarity score S to the determination means 22.

[0060] For example, the score calculation means 21 calculates the L2 distance, cosine similarity, or NN-based similarity between the ground feature vector AV and the aerial feature vector BV as the similarity score S. The NN-based similarity is calculated by an NN trained to calculate the similarity between two pieces of input data (the similarity between the ground feature vector AV and the aerial feature vector BV).

[0061] The determination means 22 receives the similarity score S from the score calculation means 21 and determines whether or not the ground images A1,...,AN match the aerial image B based on the similarity score S (step S803). Then, the determination means 22 generates and outputs a determination result (step S804).

[0062] For example, if the similarity score S is equal to or greater than a preset threshold, the determination means 22 determines that the ground images A1,...,AN match the aerial image B, and if the similarity score S is less than the threshold, the determination means 22 determines that the ground images A1,...,AN do not match the aerial image B. Note that the higher the similarity calculated based on the ground feature vector AV and the aerial feature vector BV, the larger the similarity score S, and the lower the similarity, the smaller the similarity score S.

[0063] As described above, according to the image matching device 1 of the embodiment of the present invention, the ground image feature extraction means 11 obtains features as a ground feature vector AV based on the ground images A1,...,AN by using the ground NN 13. The aerial image feature extraction means 12 obtains features as an aerial feature vector BV based on the aerial image B by using the aerial NN 14.

[0064] The matching determination means 20 calculates a similarity score S based on the ground feature vector AV and the aerial feature vector BV, and determines based on the similarity score S whether the ground images A1, . . . , AN match the aerial image B.

[0065] In this way, even for ground image A with a narrow viewing angle, by utilizing multiple ground images A1,...,AN taken at the same location from different viewpoints, it is possible to obtain a ground feature vector AV that integrates these. This ground feature vector AV reflects the features of ground images A1,...,AN, and therefore is data that reflects the features of the shooting location more effectively than a vector that reflects the features of a single ground image A. A similarity score S is calculated using such ground feature vector AV, and matching is determined based on the similarity score S, thereby improving the matching accuracy between ground image A and aerial image B.

[0066] [Image matching model training device / first example] Next, a first image matching model training apparatus according to an embodiment of the present invention will be described. Fig. 9 is a block diagram showing an example of the configuration of the first image matching model training apparatus according to an embodiment of the present invention, and Fig. 10 is a flowchart showing an example of the processing performed by the first image matching model training apparatus according to the embodiment of the present invention.

[0067] This image matching model training device 2-1 corresponds to the image matching device 1 equipped with the first ground image feature extraction means 11 shown in Figures 1 and 3(1), and is a device for acquiring parameters used for the ground NN 13 (13-1) and the aerial photography NN 14 equipped in the image matching device 1.

[0068] The ground NN 13-1 and the aerial NN 14 provided in the image matching device 1 correspond to the ground NN 31-1 and the aerial NN 32-1 (described later) provided in the image matching model training device 2-1. In order to acquire the respective parameters of the ground NN 13-1 and the aerial NN 14 provided in the image matching device 1, it is necessary to train the ground NN 31-1 and the aerial NN 32-1 (described later) provided in the image matching model training device 2-1 using a training data set including a pair of ground image A and aerial image B.

[0069] The ground NN31-1 and the aerial NN32-1 are trained based on the loss between the ground feature vector AV' and the aerial feature vector BV' calculated from a training data set, similar to when training a Siamese network, for example. For details about Siamese networks, see, for example, Non-Patent Document 1. The parameters of the ground NN31-1 and the aerial NN32-1 are then updated based on the loss calculated using a predetermined loss function.

[0070] The same applies to the terrestrial NN 31-2 and the aerial photography NN 32-2 (corresponding to the terrestrial NN 13 (13-2) and the aerial photography NN 14 shown in Figures 1 and 3(2)) trained by the image matching model training device 2-2 shown in Figures 11 and 12, which will be described later.The same applies to the terrestrial NN 31-3 and the aerial photography NN 32-3 (corresponding to the terrestrial NN 13 (13-3) and the aerial photography NN 14 shown in Figures 1 and 3(3)) trained by the image matching model training device 2-3 shown in Figures 13 and 14, which will be described later.

[0071] 9 and 10, the image matching model training device 2-1 includes a single image generating means 30, a ground NN 31-1, an aerial NN 32-1, and an updating means 33.

[0072] The image matching model training device 2-1 inputs a training data set consisting of ground images A1',...,AM' for training the ground NN 31-1 and aerial images B' for training the aerial NN 32-1 (step S1001). The ground images A1',...,AM' are M images (M is an integer equal to or greater than 2) taken at the same location from different viewpoints. M indicates the number of ground images.

[0073] Note that the image matching model training device 2-1 is assumed to function even when M = 1. In this case, the image matching model training device 2-1 inputs one ground image A' and one aerial image B' as training data sets, and updates the parameters of the ground NN 31-1 and the aerial NN 32-1 using the ground NN 31-1, the aerial NN 32-1, and the update means 33. The same applies to the image matching model training devices 2-2 and 2-3, which will be described later.

[0074] The single image generating means 30 generates a single ground image AA' from the ground images A1', . . . , AM' (step S1002), and outputs the ground image AA' to the ground NN 31-1.

[0075] For example, the single image generation means 30 generates one ground image AA' by combining and synthesizing the ground images A1',...,AM' through image mosaicing processing, similar to the single image generation means 15 shown in Fig. 3(1). Also, for example, the single image generation means 30 generates one ground image AA' by arranging the ground images A1',...,AM' through concatenation processing, similar to the single image generation means 15 shown in Fig. 3(1).

[0076] The ground NN 31-1 receives the ground image AA' from the single image generation means 30. Then, similar to the ground NN 13-1 shown in FIG. 3(1), the ground NN 31-1 uses the ground image AA' as input data and the ground feature vector AV' as output data, and performs calculations of the ground NN 31-1 using the parameters it holds to obtain the ground feature vector AV' from the ground image AA' (step S1003). The ground NN 31-1 outputs the ground feature vector AV' to the update means 33.

[0077] 1, the aerial photography NN 32-1 uses the aerial image B' as input data and the aerial feature vector BV' as output data, and performs calculations within the aerial photography NN 32-1 using the parameters it holds to determine the aerial feature vector BV' from the aerial image B' (step S1004).The aerial photography NN 32-1 outputs the aerial feature vector BV' to the update means 33.

[0078] The updating means 33 inputs the ground feature vector AV' from the ground NN 31-1 and the aerial photography feature vector BV' from the aerial photography NN 32-1. Then, the updating means 33 calculates the loss between the ground feature vector AV' and the aerial photography feature vector BV' using a predetermined loss function based on the ground feature vector AV' and the aerial photography feature vector BV' (step S1005).

[0079] The update means 33 updates the parameters of the terrestrial NN 31-1 and the aerial photography NN 32-1 by maximizing or minimizing the loss based on the loss (step S1006).

[0080] This optimizes the parameters held in the terrestrial NN 31-1 and the aerial photography NN 32-1. The optimized parameters are then stored in the terrestrial NN 13 (13-1) and the aerial photography NN 14 shown in Figures 1 and 3(1).

[0081] The training data set consisting of the ground images A1', . . . , AM' and the aerial image B' may be generated by extending a training data set including a pair of the ground image A' and the aerial image B'.

[0082] In this case, the image matching model training device 2-1 is equipped with an expansion means, which inputs m (m is an integer greater than or equal to 1 and less than M) ground images A' and one aerial image B' and expands the m ground images A' to generate M ground images A1',...,AM'.The expansion means then outputs the M ground images A1',...,AM' to the single image generation means 30, and outputs one aerial image B' to the aerial NN 32-1.

[0083] For example, the extension means generates M-1 ground images A2',...,AM' from other viewpoints taken at the same location from one ground image A1' taken from a specific viewpoint. Then, the extension means outputs the input ground image A1' and the generated M-1 ground images A2',...,AM' as M ground images A1',...,AM'.

[0084] Furthermore, for example, the expansion means generates M-1 ground images A2',...,AM' from one ground image A1' using a technique such as NeRF (Neural Radiance Fields), and outputs these as M ground images A1',...,AM'. Furthermore, for example, the expansion means cuts out and outputs M ground images A1',...,AM' with a field of view from one panoramic ground image A'. This makes it possible to increase the number of ground images A'. The same applies to the image matching model training devices 2-2 and 2-3 described later.

[0085] As described above, according to the image matching model training device 2-1 of the first embodiment of the present invention, the single image generation means 30 generates a single ground image AA' from the ground images A1',...,AM' included in the training data set, and the ground NN 31-1 obtains a ground feature vector AV' from the ground image AA'.

[0086] The aerial photography NN 32-1 obtains an aerial photography feature vector BV' from an aerial photography image B' included in the training data set.

[0087] The update means 33 calculates the loss between the ground feature vector AV' and the aerial photography feature vector BV' based on these, and updates the parameters of the ground NN 31-1 and the aerial photography NN 32-1 based on the loss.

[0088] As a result, the parameters held in the terrestrial NN 31-1 and the parameters held in the aerial photography NN 32-1 are optimized. Then, the optimized parameters are stored in the terrestrial NN 13 (13-1) and the aerial photography NN 14 provided in the image matching device 1, and are used for matching determination.

[0089] Therefore, the image matching device 1 can utilize multiple ground images A1,...,AN taken from different viewpoints at the same location, and can improve the matching accuracy between the ground image A and the aerial image B compared to when a single ground image A is used.

[0090] [Image matching model training device / Example 2] Next, a second image matching model training apparatus according to an embodiment of the present invention will be described. Fig. 11 is a block diagram showing an example of the configuration of the second image matching model training apparatus according to an embodiment of the present invention, and Fig. 12 is a flowchart showing an example of the processing performed by the second image matching model training apparatus according to an embodiment of the present invention.

[0091] This image matching model training device 2-2 corresponds to the image matching device 1 equipped with the second ground image feature extraction means 11 shown in Figures 1 and 3(2), and is a device for acquiring parameters used for the ground NN 13 (13-2) and the aerial photography NN 14 equipped in the image matching device 1.

[0092] The image matching model training device 2-2 includes a ground NN 31-2, an aerial NN 32-2, a memory 34, a synthesis means 35, and an update means 36.

[0093] The image matching model training device 2-2 receives as input a training data set consisting of ground images A1', . . . , AM' for training the ground NN 31-2 and an aerial image B' for training the aerial NN 32-2 (step S1201).

[0094] 3(2), the ground NN31-2 uses each of the ground images A1',...,AM' as input data and each of the ground feature vectors A1V',...,AMV' as output data, and performs calculations of the ground NN31-2 using the parameters stored therein to obtain each of the ground feature vectors A1V',...,AMV' from each of the ground images A1',...,AM' (step S1202). The ground NN31-2 stores the ground feature vectors A1V',...,AMV' in the memory 34.

[0095] The combining means 35 reads out the ground feature vectors A1V',...,AMV' from the memory 34 and combines the ground feature vectors A1V',...,AMV' to obtain a single ground feature vector AV' (step S1203). Then, the combining means 35 outputs the ground feature vector AV' to the updating means 36.

[0096] 1, the aerial photography NN 32-2 uses the aerial image B' as input data and the aerial feature vector BV' as output data, and performs calculations within the aerial photography NN 32-2 using the parameters it holds to determine the aerial feature vector BV' from the aerial image B' (step S1204).The aerial photography NN 32-2 outputs the aerial feature vector BV' to the update means 36.

[0097] The updating means 36 receives the ground feature vector AV' from the combining means 35 and also receives the aerial photography feature vector BV' from the aerial photography NN 32-2. Then, the updating means 36 calculates the loss between the ground feature vector AV' and the aerial photography feature vector BV' using a predetermined loss function based on the ground feature vector AV' and the aerial photography feature vector BV' (step S1205).

[0098] The update means 36 updates the parameters of the terrestrial NN 31-2 and the aerial photography NN 32-2 by maximizing or minimizing the loss based on the loss (step S1206).

[0099] This optimizes the parameters held in the terrestrial NN 31-2 and the aerial photography NN 32-2. The optimized parameters are then stored in the terrestrial NN 13 (13-2) and the aerial photography NN 14 shown in Figures 1 and 3(2).

[0100] As described above, according to the image matching model training device 2-2 of the second embodiment of the present invention, the ground NN 31-2 obtains ground feature vectors A1V',...,AMV' from the ground images A1',...,AM' included in the training data set, and the synthesis means 35 obtains a single ground feature vector AV' by synthesizing the ground feature vectors A1V',...,AMV'.

[0101] The aerial photography NN 32-2 obtains an aerial photography feature vector BV' from the aerial photography image B' included in the training data set.

[0102] The update means 36 calculates the loss between the ground feature vector AV' and the aerial photography feature vector BV' based on these, and updates the parameters of the ground NN 31-2 and the aerial photography NN 32-2 based on the loss.

[0103] As a result, the parameters held in the terrestrial NN 31-2 and the parameters held in the aerial photography NN 32-2 are optimized. Then, the optimized parameters are stored in the terrestrial NN 13 (13-2) and the aerial photography NN 14 provided in the image matching device 1, and are used for matching determination.

[0104] Therefore, the image matching device 1 can utilize multiple ground images A1,...,AN taken from different viewpoints at the same location, and can improve the matching accuracy between the ground image A and the aerial image B compared to when a single ground image A is used.

[0105] [Image matching model training device / Example 3] Next, a third image matching model training apparatus according to an embodiment of the present invention will be described. Fig. 13 is a block diagram showing an example of the configuration of the third image matching model training apparatus according to an embodiment of the present invention, and Fig. 14 is a flowchart showing an example of the processing performed by the third image matching model training apparatus according to an embodiment of the present invention.

[0106] This image matching model training device 2-3 corresponds to the image matching device 1 equipped with the third ground image feature extraction means 11 shown in Figures 1 and 3(3), and is a device for acquiring parameters used for the ground NN 13(13-3) and the aerial photography NN 14 equipped in the image matching device 1.

[0107] The image matching model training device 2-3 includes a ground NN 31-3, an aerial NN 32-3, and an updating means 37.

[0108] The image matching model training device 2-3 receives as input a training data set consisting of ground images A1', . . . , AM' for training the ground NN 31-3 and an aerial image B' for training the aerial NN 32-3 (step S1401).

[0109] 3(3), the ground NN31-3 uses the ground images A1',...,AM' as input data and the ground feature vector AV' as output data, and performs calculations of the ground NN31-3 using the parameters held therein to obtain the ground feature vector AV' from the ground images A1',...,AM' (step S1402). The ground NN31-3 outputs the ground feature vector AV' to the update means 37.

[0110] 1, the aerial photography NN 32-3 uses the aerial image B' as input data and the aerial feature vector BV' as output data, and performs calculations within the aerial photography NN 32-3 using the parameters it holds to determine the aerial feature vector BV' from the aerial image B' (step S1403).The aerial photography NN 32-3 outputs the aerial feature vector BV' to the update means 37.

[0111] The updating means 37 inputs the ground feature vector AV' from the ground NN 31-3 and the aerial photography feature vector BV' from the aerial photography NN 32-3. Then, based on the ground feature vector AV' and the aerial photography feature vector BV', the updating means 37 calculates the loss between the ground feature vector AV' and the aerial photography feature vector BV' using a predetermined loss function (step S1404).

[0112] The update means 37 updates the parameters of the terrestrial NN 31-3 and the aerial photography NN 32-3 by maximizing or minimizing the loss based on the loss (step S1405).

[0113] This optimizes the parameters held in the terrestrial NN 31-3 and the aerial photography NN 32-3. The optimized parameters are then stored in the terrestrial NN 13 (13-3) and the aerial photography NN 14 shown in Figures 1 and 3(3).

[0114] As described above, according to the image matching model training device 2-3 of the third embodiment of the present invention, the ground NN31-3 obtains a ground feature vector AV' from the ground images A1',...,AM' included in the training data set, and the aerial NN32-3 obtains an aerial feature vector BV' from the aerial image B' included in the training data set.

[0115] The update means 37 calculates the loss between the ground feature vector AV' and the aerial feature vector BV' based on these, and updates the parameters of the ground NN 31-3 and the aerial NN 32-3 based on the loss.

[0116] As a result, the parameters held in the terrestrial NN 31-3 and the parameters held in the aerial photography NN 32-3 are optimized. Then, the optimized parameters are stored in the terrestrial NN 13 (13-3) and the aerial photography NN 14 provided in the image matching device 1, and are used for matching determination.

[0117] Therefore, the image matching device 1 can utilize multiple ground images A1,...,AN taken from different viewpoints at the same location, and can improve the matching accuracy between the ground image A and the aerial image B compared to when a single ground image A is used.

[0118] [Filming location estimation system] Next, a shooting location estimation system including a shooting location estimation device according to an embodiment of the present invention will be described. There are various applications for the image matching device 1 shown in Fig. 1. For example, the image matching device 1 can be used as part of a system that performs image geolocalization (hereinafter referred to as a "shooting location estimation system"). Image geolocalization is a technology for identifying the shooting location of an input image.

[0119] 15 is a schematic diagram showing an example of the overall configuration of a photography location estimation system including a photography location estimation device according to an embodiment of the present invention. This photography location estimation system 3 is a system that inputs ground images A1,...,AN in accordance with user operations and estimates position information P indicating the photography locations of these images.

[0120] The shooting location estimation system 3 is configured to include a user terminal 4 that functions as a client, a shooting location estimation device 5 that functions as a server, and a location DB (database) 6. The user terminal 4 and the shooting location estimation device 5 are connected via a network 7, and the shooting location estimation device 5 and the location DB 6 are also connected via the network 7.

[0121] As described above, the ground images A1,...,AN are N images taken from different viewpoints at the same location. Note that the shooting location estimation system 3 will function even if N=1. In this case, the shooting location estimation system 3 inputs one ground image A and estimates position information P, which is the shooting location.

[0122] The user terminal 4 inputs ground images A1,...,AN in accordance with the operation of the user using the user terminal 4, generates a query including the ground images A1,...,AN, and transmits the query to the shooting location estimation device 5 via the network 7.

[0123] Furthermore, the user terminal 4 receives a response to the query from the photography location estimation device 5 via the network 7, extracts the aerial image B and the location information P from the response, and displays them on a screen, for example. This makes it possible to present the location information P to the user as the photography location of the ground image A1,...,AN, and also present the aerial image B corresponding to the ground image A1,...,AN.

[0124] The photography location estimation device 5 receives a query from the user terminal 4 via the network 7. Then, the photography location estimation device 5 searches the position DB 6 for an aerial image B that matches the ground images A1,...,AN included in the query, and identifies the location information P corresponding to the found aerial image B as the photography location. The photography location estimation device 5 generates a response including the aerial image B and the location information P, and transmits the response to the user terminal 4 via the network 7. Details of the photography location estimation device 5 will be described later.

[0125] The position DB 6 stores Q (Q is an integer equal to or greater than 2) aerial images B1, . . . , BQ and the corresponding Q pieces of position information P1, . . . , PQ. Q indicates the number of aerial images.

[0126] 16 is a diagram showing an example of the data configuration of the position DB 6. The position DB 6 is made up of multiple sets of data, each set consisting of one aerial-photographed image Bq and its corresponding position information Pq, where q=1, . . . , Q. The position information Pq is information (e.g., GPS (Global Positioning System) coordinates) for identifying the location of a subject, scenery, etc. included in the aerial-photographed image Bq.

[0127] In this way, by using the position DB6, the photography location estimation device 5 searches for an aerial image B that matches the terrestrial image A1,...,AN, and identifies the location information P corresponding to the aerial image B as the photography location. In other words, since the position DB6 associates the aerial image B with the location information P, the photography location of the terrestrial images A1,...,AN is indicated by the location information P associated with the aerial image B that matches the terrestrial image A1,...,AN.

[0128] (Filming location estimation device 5) Next, the shooting location estimation device 5 shown in Fig. 15 will be described in detail. Fig. 17 is a block diagram showing an example of the configuration of the shooting location estimation device 5 according to an embodiment of the present invention. This shooting location estimation device 5 includes image matching means 40 and shooting location estimation means 41.

[0129] The image matching means 40 corresponds to the image matching device 1 shown in Fig. 1. The image matching means 40 receives the ground images A1,...,AN and the aerial image Bq from the photography location estimation means 41, generates a determination result indicating whether or not the ground images A1,...,AN match the aerial image Bq, and outputs the determination result to the photography location estimation means 41. The processing of the image matching means 40 is similar to that of the image matching device 1, and therefore a detailed description thereof will be omitted here.

[0130] The image matching means 40 generates and outputs a determination result including the similarity score S used when determining whether the ground images A1, . . . , AN match the aerial image Bq.

[0131] When the photography location estimation means 41 receives a query including ground images A1,...,AN from the user terminal 4, it receives aerial images B1,...,BQ and corresponding position information P1,...,PQ from the position DB6.

[0132] The photography location estimation means 41 outputs the ground images A1,...,AN and the aerial image Bq included in the query to the image matching means 40, and inputs the determination result from the image matching means 40. Then, based on the determination result, the photography location estimation means 41 identifies one or more aerial images B that match the ground images A1,...,AN, and identifies location information P corresponding to the aerial images B.

[0133] The photography location estimation means 41 generates a response including the aerial image B and the location information P, and transmits the response to the user terminal 4.

[0134] 18 is a flowchart showing an example of processing by the photography location estimating means 41. When the photography location estimating means 41 receives a query including ground images A1,...,AN from the user terminal 4 (step S1801), it extracts the ground images A1,...,AN from the query and sets 1 to the number q (step S1802: q←1).

[0135] The aerial photography location estimation means 41 receives (reads out) the aerial image Bq with number q and the corresponding position information Pq from the position DB 6 (step S1803), and outputs the ground images A1, ···, AN and the aerial image Bq to the image matching means 40 (step S1804).

[0136] Thereby, the image matching means 40 calculates the similarity score S between the ground images A1, ···, AN and the aerial image Bq, performs matching, and generates a determination result.

[0137] The aerial photography location estimation means 41 inputs the determination result from the image matching means 40 (step S1805), and determines whether the number q is the same as the number Q of aerial images B (q = Q?) (step S1806).

[0138] In step S1806, when the aerial photography location estimation means 41 determines that the number q is not the same as the number Q of aerial images B (step S1806: N: q < Q), it adds 1 to the number q (step S1807: q ← q + 1) and proceeds to step S1803.

[0139] On the other hand, in step S1806, when the aerial photography location estimation means 41 determines that the number q is the same as the number Q of aerial images B (step S1806: Y: q = Q), assuming that the matching determination between the ground image A and the aerial images Bq (q = 1, ···, Q) by the image matching means 40 is completed, it proceeds to step S1808.

[0140] Thereby, the aerial photography location estimation means 41 has obtained Q determination results (including the similarity score S) indicating whether the determination results for each aerial image Bq, that is, whether the ground images A1, ···, AN and the aerial images Bq (q = 1, ···, Q) match.

[0141] The shooting location estimation means 41 identifies one or more aerial images B and location information P that match the ground images A1,...,AN based on the judgment results for each aerial image Bq, and estimates the location information P as the shooting location of the ground images A1,...,AN (step S1808).

[0142] For example, the photography location estimation means 41 identifies one aerial image B from among the aerial images B1,...,BQ that has the highest similarity score S included in the determination result, and identifies location information P corresponding to the aerial image B. The location information P identified in this way is treated as the photography location of the ground image A.

[0143] Furthermore, for example, the photography location estimation means 41 identifies a predetermined number of aerial images B from among the aerial images B1,...,BQ in descending order of the similarity score S included in the determination result, and identifies a number of pieces of location information P corresponding to the plurality of aerial images B. The plurality of pieces of location information P identified in this way will be treated as the photography locations of the ground image A.

[0144] The photography location estimation means 41 generates a response including the identified aerial image B and location information P, and transmits the response to the user terminal 4 (step S1809). In the example of identifying a predetermined number of aerial images B in descending order of similarity score S described above, the photography location estimation means 41 generates a response in a ranking format including combinations of aerial images B and location information P in descending order of similarity score S, and transmits the response to the user terminal 4.

[0145] In steps S1803 to S1807, the photography location estimating means 41 increments the number q while inputting the determination result from the image matching means 40. In step S1805, if the determination result indicates that the ground image A and the aerial image Bq match, the photography location estimating means 41 may proceed to step S1808.

[0146] Then, in step S1808, the photography location estimation means 41 identifies the aerial image Bq that is the first to match the ground image A with the aerial image Bq as the aerial image B, and estimates the corresponding location information P as the photography location of the ground image A. In step S1809, a response including the aerial image B and location information P identified in this way is transmitted to the user terminal 4.

[0147] As described above, according to the shooting location estimation device 5 of an embodiment of the present invention, the shooting location estimation means 41 receives a query including ground images A1,...,AN from the user terminal 4, and receives aerial images B1,...,BQ and corresponding location information P1,...,PQ from the location DB6.

[0148] The image matching means 40 functioning as the image matching device 1 determines whether or not the ground images A1, . . . , AN match with each of the aerial images B1, . . . , BQ.

[0149] Based on the judgment result, the shooting location estimation means 41 identifies an aerial image B and location information P that match the ground image A, thereby estimating the location information P as the shooting location of the ground image A, and sends a response including the aerial image B and location information P to the user terminal 4.

[0150] In this way, even for ground image A with a narrow viewing angle, by utilizing multiple ground images A1,...,AN taken at the same location from different viewpoints, the matching accuracy between ground image A and aerial image B can be improved compared to when using a single ground image A. Furthermore, by using position information P corresponding to aerial image B, the shooting location of ground image A can be estimated with high accuracy.

[0151] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0152] 15 is configured to include a user terminal 4, a shooting location estimation device 5, and a position DB 6, but the shooting location estimation device 5 may be configured to include the position DB 6. In this case, the shooting location estimation system 3 is configured to include a user terminal 4 and a shooting location estimation device 5.

[0153] It should be noted that a normal computer can be used as the hardware configuration of the image matching device 1, the image matching model training devices 2-1, 2-2, and 2-3, and the photography location estimation device 5 according to the embodiment of the present invention.

[0154] The image matching device 1, the image matching model training devices 2-1, 2-2, and 2-3, and the shooting location estimation device 5 are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, and the like.

[0155] The functions of the feature extraction means 10, which has a ground image feature extraction means 11 and an aerial image feature extraction means 12, and the matching determination means 20 provided in the image matching device 1 are each realized by having a CPU execute a program that describes these functions.

[0156] In addition, the functions of the single image generation means 30, ground NN 31-1, aerial photography NN 32-1, and update means 33 provided in the image matching model training device 2-1 are each realized by having the CPU execute a program that describes these functions.

[0157] In addition, the functions of the ground NN 31-2, the aerial NN 32-2, the memory 34, the synthesis means 35, and the update means 36 provided in the image matching model training device 2-2 are each realized by having the CPU execute a program that describes these functions.

[0158] Furthermore, the functions of the ground NN 31-3, the aerial NN 32-3, and the update means 37 provided in the image matching model training device 2-3 are also realized by causing the CPU to execute a program describing these functions.

[0159] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (HDD, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory (SSD, etc.) and distributed, or can be transmitted and received via a network. [Explanation of symbols]

[0160] 1. Image matching device 2-1, 2-2, 2-3 Image matching model training device 3. Shooting location estimation system 4. User terminal 5. Shooting location estimation device 6 Position DB 7 Network 10 Feature extraction method 11 Ground image feature extraction method 12. Aerial image feature extraction method 13,13-1,13-2,13-3,31-1,31-2,31-3 Ground NN 14,32-1,32-2,32-3 Aerial photography NN 15,30 Single image generation means 16,34 memory 17,35 Synthesis means 20 Matching Determination Method 21 Score calculation method 22 Judgment means 33,36,37 Update means 40 Image Matching Method 41 Method of estimating the location of the photo A,A1,···,AN,A',A1',···,AM' Ground image N,M Number of ground images AA,AA' Single ground image B,B',B1,···,BQ,Bq Aerial image Q Number of aerial images AV,A1V,···,ANV,AV' ,A1V',···,AMV' ground feature vectors BV,BV' Aerial image feature vector S Similarity Score P,P1,...,PQ,Pq Location information q number

Claims

1. An image matching device for determining a match between a ground image and an aerial image, a ground image feature extraction means for extracting features of a plurality of ground images based on the plurality of ground images taken at the same location from different viewpoints, and determining a ground feature vector AV; an aerial image feature extraction means for extracting features of one aerial image based on the one aerial image and determining an aerial feature vector BV; a matching determination means for determining the matching based on the ground feature vector AV obtained by the ground image feature extraction means and the aerial feature vector BV obtained by the aerial image feature extraction means, and generating a determination result; The ground image feature extraction means a single image generating means for generating a single ground image based on the plurality of ground images; a ground NN (neural network) that determines the ground feature vector AV from the single ground image generated by the single image generation means, The aerial image feature extraction means An image matching device comprising an aerial photography NN that determines the aerial photography feature vector BV from the one aerial photography image.

2. An image matching device for determining a match between a ground image and an aerial image, a ground image feature extraction means for extracting features of a plurality of ground images based on the plurality of ground images taken at the same location from different viewpoints, and determining a ground feature vector AV; an aerial image feature extraction means for extracting features of one aerial image based on the one aerial image and determining an aerial feature vector BV; a matching determination means for determining the matching based on the ground feature vector AV obtained by the ground image feature extraction means and the aerial feature vector BV obtained by the aerial image feature extraction means, and generating a determination result; The ground image feature extraction means a ground neural network (NN) that determines a ground feature vector from each of the plurality of ground images; a synthesis means for generating the ground feature vector AV by synthesizing the ground feature vectors for each of the plurality of ground images obtained by the ground NN, The aerial image feature extraction means An image matching device comprising an aerial photography NN that determines the aerial photography feature vector BV from the one aerial photography image.

3. 2. An image matching model training device for training a model for generating parameters of a ground-based NN (Neural Network) and parameters of an aerial NN used in the image matching device according to claim 1, a single image generating means for generating a single ground image based on a plurality of ground images taken at the same location from different viewpoints; a ground neural network (NN) for determining a ground feature vector AV' indicating the features of the single ground image generated by the single image generating means; an aerial photography neural network that calculates an aerial photography feature vector BV' that indicates the features of a single aerial photography image; an update means for calculating a loss based on the ground feature vector AV' calculated by the ground neural network and the aerial photography feature vector BV' calculated by the aerial neural network, and updating the parameters of the ground neural network and the aerial neural network based on the loss; An image matching model training device comprising:

4. 3. An image matching model training device for training a model for generating parameters of a ground-based NN (Neural Network) and parameters of an aerial NN used in the image matching device according to claim 2, a ground neural network that calculates a ground feature vector that indicates the characteristics of each of a plurality of ground images taken at the same location from different viewpoints; a synthesis means for generating a ground feature vector AV′ by synthesizing the ground feature vectors for each of the plurality of ground images obtained by the ground NN; an aerial photography neural network that calculates an aerial photography feature vector BV' that indicates the features of a single aerial photography image; an update means for calculating a loss based on the ground feature vector AV' generated by the synthesis means and the aerial photography feature vector BV' calculated by the aerial photography neural network, and updating parameters of the ground neural network and the aerial photography neural network based on the loss; An image matching model training device comprising:

5. 1. A photography location estimation device that determines a match between a ground image and an aerial image and estimates a photography location of the ground image based on a result of the determination, The image matching device according to claim 1 or 2; a photography location estimation means for inputting a plurality of ground images taken at the same location from different viewpoints, identifying an aerial image that matches the plurality of ground images based on the determination result generated by the image matching device, and estimating location information of the aerial image as the photography location of the ground image, The shooting location estimation means A single aerial photograph and the location information of the single aerial photograph are treated as a set, and each of the multiple sets of data is read out from a location DB (database) that is configured with multiple sets of data; For each of the plurality of sets of data, the determination result for the plurality of ground images and the one aerial image in the set of data is obtained from the image matching device; A shooting location estimation device characterized by identifying an aerial image that matches the multiple ground images based on the judgment result, and extracting the location information corresponding to the identified aerial image included in the set of data, thereby estimating the location information as the shooting location.

6. A program for causing a computer to function as the image matching device according to claim 1 or 2.

7. A program for causing a computer to function as the image matching model training device according to claim 3 or 4.

Images