IMAGE MATCHING DEVICE, IMAGE MATCHING METHOD, AND PROGRAM

Abstract

The image matching device (2000) acquires a ground image (20), an aerial image (30), and class information (40). The class information (40) indicates the distribution of classes of objects in the ground image (20), the aerial image (30), or both. The image matching device (2000) calculates a ground image feature (60) by extracting a feature from the ground image (20), calculates an aerial image feature (70) by extracting a feature from the aerial image (30), and calculates a class feature (80) by extracting a feature from the class information (40). The image matching device (2000) determines whether the ground image (20) and the aerial image (30) match each other based on the ground image feature (60), the aerial image feature (70), and the class feature (80).

Description

[Technical field]

[0001] The present disclosure generally relates to an image matching apparatus, an image matching method, and a non-transitory computer-readable storage medium. [Background technology]

[0002] A computer system that performs ground-to-aerial cross-view matching (matching of ground images and aerial images) has been developed. For example, Non-Patent Document 1 discloses a system that includes a pair of CNNs (Convolutional Neural Networks) for matching ground images with a field of view of less than 360 degrees and aerial images. Specifically, one of the CNNs acquires a set of a ground image and a direction map indicating the direction (azimuth and altitude) of each position captured in the ground image, and extracts features therefrom. The other CNN acquires a set of an aerial image and a direction map indicating the direction (azimuth and altitude) of each position captured in the aerial image, and extracts features therefrom. The system then determines whether the ground image matches the aerial image based on the extracted features. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Liu Liu and Hongdong Li, “Lending Orientation to Neural Networks for Cross-view Geo-localization”, [online], March 29, 2019, arXiv, [Retrieved September 24, 2021],<URL:https: / / arxiv.org / pdf / 1903.12351> [Non-Patent Document 2] Jeffrey Pennington, Richard Socher, and Christopher D. Manning, "GloVe: Global Vectors for Word Representation," Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, October 25, 2014. [Non-Patent Document 3] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean, "Distributed Representations of Words and Phrases and their Compositionality," [online], October 16, 2013, [Retrieved March 10, 2022],<arXiv,https: / / arxiv.org / pdf / 1310.4546.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] In Non-Patent Document 1, extraction of features from images other than RGB images or their direction maps is not taken into consideration. The present disclosure aims to provide a new technology for determining whether a ground image and an aerial image match each other. [Means for solving the problem]

[0005] The present disclosure provides at least one memory configured to store instructions and at least one processor configured to: acquire a ground image, an aerial image, and class information indicating a distribution of classes of objects in the ground image, the aerial image, or both, calculate ground image features by extracting features from the ground image, calculate aerial image features by extracting features from the aerial image, calculate class features by extracting features from the class information, and determine whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

[0006] The present disclosure provides a computer-implemented image matching method, the image matching method including the steps of acquiring a ground image, an aerial image, and class information indicating a distribution of classes of objects in the ground image, the aerial image, or both, calculating ground image features by extracting features from the ground image, calculating aerial image features by extracting features from the aerial image, calculating class features by extracting features from the class information, and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

[0007] The present disclosure provides a non-transitory computer-readable storage medium storing a program, the program causing a computer to execute the steps of acquiring a ground image, an aerial image, and class information indicating a distribution of classes of objects in the ground image, the aerial image, or both, calculating a ground image feature by extracting a feature from the ground image, calculating an aerial image feature by extracting a feature from the aerial image, calculating a class feature by extracting a feature from the class information, and determining whether the ground image and the aerial image match each other based on the ground image feature, the aerial image feature, and the class feature.

[0008] According to the present disclosure, a novel technique can be provided for determining whether a ground image and an aerial image match each other. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview of an image matching device. [Diagram 2] 1A and 1B are diagrams illustrating an example of a ground image and an aerial image. [Diagram 3] 1 is a block diagram showing an example of a functional configuration of an image matching device; [Figure 4] 1 is a block diagram showing an example of a hardware configuration of an image matching device. [Diagram 5] 4 is a flowchart illustrating an exemplary flow of a process performed by the image matching device. [Figure 6] FIG. 1 illustrates a geolocalization system including an image matching device. [Figure 7] FIG. 2 is a diagram showing an example of a portion of the configuration of an image matching device for comparing ground features with aerial features. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An embodiment according to the present disclosure will be described below with reference to the drawings. The same elements are assigned the same reference numerals throughout the drawings, and redundant descriptions are omitted as necessary. In addition, unless otherwise specified, predetermined information (e.g., a predetermined value or a predetermined threshold value) is stored in advance in a storage device accessible by a computer that uses the information.

[0011] EMBODIMENT 1 <Summary> Fig. 1 shows an overview of an image matching device 2000 according to the first embodiment. The image matching device 2000 functions as a classifier that matches a ground image 20 with an aerial image 30 (so-called ground-to-aerial cross-view matching). Fig. 2 shows an example of the ground image 20 and the aerial image 30.

[0012] The ground image 20 is a digital image (e.g., an RGB image of the ground view) that includes a ground view of a location. The ground image 20 is generated by a ground camera, for example, held by a pedestrian or mounted on a vehicle. The ground image may be panoramic (360 degree field of view) or may have a limited field of view (less than 360).

[0013] The aerial image 30 is an image that includes a planar view of a location (e.g., an RGB image of a view from above). For example, the aerial image 30 may be generated by an aerial camera mounted on a drone, an airplane, or a satellite.

[0014] In addition to the ground images 20 and the aerial images 30, the image matching device 2000 uses class information 40, which indicates the distribution of classes of objects (e.g., "buildings", "roads", "sidewalks", etc.) in the ground images 20, the aerial images 30, or both. The class information 40 may include a segmented image, each pixel of which represents the class of an object imaged in a corresponding region (i.e., one or more corresponding pixels) of the original image (i.e., the ground image 20 or the aerial image 30).

[0015] It should be noted that the data included in the class information 40 is not limited to segmented images. Additionally or alternatively, the class information 40 may include a keyword matrix, where each element of the keyword matrix represents a class of an object captured in a corresponding region (i.e., one or more corresponding pixels) of the original image (i.e., the terrestrial image 20 or the aerial image 30).

[0016] Although details will be described later, class information 40 may be generated within image matching device 2000 instead of being obtained from outside image matching device 2000.

[0017] The image matching device 2000 extracts features from each of the acquired data, i.e., the ground image 20, the aerial image 30, and the class information 40. Specifically, the image matching device 2000 extracts features from the ground image 20 to generate ground image features 60. The image matching device 2000 extracts features from the aerial image 30 to generate aerial image features 70. The image matching device 2000 extracts features from the class information 40 to generate class features 80.

[0018] When the class information 40 includes data representing the distribution of classes in the ground image 20, the class features 80 include features called “ground class features” extracted from the data. The ground class features represent the characteristics of the distribution of classes in the ground image 20. When the class information 40 includes data representing the distribution of classes in the aerial image 30, the class features 80 include features called “aerial class features” extracted from the data. The aerial class features represent the characteristics of the distribution of classes in the aerial image 30.

[0019] After extracting the above-mentioned features, the image matching device 2000 uses the ground image features 60, the aerial image features 70, and the class features 80 to determine whether the ground image 20 and the aerial image 30 match each other.

[0020] <Examples of effects> According to the image matching device 2000 of the first embodiment, not only the feature amounts extracted from the ground image 20 and the aerial image 30 but also the feature amounts extracted from the class information 40 are used to determine whether the ground image 20 and the aerial image 30 match each other. By using the feature amounts extracted from the class information 40, i.e., the class feature amounts 80, it is possible to compare the ground image 20 and the aerial image 30 based on not only the similarity in appearance but also the similarity in class distribution. This allows the image matching device 2000 to perform ground-to-aerial cross-view matching more accurately than when the class feature amounts 80 are not used.

[0021] The image matching device 2000 will now be described in more detail.

[0022] <Example of functional configuration> 3 is a block diagram showing an example of a functional configuration of the image matching device 2000. The image matching device 2000 includes an acquisition unit 2020, a ground image feature amount extraction unit 2040, an aerial image feature amount extraction unit 2060, a class feature amount extraction unit 2080, and a determination unit 2100.

[0023] The acquisition unit 2020 acquires ground images 20, aerial images 30, and class information 40. The ground image feature extraction unit 2040 extracts features from the ground images 20, thereby obtaining ground image features 60. The aerial image feature extraction unit 2060 extracts features from the aerial image 30, thereby obtaining aerial image features 70. The class feature extraction unit 2080 extracts features from the class information 40, thereby obtaining class features 80. The determination unit 2100 uses the ground image features 60, the aerial image features 70, and the class features 80 to determine whether the ground images 20 and the aerial images 30 match each other.

[0024] <Example of hardware configuration> Image matching device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer manufactured for realizing image matching device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0025] Image matching device 2000 may be realized by installing an application in one or more computers. The application is realized by a program for causing one or more computers to function as image matching device 2000. In other words, the program is an implementation of the functional components of image matching device 2000.

[0026] Fig. 4 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes the image matching device 2000. In Fig. 4, the computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0027] The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage element such as a random access memory (RAM) or a read only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and a peripheral device (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0028] The above-mentioned programs may be stored in the storage device 1080. The processor 1040 reads the programs from the storage device 1080 and executes the programs to realize the respective functional components of the image matching device 2000.

[0029] The hardware configuration of the computer 1000 is not limited to that shown in Fig. 4. For example, as described above, the image matching device 2000 may be realized by a plurality of computers. In this case, the computers may be connected to each other via a network.

[0030] <Processing flow> 5 is a flowchart showing an exemplary flow of processing performed by the image matching device 2000. The acquisition unit 2020 acquires the ground image 20, the aerial image 30, and the class information 40 (S102). The ground image feature amount extraction unit 2040 extracts features from the ground image 20 and calculates the ground image feature amount 60 (S104). The aerial image feature amount extraction unit 2060 extracts features from the aerial image 30 and calculates the aerial image feature amount 70 (S106). The class feature amount extraction unit 2080 extracts features from the class information 40 and calculates the class feature amount 80 (S108). The determination unit 2100 uses the ground image feature amount 60, the aerial image feature amount 70, and the class feature amount 80 to determine whether the ground image 20 and the aerial image 30 match each other (S110).

[0031] <Examples of Applications of the Image Matching Device 2000> There are various possible applications for the image matching device 2000. For example, the image matching device 2000 can be used as part of a system for performing image geolocalization (hereinafter, referred to as a geolocalization system). Image geolocalization is a technology for identifying the location where an input image was taken. The geolocalization system 200 can be implemented in one or more arbitrary computers (such as the one shown in FIG. 4). Here, the geolocalization system is merely an example of an application of the image matching device 2000, and the application of the image matching device 2000 is not limited to being used in a geolocalization system.

[0032] 6 illustrates a geolocalization system 200 including an image matching device 2000. The geolocalization system 200 includes the image matching device 2000 and a location database 300. The location database 300 includes a plurality of aerial images, each having location information attached thereto. An example of the location information may be the Global Positioning System (GPS) coordinates of the location where the corresponding aerial image was taken.

[0033] The geolocalization system 200 receives a query including a ground image and a set of ground class information from a client (e.g., a user terminal). The ground class information is one aspect of class information 40 indicating class distribution in a ground image, such as a segmented image of the ground image 20. The geolocalization system 200 then searches the location database 300 for an aerial image that matches the ground image in the received query, thereby identifying the capture location of the ground image. Specifically, until an aerial image that matches the ground image in the query is detected, the geolocalization system 200 repeatedly executes the process of "obtaining one of the aerial images from the location database 300, inputting the set of the ground image and ground class information, as well as the obtained set of the aerial image and the aerial class information, to the image matching device 2000, and determining whether the output from the image matching device 2000 indicates that the ground image matches the aerial image." Note that the aerial class information is one aspect of class information 40 indicating class distribution in an aerial image, such as a segmented image of the aerial image 30.

[0034] By repeatedly performing the above process, the geolocalization system 200 can find an aerial image that includes the capture location of the ground image. Because the detected aerial image is associated with location information such as GPS coordinates, the geolocalization system 200 can determine that the capture location of the ground image is the location indicated by the location information associated with the aerial image that matches the ground image.

[0035] 6, both the class distribution in the ground image 20 and the class distribution in the aerial image 30 are considered. However, in another embodiment, the image matching device 2000 may use either one of them in some embodiments.

[0036] 6, the ground class information is included in the query. However, in another implementation, the geolocalization system 200 may receive a query that does not include the ground class information. In this case, the geolocalization system 200 may generate the ground class information from the ground imagery 20, for example, by performing semantic segmentation on the ground imagery 20.

[0037] 6, the aerial class information is stored in the location database 300 in association with the aerial image 30. However, in another embodiment, the location database 300 may not store the aerial class information. In this case, the geo-localization system 200 may generate the aerial class information from the aerial image 30, for example, by performing semantic segmentation on the aerial image 30.

[0038] It should be noted that terrestrial and aerial imagery may be used in the opposite manner in geolocalization system 200, where location database 300 stores a number of terrestrial images, each with location information attached thereto, and geolocalization system 200 receives a query containing an aerial image and searches location database 300 for terrestrial images that match the aerial image in the query, thereby identifying the location of where the aerial image was taken.

[0039] <About class information> The class information 40 may include one or more pieces of information that represent the distribution of classes of objects in the terrestrial imagery 20, the aerial imagery 30, or both. As mentioned above, the class information 40 may include a segmented image or a keyword matrix. Examples of these pieces of class information 40 are described below.

[0040] <<Segmented image>> A segmented image is an image in which each pixel represents by its color (i.e., pixel value) the class of the object captured in the corresponding region in the original image from which the segmented image was generated. Assume that there are five predefined classes: "sky", "building", "road", "sidewalk" and "other", and the colors "yellow", "blue", "green", "red" and "gray" are assigned to these classes in order. In this case, for example, if sky is captured in some pixels in the original image, the corresponding pixels in the segmented image are filled with yellow. Similarly, if a building is captured in some pixels in the original image, the corresponding pixels in the segmented image are filled with blue.

[0041] It should be noted that the dimensions (i.e. width and height) of the segmented image may be the same as those of the original image, or may differ from those of the original image, in which case a pixel of the segmented image indicates the class of the object imaged at the corresponding pixel in the original image.

[0042] In the latter case, a pixel of the segmented image indicates the class of the object imaged at the corresponding pixels in the original image. For example, a segmented image may be generated such that each pixel corresponds to a region containing NxM pixels in the original image, where N, M, or both are greater than 1.

[0043] The segmented image may be generated by performing semantic segmentation on the original image. There are various well-known methods for performing semantic segmentation on an image, any of which may be applied to generate the segmented image of the terrestrial image 20, the aerial image 30, or both. Furthermore, if the dimensions of the segmented image are smaller than those of the original image, the image matching device 2000 may perform subsampling (e.g., average pooling or max pooling) on ​​the segmented image to reduce the dimensions of the segmented image. Note that it is not necessary for the image matching device 2000 to generate the segmented image.

[0044] <<Keyword matrix>> The keyword matrix is ​​a matrix in which each element indicates a vector named "keyword vector", and the keyword vector indicates the class of the object captured in the corresponding region in the original image from which the keyword matrix was generated. It is assumed that there are five predetermined classes, "sky", "building", "road", "sidewalk" and "other", and these classes are assigned one-hot vectors (0,0,0,0,1), (0,0,0,1,0), (0,0,1,0,0), (0,1,0,0,0), (1,0,0,0,0), in that order. In this case, for example, if the sky is captured in some pixels in the original image, the corresponding elements of the keyword matrix indicate (0,0,0,0,1). Similarly, if a building is captured in some pixels in the original image, the corresponding elements of the keyword matrix indicate (0,0,0,1,0).

[0045] Note that, like the segmented image, the width and height of the keyword matrix may be the same as or different from the width and height of the original image, in which case the elements of the keyword matrix represent keyword vectors that represent the class of the object captured at the corresponding pixel in the original image.

[0046] In the latter case, the elements of the keyword matrix represent keyword vectors that represent classes of objects captured at corresponding pixels in the source image. For example, a keyword matrix may be generated such that each element corresponds to a region of NxM pixels in the source image, where N, M, or both are greater than 1.

[0047] The keyword vector is not limited to a one-hot vector. For example, the set of keyword vectors is defined using knowledge about classes (e.g., similarity between classes). The knowledge can be embedded in the distance between keyword vectors. For example, the set of keyword vectors is defined such that the similarity between classes is represented by the distance between the keyword vectors of those classes. Conceptually, the more similar two classes are, the shorter the distance between their keyword vectors will be. Such a set of keywords can be defined by the technology disclosed in, for example, Non-Patent Document 2 or Non-Patent Document 3.

[0048] To generate the keyword matrix, the image matching device 2000 first identifies the class of the object captured at each pixel of the original image. This identification may be performed using semantic segmentation. The image matching device 2000 then assigns to each pixel a keyword vector representing the class of the object captured at that pixel. Furthermore, if the dimensions of the keyword matrix are smaller than those of the original image, the image matching device 2000 may perform subsampling (e.g., average pooling or max pooling) on ​​the keyword matrix to reduce the dimensions of the keyword matrix. Note that the image matching device 2000 does not necessarily have to generate the keyword matrix.

[0049] <Data Acquisition: S102> The acquisition unit 2020 acquires the ground image 20, the aerial image 30, and the class information 40 (S102). There are various methods for acquiring these data. In some embodiments, the acquisition unit 2020 may receive the data transmitted from another computer. In other embodiments, the acquisition unit 2020 may acquire the data from an accessible storage device.

[0050] Regarding the class information 40, the image matching device 2000 may generate the class information 40 based on the ground image 20, the aerial image 30, or both, and the acquisition unit 2020 may acquire the class information 40 generated inside the image matching device 2000. A specific method for generating the class information 40 has been described above.

[0051] <Extraction of 60 ground image features: S104> The ground image feature extraction unit 2040 extracts features from the ground image 20 to calculate the ground image feature 60 (S104). There are various methods for extracting features from an image, and any of these methods may be adopted to configure the ground image feature extraction unit 2040. For example, the ground image feature extraction unit 2040 may be realized by a machine learning-based model such as a neural network. More specifically, a feature extraction layer of a Convolutional Neural Network (CNN) may be used to configure the ground image feature extraction unit 2040.

[0052] <Extraction of 70 aerial image features: S106> The aerial image feature extraction unit 2060 extracts features from the aerial image 30 and calculates the aerial image feature 70 (S106). As described above, there are various methods for extracting features from an image. Therefore, any of these may be adopted to configure the aerial image feature extraction unit 2060. For example, the aerial image feature extraction unit 2060 may be realized by a machine learning-based model such as a neural network. More specifically, a feature extraction layer of a Convolutional Neural Network (CNN) may be used to configure the aerial image feature extraction unit 2060.

[0053] <Extraction of class feature 80: S108> The class feature extraction unit 2080 extracts features from the class information 40 to calculate class features 80 (S108). The features of the class information 40 may be extracted from the class information 40 in a manner similar to the manner in which the ground image features 60 are extracted from the ground images 20, or the manner in which the aerial image features 70 are extracted from the aerial images 30. For example, the class feature extraction unit 2080 may be realized by a machine learning-based model such as a neural network.

[0054] When the class information 40 is configured to include a plurality of types of information, such as a segmented image of the ground image 20 and a keyword matrix, the class information 40 includes a feature extraction unit for each of these data.

[0055] <Matching of ground image 20 and aerial image 30: S110> The determination unit 2100 uses the ground image feature 60, the aerial image feature 70, and the class feature 80 to determine whether the ground image 20 and the aerial image 30 match each other (S110). Specifically, the determination unit 2100 makes the determination by comparing a feature related to the ground image 20 (called a "ground feature") with a feature related to the aerial image 30 (called an "aerial feature"). If the class feature 80 includes a ground class feature, the determination unit 2100 calculates a combined feature of the ground image feature 60 and the ground class feature, and uses the calculated feature as the ground feature. On the other hand, if the class feature 80 does not include a ground class feature, the determination unit 2100 uses the ground image feature 60 as the ground feature. Similarly, if the class feature 80 includes an aerial class feature, the determination unit 2100 calculates a combined feature of the aerial image feature 70 and the aerial class feature, and uses the calculated feature as the aerial feature. On the other hand, if the class features 80 do not include the aerial photography class features, the determining unit 2100 uses the aerial photography image features 70 as the aerial photography features.

[0056] 7 shows an example of a part of the configuration of an image matching device 2000 for comparing ground features with aerial features. In this example, the class information 40 includes segmented images and keyword matrices for both the ground image 20 and the aerial image 30.

[0057] The image matching device 2000 includes networks 100, 110, 120, 130, 140, and 150. The network 100 included in the ground image feature extraction unit 2040 extracts a ground image feature 60 from the ground image 20. The network 110 included in the class feature extraction unit 2080 extracts a feature 160 from the segmented image 22 of the ground image 20. The network 120 included in the class feature extraction unit 2080 extracts a feature 170 from the keyword matrix 24 of the ground image 20. In this case, the ground class feature includes the features 160 and 170. Next, the ground image feature 60, the feature 160, and the feature 170 are combined with each other to calculate a ground feature 65.

[0058] There are various methods for combining a plurality of features, and by applying any of the methods, the ground image feature 60, the feature 160, and the feature 170 can be combined to calculate the ground feature 65. For example, the features 160 and 170 are concatenated with the ground image feature 60 to calculate the ground feature 65. In another example, the ground image feature 60, the feature 160, and the feature 170 are supplied to a feature extraction unit such as a neural network, and the output from the feature extraction unit is used as the ground feature 65.

[0059] The aerial feature can be calculated in the same manner as the method of calculating the ground feature. Specifically, the network 130 included in the aerial image feature extraction unit 2060 extracts the aerial image feature 70 from the aerial image 30. The network 140 included in the class feature extraction unit 2080 extracts the feature 180 from the segmented image 32 of the aerial image 30. The network 150 included in the class feature extraction unit 2080 extracts the feature 190 from the keyword matrix 34 of the aerial image 30. In this case, the aerial class feature includes the features 180 and 190. Next, the aerial image feature 70, the feature 180, and the feature 190 are combined with each other to calculate the aerial feature 75. Note that the aerial image feature 70, the feature 180, and the feature 190 can be combined in the same manner as the method of combining the ground image feature 60, the feature 160, and the feature 170.

[0060] After combining the features, the determination unit 2100 may calculate a similarity score representing the similarity between the ground feature and the aerial image feature. There are various indices for quantifying the similarity between the features, and the similarity score may be calculated using any one of them. For example, the similarity score may be calculated as one of various types, such as distance (e.g., L2 distance), correlation, cosine similarity, or NN (Neural Network)-based similarity between the ground feature and the aerial image feature. The NN-based similarity is a similarity calculated by a neural network trained to calculate the similarity between two input data (in this case, the ground feature and the aerial image feature).

[0061] The determination unit 2100 determines whether the ground image 20 and the aerial image 30 match each other based on the calculated similarity score. Conceptually, the higher the similarity between the ground feature amount and the aerial feature amount, the higher the probability that the ground image 20 and the aerial image 30 match each other. Therefore, for example, the determination unit 2100 determines whether the similarity score is equal to or greater than a predetermined threshold. If the similarity score is equal to or greater than the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 match each other. On the other hand, if the similarity score is less than the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 do not match each other.

[0062] In the above case, it is assumed that the higher the similarity between the ground feature and the aerial photography feature, the larger the similarity score. Therefore, when an index (e.g., distance) is used that has a smaller value calculated for the ground feature and the aerial photography feature as the similarity between them is higher, the similarity score can be defined as the reciprocal of the value calculated for the ground feature and the aerial photography feature.

[0063] In another example, when the similarity score decreases as the similarity between the ground feature amount and the aerial image feature amount increases, the determination unit 2100 may determine whether the similarity score is equal to or smaller than a predetermined threshold. When the similarity score is equal to or smaller than the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 match each other. On the other hand, when the similarity score exceeds the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 do not match each other.

[0064] <Output of Image Matching Device 2000> The image matching device 2000 may output information indicating the result of the determination (hereinafter, output information). For example, the output information may indicate whether the ground image 20 and the aerial image 30 match each other. Furthermore, as described with reference to FIG. 6, the output information may further include location information indicating the location where the query image (the ground image 20 or the aerial image 30) was captured.

[0065] There are various ways to output the output information. For example, the image matching device 2000 may store the output information in a storage device. In another example, the image matching device 2000 may output the output information to a display device, which may display the contents of the output information. In another example, the image matching device 2000 may output the output information to another computer (such as a computer included in the geolocalization system 200 shown in FIG. 6).

[0066] <Model training> As described above, the image matching device 2000 may include one or more machine learning based models, such as a neural network. For example, as described with reference to Fig. 7, the ground image feature extractor 2040, the aerial image feature extractor 2060, and the class feature extractor 2080 may include neural networks. When the image matching device 2000 is implemented with machine learning based models, the models are trained using a training data set prior to operation of the image matching device 2000.

[0067] In some embodiments, a computer that trains a model (hereinafter, a training device) may repeatedly execute a process of "calculating a loss (e.g., triplet loss or contrastive loss) using a training dataset, and updating trainable parameters of the model based on the calculated loss." The training device may be implemented in the computer 1000 in which the image matching device 2000 is implemented, or in another computer. In the former case, it can be said that the image matching device 2000 also has the function of a training device described later. In the latter case, the training device may be implemented using one or more computers whose hardware configuration can be exemplified in FIG. 4, similar to the image matching device 2000.

[0068] When the triplet loss is used to train the model, the training dataset may include an anchor image, a positive example image, and a negative example image. The positive example image is an image of a different type (ground-view or aerial-view) from the anchor image and matches the anchor image. The negative example image is an image of a different type from the anchor image but the same type as the positive example image and does not match the anchor image. If the training dataset includes ground images as anchor images, the training dataset includes aerial images that match the anchor images as positive example images and another aerial image that does not match the anchor image as negative example images. On the other hand, if the training dataset includes aerial images as anchor images, the training dataset includes ground images that match the anchor images as positive example images and another ground image that does not match the anchor image as negative example images.

[0069] The training data set may also include class information for each of the anchor images, the positive example images, and the negative example images. Specifically, the training data set may include a segmented image and a keyword matrix for each of the anchor images, the positive example images, and the negative example images. However, as mentioned above, the class information may be generated instead of being obtained externally.

[0070] The training device obtains features from the anchor image, the positive example image, the negative example image, and the class information in the training data set using the ground image feature extraction unit 2040, the aerial image feature extraction unit 2060, and the class feature extraction unit 2080. Assume that the image matching device 2000 has the structure shown in FIG. 7. Assume further that the training data set includes a ground image as an anchor image. In this case, the training device inputs the anchor image, the segmented image of the anchor image, and the keyword matrix of the anchor image to the networks 100, 110, and 120, respectively. As a result, the training device obtains ground features of the anchor image, which are combinations of the ground image features of the anchor image, the features of the segmented image of the anchor image, and the features of the keyword matrix of the anchor image.

[0071] Furthermore, the training device inputs the positive example image, the segmented image of the positive example image, and the keyword matrix of the positive example image to networks 130, 140, and 150, respectively. As a result, the training device obtains the aerial image features of the positive example image, which are the combination of the aerial image features of the positive example image, the features of the segmented image of the positive example image, and the features of the keyword matrix of the positive example image.

[0072] Similarly, the training device inputs the negative example image, the segmented image of the negative example image, and the keyword matrix of the negative example image to networks 130, 140, and 150, respectively. As a result, the training device obtains aerial features of the negative example image, which are the combination of the aerial image features of the negative example image, the features of the segmented image of the negative example image, and the features of the keyword matrix of the negative example image.

[0073] The training device calculates a triplet loss based on the ground features of the anchor images, the aerial features of the positive example images, and the aerial features of the negative example images. The training device then updates trainable parameters of the models based on the obtained triplet loss. Note that there are various known methods for updating trainable parameters of one or more machine learning based models based on triplet losses calculated based on the outputs from these models, any one of which may be adopted by the training device.

[0074] Note that triplet loss is just an example of a loss that can be used to train the model; any other type of loss may be used to train the model.

[0075] The program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs). The program may also be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0076] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present invention.

[0077] All or part of the above-described embodiment may be described as, but is not limited to, the following supplementary notes. (Appendix 1) An image matching device, comprising: at least one memory configured to store instructions; Executing the instructions, Acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; Calculating ground image feature amounts by extracting feature amounts from the ground image; Calculating an aerial image feature by extracting a feature from the aerial image; Calculating class features by extracting features from the class information; Based on the ground image feature amount, the aerial image feature amount, and the class feature amount, it is determined whether the ground image and the aerial image match each other. and at least one processor configured to: (Appendix 2) 2. The image matching device of claim 1, wherein the class information includes a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 3) The image matching device of Appendix 1, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 4) 4. The image matching device of claim 3, wherein the keyword vectors are defined such that similarity between classes is represented by the distance between the keyword vectors corresponding to those classes. (Appendix 5) The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between ground features and aerial photography features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; When the class information includes ground class information indicating a distribution of classes of objects in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, An image matching device as described in any one of Appendix 1 to 4, wherein when the class information includes aerial class information indicating a distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information. (Appendix 6) 1. A computer-implemented method for image matching, comprising: acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; A step of calculating a ground image feature by extracting a feature from the ground image; Calculating an aerial image feature by extracting a feature from the aerial image; Calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features. (Appendix 7) 7. The image matching method of claim 6, wherein the class information includes a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 8) 7. The image matching method of claim 6, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 9) 9. The image matching method of claim 8, wherein the keyword vectors are defined such that similarity between classes is represented by the distance between the keyword vectors corresponding to those classes. (Appendix 10) The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between ground features and aerial photography features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; When the class information includes ground class information indicating a distribution of classes of objects in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, An image matching method according to any one of Appendix 6 to 9, wherein when the class information includes aerial class information indicating a distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information. (Appendix 11) A non-transitory computer-readable storage medium for storing a program, On the computer, acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; A step of calculating a ground image feature by extracting a feature from the ground image; Calculating an aerial image feature by extracting a feature from the aerial image; Calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features. (Appendix 12) 12. The storage medium of claim 11, wherein the class information includes a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image. (Appendix 13) 12. The storage medium of claim 11, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 14) 14. The storage medium of claim 13, wherein the keyword vectors are defined such that similarity between classes is represented by the distance between the keyword vectors corresponding to those classes. (Appendix 15) The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between ground features and aerial photography features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; When the class information includes ground class information indicating a distribution of classes of objects in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, A storage medium described in any one of Appendices 11 to 14, wherein when the class information includes aerial class information indicating a distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information. [Explanation of symbols]

[0078] 20 Ground Images 22 Segmented ground image 24 Keyword matrix for ground images 30 Aerial Images 32 Segmented image of aerial image 34 Keyword matrix for aerial images 40 Class Information 60 Ground Image Features 65 Ground Features 70 Aerial Image Features 75 Aerial Feature Values 80 class features 100,110,120,130,140,150 Network 160,170,180,190 features 200 Geolocalization System 300 Location Database 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Image Matching Device 2020 Acquisition Department 2040 Ground image feature extraction unit 2060 Aerial Image Feature Extraction Unit 2080 Class feature extraction unit 2100 Judgment section

Claims

1. An image matching device, comprising: at least one memory configured to store instructions; Executing the instructions, Acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; Calculating ground image feature amounts by extracting feature amounts from the ground image; Calculating an aerial image feature by extracting a feature from the aerial image; Calculating class features by extracting features from the class information; Based on the ground image feature amount, the aerial image feature amount, and the class feature amount, it is determined whether the ground image and the aerial image match each other. and at least one processor configured to:

2. The image matching apparatus of claim 1 , wherein the class information comprises a segmented image, each pixel of which indicates the class of the object imaged at one or more corresponding pixels in the terrestrial or aerial image.

3. The image matching device of claim 1 , wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image.

4. The image matching apparatus according to claim 3 , wherein the keyword vectors are defined such that a similarity between classes is represented by a distance between the keyword vectors corresponding to those classes.

5. The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between ground features and aerial photography features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; When the class information includes ground class information indicating a distribution of classes of objects in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, 5. The image matching device according to claim 1, wherein when the class information includes aerial class information indicating a distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information.

6. 1. A computer-implemented method for image matching, comprising: acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; A step of calculating a ground image feature by extracting a feature from the ground image; Calculating an aerial image feature by extracting a feature from the aerial image; Calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

7. The method of claim 6 , wherein the class information comprises a segmented image, each pixel of which indicates the class of the object imaged at one or more corresponding pixels in the terrestrial or aerial image.

8. The image matching method of claim 6, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image.

9. The image matching method according to claim 8 , wherein the keyword vectors are defined such that the similarity between classes is represented by the distance between the keyword vectors corresponding to those classes.

10. The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between ground features and aerial photography features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; When the class information includes ground class information indicating a distribution of classes of objects in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, The image matching method according to any one of claims 6 to 9, wherein when the class information includes aerial class information indicating a distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information.

11. A non-transitory computer-readable storage medium for storing a program, On the computer, acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; A step of calculating a ground image feature by extracting a feature from the ground image; Calculating an aerial image feature by extracting a feature from the aerial image; Calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

12. The storage medium of claim 11 , wherein the class information comprises a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the terrestrial or aerial image.

13. 12. The storage medium of claim 11, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image.

14. The storage medium of claim 13 , wherein the keyword vectors are defined such that a similarity between classes is represented by a distance between the keyword vectors corresponding to those classes.

15. The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between ground features and aerial photography features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; When the class information includes ground class information indicating a distribution of classes of objects in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, A storage medium according to any one of claims 11 to 14, wherein when the class information includes aerial class information indicating a distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information.

Images