Image matching device, image matching method and program

Abstract

The image matching device (2000) acquires a ground image (20), an aerial image (30), and class information (40). The class information (40) indicates the distribution of classes of objects in the ground image (20), the aerial image (30), or both. The image matching device (2000) calculates a ground image feature (60) by extracting a feature from the ground image (20), calculates an aerial image feature (70) by extracting a feature from the aerial image (30), and calculates a class feature (80) by extracting a feature from the class information (40). The image matching device (2000) determines whether the ground image (20) and the aerial image (30) match each other based on the ground image feature (60), the aerial image feature (70), and the class feature (80).

Description

[Technical Field]

[0001] The present disclosure generally relates to an image matching device, an image matching method, and a non-transitory computer-readable storage medium. [Background technology]

[0002] Computer systems for ground-to-aerial cross-view matching (matching of ground images and aerial images) have been developed. For example, Non-Patent Document 1 discloses a system that includes a pair of convolutional neural networks (CNNs) for matching ground-view images with a field of view of less than 360 degrees and aerial-view images. Specifically, one CNN acquires a set of ground images and a direction map indicating the direction (azimuth and altitude) of each location captured in the ground images, and extracts features from them. The other CNN acquires a set of aerial images and a direction map indicating the direction (azimuth and altitude) of each location captured in the aerial images, and extracts features from them. The system then determines whether the ground image matches the aerial image based on the extracted features. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Liu Liu and Hongdong Li, “Lending Orientation to Neural Networks for Cross-view Geo-localization”, [online], March 29, 2019, arXiv, [Retrieved September 24, 2021],<URL:https: / / arxiv.org / pdf / 1903.12351> [Non-patent document 2] Jeffrey Pennington, Richard Socher, and Christopher D. Manning, "GloVe: Global Vectors for Word Representation," Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, October 25, 2014. [Non-patent document 3] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean, "Distributed Representations of Words and Phrases and Their Compositionality," [online], October 16, 2013, [retrieved March 10, 2022].<arXiv,https: / / arxiv.org / pdf / 1310.4546.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] Non-Patent Document 1 does not consider extracting features from images other than RGB images or their orientation maps. The purpose of this disclosure is to provide a novel technique for determining whether a ground image and an aerial image match each other. [Means for solving the problem]

[0005] The present disclosure provides at least one memory configured to store instructions and at least one processor configured to acquire a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both, calculate ground image features by extracting features from the ground image, calculate aerial image features by extracting features from the aerial image, calculate class features by extracting features from the class information, and determine whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

[0006] The present disclosure provides a computer-implemented image matching method, which includes the steps of acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both, calculating ground image features by extracting features from the ground image, calculating aerial image features by extracting features from the aerial image, calculating class features by extracting features from the class information, and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

[0007] The present disclosure provides a non-transitory computer-readable storage medium storing a program that causes a computer to execute the steps of acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both, calculating ground image features by extracting features from the ground image, calculating aerial image features by extracting features from the aerial image, calculating class features by extracting features from the class information, and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

[0008] According to the present disclosure, a novel technique can be provided for determining whether a ground image and an aerial image match each other. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an image matching device. [Figure 2] 1A and 1B are diagrams illustrating examples of a ground image and an aerial image. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image matching device. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of the image matching device. [Figure 5] 10 is a flowchart illustrating an exemplary flow of processing performed by the image matching device. [Figure 6] FIG. 1 illustrates a geolocalization system including an image matching device. [Figure 7] FIG. 1 is a diagram illustrating an example of a portion of the configuration of an image matching device for comparing ground features with aerial features. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments according to the present disclosure will be described below with reference to the drawings. The same elements are assigned the same reference numerals throughout the drawings, and redundant descriptions will be omitted as necessary. Furthermore, unless otherwise specified, predetermined information (e.g., predetermined values ​​or predetermined threshold values) is pre-stored in a storage device accessible by a computer that uses the information.

[0011] Embodiment 1 <Summary> Fig. 1 shows an overview of an image matching device 2000 according to the first embodiment. The image matching device 2000 functions as a classifier that matches a ground image 20 with an aerial image 30 (so-called ground-to-aerial cross-view matching). Fig. 2 shows an example of the ground image 20 and the aerial image 30.

[0012] The ground image 20 is a digital image (e.g., an RGB image of the ground scene) containing a ground view of a location. The ground image 20 is generated by a ground camera, for example, held by a pedestrian or mounted on a vehicle. The ground image may be panoramic (360 degree field of view) or may have a limited field of view (less than 360 degrees).

[0013] The aerial image 30 is an image containing a planar view of a location (e.g., an RGB image of a view from above). For example, the aerial image 30 may be generated by an aerial camera mounted on a drone, an airplane, or a satellite.

[0014] In addition to the ground image 20 and the aerial image 30, the image matching device 2000 uses class information 40, which indicates the distribution of object classes (e.g., "buildings," "roads," "sidewalks," etc.) in the ground image 20, the aerial image 30, or both. The class information 40 may include a segmented image, each pixel of which represents the class of the object imaged in the corresponding region (i.e., one or more corresponding pixels) of the original image (i.e., the ground image 20 or the aerial image 30).

[0015] It should be noted that the data included in the class information 40 is not limited to segmented images. Additionally or alternatively, the class information 40 may include a keyword matrix, each element of which represents the class of an object captured in a corresponding region (i.e., one or more corresponding pixels) of the original image (i.e., the terrestrial image 20 or the aerial image 30).

[0016] As will be described in detail later, class information 40 may be generated within image matching device 2000 instead of being acquired from outside image matching device 2000.

[0017] The image matching device 2000 extracts features from each of the acquired data, i.e., the ground image 20, the aerial image 30, and the class information 40. Specifically, the image matching device 2000 extracts features from the ground image 20 to generate ground image features 60. The image matching device 2000 extracts features from the aerial image 30 to generate aerial image features 70. The image matching device 2000 extracts features from the class information 40 to generate class features 80.

[0018] When the class information 40 includes data representing the distribution of classes in the ground image 20, the class feature 80 includes features called "ground class features" extracted from that data. The ground class features represent the characteristics of the distribution of classes in the ground image 20. When the class information 40 includes data representing the distribution of classes in the aerial image 30, the class feature 80 includes features called "aerial class features" extracted from that data. The aerial class features represent the characteristics of the distribution of classes in the aerial image 30.

[0019] After extracting the above-mentioned features, the image matching device 2000 uses the ground image features 60, the aerial image features 70, and the class features 80 to determine whether the ground image 20 and the aerial image 30 match each other.

[0020] <Examples of effects> According to the image matching device 2000 of the first embodiment, it is determined whether the ground image 20 and the aerial image 30 match each other by using not only the feature amounts extracted from the ground image 20 and the aerial image 30 but also the feature amounts extracted from the class information 40. By using the feature amounts extracted from the class information 40, i.e., the class feature amounts 80, it is possible to compare the ground image 20 and the aerial image 30 based not only on the similarity of their appearance but also on the similarity of their class distributions. This allows the image matching device 2000 to perform ground-to-aerial cross-view matching more accurately than when the class feature amounts 80 are not used.

[0021] The image matching device 2000 will be described in more detail below.

[0022] <Example of functional configuration> 3 is a block diagram showing an example of the functional configuration of the image matching device 2000. The image matching device 2000 includes an acquisition unit 2020, a ground image feature extraction unit 2040, an aerial image feature extraction unit 2060, a class feature extraction unit 2080, and a determination unit 2100.

[0023] The acquisition unit 2020 acquires ground images 20, aerial images 30, and class information 40. The ground image feature extraction unit 2040 extracts features from the ground images 20, thereby obtaining ground image features 60. The aerial image feature extraction unit 2060 extracts features from the aerial images 30, thereby obtaining aerial image features 70. The class feature extraction unit 2080 extracts features from the class information 40, thereby obtaining class features 80. The determination unit 2100 uses the ground image features 60, the aerial image features 70, and the class features 80 to determine whether the ground images 20 and the aerial images 30 match each other.

[0024] <Example of hardware configuration> Image matching device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer manufactured for realizing image matching device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0025] Image matching device 2000 may be realized by installing an application on one or more computers. The application is realized by a program that causes one or more computers to function as image matching device 2000. In other words, the program is an implementation of the functional components of image matching device 2000.

[0026] 4 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes the image matching device 2000. In FIG. 4, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0027] The bus 1020 is a data transmission path through which the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main memory element such as a random access memory (RAM) or a read-only memory (ROM). The storage device 1080 is an auxiliary memory element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0028] The above-mentioned programs may be stored in the storage device 1080. The processor 1040 reads the programs from the storage device 1080 and executes the programs to realize the respective functional components of the image matching device 2000.

[0029] The hardware configuration of computer 1000 is not limited to that shown in Fig. 4. For example, as described above, image matching device 2000 may be realized by multiple computers. In this case, the computers may be connected to each other via a network.

[0030] <Processing flow> 5 is a flowchart showing an exemplary flow of processing performed by the image matching device 2000. The acquisition unit 2020 acquires ground images 20, aerial images 30, and class information 40 (S102). The ground image feature extraction unit 2040 extracts features from the ground images 20 and calculates ground image features 60 (S104). The aerial image feature extraction unit 2060 extracts features from the aerial images 30 and calculates aerial image features 70 (S106). The class feature extraction unit 2080 extracts features from the class information 40 and calculates class features 80 (S108). The determination unit 2100 uses the ground image features 60, aerial image features 70, and class features 80 to determine whether the ground images 20 and the aerial images 30 match each other (S110).

[0031] <Examples of applications of the image matching device 2000> The image matching device 2000 has a variety of possible applications. For example, the image matching device 2000 can be used as part of a system that performs image geolocalization (hereinafter referred to as a geolocalization system). Image geolocalization is a technology for identifying the location where an input image was taken. The geolocalization system 2000 can be implemented on one or more arbitrary computers (such as the one shown in FIG. 4). Here, the geolocalization system is merely an example of an application of the image matching device 2000, and the application of the image matching device 2000 is not limited to being used in a geolocalization system.

[0032] 6 shows a geolocalization system 200 including an image matching device 2000. The geolocalization system 200 includes the image matching device 2000 and a location database 300. The location database 300 includes a plurality of aerial images, each with location information attached. An example of the location information may be the Global Positioning System (GPS) coordinates of the location where the corresponding aerial image was taken.

[0033] The geolocalization system 200 receives a query including a ground image and a set of ground class information from a client (e.g., a user terminal). The ground class information is one aspect of class information 40 indicating class distribution in a ground image, such as a segmented image of the ground image 20. The geolocalization system 200 then searches the location database 300 for aerial images that match the ground image in the received query, thereby identifying the location where the ground image was captured. Specifically, until an aerial image that matches the ground image in the query is found, the geolocalization system 200 repeatedly performs the following process: acquire one of the aerial images from the location database 300; input the set of ground image and ground class information, as well as the acquired set of aerial image and aerial class information, into the image matching device 2000; and determine whether the output from the image matching device 2000 indicates that the ground image matches the aerial image. Note that the aerial class information is one aspect of class information 40 indicating class distribution in aerial images, such as a segmented image of the aerial image 30.

[0034] By repeatedly performing the above process, the geolocalization system 200 can find an aerial image that includes the location where the ground image was captured. Because the detected aerial image is associated with location information such as GPS coordinates, the geolocalization system 200 can determine that the location where the ground image was captured is the location indicated by the location information associated with the aerial image that matches the ground image.

[0035] 6, both the class distribution in the ground image 20 and the class distribution in the aerial image 30 are taken into consideration. However, in another embodiment, the image matching device 2000 ,So Either one of these may be used.

[0036] 6, the ground class information is included in the query. However, in another implementation, the geolocalization system 200 may receive a query that does not include the ground class information. In this case, the geolocalization system 200 may generate the ground class information from the ground imagery 20, for example, by performing semantic segmentation on the ground imagery 20.

[0037] 6, the aerial class information is stored in the location database 300 in association with the aerial image 30. However, in another embodiment, the location database 300 may not store the aerial class information. In this case, the geo-localization system 200 may generate the aerial class information from the aerial image 30, for example, by performing semantic segmentation on the aerial image 30.

[0038] Note that terrestrial imagery and aerial imagery may be used in reverse in geolocalization system 200. In this case, location database 300 stores multiple terrestrial images, each with location information. Geolocalization system 200 receives a query containing aerial images and searches location database 300 for terrestrial images that match the aerial images in the query, thereby identifying the location of where the aerial images were taken.

[0039] <About class information> The class information 40 may include one or more pieces of information that represent the distribution of classes of objects in the terrestrial imagery 20, the aerial imagery 30, or both. As mentioned above, the class information 40 may include a segmented image or a keyword matrix. Examples of these pieces of class information 40 are described below.

[0040] <<Segmented image>> A segmented image is an image in which each pixel, by its color (i.e., pixel value), represents the class of object captured in the corresponding region of the original image from which the segmented image was generated. Assume that there are five predefined classes: "sky," "building," "road," "sidewalk," and "other," and that these classes are assigned the colors "yellow," "blue," "green," "red," and "gray," in that order. In this case, for example, if some pixels in the original image capture sky, the corresponding pixels in the segmented image are filled with yellow. Similarly, if some pixels in the original image capture buildings, the corresponding pixels in the segmented image are filled with blue.

[0041] It should be noted that the dimensions (i.e., width and height) of the segmented image may be the same as or different from the dimensions of the original image, in which case the pixels of the segmented image indicate the class of the object imaged at the corresponding pixel in the original image.

[0042] In the latter case, pixels in the segmented image indicate the class of the object imaged in the corresponding pixels in the original image. For example, a segmented image may be generated such that each pixel corresponds to a region containing NxM pixels in the original image, where N, M, or both are greater than 1.

[0043] The segmented image may be generated by performing semantic segmentation on the original image. There are various well-known methods for performing semantic segmentation on an image, and any of these methods may be applied to generate a segmented image of the terrestrial image 20, the aerial image 30, or both. Furthermore, if the dimensions of the segmented image are smaller than those of the original image, the image matching device 2000 may perform subsampling (e.g., average pooling or max pooling) on ​​the segmented image to reduce the dimensions of the segmented image. Note that the image matching device 2000 does not necessarily have to generate the segmented image.

[0044] <<Keyword Matrix>> A keyword matrix is ​​a matrix in which each element represents a vector named "keyword vector." A keyword vector represents the class of the object captured in the corresponding region of the original image from which the keyword matrix was generated. Assume that there are five predefined classes: "sky," "building," "road," "sidewalk," and "other." These classes are assigned the one-hot vectors (0,0,0,0,1), (0,0,0,1,0), (0,0,1,0,0), (0,1,0,0,0), and (1,0,0,0,0), in that order. In this case, for example, if sky is captured in some pixels in the original image, the corresponding element of the keyword matrix will be (0,0,0,0,1). Similarly, if a building is captured in some pixels in the original image, the corresponding element of the keyword matrix will be (0,0,0,1,0).

[0045] As with the segmented image, the width and height of the keyword matrix may be the same as or different from the width and height of the original image, where each element of the keyword matrix represents a keyword vector that represents the class of the object captured at the corresponding pixel in the original image.

[0046] In the latter case, the elements of the keyword matrix represent keyword vectors that represent the classes of objects captured in corresponding pixels of the original image. For example, a keyword matrix may be generated such that each element corresponds to an area of ​​NxM pixels in the original image, where N, M, or both are greater than 1.

[0047] The keyword vector is not limited to a one-hot vector. For example, a set of keyword vectors may be defined using knowledge about classes (e.g., similarity between classes). The knowledge may be embedded in the distance between keyword vectors. For example, a set of keyword vectors may be defined such that the similarity between classes is represented by the distance between the keyword vectors of those classes. Conceptually, the more similar two classes are, the shorter the distance between their keyword vectors will be. Such a set of keywords may be defined by the techniques disclosed in, for example, Non-Patent Document 2 or Non-Patent Document 3.

[0048] To generate the keyword matrix, the image matching device 2000 first identifies the class of object captured at each pixel of the original image. This identification may be performed using semantic segmentation. The image matching device 2000 then assigns, for each pixel, a keyword vector representing the class of object captured at that pixel. Furthermore, if the dimensions of the keyword matrix are smaller than those of the original image, the image matching device 2000 may perform subsampling (e.g., average pooling or max pooling) on ​​the keyword matrix to reduce the dimensions of the keyword matrix. Note that the keyword matrix does not necessarily need to be generated by the image matching device 2000.

[0049] <Data Acquisition: S102> The acquisition unit 2020 acquires the ground image 20, the aerial image 30, and the class information 40 (S102). There are various methods for acquiring this data. In some embodiments, the acquisition unit 2020 may receive the data transmitted from another computer. In other embodiments, the acquisition unit 2020 may acquire the data from an accessible storage device.

[0050] With regard to the class information 40, the image matching device 2000 may generate the class information 40 based on the ground image 20, the aerial image 30, or both, and the acquisition unit 2020 may acquire the class information 40 generated inside the image matching device 2000. A specific method for generating the class information 40 has been described above.

[0051] <Extraction of ground image feature 60: S104> The ground image feature extraction unit 2040 extracts features from the ground image 20 and calculates the ground image feature 60 (S104). There are various methods for extracting features from an image, and any of these methods may be adopted to configure the ground image feature extraction unit 2040. For example, the ground image feature extraction unit 2040 may be realized by a machine learning-based model such as a neural network. More specifically, a feature extraction layer of a convolutional neural network (CNN) may be used to configure the ground image feature extraction unit 2040.

[0052] <Extraction of 70 aerial image features: S106> The aerial image feature extraction unit 2060 extracts features from the aerial image 30 and calculates the aerial image feature 70 (S106). As described above, there are various methods for extracting features from an image. Therefore, the aerial image feature extraction unit 2060 may be configured using any of these methods. For example, the aerial image feature extraction unit 2060 may be implemented using a machine learning-based model such as a neural network. More specifically, the feature extraction layer of a convolutional neural network (CNN) may be used to configure the aerial image feature extraction unit 2060.

[0053] <Extraction of class feature 80: S108> The class feature extraction unit 2080 extracts features from the class information 40 to calculate class features 80 (S108). The features of the class information 40 may be extracted from the class information 40 in a manner similar to the way ground image features 60 are extracted from ground images 20 or the way aerial image features 70 are extracted from aerial images 30. For example, the class feature extraction unit 2080 can be realized by a machine learning-based model such as a neural network.

[0054] When the class information 40 is configured to include multiple types of information such as segmented images of the ground image 20 and keyword matrices, the class information 40 includes a feature extraction unit for each of these types of data.

[0055] <Matching of ground image 20 and aerial image 30: S110> The determination unit 2100 determines whether the ground image 20 and the aerial image 30 match each other using the ground image features 60, the aerial image features 70, and the class features 80 (S110). Specifically, the determination unit 2100 makes the determination by comparing features related to the ground image 20 (referred to as "ground features") with features related to the aerial image 30 (referred to as "aerial features"). If the class features 80 include a ground class feature, the determination unit 2100 calculates a combined feature of the ground image features 60 and the ground class feature, and uses the calculated feature as the ground feature. On the other hand, if the class features 80 do not include a ground class feature, the determination unit 2100 uses the ground image features 60 as the ground feature. Similarly, if the class features 80 include an aerial class feature, the determination unit 2100 calculates a combined feature of the aerial image features 70 and the aerial class feature, and uses the calculated feature as the aerial feature. On the other hand, if the class features 80 do not include the aerial photography class features, the determining unit 2100 uses the aerial photography image features 70 as the aerial photography features.

[0056] 7 shows an example of a portion of the configuration of an image matching device 2000 for comparing ground features with aerial features. In this example, the class information 40 includes segmented images and keyword matrices for both the ground image 20 and the aerial image 30.

[0057] The image matching device 2000 includes networks 100, 110, 120, 130, 140, and 150. The network 100 included in the ground image feature extraction unit 2040 extracts ground image features 60 from the ground image 20. The network 110 included in the class feature extraction unit 2080 extracts features 160 from the segmented image 22 of the ground image 20. The network 120 included in the class feature extraction unit 2080 extracts features 170 from the keyword matrix 24 of the ground image 20. In this case, the ground class features include the features 160 and 170. Next, the ground image features 60, the features 160, and the features 170 are combined with each other to calculate a ground feature 65.

[0058] There are various methods for combining multiple feature quantities, and by applying any of these methods, the ground image feature quantity 60, the feature quantity 160, and the feature quantity 170 can be combined to calculate the ground feature quantity 65. For example, the feature quantities 160 and 170 are concatenated with the ground image feature quantity 60 to calculate the ground feature quantity 65. In another example, the ground image feature quantity 60, the feature quantity 160, and the feature quantity 170 are supplied to a feature quantity extraction unit such as a neural network, and the output from this feature quantity extraction unit is used as the ground feature quantity 65.

[0059] Aerial image features can be calculated using a method similar to that for calculating ground features. Specifically, network 130 included in aerial image feature extraction unit 2060 extracts aerial image features 70 from aerial image 30. Network 140 included in class feature extraction unit 2080 extracts features 180 from segmented image 32 of aerial image 30. Network 150 included in class feature extraction unit 2080 extracts features 190 from keyword matrix 34 of aerial image 30. In this case, the aerial class features include features 180 and 190. Next, aerial image features 70, features 180, and features 190 are combined to calculate aerial feature 75. Note that aerial image features 70, features 180, and features 190 can be combined using a method similar to that for combining ground image features 60, features 160, and features 170.

[0060] After combining the features, the determination unit 2100 may calculate a similarity score representing the similarity between the ground features and the aerial image features. There are various indices for quantifying the similarity between features, and the similarity score can be calculated using any one of them. For example, the similarity score can be calculated as one of various types, such as the distance (e.g., L2 distance) between the ground features and the aerial image features, correlation, cosine similarity, or neural network (NN)-based similarity. NN-based similarity is calculated by a neural network trained to calculate the similarity between two input data (in this case, ground features and aerial image features).

[0061] The determination unit 2100 determines whether the ground image 20 and the aerial image 30 match each other based on the calculated similarity score. Conceptually, the higher the similarity between the ground feature amount and the aerial feature amount, the higher the probability that the ground image 20 and the aerial image 30 match each other. Therefore, for example, the determination unit 2100 determines whether the similarity score is equal to or greater than a predetermined threshold. If the similarity score is equal to or greater than the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 match each other. On the other hand, if the similarity score is less than the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 do not match each other.

[0062] In the above case, it is assumed that the higher the similarity between the ground feature amount and the aerial photography feature amount, the larger the similarity score. Therefore, when an index (e.g., distance) is used such that the value calculated for the ground feature amount and the aerial photography feature amount becomes smaller as the similarity between them increases, the similarity score can be defined as the reciprocal of the value calculated for the ground feature amount and the aerial photography feature amount.

[0063] In another example, when the similarity score decreases as the similarity between the ground feature amount and the aerial feature amount increases, the determination unit 2100 may determine whether the similarity score is equal to or less than a predetermined threshold. If the similarity score is equal to or less than the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 match each other. On the other hand, if the similarity score exceeds the predetermined threshold, the determination unit 2100 determines that the ground image 20 and the aerial image 30 do not match each other.

[0064] <Output of image matching device 2000> The image matching device 2000 may output information indicating the result of the determination (hereinafter, referred to as output information). For example, the output information may indicate whether the ground image 20 and the aerial image 30 match each other. Furthermore, as described with reference to FIG. 6, the output information may further include location information indicating the location where the query image (the ground image 20 or the aerial image 30) was captured.

[0065] There are various ways to output the output information. For example, the image matching device 2000 may store the output information in a storage device. In another example, the image matching device 2000 may output the output information to a display device, which can then display the contents of the output information. In another example, the image matching device 2000 may output the output information to another computer (such as a computer included in the geolocalization system 200 shown in FIG. 6).

[0066] <Model training> As described above, the image matching device 2000 may include one or more machine learning-based models, such as a neural network. For example, as described with reference to Fig. 7, the ground image feature extraction unit 2040, the aerial image feature extraction unit 2060, and the class feature extraction unit 2080 may include neural networks. When the image matching device 2000 is implemented using machine learning-based models, the models are trained using a training dataset prior to operation of the image matching device 2000.

[0067] In some embodiments, a computer that trains a model (hereinafter, referred to as a training device) may repeatedly perform the process of "calculating a loss (e.g., triplet loss or contrastive loss) using a training dataset and updating trainable parameters of the model based on the calculated loss." The training device may be implemented in the computer 1000 in which the image matching device 2000 is implemented, or in another computer. In the former case, the image matching device 2000 can be said to also have the functions of the training device described below. In the latter case, the training device may be implemented using one or more computers, the hardware configuration of which can be exemplified in FIG. 4, similar to the image matching device 2000.

[0068] When training a model using triplet loss, the training dataset may include anchor images, positive example images, and negative example images. A positive example image is an image of a different type (ground-view or aerial-view) from the anchor image and matches the anchor image. A negative example image is an image of a different type from the anchor image but the same type as the positive example image and does not match the anchor image. If the training dataset includes ground images as anchor images, the training dataset includes aerial images that match the anchor images as positive example images and other aerial images that do not match the anchor images as negative example images. On the other hand, if the training dataset includes aerial images as anchor images, the training dataset includes ground images that match the anchor images as positive example images and other ground images that do not match the anchor images as negative example images.

[0069] The training dataset may also include class information for each of the anchor images, positive example images, and negative example images. Specifically, the training dataset may include segmented images and keyword matrices for each of the anchor images, positive example images, and negative example images. However, as mentioned above, the class information may be generated instead of being obtained externally.

[0070] The training device obtains features from the anchor image, positive example images, negative example images, and class information in the training dataset using the ground image feature extraction unit 2040, the aerial image feature extraction unit 2060, and the class feature extraction unit 2080. Assume that the image matching device 2000 has the structure shown in FIG. 7 . Assume further that the training dataset includes ground images as anchor images. In this case, the training device inputs the anchor images, segmented images of the anchor images, and keyword matrices of the anchor images to the networks 100, 110, and 120, respectively. As a result, the training device obtains ground features of the anchor images, which are combinations of the ground image features of the anchor images, the features of the segmented images of the anchor images, and the features of the keyword matrices of the anchor images.

[0071] Furthermore, the training device inputs the positive example image, the segmented image of the positive example image, and the keyword matrix of the positive example image into networks 130, 140, and 150, respectively. As a result, the training device obtains aerial features of the positive example image, which are a combination of the aerial image features of the positive example image, the features of the segmented image of the positive example image, and the features of the keyword matrix of the positive example image.

[0072] Similarly, the training device inputs the negative example image, the segmented image of the negative example image, and the keyword matrix of the negative example image into networks 130, 140, and 150, respectively. As a result, the training device obtains aerial features of the negative example image, which are a combination of the aerial image features of the negative example image, the features of the segmented image of the negative example image, and the features of the keyword matrix of the negative example image.

[0073] The training device calculates a triplet loss based on the ground features of the anchor images, the aerial features of the positive example images, and the aerial features of the negative example images. The training device then updates the trainable parameters of the model based on the obtained triplet loss. Note that there are various known methods for updating the trainable parameters of one or more machine learning-based models based on the triplet loss calculated based on the outputs from these models, any one of which can be adopted by the training device.

[0074] Note that triplet loss is just an example of a loss that can be used to train a model; any other type of loss may be used to train a model.

[0075] The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and RAMs). The program may also be provided to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0076] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present invention.

[0077] All or part of the above-described embodiments may also be described as, but not limited to, the following supplementary notes. (Appendix 1) An image matching device, comprising: at least one memory configured to store instructions; Execute the instructions, acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; Calculating ground image feature amounts by extracting feature amounts from the ground image; Calculating aerial image feature amounts by extracting feature amounts from the aerial image; Calculating class features by extracting features from the class information; Based on the ground image feature amount, the aerial image feature amount, and the class feature amount, it is determined whether the ground image and the aerial image match each other. and at least one processor configured to: (Appendix 2) 2. The image matching device of claim 1, wherein the class information includes a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 3) The image matching device described in Appendix 1, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 4) 4. The image matching device of claim 3, wherein the keyword vectors are defined such that the similarity between classes is represented by the distance between the keyword vectors corresponding to those classes. (Appendix 5) The determination of whether the ground image and the aerial image match each other is performed by: Calculating a similarity between ground features and aerial features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; when the class information includes ground class information indicating a distribution of object classes in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, An image matching device described in any one of Appendix 1 to 4, wherein when the class information includes aerial class information indicating the distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information. (Appendix 6) 1. A computer-implemented image matching method, comprising: acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; calculating a ground image feature by extracting a feature from the ground image; calculating aerial image feature amounts by extracting feature amounts from the aerial image; calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features. (Appendix 7) 7. The image matching method of claim 6, wherein the class information includes a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image. (Appendix 8) 7. The image matching method of claim 6, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the ground image or the aerial image. (Appendix 9) 9. The image matching method of claim 8, wherein the keyword vectors are defined such that the similarity between classes is represented by the distance between the keyword vectors corresponding to those classes. (Appendix 10) The determination of whether the ground image and the aerial image match each other is performed by: Calculating a similarity between ground features and aerial features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; when the class information includes ground class information indicating a distribution of object classes in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, An image matching method described in any one of Appendixes 6 to 9, wherein when the class information includes aerial class information indicating the distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information. (Appendix 11) A non-transitory computer-readable storage medium for storing a program, On the computer, acquiring a ground image, an aerial image, and class information indicating a distribution of object classes in the ground image, the aerial image, or both; calculating a ground image feature by extracting a feature from the ground image; calculating aerial image feature amounts by extracting feature amounts from the aerial image; calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features. (Appendix 12) 12. The storage medium of claim 11, wherein the class information includes a segmented image in which each pixel indicates the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image. (Appendix 13) 12. The storage medium of claim 11, wherein the class information includes a keyword matrix, each element of which indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image. (Appendix 14) 14. The storage medium of claim 13, wherein the keyword vectors are defined such that the similarity between classes is represented by the distance between the keyword vectors corresponding to those classes. (Appendix 15) The determination of whether the ground image and the aerial image match each other is performed by: Calculating a similarity between ground features and aerial features; and determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold value; when the class information includes ground class information indicating a distribution of object classes in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, A storage medium described in any one of Appendices 11 to 14, wherein when the class information includes aerial class information indicating the distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information. [Explanation of symbols]

[0078] 20 Ground images 22 Segmented ground image 24 Keyword matrix for ground images 30 Aerial Images 32 Segmented image of aerial image 34 Keyword matrix for aerial images 40 Class Information 60 Ground Image Features 65 Ground Features 70 Aerial image features 75 Aerial Feature 80 class features 100,110,120,130,140,150 Network 160,170,180,190 features 200 Geolocalization System 300 Location Database 1000 computers 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 2000 Image Matching Device 2020 Acquisition Department 2040 Ground image feature extraction unit 2060 Aerial image feature extraction unit 2080 Class feature extraction unit 2100 Judgment section

Claims

1. A system having an acquisition means for acquiring ground images, aerial images, and class information, the class information indicates a distribution of object classes in the ground image, the aerial image, or both; a ground image feature extraction means for extracting feature amounts from the ground image to calculate ground image feature amounts; an aerial image feature extraction means for calculating aerial image feature values ​​by extracting feature values ​​from the aerial image; a class feature extraction means for calculating class features by extracting features from the class information; and a determination means for determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

2. the class information includes a segmented image; The image matching device of claim 1 , wherein each pixel of the segmented image indicates the class of the object imaged at one or more corresponding pixels in the terrestrial or aerial image.

3. the class information includes a keyword matrix; The image matching device according to claim 1 , wherein each element of the keyword matrix indicates a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image.

4. 4. The image matching device according to claim 3, wherein the keyword vectors are defined such that the similarity between classes is represented by the distance between the keyword vectors corresponding to those classes.

5. The determination means Calculating the similarity between the ground feature and the aerial feature; If the calculated similarity is equal to or greater than a predetermined threshold, it is determined that the ground image and the aerial image match each other; when the class information includes ground class information indicating a distribution of object classes in the ground image, the ground feature amount is a combination of the ground image feature amount and the class feature amount extracted from the ground class information, 5. The image matching device of claim 1, wherein when the class information includes aerial class information indicating the distribution of object classes in the aerial image, the aerial feature is a combination of the aerial image feature and the class feature extracted from the aerial class information.

6. A method for detecting a ground image, an aerial image, and class information, the class information indicates a distribution of object classes in the ground image, the aerial image, or both; calculating a ground image feature by extracting a feature from the ground image; calculating aerial image feature amounts by extracting feature amounts from the aerial image; calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

7. the class information includes a segmented image; The image matching method of claim 6 , wherein each pixel of the segmented image indicates the class of the object imaged at one or more corresponding pixels in the terrestrial or aerial image.

8. the class information includes a keyword matrix; The image matching method of claim 6 , wherein each element of the keyword matrix represents a keyword vector assigned to the class of the object imaged at one or more corresponding pixels in the terrestrial image or the aerial image.

9. A method for detecting a ground image, an aerial image, and class information by a computer; the class information indicates a distribution of object classes in the ground image, the aerial image, or both; calculating a ground image feature by extracting a feature from the ground image; calculating aerial image feature amounts by extracting feature amounts from the aerial image; calculating class features by extracting features from the class information; and determining whether the ground image and the aerial image match each other based on the ground image features, the aerial image features, and the class features.

10. the class information includes a segmented image; 10. The program of claim 9, wherein each pixel of the segmented image indicates the class of the object imaged at one or more corresponding pixels in the terrestrial or aerial image.

Images