IMAGE MATCHING DEVICE, CONTROL METHOD, AND PROGRAM

Abstract

The image matching device (2000) acquires a ground image (20), an aerial image (30), a ground depth image (40), and an aerial depth image (50). The ground depth image (40) indicates the distance between each position captured in the ground image (20) and a ground camera. The aerial depth image (50) indicates the distance from a center position captured in the aerial image to each position captured in the aerial image (50). The image matching device (2000) extracts features from the ground image and the ground depth image to calculate a ground feature (60), and extracts features from the aerial image and the aerial depth image to calculate an aerial depth image (70). The image matching device (2000) uses the ground feature (60) and the aerial feature (70) to determine whether the ground image (20) and the aerial image (30) match each other.

Description

[Technical field]

[0001] The present invention relates generally to image matching, and more particularly to matching terrestrial and aerial images. [Background technology]

[0002] A computer system that performs ground-to-aerial cross-view matching (matching of ground images and aerial images) has been developed. For example, Non-Patent Document 1 discloses a system that includes a pair of CNNs (Convolutional Neural Networks) for matching ground images and aerial images with a field of view of less than 360 degrees. Specifically, one of the CNNs acquires a set of ground images and a direction map indicating the direction (azimuth and altitude) of each position captured in the ground images, and extracts features therefrom. The other CNN acquires a set of aerial images and a direction map indicating the direction (azimuth and altitude) of each position captured in the aerial images, and extracts features therefrom. The system then determines whether the ground image matches the aerial image based on the extracted features. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Liu Liu and Hongdong Li, “Lending Orientation to Neural Networks for Cross-view Geo-localization”, [online], March 29, 2019, arXiv, [Retrieved September 24, 2021],<URL:https: / / arxiv.org / pdf / 1903.12351> [Non-Patent Document 2] Zhengqi Li and Noah Snavely, "MegaDepth: Learning Single-View Depth Prediction from Internet Photos," [online], April 2, 2018, arXiv, [Retrieved September 24, 2021],<URL:https: / / arxiv.org / pdf / 1804.00607> Summary of the Invention [Problem to be solved by the invention]

[0004] In Non-Patent Document 1, extraction of features from sources other than RGB images and orientation maps is not considered. One of the objectives of this disclosure is to provide a new technique for determining whether a ground image and an aerial image match each other. [Means for solving the problem]

[0005] The present disclosure provides a training device having at least one processor and at least one storage element configured to store instructions, the at least one processor being configured to execute the instructions to acquire a ground image, an aerial image, a ground depth image, and an aerial depth image, the ground depth image being an image showing a distance between each position captured in the ground image and a ground camera, the aerial depth image being an image showing a distance from a center position captured in the aerial image to each position captured in the aerial image, extracting features from the ground image and the ground depth image to calculate ground features, extracting features from the aerial image and the aerial depth image to calculate aerial features, and determining whether the ground image and the aerial image match each other based on the ground features and the aerial features.

[0006] The present disclosure further provides a control method executed by a computer, the control method including: acquiring a ground image, an aerial image, a ground depth image, and an aerial depth image, the ground depth image being an image showing a distance between each position captured in the ground image and a ground camera, the aerial depth image being an image showing a distance from a center position captured in the aerial image to each position captured in the aerial image, extracting feature amounts from the ground image and the ground depth image to calculate ground feature amounts, extracting feature amounts from the aerial image and the aerial depth image to calculate aerial feature amounts, and determining whether the ground image and the aerial image match each other based on the ground feature amounts and the aerial feature amounts.

[0007] The present disclosure further provides a computer-readable medium having a program stored therein, the program causing a computer to execute the following: acquiring a ground image, an aerial image, a ground depth image, and an aerial depth image, the ground depth image being an image showing a distance between each position captured in the ground image and a ground camera, the aerial depth image being an image showing a distance from a center position captured in the aerial image to each position captured in the aerial image, extracting feature amounts from the ground image and the ground depth image to calculate ground feature amounts, extracting feature amounts from the aerial image and the aerial depth image to calculate aerial feature amounts, and determining whether the ground image and the aerial image match each other based on the ground feature amounts and the aerial feature amounts. Effect of the Invention

[0008] In accordance with the present disclosure, new techniques are provided for determining whether terrestrial and aerial images match each other. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows an overview of an image matching device 2000 according to the first embodiment. [Diagram 2] FIG. 2 shows an example of a terrestrial image 20 and an aerial image 30 . [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the image matching device. [Figure 4] FIG. 4 is a block diagram illustrating an example of a hardware configuration of the image matching device. [Diagram 5] FIG. 5 is a flowchart illustrating an example of the flow of processing executed by the image matching device. [Figure 6] FIG. 6 illustrates a geolocalization system that includes an image matching device. [Figure 7] FIG. 7 shows an aerial depth image. [Figure 8] FIG. 8 illustrates an example of the configuration of the ground feature extraction unit. [Figure 9] FIG. 9 illustrates an example of the configuration of the aerial photography feature extraction unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An example of an embodiment according to the present disclosure will be described below with reference to the drawings. Note that the same elements in the drawings are given the same reference numerals, and redundant explanations are omitted as necessary. In addition, unless otherwise specified, the predetermined information (e.g., a predetermined value or a predetermined threshold value) is stored in advance in a storage device that can be accessed by a computer that uses the information.

[0011] EMBODIMENT 1 <Summary> Fig. 1 is a diagram showing an overview of an image matching device 2000 according to the first embodiment. The image matching device 2000 functions as a classifier that performs matching between a ground image 20 and an aerial image 30 (so-called ground-to-aerial cross-view matching). Fig. 2 shows an example of the ground image 20 and the aerial image 30.

[0012] The ground image 20 is a digital image (e.g., an RGB image of the ground view) that includes a ground view of a location. The ground image 20 is generated by a ground camera, for example, held by a pedestrian or mounted on a vehicle. The ground image may be panoramic (360 degree field of view) or may have a limited field of view (less than 360).

[0013] The aerial image 30 is an image including a planar view of a location (e.g., an RGB image of a view from above). For example, the aerial image 30 may be generated by an aerial camera mounted on a drone, an airplane, or a satellite.

[0014] In addition to the ground image 20 and the aerial image 30, the image matching device 2000 uses a depth image called a "ground depth image 40" corresponding to the ground image 20 and a depth image called an "aerial depth image 50" corresponding to the aerial image 30. The ground depth image 40 indicates the approximate distance from the ground camera to each position captured in the ground image 20. On the other hand, the aerial depth image 50 indicates the approximate distance from the center position captured in the aerial image 30 to each position captured in the aerial image 30. When the ground image 20 and the aerial image 30 match, the center position captured in the aerial image 30 may approximately match the position of the ground camera in a planar view. Therefore, the ground depth image 4 0 and aerial depth image 5 Each 0 indicates the approximate distance to each location as imaged from a ground camera.

[0015] As will be described in more detail below, the ground depth image 40, the aerial depth image 50, or both, may be generated within the image matching device 2000 instead of being obtained from outside the image matching device 2000.

[0016] The image matching device 2000 extracts features from the above-mentioned images (ground image 20, aerial image 30, depth above ground image 40, and aerial depth image 50). Specifically, the image matching device 2000 extracts features from the ground image 20 and depth above ground image 40 to obtain a "ground feature 60" representing the combined feature of the ground image 20 and the depth above ground image 40. Similarly, the image matching device 2000 extracts features from the aerial image 30 and the aerial depth image 50 to obtain an "aerial feature 70" representing the combined feature of the aerial image 30 and the aerial depth image 50.

[0017] After extracting the features, the image matching device 2000 compares the ground features 60 with the aerial features 70 to determine whether the ground image 20 and the aerial image 30 match. If the similarity between the ground features 60 and the aerial features 70 is high, the image matching device 2000 determines that the ground image 20 and the aerial image 30 match. On the other hand, if the similarity between the ground features 60 and the aerial features 70 is low, the image matching device 2000 determines that the ground image 20 and the aerial image 30 do not match.

[0018] <Examples of effects> According to the image matching device 2000 of the first embodiment, by comparing the combined feature amount of the ground image 20 and the ground depth image 40 with the combined feature amount of the aerial image 30 and the aerial depth image 50, it is determined whether the ground image 20 and the aerial image 30 match. In this way, by using the depth image, it is possible to compare the object captured in the ground image 20 and the object captured in the aerial image 30 based on not only the similarity of their appearance but also the similarity of their positions in space. This allows the image matching device 2000 to perform ground-to-aerial cross-view matching more accurately than when a depth image is not used.

[0019] In particular, when there is an object that is visible in a ground view but not in a planar view, or vice versa, it is effective to consider the similarity of spatial positions. For example, assume that a ground camera and an aerial camera capture a scene in which a tall building is located far away from the ground camera. In this case, the building may be visible from the ground camera but not from the aerial camera. Therefore, the building may be included in the ground image 20 but not included in the aerial image 30. If these images are compared based only on visual similarity, it may be difficult to determine whether these images match.

[0020] In another example, a ground camera and an aerial camera may capture a view of a location where there is a parking lot located away from the ground camera and there are tall objects such as trees or buildings between the parking lot and the ground camera. In this case, the parking lot may be hidden by the trees or buildings from a ground perspective, so the parking lot may be visible from the aerial camera but not from the ground camera. Thus, the parking lot may be included in the aerial image 30 but not in the ground image 20. If these images are compared based solely on visual similarity, it may be difficult to determine whether the images match.

[0021] Even in the above-mentioned case, the image matching device 2000 compares the ground image 20 and the aerial image 30 taking into account spatial similarity, and therefore can correctly determine that these images match.

[0022] A more detailed description of the image matching device 2000 is provided below.

[0023] <Example of functional configuration> 3 is a diagram showing an example of the functional configuration of the image matching device 2000. The image matching device 2000 includes an acquisition unit 2020, a ground feature amount extraction unit 2040, an aerial photography feature amount extraction unit 2060, and a determination unit 2080.

[0024] The acquisition unit 2020 acquires the ground image 20, the aerial image 30, the ground depth image 40, and the ground depth image 50. The ground feature amount extraction unit 2040 extracts features from the ground image 20 and the ground depth image 40 to calculate a ground feature amount 60. The aerial photography feature amount extraction unit 2060 extracts features from the aerial photography image 30 and the aerial photography depth image 50 to calculate an aerial photography feature amount 70. The determination unit 2080 compares the ground feature amount 60 with the aerial photography feature amount 70 to determine whether the ground image 20 and the aerial photography image 30 match.

[0025] <Example of hardware configuration> Image matching device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer created for realizing image matching device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0026] Image matching device 2000 can be realized by installing an application on a computer. The application is realized by a program that causes the computer to function as image matching device 2000. In other words, the program implements the functional components of image matching device 2000.

[0027] 4 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes the image matching device 2000. In FIG. 4, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120.

[0028] The bus 1020 is a data communication path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage element such as a random access memory (RAM) or a read only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and a peripheral device (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN). The storage device 1080 may store the above-mentioned programs. The processor 1040 executes the program to realize each functional component of the image matching device 2000 .

[0029] The hardware configuration of the computer 1000 is not limited to the configuration shown in FIG. For example, as described above, the image matching device 2000 may be realized by a plurality of computers, which may be connected to each other via a network.

[0030] <Processing flow> 5 is a flowchart showing an example of the process flow of the image matching device 2000. The acquisition unit 2020 acquires the ground image 20, the aerial image 30, the ground depth image 40, and the aerial depth image 50 (S102). The ground feature amount extraction unit 2040 extracts features from the ground image 20 and the ground depth image 40, and calculates the ground feature amount 60 (S104). The aerial feature amount extraction unit 2060 extracts features from the aerial image 30 and the aerial depth image 50, and calculates the aerial feature amount 70 (S106). The determination unit 2080 uses the ground feature amount 60 and the aerial feature amount 70 to determine the relationship between the ground image 20 and the aerial image 20. 3 It is determined whether it matches 0 (S108).

[0031] <Application example of the image matching device 2000> There are various applications of the image matching device 2000. For example, the image matching device 2000 can be used as part of a system that performs image geolocalization (hereinafter, referred to as a geolocalization system). Image geolocalization is a technology for identifying the location where an input image was taken. The geolocalization system 200 can be implemented in one or more arbitrary computers (such as the one shown in FIG. 4). Here, the geolocalization system is merely one example of an application of the image matching device 2000, and the application of the image matching device 2000 is not limited to being used in a geolocalization system.

[0032] 6 illustrates a geolocalization system 200 including an image matching device 2000. The geolocalization system 200 includes the image matching device 2000 and a location database 300. The location database 300 includes a plurality of aerial images, each having location information attached thereto. An example of the location information may be the Global Positioning System (GPS) coordinates of the location where the corresponding aerial image was taken.

[0033] The geolocalization system 200 receives a query including a ground image and a ground depth image from a client (e.g., a user terminal), searches the location database 300 for an aerial image that matches the ground image included in the received query, and thereby identifies the capture location of the ground image. Specifically, until an aerial image that matches the ground image included in the query is detected, the geolocalization system 200 repeats the process of "obtaining one aerial image from the location database 300, inputting a set of the ground image and the ground depth image and the acquired set of the aerial image and the aerial depth image to the image matching device 2000, and determining whether the output from the image matching device 2000 indicates that the ground image and the aerial image match." In this way, the geolocalization system 200 can find an aerial image that includes the capture location of the ground image. Since the detected aerial image is associated with location information such as GPS coordinates, the geolocalization system 200 can determine that the capture location of the ground image is the location indicated by the location information associated with the aerial image that matches the ground image.

[0034] As will be described in more detail below, there are various ways to obtain the aerial depth image and the ground depth image. In some implementations, the aerial depth image may be pre-generated and stored in the location database 300 in association with the aerial image. In other implementations, the geo-localization system 200 may generate the aerial depth image based on the aerial image obtained from the location database 300. Similarly, the geo-localization system 200 may generate the ground depth image from the ground image instead of obtaining the ground depth image from the query.

[0035] In addition, in the geolocalization system 200, the terrestrial images and the aerial images may be used in the opposite manner. In this case, the location database 300 stores a plurality of terrestrial images, each of which has location information added thereto. The geolocalization system 200 acquires a query that includes an aerial image, and searches the location database 300 for a terrestrial image that matches the aerial image in the query, thereby identifying the location of the place captured in the aerial image.

[0036] <About ground depth images> The ground depth image 40 indicates the approximate distance from the ground camera to each position captured in the ground image 20. For example, a pixel of the ground depth image 40 has a larger pixel value as the distance between the position corresponding to that pixel and the ground camera becomes smaller. Note that the pixel value of the ground depth image 40 is preferably normalized to be within a predetermined pixel range (e.g., [0, 1]).

[0037] There are various methods for generating the ground depth image 40. In one implementation, the ground depth image 40 may be generated using a distance sensor (e.g., a depth camera, LiDAR (light detection and ranging), etc.) that measures the distance to each position within the sensing range of the distance sensor. In this case, it is preferable to place the ground camera and the distance sensor in close proximity so that the ground depth image generated by the distance sensor indicates the approximate distance from the ground camera to each position. For example, the depth sensor may be incorporated into the housing of the ground camera.

[0038] In another implementation, the ground depth image 40 is generated from the ground image 20. For example, a CNN-based technique for estimating depth from an RGB image, such as that disclosed by NPL2, can be used to generate the ground depth image 40 from the ground image 20. This generation of the ground depth image 40 from the ground image 20 may be performed by the image matching device 2000 or by other devices.

[0039] <About aerial depth images> The aerial depth image 50 indicates the approximate distance from the center position captured in the aerial image 30 to each position captured in the aerial image 30. The aerial depth image 50 may be generated by the image matching device 2000 or by another device. As with the ground depth image 40, the pixel value of a pixel in the aerial depth image 50 increases as the distance from the center position of the aerial image 30 to the corresponding position decreases. The pixel values ​​of the aerial depth image 50 are preferably normalized so that the pixel range is within a predetermined range (e.g., [0, 1]).

[0040] Theoretically, the distance from position L captured in the aerial image 30 to center position C captured in the aerial image 30 increases as the distance from the pixel of the aerial image 30 corresponding to position L to the center of the aerial image 30 corresponding to position C increases. Therefore, the pixel value of each pixel of the aerial depth image 50 can be determined based on the distance from each pixel to the center of the image.

[0041] Based on this idea, the aerial depth image 50 can be generated so that each pixel of the aerial depth image 50 has a value proportional to the distance between the pixel and the center of the aerial depth image 50. Specifically, for example, the image matching device 2000 generates the aerial depth image 50 by 3 The image may be initialized to the same size as 0, and for each pixel of the initialized aerial depth image 50, a value proportional to the distance from the image to the center of the aerial depth image 50 may be calculated using a linear function. The proportionality constant of the linear function is set to a negative value in order to obtain a larger value for a pixel that is closer to the center of the aerial depth image 50. The image matching device 2000 then normalizes the value obtained from the linear function, and uses the normalized value as the pixel value of the corresponding pixel.

[0042] Here, the aerial depth image 50 is initialized to the same size as the aerial image 30, but the size of the aerial depth image 50 does not have to be the same as the size of the aerial image 30. The same applies to the ground depth image 40, i.e., the size of the ground depth image 40 does not have to be the same as the size of the ground image 20.

[0043] 7 is a diagram illustrating an aerial depth image 50. In this diagram, lower density dots represent regions of pixels with higher pixel values. In addition, the pixel value of a pixel is set to be higher the closer the pixel is to the center of the aerial depth image 50. Therefore, the density of dots in a region is drawn lower the closer the region is to the center of the aerial depth image 50.

[0044] <Image acquisition: S102> The acquisition unit 2020 acquires the ground image 20, the aerial image 30, the ground depth image 40, and the aerial depth image 50 (S102). There are various ways to acquire these images. In some implementations, the acquisition unit 2020 may receive these images from another computer. In other implementations, the acquisition unit 2020 may acquire these images from an accessible storage device.

[0045] The ground depth image 40 and the aerial depth image 50 can be generated by the image matching device 2000 based on the ground image 20 and the aerial image 30, and the acquisition unit 2020 acquires these depth images generated in the image matching device 2000. The specific method of generating the ground depth image 40 and the aerial depth image 50 has been described above.

[0046] In addition, since the pixel values ​​of the aerial depth image 50 do not depend on the pixel values ​​of the aerial image 30, the image matching device 200 0 may use an aerial depth image 50 common to different aerial images 30. For example, the aerial depth image 50 is prepared in advance and stored in a storage device accessible by the image matching device 2000. The image matching device 2000 may read out the aerial depth image 50 from the storage device in response to the acquisition of an aerial image.

[0047] <Extraction of ground feature 60: S104> The ground feature amount extraction unit 2040 calculates a ground feature amount 60 based on the ground image 20 and the aerial depth image 50 (S104). The ground feature amount 60 is calculated based on the feature amount extracted from the ground image 20 and the ground depth image 50. 40 and a feature extracted from the image. There are various methods for extracting features from an image, and one of these may be adopted in the ground feature extraction unit 2040. For example, the ground feature extraction unit 2040 may be realized by a machine learning based model such as a neural network. More specifically, a feature extraction layer of a Convolutional Neural Network (CNN) may be adopted to configure the ground feature extraction unit 2040.

[0048] 8 shows an example of the configuration of the ground feature extraction unit 2040. In the upper diagram, the ground feature extraction unit 2040 has one network 100. The network 100 receives a ground image 20 and a ground depth image 4 A concatenation with 0 is treated as an input, features are extracted from this concatenation data, and the extracted features are output as ground features 60.

[0049] Meanwhile, in the lower diagram, the ground feature extraction unit 2040 has three networks 110, 120, and 130. The network 110 takes a ground image 20 as input, extracts features from it, and outputs the extracted features. Similarly, the network 120 takes a ground depth image 20 as input, 4 The network 130 takes as input the features extracted from the ground image 20 and the features extracted from the ground depth image 50, combines them, and outputs the combined features as ground features 60.

[0050] <Extraction of 70 aerial photography features: S106> The aerial photography feature extraction unit 2060 extracts the aerial photography image 30 and the aerial photography depth image 5 0, the aerial photography feature quantity 70 is calculated (S106). The aerial photography feature quantity 70 is a combination of the feature quantity extracted from the ground image 20 and the feature quantity extracted from the ground depth image 50.

[0051] 9 shows an example of the configuration of the aerial photography feature extraction unit 2060. The aerial photography feature extraction unit 2060 can be configured in the same manner as the determination unit 2080, except that the aerial photography feature extraction unit 2060 handles the aerial photography image 30 and the aerial photography depth image 60 as inputs instead of the ground image 20 and the ground depth image 50.

[0052] Specifically, in the upper diagram, the aerial feature extraction unit 2060 has a network 140 that handles the concatenation of the aerial image 30 and the aerial depth image 50 as input, and outputs the combined features of the aerial image 30 and the aerial depth image 50. In the lower diagram, the aerial feature extraction unit 2060 has networks 150, 160, and 170. The network 150 handles the aerial image 30 as input, and outputs its features. The network 160 handles the aerial depth image 50 as input, and outputs its features. The network 170 handles the features of the aerial image 30 and the features of the aerial depth image 50 as input, and outputs their combined features.

[0053] <Matching of ground image 20 and aerial image 30: S108> The determination unit 2080 determines whether the ground image 20 and the aerial image 30 match by comparing the ground feature amount 60 and the aerial feature amount 70 (S108). For the comparison, the determination unit 2080 may calculate a similarity score representing the similarity between the ground image 20 and the aerial image 30.

[0054] Various measures may be used to calculate the similarity score. For example, the similarity may be calculated as one of various types of distance (e.g., L2 distance), correlation, cosine similarity, or a neural network (NN)-based similarity between the ground features 60 and the aerial features 70. The NN-based similarity is calculated by a neural network trained to calculate the similarity between two input data (in this case, the ground features 60 and the aerial features 70).

[0055] The determination unit 2080 determines whether the ground image 20 and the aerial image 30 match each other based on the similarity calculated therefor. Conceptually, the higher the similarity between the ground image 20 and the aerial image 30, the higher the probability that the ground image 20 and the aerial image 30 match each other. Therefore, for example, the determination unit 2080 determines whether the similarity score is equal to or greater than a predetermined threshold. If the similarity score is equal to or greater than the predetermined threshold, the determination unit 2080 determines that the ground image 20 and the aerial image 30 match each other. On the other hand, if the similarity score is smaller than the predetermined threshold, the determination unit 2080 determines that the ground image 20 and the aerial image 30 do not match each other.

[0056] Here, it is assumed that the similarity score in the above-mentioned case increases as the similarity between the ground feature 60 and the aerial photography feature 70 increases. Therefore, when a measure (e.g., distance) is used in which the value calculated for the ground feature 60 and the aerial photography feature 70 decreases as the similarity between the ground feature 60 and the aerial photography feature 70 increases, the similarity score is defined as the reciprocal of the value calculated for the ground feature 60 and the aerial photography feature 70.

[0057] Alternatively, for example, if the similarity score decreases as the similarity between the ground feature 60 and the aerial image feature 70 increases, the determination unit 2080 determines whether the similarity score is equal to or smaller than a predetermined threshold. If the similarity score is equal to or smaller than the predetermined threshold, the determination unit 2080 determines that the ground image 20 and the aerial image 30 match each other. On the other hand, if the similarity score is greater than the predetermined threshold, the determination unit 2080 determines that the ground image 20 and the aerial image 30 do not match each other.

[0058] <Output from Image Matching Device 2000> The image matching device 2000 may output information indicating the result of the determination (hereinafter, output information). For example, the output information may indicate whether the ground image 20 and the aerial image 30 match each other.

[0059] There are various ways to output the output information. For example, the image matching device 2000 may store the output information in a storage device. For another example, the image matching device 2000 may output the output information to a display device so that the contents of the output information are displayed on the display device. For another example, the image matching device 2000 may output the output information to another device, such as a device included in the geolocalization system 200 shown in FIG. 6.

[0060] <Model training> The image matching device 2000 may include one or more machine learning based models, such as a neural network. For example, as described above, the ground feature extractor 2040 and the aerial image feature extractor 2060 may include a neural network (such as a feature extraction layer of a CNN). When the image matching device 2000 is implemented using machine learning based models, the models are pre-trained using a training dataset.

[0061] In some implementations, a computer that trains a model (hereinafter, a training device) repeatedly performs the following steps: "using a training data set to calculate a loss (e.g., triplet loss), and updating trainable parameters of the model based on the calculated loss." Here, the computer in which the image matching device 2000 is implemented 10 00, or may be implemented in another computer. In the former case, it can be said that image matching device 2000 also has the functions of a training device, which will be described later. In the latter case, the training device can be implemented using one or more computers, the hardware configuration of which can be exemplified in FIG. 4, similar to image matching device 2000.

[0062] When a triplet loss is used, the training dataset may include anchor images, positive example images, and negative example images. The positive example images are images of a different type from the anchor images that match the anchor images. The negative example images are images of a different type from the anchor images but of the same type as the positive example images that do not match the anchor images. When the training dataset includes ground images as anchor images, it includes aerial images that match the anchor images as positive example images and aerial images that do not match the anchor images as negative example images. On the other hand, when the training dataset includes aerial images as anchor images, it includes ground images that match the anchor images as positive example images and ground images that do not match the anchor images as negative example images.

[0063] The training data set may further include a ground depth image and an aerial depth image. If the positive example image and the negative example image are ground images, the training data set may include a ground depth image corresponding to the positive example image and a ground depth image corresponding to the negative example image. However, as mentioned above, the ground depth image, the aerial depth image, or both may be generated instead of being obtained externally.

[0064] The training device utilizes the ground feature extraction unit 2040 and the aerial feature extraction unit 2060 to obtain features from each image in the training dataset. Assume that the training dataset includes a ground image as an anchor image. In this case, the training device inputs the anchor image and the ground depth image to the ground feature extraction unit 2040 to obtain ground features of the anchor image. In addition, the training device inputs the positive example image and the aerial depth image to the aerial feature extraction unit 2060 to obtain aerial features of the positive example image. Similarly, the training device inputs the negative example image and the aerial depth image to the aerial feature extraction unit 2060 to obtain aerial features of the negative example image.

[0065] After obtaining the features, the training device calculates a triplet loss based on the ground features, the aerial features of the positive example images, and the aerial features of the negative example images. The training device then updates trainable parameters of the models based on the obtained triplet loss. There are various known techniques for updating trainable parameters of one or more machine learning based models based on triplet losses calculated based on the outputs of the models, and the training device may adopt any one of them.

[0066] When the training data set includes an aerial image as an anchor image, the training apparatus inputs the anchor image and the aerial depth image to the aerial feature extraction unit 2060 to obtain the aerial feature of the anchor image. In addition, the training apparatus inputs the positive example image and the ground depth image corresponding to the positive example image to the ground feature extraction unit 2040 to obtain the ground feature of the positive example image. Similarly, the training apparatus inputs the negative example image and the ground depth image corresponding to the negative example image to the ground feature extraction unit 2040 to obtain the ground feature of the negative example image. Then, the training apparatus calculates a triplet loss based on the aerial feature, the ground feature of the positive example image, and the ground feature of the negative example image, and updates the trainable parameters of the model based on the calculated triplet loss.

[0067] It should be noted that triplet loss is just one example of a loss that can be used to train the model; any other type of loss can be used to train the model.

[0068] <Changes to the training dataset> The training device may modify the training dataset so that it is more suitable for training the model of the image matching device 2000. In addition to the modified training dataset, the original training dataset may also be used to train the model. In this case, the number of training datasets is increased by modifying the original training dataset. In other words, data augmentation is performed by modifying the original training dataset.

[0069] In some implementations, the training device may modify pixels of the ground image to represent locations farther from the ground camera. Specifically, the training device identifies pixels in the ground depth image that have pixel values ​​less than a threshold (i.e., pixels that correspond to locations that are greater than a predetermined distance from the ground camera). The training device then modifies pixel values ​​of pixels in the ground image that correspond to the identified pixels in the ground depth image.

[0070] The pixel values ​​of the ground image are altered so that the objects represented by those pixels are obscured compared to the objects represented by the original pixels. Examples of ways to do this are blurring, adding noise, or blacking out (i.e. changing the pixel values ​​to 0).

[0071] From the perspective of ground-to-aerial cross-view matching, the closer the objects captured in the ground image are to the ground image, the more useful their features are for matching. Therefore, it is preferable for the feature extractor to extract features such that the features of objects relatively close to the ground camera dominate over the features of objects relatively far from the ground camera.

[0072] By modifying the ground image so that objects relatively far from the ground camera are obscured, features extracted from pixels representing objects relatively close to the ground camera become dominant in the ground features extracted by the ground feature extractor 2040. Therefore, the ground feature extractor 2040 can be trained to extract features of the ground image that are more useful for ground-to-aerial cross-view matching.

[0073] In a similar manner, the training device may modify pixels of the aerial image that represent objects far from the center location captured in the aerial image. Specifically, the training device identifies pixels of the aerial image whose distance from the center of the aerial image is greater than a predetermined threshold. The training device then modifies the pixel values ​​of the identified pixels of the aerial image. As with the modification of the ground image, examples of methods for modifying the aerial image include blurring, adding noise, or blacking out. By modifying the aerial image in the above-described manner, the aerial feature extractor 2060 can be trained to extract aerial image features that are more useful for ground-to-aerial cross-view matching.

[0074] Here, the training device may modify the terrestrial images, the aerial images, or both, in the training data set.

[0075] The program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs). The program may also be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0076] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0077] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. <Additional Notes> (Appendix 1) at least one storage element configured to store instructions; By executing the instructions, Acquiring a ground image, an aerial image, a ground depth image, and an aerial depth image, the ground depth image being an image showing a distance between each position captured in the ground image and a ground camera, and the aerial depth image being an image showing a distance from a center position captured in the aerial image to each position captured in the aerial image; Extracting features from the ground image and the ground depth image to calculate ground features; Extracting features from the aerial image and the aerial depth image to calculate aerial features; and at least one processor configured to determine whether the ground image and the aerial image match each other based on the ground features and the aerial features. (Appendix 2) The acquisition of the aerial depth image includes: Calculate the distance between the center of the aerial image and each pixel of the aerial image, and for each pixel of the aerial depth image, calculate the distance between the pixel and the aerial depth image. Photography generating the aerial depth image by setting a value proportional to the calculated distance between the center of the image; and acquiring the generated aerial depth image. (Appendix 3) The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between the ground feature amount and the aerial photography feature amount; An image matching device as described in Appendix 1 or 2, further comprising determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold. (Appendix 4) The at least one storage element is further configured to store a first model and a second model, the first model being trained to extract features from the ground image and the ground depth image and output the ground features, and the second model being trained to extract features from the aerial image and the aerial depth image and output the aerial features; The at least one processor executes the instructions to: Obtaining a training dataset including the ground image, the ground depth image, the aerial image of a positive example, the aerial image of a negative example, and the aerial depth image; inputting the ground image and the ground depth image into the first model to obtain the ground feature amount; inputting the positive example and the aerial depth image into the second model to obtain an aerial feature of the positive example; inputting the negative example and the aerial depth image into the second model to obtain an aerial feature of the negative example; 4. The image matching device of claim 1, further configured to update trainable parameters of the first model and the second model based on the ground features, the aerial image features of the positive examples, and the aerial image features of the negative examples. (Appendix 5) The at least one storage element is further configured to store a first model and a second model, the first model being trained to extract features from the ground image and the ground depth image and output the ground features, and the second model being trained to extract features from the aerial image and the aerial depth image and output the aerial features; The at least one processor executes the instructions to: The aerial image, the ground image of a positive example, and the negative example above groundObtain a training dataset including an image, the aerial depth image, a ground depth image corresponding to the positive example, and a ground depth image corresponding to the negative example; inputting the aerial image and the aerial depth image into the second model to obtain the aerial feature amount; inputting the positive example and the ground depth image corresponding to the positive example into the first model to obtain a ground feature amount of the positive example; inputting the negative example and the ground depth image corresponding to the negative example into the first model to obtain a ground feature amount of the negative example; 4. The image matching device of claim 1, further configured to update trainable parameters of the first model and the second model based on the aerial photography features, the ground features of the positive examples, and the ground features of the negative examples. (Appendix 6) The image matching device of claim 4 or 5, wherein the at least one processor is further configured to execute the instructions to identify pixels in the ground depth image of the training data set that exhibit values ​​greater than a predetermined threshold, and to modify the ground image of the training data set by modifying pixels of the ground image that correspond to the identified pixels of the ground depth image. (Appendix 7) 6. The image matching apparatus of claim 4 or 5, wherein the at least one processor is further configured to modify the aerial images of the training dataset by executing the instructions to modify pixels in the aerial images that are a distance from a center of the aerial image that is greater than a predetermined threshold. (Appendix 8) acquiring a ground image, an aerial image, a ground depth image, and an aerial depth image, the ground depth image being an image showing a distance between each position captured in the ground image and a ground camera, the aerial depth image being an image showing a distance from a center position captured in the aerial image to each position captured in the aerial image; Extracting features from the ground image and the ground depth image to calculate ground features; Extracting features from the aerial image and the aerial depth image to calculate aerial features; and determining whether the ground image and the aerial image match each other based on the ground features and the aerial features. (Appendix 9) The acquisition of the aerial depth image includes: Calculate the distance between the center of the aerial image and each pixel of the aerial image, and for each pixel of the aerial depth image, calculate the distance between the pixel and the aerial depth image. Photography generating the aerial depth image by setting a value proportional to the calculated distance between the center of the image and the and acquiring the generated aerial depth image. (Appendix 10) The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between the ground feature amount and the aerial photography feature amount; A control method as described in Appendix 8 or 9, comprising determining that the ground image and the aerial image match each other if the calculated similarity is greater than or equal to a predetermined threshold. (Appendix 11) The computer is configured to store a first model and a second model, the first model being trained to extract features from the ground image and the ground depth image and output the ground features, and the second model being trained to extract features from the aerial image and the aerial depth image and output the aerial features; The control method includes: Obtaining a training dataset including the ground image, the ground depth image, the aerial images of positive examples, the aerial images of negative examples, and the aerial depth image; inputting the ground image and the ground depth image into the first model to obtain the ground feature amount; inputting the positive example and the aerial depth image into the second model to obtain an aerial feature of the positive example; inputting the negative example and the aerial depth image into the second model to obtain an aerial feature of the negative example; updating trainable parameters of the first model and the second model based on the ground features, the aerial features of the positive examples, and the aerial features of the negative examples. 8 11. A control method according to any one of claims 1 to 10. (Appendix 12) The computer is configured to store a first model and a second model, the first model being trained to extract features from the ground image and the ground depth image and output the ground features, and the second model being trained to extract features from the aerial image and the aerial depth image and output the aerial features; The control method includes: The aerial image, the ground image of a positive example, and the negative example above ground obtaining a training dataset including an image, the aerial depth image, a ground depth image corresponding to the positive example, and a ground depth image corresponding to the negative example; inputting the aerial image and the aerial depth image into the second model to obtain the aerial feature quantity; inputting the positive example and the ground depth image corresponding to the positive example into the first model to obtain a ground feature amount of the positive example; inputting the negative example and the ground depth image corresponding to the negative example into the first model to obtain ground features of the negative example; updating trainable parameters of the first model and the second model based on the aerial image features, the ground features of the positive examples, and the ground features of the negative examples. 8 11. A control method according to any one of claims 1 to 10. (Appendix 13) The control method of claim 11 or 12, further comprising modifying the ground image of the training data set by identifying pixels in the ground depth image of the training data set that exhibit values ​​greater than a predetermined threshold, and modifying pixels of the ground image that correspond to the identified pixels of the ground depth image. (Appendix 14) 13. The method of claim 11 or 12, further comprising modifying the aerial images of the training data set by modifying pixels in the aerial images that are a distance from a center of the aerial image that is greater than a predetermined threshold. (Appendix 15) acquiring a ground image, an aerial image, a ground depth image, and an aerial depth image, the ground depth image being an image showing a distance between each position captured in the ground image and a ground camera, the aerial depth image being an image showing a distance from a center position captured in the aerial image to each position captured in the aerial image; Extracting features from the ground image and the ground depth image to calculate ground features; Extracting features from the aerial image and the aerial depth image to calculate aerial features; A non-transitory computer-readable storage medium storing a program for causing a computer to execute the steps of: determining whether the ground image and the aerial image match each other based on the ground features and the aerial features. (Appendix 16) The acquisition of the aerial depth image includes: Calculate the distance between the center of the aerial image and each pixel of the aerial image, and for each pixel of the aerial depth image, calculate the distance between the pixel and the aerial depth image. Photography generating the aerial depth image by setting a value proportional to the calculated distance between the center of the image and the and acquiring the generated aerial depth image. (Appendix 17) The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between the ground feature amount and the aerial photography feature amount; A storage medium as described in Appendix 15 or 16, further comprising determining that the ground image and the aerial image match each other if the calculated similarity is equal to or greater than a predetermined threshold. (Appendix 18) A first model and a second model are further stored, the first model is trained to extract features from the ground image and the ground depth image and output the ground features, and the second model is trained to extract features from the aerial image and the aerial depth image and output the aerial features; The program is Obtaining a training dataset including the ground image, the ground depth image, the aerial images of positive examples, the aerial images of negative examples, and the aerial depth image; inputting the ground image and the ground depth image into the first model to obtain the ground feature amount; inputting the positive example and the aerial depth image into the second model to obtain an aerial feature of the positive example; inputting the negative example and the aerial depth image into the second model to obtain an aerial feature of the negative example; The storage medium according to any one of appendixes 15 to 17, further causing the computer to execute the steps of: updating trainable parameters of the first model and the second model based on the ground features, the aerial features of the positive examples, and the aerial features of the negative examples. (Appendix 19) A first model and a second model are further stored, the first model is trained to extract features from the ground image and the ground depth image and output the ground features, and the second model is trained to extract features from the aerial image and the aerial depth image and output the aerial features; The program is The aerial image, the ground image of a positive example, and the negative example above ground obtaining a training dataset including an image, the aerial depth image, a ground depth image corresponding to the positive example, and a ground depth image corresponding to the negative example; inputting the aerial image and the aerial depth image into the second model to obtain the aerial feature quantity; inputting the positive example and the ground depth image corresponding to the positive example into the first model to obtain a ground feature amount of the positive example; inputting the negative example and the ground depth image corresponding to the negative example into the first model to obtain ground features of the negative example; The storage medium according to any one of appendixes 15 to 17, further causing the computer to execute: updating trainable parameters of the first model and the second model based on the aerial photography features, the ground features of the positive examples, and the ground features of the negative examples. (Appendix 20) The storage medium of claim 18 or 19, wherein the program further causes the computer to modify the ground image of the training data set by identifying pixels in the ground depth image of the training data set that exhibit values ​​greater than a predetermined threshold, and modifying pixels of the ground image that correspond to the identified pixels of the ground depth image. (Appendix 21) 20. The control method of claim 18 or 19, wherein the program further causes the computer to modify the aerial images of the training data set by modifying pixels in the aerial images that are a distance from the center of the aerial images that is greater than a predetermined threshold. [Explanation of symbols]

[0078] 20 Ground Images 30 Aerial Images 40 Ground Depth Images 50 Aerial Depth Images 60 Ground Features 70 Aerial Feature Values 100, 110, 120, 130, 140, 150, 160, 170 Network 200 Geolocalization System 300 Location Database 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Image Matching Device 2020 Acquisition Department 2040 Ground feature extraction unit 2060 Aerial Feature Extraction Unit 2080 Judgment section

Claims

1. The apparatus has an acquisition unit that acquires a ground image, an aerial image having an identifiable center, ground depth data, and aerial depth data, the ground depth data is data indicating a distance between each position captured in the ground image and a ground camera; The aerial depth data is data indicating a distance from a center position, which is a position captured at the center of the aerial image, to each position captured in the aerial image, a ground feature extraction unit that extracts features from the ground image and the ground depth data to calculate ground features; an aerial photography feature extraction unit that extracts features from the aerial photography image and the aerial photography depth data to calculate aerial photography feature values; and a determination unit that determines whether the ground image and the aerial image match each other based on the ground feature amount and the aerial feature amount.

2. The aerial depth data includes values ​​corresponding to each pixel of the aerial image, The acquisition of the aerial depth data includes: Calculating a distance between a center of the aerial image and each pixel of the aerial image; generating the aerial depth data including a value corresponding to each pixel of the aerial image that is proportional to the calculated distance between the pixel and a center of the aerial image; and acquiring the generated aerial depth data.

3. The determination of whether the ground image and the aerial image match each other includes: Calculating a similarity between the ground feature amount and the aerial photography feature amount; The image matching device according to claim 1 , further comprising: determining that the ground image and the aerial image match each other when the calculated similarity is equal to or greater than a predetermined threshold value.

4. Further comprising a first model and a second model, The first model is trained to extract features from the ground image and the ground depth data and output the ground features; The second model is trained to extract features from the aerial image and the aerial depth data and output the aerial features; Obtaining a training data set including the ground image, the ground depth data, the aerial image of a positive example, the aerial image of a negative example, and the aerial depth data; inputting the ground image and the ground depth data into the first model to obtain the ground feature amount; inputting the positive example and the aerial depth data into the second model to obtain an aerial feature of the positive example; inputting the negative example and the aerial depth data into the second model to obtain an aerial feature of the negative example; The image matching device according to claim 1 , further comprising: updating trainable parameters of the first model and the second model based on the ground features, the aerial photography features of the positive examples, and the aerial photography features of the negative examples.

5. Further comprising a first model and a second model, The first model is trained to extract features from the ground image and the ground depth data and output the ground features; The second model is trained to extract features from the aerial image and the aerial depth data and output the aerial features; Obtaining a training data set including the aerial image, the ground image of a positive example, the ground image of a negative example, the aerial depth data, ground depth data corresponding to the positive example, and ground depth data corresponding to the negative example; inputting the aerial image and the aerial depth data into the second model to obtain the aerial feature; inputting the positive example and the ground depth data corresponding to the positive example into the first model to obtain a ground feature amount of the positive example; inputting the negative example and the ground depth data corresponding to the negative example into the first model to obtain a ground feature amount of the negative example; The image matching device according to claim 1 , further comprising: updating trainable parameters of the first model and the second model based on the aerial photography features, the ground features of the positive examples, and the ground features of the negative examples.

6. 6. The image matching device of claim 4 or 5, further comprising: identifying values ​​in the ground depth data of the training data set that are greater than a predetermined threshold; and modifying the ground image of the training data set by modifying pixels of the ground image that correspond to the identified values ​​of the ground depth data.

7. 6. An image matching apparatus according to claim 4 or 5, further comprising modifying the aerial images of the training data set by modifying pixels in the aerial images that are at a distance greater than a predefined threshold from the centre of the aerial image.

8. obtaining a ground image, a center-identifiable aerial image, ground depth data, and aerial depth data; the ground depth data indicates a distance between each position captured in the ground image and a ground camera; The aerial depth data indicates a distance from a center position, which is a position captured at the center of the aerial image, to each position captured in the aerial image, Extracting features from the ground image and the ground depth data to calculate ground features; Extracting features from the aerial image and the aerial depth data to calculate aerial features; and determining whether the ground image and the aerial image match each other based on the ground features and the aerial features.

9. The aerial depth data includes values ​​corresponding to each pixel of the aerial image, The acquisition of the aerial depth data includes: Calculating a distance between a center of the aerial image and each pixel of the aerial image; generating the aerial depth data including a value corresponding to each pixel of the aerial image that is proportional to the calculated distance between the pixel and a center of the aerial image; and acquiring the generated aerial depth data.

10. causing a computer to acquire ground images, center-identifiable aerial images, ground depth data, and aerial depth data; the ground depth data is data indicating a distance between each position captured in the ground image and a ground camera; The aerial depth data is data indicating a distance from a center position, which is a position captured at the center of the aerial image, to each position captured in the aerial image, Extracting features from the ground image and the ground depth data to calculate ground features; Extracting features from the aerial image and the aerial depth data to calculate aerial features; and determining whether the ground image and the aerial image match each other based on the ground feature amount and the aerial feature amount.

Images