Training device, control method, and program

Abstract

The training device (2000) acquires a ground image (20), an aerial image (30), and positioning information (50) to train a classifier (10), and the classifier (10) extracts a plurality of partial aerial regions (40) from the aerial image (30), and determines that the ground image (20) and the aerial image (30) match when a partial aerial region (40) that matches the ground image (20) exists. The training device (2000) extracts a properly positioned partial aerial region (42) and an incorrectly positioned partial aerial region (44) from the aerial image (30). The properly positioned partial aerial region (42) is indicated by the positioning information (50) and corresponds to the ground image (20). The incorrectly positioned partial aerial region (44) partially overlaps with the properly positioned partial aerial region (42). The training device (2000) trains the classifier (10) using both the correctly positioned partial aerial photograph area (42) and the incorrectly positioned partial aerial photograph area (44) as positive examples.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to image matching, and more particularly to matching between terrestrial and aerial images. [Background technology]

[0002] Computer systems that perform ground-to-aerial cross-view matching (matching of ground images and aerial images) have been developed. For example, Non-Patent Document 1 discloses a system that includes a set of Convolutional Neural Networks (CNNs) for matching ground images and aerial images with a field of view of less than 360 degrees.

[0003] Specifically, one of the CNNs takes a ground image and extracts features from it; the other takes a polar transformed aerial image and extracts features from it. The system then estimates the directional alignment of the ground image to the aerial image by computing the correlation between the features extracted from the ground image and the features extracted from the aerial image. Based on the directional alignment, the system shifts and crops a portion of the features of the aerial image that are estimated to correspond to the ground image. The features of the ground image are then compared to the cropped features of the aerial image to determine whether the ground image matches the aerial image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li, "Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching," Computer Research Repository, arXiv:2005.03860, May 8, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] In Non-Patent Document 1, it is assumed that all training data is prepared in advance to train a group of networks that perform matching between ground images and aerial images with a viewing angle of less than 360 degrees. The objective of this disclosure is to provide an effective method for training a classifier that performs matching between ground images and aerial images with a viewing angle of less than 360 degrees. [Means for solving the problem]

[0006] The training device provided by the present disclosure includes at least one processor and a storage element in which a sequence of instructions is stored. The at least one processor executes the sequence of instructions to acquire a ground image having a viewing angle of less than 360 degrees, an aerial image, and configuration information, and trains a classifier that determines whether the ground image and the aerial image match, and the classifier extracts a plurality of partial aerial regions from the aerial image, and determines that the ground image and the aerial image match when the partial aerial region that matches the ground image exists. Training the classifier includes extracting a correctly positioned partial aerial area and a misplaced partial aerial area from the aerial image, where the correctly positioned partial aerial area is the partial aerial area indicated by the placement information and corresponding to the ground image, and the misplaced partial aerial area is the partial aerial area that partially overlaps with the correctly positioned partial aerial area, and training the classifier using a set of the ground image and the correctly positioned partial aerial area and a set of the ground image and the misplaced partial aerial area, and both the correctly positioned partial aerial area and the misplaced partial aerial area are used as the partial aerial areas of positive examples.

[0007] The present disclosure further provides a control method executed by a computer, the control method including: acquiring a ground image having a viewing angle of less than 360 degrees, an aerial image, and configuration information; training a classifier that determines whether the ground image and the aerial image match, the classifier extracting a plurality of partial aerial regions from the aerial image, and determining that the ground image and the aerial image match when the partial aerial region that matches the ground image exists. Training the classifier includes extracting a correctly positioned partial aerial area and a misplaced partial aerial area from the aerial image, where the correctly positioned partial aerial area is the partial aerial area indicated by the placement information and corresponding to the ground image, and the misplaced partial aerial area is the partial aerial area that partially overlaps with the correctly positioned partial aerial area, and training the classifier using a set of the ground image and the correctly positioned partial aerial area and a set of the ground image and the misplaced partial aerial area, and both the correctly positioned partial aerial area and the misplaced partial aerial area are used as the partial aerial areas of positive examples.

[0008] The present disclosure further provides a non-transitory computer-readable storage medium storing a program that causes a computer to obtain a ground image having a viewing angle of less than 360 degrees, an aerial image, and configuration information, and train a classifier that determines whether the ground image and the aerial image match. The classifier extracts a plurality of partial aerial regions from the aerial image, and determines that the ground image and the aerial image match when the partial aerial region that matches the ground image exists. Training the classifier includes extracting a correctly positioned partial aerial area and a misplaced partial aerial area from the aerial image, where the correctly positioned partial aerial area is the partial aerial area indicated by the placement information and corresponding to the ground image, and the misplaced partial aerial area is the partial aerial area that partially overlaps with the correctly positioned partial aerial area, and training the classifier using a set of the ground image and the correctly positioned partial aerial area and a set of the ground image and the misplaced partial aerial area, and both the correctly positioned partial aerial area and the misplaced partial aerial area are used as the partial aerial areas of positive examples. Effect of the Invention

[0009] According to the present disclosure, a method is provided for effectively training a classifier for matching ground images and aerial images with a viewing angle of less than 360 degrees. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 shows a classifier trained by the training device of the first embodiment. [Diagram 2] FIG. 2 shows an overview of the training device of the first embodiment. [Diagram 3] FIG. 3 shows an example of a properly placed aerial photography area and a misplaced aerial photography area. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of the training device. [Diagram 5] FIG. 5 is a block diagram showing an example of a hardware configuration of a training device. [Figure 6] FIG. 6 is a flowchart showing the flow of the process executed by the training device. [Figure 7] FIG. 7 illustrates a geolocalization system that includes a classifier. [Figure 8] FIG. 8 shows a partial aerial view obtained using radial lines. [Figure 9] FIG. 9 shows a case where a partial aerial region is extracted from a circular region in an aerial image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An example of an embodiment according to the present disclosure will be described below with reference to the drawings. Note that the same elements in the drawings are given the same reference numerals, and redundant explanations are omitted as necessary. In addition, unless otherwise specified, the predetermined information (e.g., a predetermined value or a predetermined threshold value) is stored in advance in a storage device that can be accessed by a computer that uses the information.

[0012] EMBODIMENT 1 <Summary> The training device of the first embodiment trains a classifier that matches ground images with aerial images (so-called ground-to-aerial cross-view matching). FIG. 1 is a diagram showing a classifier trained by the training device of the first embodiment. The classifier 10 acquires a ground image 20 and an aerial image 30, and determines whether the ground image 20 and the aerial image 30 match. Note that "the ground image and the aerial image match" means that a location captured in the ground image sufficiently overlaps with a location captured in the aerial image.

[0013] The ground image 20 is an image that includes a ground view of a location. The ground image 20 is captured, for example, by a camera held by a pedestrian or a camera installed on a vehicle on a road. The ground image 20 handled by the classifier 10 has a limited field of view (less than 360 degrees). The aerial image 30 is an image that includes a planar view of a location. The aerial image 30 is captured, for example, by a camera mounted on a drone, an aircraft, or an artificial satellite.

[0014] Since the angle of view of the ground image 20 is less than 360 degrees, the location captured in the ground image 20 is a part of the location captured in the aerial image 30. Therefore, the classifier 10 acquires the ground image 20 and the aerial image 30, and determines whether or not the acquired aerial image 30 includes a partial aerial region 40 (i.e., a part of the aerial image 30) that matches the acquired ground image 20. If the acquired aerial image 30 includes a partial aerial region 40 that matches the acquired ground image 20, it is determined that the ground image 20 matches the aerial image 30. On the other hand, if the acquired aerial image 30 does not include a partial aerial region 40 that matches the acquired ground image 20, it is determined that the ground image 20 does not match the aerial image 30. Note that the "ground image that matches the partial aerial region" means that the location captured in the ground image sufficiently overlaps with the location captured in the partial aerial region.

[0015] FIG. 2 is a diagram showing an overview of the training device 2000 of the first embodiment. In order to train the classifier 10 that operates as described above, the training device 2000 operates as follows. The training device 2000 acquires a ground image 20 and an aerial image 30 that includes a partial aerial photographed area 40 that matches the ground image 20. The training device 2000 also acquires arrangement information 50 indicating which part of the aerial photographed image 30 corresponds to the ground image 20. In other words, the arrangement information 50 indicates the partial aerial photographed area 40 that corresponds to the ground image 20. Hereinafter, the partial aerial photographed area 40 indicated by the arrangement information 50 is described as a "properly arranged partial aerial photographed area 42." Using this term, it can be rephrased as "the arrangement information 50 indicates which part of the aerial photographed image 30 should be extracted as the properly arranged partial aerial photographed area 42." The properly arranged partial aerial photographed area 42 may be the most similar to the ground image 20 among the partial aerial photographed areas 40 extracted by the classifier 10.

[0016] The training device 2000 extracts the properly arranged partial aerial-photographed area 42 from the aerial-photographed image 30 using the arrangement information 50. The training device 2000 also extracts the improperly arranged partial aerial-photographed area 43 from the aerial-photographed image 30 using the arrangement information 50. 4The misplaced partial aerial photography area 44 is a partial aerial photography area that partially overlaps with the properly placed partial aerial photography area 42. Fig. 3 shows an example of the properly placed partial aerial photography area 42 and the misplaced partial aerial photography area 44. The misplaced partial aerial photography area 44 shown in Fig. 3 is obtained by slightly rotating the properly placed partial aerial photography area 42.

[0017] Conceptually, the training device 2000 trains the classifier 10 so that not only the correctly positioned partial aerial photography area 42 but also the misaligned partial aerial photography area 44 matches the ground image 20. To realize this feature, the training device 2000 uses both the set of the ground image 20 and the correctly positioned partial aerial photography area 42 and the set of the ground image 20 and the misaligned partial aerial photography area 44 as training data sets having positive example partial aerial photography areas 40 to train the classifier 10. That is, the training device 2000 uses both the correctly positioned partial aerial photography area 42 and the misaligned partial aerial photography area 44 as positive examples of the partial aerial photography area 40 in training the classifier 10. Note that the positive examples of the partial aerial photography area 40 are the partial aerial photography areas 40 that are determined by the classifier 10 to match the ground image 20.

[0018] <Examples of effects> According to the training device 2000 of the first embodiment, the training device 2000 extracts not only the properly positioned partial aerial-photographed region 42 but also the misplaced partial aerial-photographed region 44 from the acquired aerial-photographed image 30, thereby acquiring a plurality of training data sets from a single aerial-photographed image 30. This data expansion makes it easier to create a training data set for training the classifier 10. In addition, by training the classifier to determine that not only the properly positioned partial aerial-photographed region 42 but also the misplaced partial aerial-photographed region 44 match the ground image 20, the classifier 10 can tolerate some degree of misalignment of the partial aerial-photographed region, thereby reducing the calculation load of the classifier 10. This point will be described later.

[0019] A more detailed description of training device 2000 is provided below.

[0020] <Example of functional configuration> 4 is a block diagram showing an example of a functional configuration of the training device 2000. The training device 2000 has an acquisition unit 2020, a partial aerial photography area extraction unit 2040, and a training execution unit 2060. The acquisition unit 2020 acquires a ground image 20, an aerial photography image 30, and arrangement information 50. The partial aerial photography area extraction unit 2040 extracts a properly arranged partial aerial photography area 42 and an incorrectly arranged partial aerial photography area 44 based on the arrangement information 50. The training execution unit 20 6 0 performs training of the classifier 10 using a set of the ground image 20 and the properly positioned partial aerial photographed region 42 and a set of the ground image 20 and the mispositioned partial aerial photographed region 44 as training data sets using positive examples of the partial aerial photographed region 40. Both the properly positioned partial aerial photographed region 42 and the mispositioned partial aerial photographed region 44 are used as positive examples of the partial aerial photographed region 40.

[0021] <Example of hardware configuration> Training device 2000 may be realized by one or more computers, each of which may be a dedicated computer created for implementing training device 2000, or a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0022] Training device 2000 can be realized by installing an application on a computer. The application is realized by a program that causes the computer to function as training device 2000. In other words, the program implements the functional components of training device 2000.

[0023] 5 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes a training device 2000. In FIG. 5, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120.

[0024] The bus 1020 is a data communication path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage element such as a random access memory (RAM) or a read only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and a peripheral device (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN). The storage device 1080 may store the above-mentioned programs. The processor 1040 executes the program to realize each functional component of the training device 2000 .

[0025] The hardware configuration of the computer 1000 is not limited to the configuration shown in FIG. For example, as previously described, training device 2000 may be implemented with multiple computers, which may be connected to one another via a network.

[0026] <Processing flow> 6 is a flowchart showing an example of the flow of processing executed by the training device 2000. The acquisition unit 2020 acquires the ground image 20, the aerial image, and the arrangement information 50 (S102). The partial aerial area extraction unit 2040 extracts the properly arranged partial aerial area 42 from the aerial image 30 (S104). The partial aerial area extraction unit 2040 extracts the incorrectly arranged partial aerial area 44 from the aerial image 30 (S106). The training execution unit 2060 executes training of the classifier 10 using a training data set including the ground image 20 and the properly arranged partial aerial area 42 and a training data set including the ground image 20 and the incorrectly arranged partial aerial area 44 (S108).

[0027] The flow shown in Fig. 6 is an example of the flow of the operation executed by the training device 2000, and the flow of the operation is not limited to that shown in Fig. 6. For example, the training device 2000 can use a training data set including the properly positioned partial aerial-photographed area 42 to extract the properly positioned partial aerial-photographed area 42 and train the classifier 10, and then use a training data set including the mispositioned partial aerial-photographed area 44 to extract the mispositioned partial aerial-photographed area 44 and train the classifier 10.

[0028] <Application example of Classifier 10> There are various applications for the classifier 10. For example, the classifier 10 can be used as part of a system that performs image geolocalization (hereinafter, geolocalization system). Image geolocalization is a technology for identifying the location where an input image was captured. Here, the geolocalization system is merely one example of an application of the classifier 10, and the application of the classifier 10 is not limited to being used in a geolocalization system.

[0029] 7 shows a geolocalization system 200 including the classifier 10. The geolocalization system 200 includes the classifier 10 and a location database 300. The location database 300 includes a plurality of aerial images, each of which has location information attached thereto. An example of the location information can be the Global Positioning System (GPS) coordinates of the location where the corresponding aerial image was taken.

[0030] The geolocalization system 200 receives a query including a ground image from a client (e.g., a user terminal), searches the location database 300 for an aerial image that matches the ground image included in the received query, and thereby identifies the capture location of the ground image. Specifically, until an aerial image that matches the ground image included in the query is detected, the geolocalization system 200 repeats the process of "obtaining one aerial image from the location database 300, inputting the ground image and the obtained aerial image to the classifier 10, and determining whether the output from the classifier 10 indicates that the ground image and the aerial image match." In this way, the geolocalization system 200 can find an aerial image that includes the capture location of the ground image. Since the detected aerial image is associated with location information such as GPS coordinates, the geolocalization system 200 can determine that the capture location of the ground image is the location indicated by the location information associated with the aerial image that matches the ground image.

[0031] It should be noted that the geolocalization system 200 may be implemented in any one or more computers, such as the computer shown in FIG.

[0032] <Detailed Description of the Classifier 10> As described above, the classifier 10 acquires the ground image 20 and the aerial image 30, and 3 0. Specifically, the classifier 10 extracts a plurality of partial aerial-photographed regions 40 from the aerial-photographed image 30, and compares each partial aerial-photographed region 40 with the ground image 20.

[0033] For the purpose of comparison, the classifier 10 extracts features from the ground image 20 and each partial aerial photography area 40. Then, for each partial aerial photography area 40, the classifier 10 determines whether the features of the partial aerial photography area 40 are sufficiently similar to the features of the ground image 20. To this end, the classifier 10 calculates an index (hereinafter, similarity score) indicating the degree of similarity between the features of the ground image 20 and the features of the partial aerial photography area 40.

[0034] If a partial aerial photographed region 40 exists having features sufficiently similar to those of the ground image 20 (for example, a similarity score calculated for the features of the ground image 20 and the partial aerial photographed region 40 is equal to or greater than a predetermined threshold), the classifier 10 determines that the ground image 20 matches the aerial photographed image 30. On the other hand, if a partial aerial photographed region 40 having features sufficiently similar to those of the ground image 20 does not exist, the classifier 10 determines that the ground image 20 and the aerial photographed image 30 do not match.

[0035] There are various methods for the classifier 10 to extract the partial aerial regions 40 from the aerial image 30. For example, the classifier 10 divides the aerial image 30 into a plurality of partial aerial regions 40 using radial lines. FIG. 8 shows a partial aerial region 40 obtained using radial lines. In FIG. 8, the center of the aerial image 30 is set as the origin, and a plurality of radial lines are drawn from the origin. The angle between two adjacent radial lines is 45 degrees. As a result, eight partial aerial regions 40-1 to 40-8 are extracted from the aerial image 30.

[0036] The magnitude of the angle between two adjacent radial lines may be set in advance, may be specified by a user of the classifier 10, or may be determined based on the size of the field of view of the ground image 20. The magnitude of the angle between two adjacent radial lines is preferably set to the same as the size of the field of view of the ground image 20. For example, when the size of the field of view of the ground image 20 is 45 degrees, the magnitude of the angle between two adjacent radial lines is preferably set to 45 degrees.

[0037] The classifier 10 may use multiple sizes for the angle between two adjacent radial lines. Assume that 45 degrees and 60 degrees are used as the angles between two adjacent radial lines. In this case, for example, the classifier 10 first sets the angle between two adjacent radial lines to 45 degrees and extracts eight partial aerial photography regions 40. Next, the classifier 10 sets the angle between two adjacent radial lines to 60 degrees and extracts six partial aerial photography regions 40. As a result, the classifier 10 obtains a total of 14 partial aerial photography regions 40. This allows the classifier 10 to find partial aerial photography regions 40 that match the ground image 20 without knowing the field of view of the ground image 20.

[0038] The classifier 10 can extract the partial aerial photography region 40 such that a portion of the partial aerial photography region 40 overlaps with a portion of one or more other partial aerial photography regions 40. In this case, for example, the classifier 10 can define radial lines for extracting the partial aerial photography region 40 from the aerial photography image 30 using a sliding window method. Hereinafter, the two radial lines for extracting the partial aerial photography region 40 from the aerial photography image 30 are referred to as a "first radial line" and a "second radial line", respectively. Note that the angle between the first radial line and the x-axis is smaller than the angle between the second radial line and the x-axis.

[0039] In this case, for each partial aerial region 40, the classifier 10 can define the first radial line and the second radial line as follows.

number

[0040] The smaller the stride size is set, the more precisely the classifier 10 can match between the ground image 20 and the partial aerial image 40. Therefore, from the viewpoint of the accuracy of cross-view matching between the ground image and the aerial image, it is preferable to set a very small value for the stride. However, a smaller stride size increases the calculation load of the classifier 10.

[0041] The training device 2000 alleviates this problem by training the classifier 10 to determine that the misaligned partial aerial region 44 matches the ground image 20. This training allows the classifier 10 to tolerate some degree of misalignment of the partial aerial region 40. That is, the classifier 10 does not need to extract a perfectly aligned partial aerial region to determine whether the ground image 20 matches the aerial image 20. This can improve the accuracy of the ground-to-aerial cross-view matching performed by the classifier 10 without setting a very small value for the stride, thereby avoiding a large computational load on the classifier 10.

[0042] Instead of extracting the partial aerial region 40 from the entire aerial image 30, the classifier 10 may extract the partial aerial region 40 from a part of the aerial image 30. For example, the classifier 10 crops a circular region from the aerial image 30, and then extracts multiple partial aerial regions 40 from the circular region using radial lines. This makes it possible to obtain multiple partial aerial regions 40 that are the same in size and shape. FIG. 9 shows partial aerial regions 40 extracted from a circular region of the aerial image 30. 0 9, a circular region 32 is a circle inscribed in the aerial image 30, and its center corresponds to the center of the aerial image 30.

[0043] The classifier 10 is configured in a trainable manner. For example, the classifier 10 has one or more machine learning-based models that include trainable parameters (such as neural network weights). For example, the classifier 10 has two feature extractors, each configured as a machine learning-based model. One of them is supplied with a ground image 20 and outputs features of the ground image 20. The other one obtains a partial aerial photographed region 40 and outputs features of the partial aerial photographed region 40. One example of a machine learning-based model that can be employed to implement a feature extractor is a feature extraction layer of a Convolutional Neural Network (CNN).

[0044] <Getting image and location information: S102> The acquisition unit 2020 acquires the ground image 20, the aerial image 30, and the arrangement information 50. There are various methods for the acquisition unit 2020 to acquire these data. For example, the ground image 20, the aerial image 30, and the arrangement information 50 are stored in advance in a storage device. In this case, the acquisition unit 2020 accesses the storage device and reads out the ground image 20, the aerial image 30, and the arrangement information 50. In another example, a set of the ground image 20, the aerial image 30, and the arrangement information 50 is input to the training device 2000 by a user. In another example, the acquisition unit 2020 may receive a set of the ground image 20, the aerial image 30, and the arrangement information 50 transmitted from another device.

[0045] The configuration information 50 may indicate in various ways which portion of the aerial image 30 should be extracted as the orthogonally positioned partial aerial region 42. For example, the configuration information 50 may indicate a first radial line and a second radial line from which the orthogonally positioned partial aerial region 42 can be extracted from the aerial image 30. The configuration information 50 may define the first radial line by an angle between the first radial line and the x-axis. Similarly, the configuration information 50 may define the second radial line by an angle between the second radial line and the x-axis. In another example, the configuration information 50 may define the second radial line by an angle between the first radial line and the second radial line.

[0046] <Extraction of partial aerial photography area: S104, S104> The partial aerial photography area extraction unit 2040 extracts the properly positioned partial aerial photography area 42 and the incorrectly positioned partial aerial photography area 44 from the aerial photography image 30 (S104, S106). For the properly positioned partial aerial photography area 42, the partial aerial photography area extraction unit 2040 may extract the partial aerial photography area 40 indicated as the properly positioned partial aerial photography area 42 by the arrangement information 50.

[0047] The partial aerial photography area extraction unit 2040 may crop a part of the aerial photography image 30 and extract the properly positioned partial aerial photography area 42 from the cropped area. For example, as shown in Fig. 9, the partial aerial photography area extraction unit 2040 may crop a circular area that is inscribed in the aerial photography image 30 and whose center corresponds to the center of the aerial photography image 30, and extract the properly positioned partial aerial photography area 42 from this circular area using the first radial line and the second radial line indicated by the arrangement information 50. The same applies to the extraction of the mispositioned partial aerial photography area 44.

[0048] For the misplaced partial aerial photography area 44, the partial aerial photography area extraction unit 2040 can calculate the first radial line and the second radial line of the misplaced partial aerial photography area 44 by rotating the properly placed partial aerial photography area 42 at a certain rotation angle. The rotation angle (i.e., the angle at which the radial lines of the properly placed partial aerial photography area 42 are rotated to obtain the radial lines of the misplaced partial aerial photography area 44) may be predefined, provided by the user, or dynamically calculated.

[0049] In the first case, the partial aerial photography area extraction unit 2040 may obtain rotation angle information indicating a rotation angle. This rotation angle information may be stored in a storage device in advance. The partial aerial photography area extraction unit 2040 rotates the first radial line and the second radial line of the properly arranged partial aerial photography area 42 by the rotation angle indicated by the rotation angle information, and obtains the first radial line and the second radial line of the misarranged partial aerial photography area 44. Then, the partial aerial photography area extraction unit 2040 extracts the misarranged partial aerial photography area 44 using the first radial line and the second radial line calculated for the misarranged partial aerial photography area 44.

[0050] When a rotation angle is provided by the user, the partial aerial photography area extraction unit 2040 may obtain a user input indicating the rotation angle. The partial aerial photography area extraction unit 2040 obtains the first radial line and the second radial line of the misplaced partial aerial photography area 44 by rotating the first radial line and the second radial line of the properly placed partial aerial photography area 42 by the rotation angle provided by the user.

[0051] When the rotation angle is dynamically determined, the partial aerial photography area extraction unit 2040 may calculate a random number as the rotation angle. For example, the rotation angle may be sampled from a zero-mean Gaussian distribution N(0,σ). The standard deviation σ may be stored in advance in a storage device or may be provided by a user. The partial aerial photography area extraction unit 2040 obtains the first radial line and the second radial line of the misplaced partial aerial photography area 44 by rotating the first radial line and the second radial line of the properly placed partial aerial photography area 42 by a randomly determined rotation angle. Sampling a random number as the rotation angle is effective in preventing the classifier 10 from overfitting the training data.

[0052] The partial aerial photography area extraction unit 2040 may extract a plurality of misarranged partial aerial photography areas 44 using a plurality of different rotation angles. When the rotation angles are defined in advance, the partial aerial photography area extraction unit 2040 may obtain rotation angle information indicating a plurality of rotation angles. The partial aerial photography area extraction unit 2040 obtains a plurality of misarranged partial aerial photography areas 44 by obtaining a misarranged partial aerial photography area 44 for each rotation angle indicated by the rotation angle information.

[0053] When a rotation angle is provided by the user, the partial aerial photography area extraction unit 2040 may acquire user input indicating a plurality of rotation angles. The partial aerial photography area extraction unit 2040 acquires a plurality of misarranged partial aerial photography areas 44 by acquiring a misarranged partial aerial photography area 44 for each rotation angle indicated in the user input.

[0054] When the rotation angle is dynamically calculated, the partial aerial photography area extraction unit 2040 may calculate multiple random numbers as the rotation angle. It is assumed that the number of misarranged partial aerial photography areas 44 to be extracted is M. In this case, the partial aerial photography area extraction unit 2040 obtains M rotation angles by sampling a random value M times from a Gaussian distribution. The partial aerial photography area extraction unit 2040 obtains M misarranged partial aerial photography areas 44 by obtaining a misarranged partial aerial photography area 44 for each of the sampled M rotation angles.

[0055] <Executing training of the classifier 10: S108> The training execution unit 2060 trains the classifier 10 using the correctly positioned partial aerial-photographed area 42 and the incorrectly positioned partial aerial-photographed area 44. Conceptually, the classifier 10 is trained to match not only the correctly positioned partial aerial-photographed area 42 but also the incorrectly positioned partial aerial-photographed area 44 with the ground image 20. For this reason, the training execution unit 2060 trains the classifier 10 using both the correctly positioned partial aerial-photographed area 42 and the incorrectly positioned partial aerial-photographed area 44 as positive examples.

[0056] For example, the training execution unit 2060 inputs a set of the ground image 20 and the properly positioned partial aerial photography area 42, and obtains a similarity score that indicates the similarity between the feature amount of the ground image 20 and the feature amount of the properly positioned partial aerial photography area 42. The training execution unit 2060 then calculates a loss based on the obtained similarity score, and updates trainable parameters (e.g., neural network weights) of the classifier 10 so as to reduce the loss. Note that there are various well-known methods for updating trainable parameters of a machine learning-based model based on the calculated loss, and any of these methods can be used to train the classifier 10.

[0057] The same can be applied to training of a classifier using the misarranged aerial-photography portion 44. Specifically, the training execution unit 2060 inputs a set of the ground image 20 and the misarranged aerial-photography portion 44, and calculates a similarity score that indicates the similarity between the feature amount of the ground image 20 and the feature amount of the misarranged aerial-photography portion 44. Then, the training execution unit 2060 calculates a loss based on the obtained similarity score, and updates the trainable parameters of the classifier 10 so as to reduce the loss.

[0058] The partial aerial photography area of ​​a negative example can also be used for training the classifier 10. For example, the training execution unit 2060 can use a set of the ground image 20, the partial aerial photography area 40 of a positive example, and the partial aerial photography area 40 of a negative example. As the positive example, the properly arranged partial aerial photography area 42 and the misarranged partial aerial photography area 44 are used, and as the negative example, another partial aerial photography area is used. Note that the aerial photography image 30 from which the partial aerial photography area 40 of the negative example is extracted may be the same aerial photography image 30 as the aerial photography image 30 from which the properly arranged partial aerial photography area 42 and the misarranged partial aerial photography area 44 are extracted, or may be a different aerial photography image 30. In the former case, the partial aerial photography area 40 of the negative example is a partial aerial photography area 40 that does not overlap with the properly arranged partial aerial photography area 42. In the latter case, the acquisition unit 2020 may acquire one or more negative example partial aerial regions 40, or may acquire one or more additional aerial images 30 from which the training execution unit 2060 may extract one or more negative example partial aerial regions 40.

[0059] There are various methods for training the classifier 10 using negative examples. For example, the training execution unit 2060 may calculate a triplet loss to train the classifier 10. Specifically, the training execution unit 2060 inputs the ground image 20 and the partial aerial photographed area 40 of the positive example that are included in the same set, and obtains a similarity score for them. The training execution unit 2060 also inputs the ground image 20 and the partial aerial photographed area 40 of the negative example that are included in the same set, and obtains a similarity score for them. Then, the training execution unit 2060 calculates a triplet loss using the similarity score calculated for the ground image 20 and the partial aerial photographed area 40 of the positive example and the similarity score calculated for the ground image 20 and the partial aerial photographed area 40 of the negative example, and updates the trainable parameters of the classifier 10 based on the calculated triplet loss. Note that there are well-known methods for updating the trainable parameters of a machine learning-based model based on the calculated triplet loss, and the classifier 10 can be trained using any of them.

[0060] <Output from Training Device 2000> The training device 2000 may output data (hereinafter, output information) obtained as a result of training the classifier 10. For example, the output information may include training parameters of the classifier 10 obtained as a result of training the classifier 10. By setting the trainable parameters output from the training device 2000 in the classifier 10, it is possible to obtain a classifier 10 that can determine that not only the correctly positioned partial aerial-photographed region 42 but also the mispositioned partial aerial-photographed region 44 matches the ground image 20. In another example, the output information may include the entire classifier 10 (for example, a program implementing the classifier 10, its non-trainable parameters, and the trainable parameters updated by the training device 2000).

[0061] There are various ways in which the training device 2000 can output the output information. For example, the training device 2000 can store the output information in a storage device. In another example, the training device 2000 can output the output information to a display device, which can display the contents of the output information. In another example, the training device 2000 can output the output information to another computer, such as a computer included in the geolocalization system 200 shown in FIG. 7.

[0062] The program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs). The program may also be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0063] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that are understandable to those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present invention.

[0064] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) At least one processor; a storage element in which a sequence of instructions is stored; The at least one processor executes the sequence of instructions to: Acquire ground images having a viewing angle of less than 360 degrees, aerial images, and location information; A classifier is trained to determine whether the ground image and the aerial image match, the classifier extracts a plurality of partial aerial regions from the aerial image, and determines that the ground image and the aerial image match when a partial aerial region that matches the ground image exists; The training of the classifier includes: extracting a properly positioned partial aerial photography area and a misplaced partial aerial photography area from the aerial photography image, the properly positioned partial aerial photography area being the partial aerial photography area indicated by the placement information and corresponding to the ground image, and the misplaced partial aerial photography area being the partial aerial photography area partially overlapping with the properly positioned partial aerial photography area; A training device comprising: training the classifier using a set of the ground image and the correctly positioned partial aerial area, and a set of the ground image and the misplaced partial aerial area, wherein both the correctly positioned partial aerial area and the misplaced partial aerial area are used as the partial aerial area of ​​a positive example. (Appendix 2) the arrangement information indicates a first radial line and a second radial line; The extraction of the normal placement partial aerial photography area includes extracting an area between the first line and the second line in the aerial photography image as the normal placement partial aerial photography area, The extraction of the misplaced aerial photography area includes: rotating the first radial line and the second radial line; The training device of claim 1, further comprising: extracting an area between the rotated first radial line and the rotated second radial line in the aerial image as the misplaced portion aerial area. (Appendix 3) extracting the misplaced portion of the aerial region includes sampling values ​​from a probability distribution; 3. The training device of claim 2, wherein the first radial line and the second radial line are rotated by an angle determined by the sampled values. (Appendix 4) 4. The training apparatus of claim 3, wherein the probability distribution is a zero-mean Gaussian distribution. (Appendix 5) The at least one processor is further configured to extract a circular region from the aerial image; The training device according to any one of appendixes 1 to 4, wherein the correctly positioned aerial photography area and the incorrectly positioned aerial photography area are extracted from the circular area. (Appendix 6) 1. A computer-implemented control method comprising: Acquiring ground images having a viewing angle of less than 360 degrees, aerial images, and location information; and training a classifier that determines whether the ground image and the aerial image match, the classifier extracting a plurality of partial aerial regions from the aerial image and determining that the ground image and the aerial image match when a partial aerial region that matches the ground image exists; The training of the classifier includes: extracting a properly positioned partial aerial photography area and a misplaced partial aerial photography area from the aerial photography image, the properly positioned partial aerial photography area being the partial aerial photography area indicated by the placement information and corresponding to the ground image, and the misplaced partial aerial photography area being the partial aerial photography area partially overlapping with the properly positioned partial aerial photography area; A control method comprising: training the classifier using a set of the ground image and the correctly positioned partial aerial area, and a set of the ground image and the misplaced partial aerial area, wherein both the correctly positioned partial aerial area and the misplaced partial aerial area are used as the partial aerial area of ​​a positive example. (Appendix 7) the arrangement information indicates a first radial line and a second radial line; The extraction of the normal placement partial aerial photography area includes extracting an area between the first line and the second line in the aerial photography image as the normal placement partial aerial photography area, The extraction of the misplaced aerial photography area includes: rotating the first radial line and the second radial line; The control method described in Appendix 6 includes extracting an area between the rotated first radial line and the rotated second radial line in the aerial image as the misplaced portion aerial area. (Appendix 8) extracting the misplaced portion of the aerial region includes sampling values ​​from a probability distribution; 8. The control method of claim 7, wherein the first radial line and the second radial line are rotated by an angle determined by the sampled value. (Appendix 9) 9. The control method of claim 8, wherein the probability distribution is a zero-mean Gaussian distribution. (Appendix 10) extracting a circular region from the aerial image; The control method according to any one of appendix 6 to 9, wherein the correctly positioned aerial-photographed area and the incorrectly positioned aerial-photographed area are extracted from the circular area. (Appendix 11) Acquiring ground images having a viewing angle of less than 360 degrees, aerial images, and location information; and training a classifier that determines whether the ground image and the aerial image match, the classifier extracting a plurality of partial aerial regions from the aerial image, and determining that the ground image and the aerial image match when a partial aerial region that matches the ground image exists; The training of the classifier includes: extracting a properly positioned partial aerial photography area and a misplaced partial aerial photography area from the aerial photography image, the properly positioned partial aerial photography area being the partial aerial photography area indicated by the placement information and corresponding to the ground image, and the misplaced partial aerial photography area being the partial aerial photography area partially overlapping with the properly positioned partial aerial photography area; A non-transitory computer-readable storage medium comprising: training the classifier using a set of the ground image and the correctly positioned partial aerial area, and a set of the ground image and the misplaced partial aerial area, wherein both the correctly positioned partial aerial area and the misplaced partial aerial area are utilized as the partial aerial area of ​​a positive example. (Appendix 12) the arrangement information indicates a first radial line and a second radial line; The extraction of the normal placement partial aerial photography area includes extracting an area between the first line and the second line in the aerial photography image as the normal placement partial aerial photography area, The extraction of the misplaced aerial photography area includes: rotating the first radial line and the second radial line; The storage medium of claim 11, further comprising: extracting an area between the rotated first radial line and the rotated second radial line in the aerial image as the misplaced portion aerial area. (Appendix 13) extracting the misplaced portion of the aerial region includes sampling values ​​from a probability distribution; 13. The storage medium of claim 12, wherein the first radial line and the second radial line are rotated by an angle determined by the sampled values. (Appendix 14) 14. The storage medium of claim 13, wherein the probability distribution is a zero-mean Gaussian distribution. (Appendix 15) The program further causes the computer to extract a circular region from the aerial image; A storage medium described in any one of Appendix 11 to 14, wherein the correctly positioned portion aerial-photography area and the incorrectly positioned portion aerial-photography area are extracted from the circular area. [Explanation of symbols]

[0065] 10 Classifier 20 Ground Images 30 Aerial Images 32 Circular Areas 40 Partial aerial photography area 42 Orthogonal Aerial Photography Area 44 Misplaced aerial photography area 50 Placement information 200 Geolocalization System 300 Location Database 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 training equipment 2020 Acquisition Department 2040 Partial aerial photography area extraction section 2060 Training Execution Department

Claims

1. An acquisition means for acquiring a ground image having a viewing angle of less than 360 degrees, an aerial image, and location information; a training execution means for training a classifier that determines whether the ground image and the aerial image match, The classifier is Extracting a plurality of partial aerial regions from the aerial image; If the partial aerial photographed area that matches the ground image exists, it is determined that the ground image matches the aerial photographed image; The training of the classifier includes: Extracting a properly placed aerial photographed area and a misplaced aerial photographed area from the aerial photographed image; Training the classifier using the set of the ground image and the properly positioned partial aerial region and the set of the ground image and the misaligned partial aerial region; The properly positioned partial aerial photography area is the partial aerial photography area indicated by the position information and corresponding to the ground image, The misplaced partial aerial photography area is the partial aerial photography area that partially overlaps with the properly placed partial aerial photography area, In training the classifier, both the properly positioned partial aerial region and the misaligned partial aerial region are used as the partially aerial region of a positive example; the placement information indicating a first radial line and a second radial line; The extraction of the normal placement partial aerial photography area includes extracting an area between the first radial line and the second radial line in the aerial photography image as the normal placement partial aerial photography area, The extraction of the misplaced aerial photography area includes: rotating the first radial line and the second radial line; A training device comprising: extracting an area between the rotated first radial line and the rotated second radial line in the aerial image as the misplaced portion aerial area.

2. extracting the misplaced portion of the aerial region includes sampling values ​​from a probability distribution; The training device of claim 1 , wherein the first radial line and the second radial line are rotated by an angle determined by the sampled values.

3. The training device of claim 2 , wherein the probability distribution is a zero-mean Gaussian distribution.

4. The training execution means extracts a circular area from the aerial image, The training device according to claim 1 , wherein the properly positioned aerial photography area and the misaligned aerial photography area are extracted from the circular area.

5. 1. A computer-implemented control method comprising: Acquiring a ground image having a viewing angle of less than 360 degrees, an aerial image, and location information; training a classifier that determines whether the ground image and the aerial image match; The classifier is Extracting a plurality of partial aerial regions from the aerial image; If the partial aerial photographed area that matches the ground image exists, it is determined that the ground image matches the aerial photographed image; The training of the classifier includes: Extracting a properly placed aerial photographed area and a misplaced aerial photographed area from the aerial photographed image; Training the classifier using the set of the ground image and the properly positioned partial aerial region and the set of the ground image and the misaligned partial aerial region; The properly positioned partial aerial photography area is the partial aerial photography area indicated by the position information and corresponding to the ground image, The misplaced partial aerial photography area is the partial aerial photography area that partially overlaps with the properly placed partial aerial photography area, In training the classifier, both the properly positioned partial aerial region and the misaligned partial aerial region are used as the partially aerial region of a positive example; the placement information indicating a first radial line and a second radial line; The extraction of the normal placement partial aerial photography area includes extracting an area between the first radial line and the second radial line in the aerial photography image as the normal placement partial aerial photography area, The extraction of the misplaced aerial photography area includes: rotating the first radial line and the second radial line; A control method comprising: extracting an area between the rotated first radial line and the rotated second radial line in the aerial image as the misplaced portion aerial area.

6. extracting the misplaced portion of the aerial region includes sampling values ​​from a probability distribution; The control method of claim 5 , wherein the first radial line and the second radial line are rotated by an angle determined by the sampled values.

7. Acquiring a ground image having a viewing angle of less than 360 degrees, an aerial image, and location information; training a classifier that determines whether the ground image and the aerial image match; The classifier is Extracting a plurality of partial aerial regions from the aerial image; If the partial aerial photographed area that matches the ground image exists, it is determined that the ground image matches the aerial photographed image; The training of the classifier includes: Extracting a properly placed aerial photographed area and a misplaced aerial photographed area from the aerial photographed image; Training the classifier using the set of the ground image and the properly positioned partial aerial region and the set of the ground image and the misaligned partial aerial region; The properly positioned partial aerial photography area is the partial aerial photography area indicated by the position information and corresponding to the ground image, The misplaced partial aerial photography area is the partial aerial photography area that partially overlaps with the properly placed partial aerial photography area, In training the classifier, both the properly positioned partial aerial region and the misaligned partial aerial region are used as the partially aerial region of a positive example; the placement information indicating a first radial line and a second radial line; The extraction of the normal placement partial aerial photography area includes extracting an area between the first radial line and the second radial line in the aerial photography image as the normal placement partial aerial photography area, The extraction of the misplaced aerial photography area includes: rotating the first radial line and the second radial line; extracting an area between the rotated first radial line and the rotated second radial line in the aerial image as the misplaced portion aerial area.

Images