Training apparatus, training method, data matching apparatus, data matching method, and storage medium

Abstract

A training apparatus performs: acquiring an anchor sample, a positive sample, and a plurality of negative samples, extracting first hard-negative samples from the plurality of the first negative samples, selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample. The feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.

Description

TRAINING APPARATUS, TRAINING METHOD, DATA MATCHING APPARATUS, DATA MATCHING METHOD, AND STORAGE MEDIUM

[0001] The present disclosure generally relates to a training apparatus, a training method, a data matching apparatus, a data matching method, and a storage medium.

[0002] A computer system for performing data matching has been developed. For example, NPL1 discloses a system comprising a set of CNNs (Convolutional Neural Networks) to match a ground-view image against an aerial-view image. Specifically, one of the CNNs acquires a set of a ground-view image and orientation maps that indicate orientations (azimuth and altitude) for each location captured on the ground-view image, and extracts features therefrom. The other one acquires a set of an aerial-view image and orientation maps that indicate orientations (azimuth and range) for each location captured on the aerial-view image, and extracts features therefrom. Then, the system determines whether the ground-view image matches the aerial-view image based on the extracted features.

[0003] NPL1: Liu Liu and Hongdong Li, "Lending Orientation to Neural Networks for Cross-view Geo-localization", [online], March 29, 2019, [retrieved on 2024-7-30], retrieved from <arXiv, https: / / arxiv.org / pdf / 1903.12351.pdf>

[0004] NPL1 does not mention how to select data to be used for the training of the system. An objective of the present disclosure is to provide a novel technique to train models used for data matching.

[0005] The present disclosure provides a training apparatus comprising: at least one memory that is configured to store instructions; and at least one processor that is configured to execute the instructions to: acquire an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample; extract first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample; select, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and update a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample. 　　The first feature extracting model being configured to take data of the first type and output a feature value thereof. The second feature extracting model being configured to take data of the second type and output a feature value thereof. The feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.

[0006] The present disclosure further provides a training method that is performed by a computer. The training method comprises: acquiring an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample; extracting first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample; selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample. 　　The first feature extracting model being configured to take data of the first type and output a feature value thereof. The second feature extracting model being configured to take data of the second type and output a feature value thereof. The feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.

[0007] The present disclosure further provides a storage medium storing a program that cause a computer to execute: acquiring an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample; extracting first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample; selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample. 　　The first feature extracting model being configured to take data of the first type and output a feature value thereof. The second feature extracting model being configured to take data of the second type and output a feature value thereof. The feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.

[0008] The present disclosure provides a data matching apparatus comprising at least one memory that is configured to store instructions and at least one processor that is configured to execute the instructions to: acquire first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data; extract a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data; perform feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value; perform feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and compare the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.

[0009] The present disclosure further provides a data matching method that is performed by a computer. The data matching method comprises: acquiring first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data; extracting a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data; performing feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value; performing feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and comparing the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.

[0010] The present disclosure further provides another storage medium storing a program that cause a computer to execute: acquiring first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data; extracting a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data; performing feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value; performing feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and comparing the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.

[0011] According to the present disclosure, it is possible to provide a novel technique to train models used for data matching.

[0012] Fig. 1 illustrates a first example configuration of data matching apparatus to be trained.Fig. 2 illustrates an overview of a training apparatus that trains the data matching apparatus 300.Fig. 3 is a block diagram showing an example of the functional configuration of the training apparatus.Fig. 4 is a block diagram illustrating an example of the hardware configuration of a computer with which the training apparatus is implemented.Fig. 5 illustrates an example of the ground-view image, the aerial-view image, and the map image.Fig. 6 illustrates an example of the 3D shape data.Fig. 7 illustrates examples of keyword matrix data.Fig. 8 illustrates a geo-localization system that includes the data matching apparatus.Fig. 9 is a flowchart illustrating an example flow of processes performed by the training apparatus.Fig. 10 illustrates the feature values of the anchor sample and the negative samples in a feature space.Fig. 11 illustrates the determination of the selected negative samples in the case where two or more anchor samples are acquired.Fig. 12 illustrates an example flow of processes performed by the training apparatus to repeatedly update the data matching apparatus.Fig. 13 illustrates a data matching apparatus to be trained.Fig. 14 illustrates an overview of a training apparatus that trains the data matching apparatus.Fig. 15 is a flowchart illustrating an example flow of processes performed by the training apparatus.Fig. 16 illustrates a first example way of updating the data matching apparatus including the third feature extracting model.Fig. 17 illustrates a second example way of updating the data matching apparatus including the third feature extracting model.Fig. 18 illustrates the data matching apparatus that takes a set of three types of representation data to be compared with the first-type representation data.Fig. 19 illustrates a first example of multi-stage feature fusion in a case where the data matching apparatus takes five types of representation data.Fig. 20 illustrates a second example of multi-stage feature fusion in a case where the data matching apparatus takes five types of representation data.Fig. 21 illustrates a representation data generated by removing a part of another representation data.Fig. 22 illustrates an overview of a training apparatus that trains the data matching apparatus.Fig. 23 is a flowchart illustrating an example flow of processes performed by the training apparatus.

[0013] Example embodiments according to the present disclosure will be described hereinafter with reference to the drawings. The same numeral signs are assigned to the same elements throughout the drawings, and redundant explanations are omitted as necessary. In addition, predetermined information (e.g., a predetermined value or a predetermined threshold) is stored in advance in a storage device to which a computer using that information has access unless otherwise described.

[0014] FIRST EXAMPLE EMBODIMENT <Overview> 　　Fig. 1 illustrates a first example configuration of data matching apparatus 300 to be trained. The data matching apparatus 300 of this example is configured to handle two types of representation data. The representation data is data that represents a specific place in the real world: e.g., a ground-view image on which a ground view of a specific place is captured. The types of representation data may include a ground-view image, an aerial-view image, a map image, a three-dimensional (3D) shape data, a keyword matrix data, location information, etc. The detailed explanations about each type of representation data will be described later.

[0015] The data matching apparatus 300 takes two types of representation data, and determine whether these two types of representation data match each other. The fact that "two types of representation data match each other" means these two types of representation data represent the same place as each other.

[0016] The two types of representation data handled by the data matching apparatus 300 is referred to as "first type" and "second type", respectively. The representation data of the first type is referred to as "first-type representation data 10", while the representation data of the second type is referred to as "second-type representation data 20".

[0017] The data matching apparatus 300 includes a feature extracting model for each type of representation data handled by the data matching apparatus 300. In the example shown by Fig. 1, the data matching apparatus 300 includes a first feature extracting model 310 and a second feature extracting model 320.

[0018] The first feature extracting model 310 is configured to take a first-type representation data 10 as input, perform computations on the first-type representation data 10, and output a value (e.g., vector or tensor) that represents the feature value of the first-type representation data 10. The first feature extracting model 310 may be implemented as a machine learning-based model, such as a neural network.

[0019] The second feature extracting model 320 is configured to take a second-type representation data 20 as input, perform computations on the second-type representation data 20, and output a value (e.g., vector or tensor) that represents the feature value of the second-type representation data 20. The second feature extracting model 320 may be implemented as a machine learning-based model, such as a neural network.

[0020] The data matching apparatus 300 compares the feature value of the first-type representation data 10 and the feature value of the second-type representation data 20 to determine whether the first-type representation data 10 and the second-type representation data 20 match each other. The first feature extracting model 310 and the second feature extracting model 320 are trained to enable the data matching apparatus 300 to make accurate determinations.

[0021] Fig. 2 illustrates an overview of a training apparatus 2000 that trains the data matching apparatus 300. It is noted that Fig. 2 does not limit operations of the training apparatus 2000, but merely show an example of possible operations of the training apparatus 2000.

[0022] The training apparatus 2000 acquires one or more anchor samples 30, one or more positive samples 40, and a plurality of negative samples 50. Unless otherwise stated, for the brevity in the description, the training apparatus 2000 is assumed to acquire a single anchor sample 30 and a single positive sample 40.

[0023] The anchor sample 30 is a first-type representation data 10. The positive sample 40 is the second-type representation data 20 that represents the place same as the place represented by the anchor sample 30. The negative sample 50 is a second-type representation data 20 that represents a place different from the place represented by the anchor sample 30.

[0024] The training apparatus 2000 selects a negative sample 50 to be used for the training of the data matching apparatus 300 from the plurality of the negative samples 50. Hereinafter, the negative sample 50 that is selected to be used for the training of the data matching apparatus 300 is referred to as "selected negative sample 80".

[0025] To select one of the negative samples 50 as the selected negative sample 80, the training apparatus 2000 extracts hard-negative samples 60 from the negative samples 50. The hard-negative samples 60 are the negative samples 50 whose feature values are so similar to the feature value of the anchor sample 30 that it is difficult to accurately determine that they do not match the anchor sample 30. Hereinafter, a set of the hard-negative samples 60 is referred to as "hard-negative group 70". Then, the training apparatus 2000 selects, as the selected negative sample 80, one of the hard-negative samples 60 based on the geographical distance between the place represented by the anchor sample 30 and the place represented by each hard-negative sample 60.

[0026] The training apparatus 2000 updates the data matching apparatus 300 based on the feature value of the anchor sample 30, the feature value of the positive sample 40, and the feature value of the selected negative sample 80. Specifically, the training apparatus 2000 computes a loss based on the feature values of the anchor sample 30, the positive sample 40, and the selected negative sample 80, and updates the first feature extracting model 310 and the second feature extracting model 320 based on the computed loss.

[0027] <Example of Advantageous Effect> 　　The training apparatus 2000 extracts hard-negative samples 60 from the negative samples 50, and select one of the hard-negative samples 60 as the selected negative sample 80 based on the geographical distance between the place represented by the anchor sample 30 and the place represented by each hard-negative sample 60. Then, the selected negative sample 80 is used to update the data matching apparatus 300. Thus, according to the training apparatus 2000, a novel technique to select the negative sample to be used for the training of an apparatus that performs data matching is provided. In particular, the training apparatus 2000 is novel and unique in that the geographical distance between the places represented by the samples is taken into consideration.

[0028] Hereinafter, a more detailed explanation of the training apparatus 2000 will be described.

[0029] <Example of Functional Configuration> 　　Fig. 3 is a block diagram showing an example of the functional configuration of the training apparatus 2000. The training apparatus 2000 includes an acquiring unit 2020, a selecting unit 2040, and an updating unit 2060. The acquiring unit 2020 acquires one or more anchor samples 30, one or more positive samples 40, and a plurality of negative samples 50. The selecting unit 2040 extracts hard-negative samples 60 from the plurality of negative samples 50. Then, the selecting unit 2040 selects one of the hard-negative samples 60 as the selected negative sample 80 based on the geographical distance between the place represented by the anchor sample 30 and the place represented by each hard-negative sample 60. The updating unit 2060 updates the data matching apparatus 300 based on the feature values of the anchor sample 30, the positive sample 40, and the selected negative sample 80.

[0030] <Example of Hardware Configuration> <<Training Apparatus 2000>> 　　The training apparatus 2000 may be implemented by one or more computers. Each of the one or more computers may be a special-purpose computer manufactured for implementing the training apparatus 2000, or may be a general-purpose computer like a personal computer (PC), a server machine, or a mobile device.

[0031] The training apparatus 2000 may be implemented by installing an application on the one or more computers. The application is implemented with a program that causes one or more computers to function as the training apparatus 2000. In other words, the program is an implementation of the functional units of the training apparatus 2000.

[0032] Fig. 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 with which the training apparatus 2000 is implemented. In Fig. 4, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0033] The bus 1020 is a data transmission channel in order for the processor 1040, the memory 1060, the storage device 1080, and the I / O interface 1100, and the network interface 1120 to mutually transmit and receive data. The processor 1040 is a processer, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), or DSP (Digital Signal Processor). The memory 1060 is a primary memory component, such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage device 1080 is a secondary memory component, such as a hard disk, an SSD (Solid State Drive), or a memory card. The I / O interface 1100 is an interface between the computer 1000 and peripheral devices, such as a keyboard, mouse, or display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0034] The storage device 1080 may store the program mentioned above. The processor 1040 reads the program from the storage device 1080, and executes the program to implement each functional unit of the training apparatus 2000.

[0035] The hardware configuration of the computer 1000 is not restricted to that shown in Fig. 4. For example, as mentioned-above, the training apparatus 2000 may be implemented by plural computers. In this case, those computers may be connected with each other through the network.

[0036] <<Data Matching Apparatus 300>> 　　The data matching apparatus 300 may be implemented by one or more computers. Each of the one or more computers may be a special-purpose computer manufactured for implementing the data matching apparatus 300, or may be a general-purpose computer like a PC, a server machine, or a mobile device.

[0037] The data matching apparatus 300 may be implemented by installing an application on the one or more computers. The application is implemented with a program that causes one or more computers to function as the training apparatus 2000. In other words, the program is an implementation of the functional units of the training apparatus 2000.

[0038] The hardware configuration of the data matching apparatus 300 may be similar to or the same as that of the training apparatus 2000 and depicted by Fig. 4. It is noted that the storage device of the data matching apparatus 300 stores the program implementing the data matching apparatus 300.

[0039] <Example of Representation data> 　　As described above, there may be various types of representation data that the training apparatus 2000 and the data matching apparatus 300 handle: e.g., ground-view image, aerial-view image, map image, 3D shape data, keyword matrix data and location information. Hereinafter, each type of representation data is explained in detail.

[0040] The ground-view image is a digital image (e.g., RGB image or grayscale image) that includes a ground view of a place. The ground-view image is generated by a ground camera that may be held by a pedestrian or installed on a vehicle. The ground-view image may be panoramic (having 360-degree field of view), or may have limited (less than 360-degree) field of view.

[0041] The aerial-view image is a digital image (e.g., RGB image or grayscale image) that includes an aerial view (top view, in other words) of a place. For example, the aerial-view image is generated by an aerial camera that may be installed on a drone, an airplane, or a satellite.

[0042] The map image is a digital image (e.g., RGB image or grayscale image) that visually describe map information of a place. In some implementations, the map image may include geometric shapes that represent various landmarks, such as buildings, roads, railways, rivers, lakes, mountains, etc. The map image may also show additional information, such as the name of each landmark, an address assigned to each section, etc.

[0043] Fig. 5 illustrates an example of the ground-view image, the aerial-view image, and the map image. In Fig. 5, the ground-view image 100, the aerial-view image 110, and the map image 120 show the same place as each other in different forms.

[0044] The 3D shape data is 3D data that represents a three-dimensional shape of a place, such as a point cloud, polygon mesh data, etc. Specifically, the 3D shape data of a place may represent 3D shapes of objects that exist in the place: e.g., 3D shapes of buildings and natural elements in the place. The natural elements may include a ground, a river, a lake, etc. Fig. 6 illustrates an example of the 3D shape data. The 3D shape data 130 shown in Fig. 6 represents 3D shapes of buildings and a 3D shape of the ground around the buildings.

[0045] Keyword matrix data is a matrix data that shows a spatial distribution of objects in a place. The keyword matrix data may show the spatial distribution of objects from a ground-level perspective or from an aerial perspective. The keyword matrix data may be generated by performing an analysis, such as semantic segmentation, on a ground-view image or an aerial-view image. In some implementations, each cell of the keyword matrix indicates a class identifier of an object that is captured on a region of the image corresponding to that cell.

[0046] Fig. 7 illustrates examples of keyword matrix data. The keyword matrix data 140-1 shows a spatial distribution of objects in the place captured on the ground-view image 100 while the keyword matrix data 140-2 shows a spatial distribution of objects in the place captured on the aerial-view image 110. In reference to the keyword matrix data 140-1, there are five types of objects that are assigned class identifiers: road (1), building (2), tree (3), ground (4), and sky (5). In reference to the keyword matrix data 140-2, there are four types of objects that are assigned class identifiers: road (1), building (2), tree (3), and ground (4).

[0047] The location information is data that represents a location of a place. One of the examples of data that represents the location of the place is GPS (Global Positioning System) coordinates.

[0048] In some implementations, the location information indicates a floating point representation of GPS coordinates of a place in which the latitude and the longitude of the place are represented in floating-point format. In another example, the location information indicates a feature value of the GPS coordinates of the place, which is obtained by converting the GPS coordinates into features. The feature value of the GPS coordinates may be extracted by inputting the GPS coordinates into a pre-trained model, which has been trained to encode the GPS coordinates input thereinto into a feature value of that GPS coordinates.

[0049] <Example Application of Data matching apparatus 300> 　　There are various possible applications of the data matching apparatus 300. For example, the data matching apparatus 300 can be used as a part of a system (hereinafter, a geo-localization system) that performs image geo-localization. Image geo-localization is a technique to determine the place at which an input image is captured. The geo-localization system 500 may be implemented by one or more arbitrary computers such as ones depicted by Fig. 4. It is noted that the geo-localization system is merely an example of the application of the data matching apparatus 300, and the application of the data matching apparatus 300 is not restricted to being used in the geo-localization system.

[0050] Fig. 8 illustrates a geo-localization system 500 that includes the data matching apparatus 300. The geo-localization system 500 includes the data matching apparatus 300 and the location database 400. The location database 400 includes a plurality of aerial-view images to each of which location information is attached. An example of the location information attached to an aerial-view image may be GPS coordinates of the place captured on the center of the corresponding aerial-view image.

[0051] In the example shown in Fig. 8, the first-type representation data 10 and the second-type representation data 20 are the ground-view image and the aerial-view image, respectively. The first feature extracting model 310 is configured to take a ground-view image as input and computes the feature value of the ground-view image input thereinto. The second feature extracting model 320 is configured to take an aerial-view image as input and computes the feature value of the aerial-view image input thereinto.

[0052] The data matching apparatus 300 receives a query that includes a ground-view image from a client (e.g., user terminal). Then, the data matching apparatus 300 searches the location database 400 for the aerial-view image that matches the ground-view image in the received query, thereby determining the place at which the ground-view image is captured. Specifically, the data matching apparatus 300 inputs the ground-view image into the first feature extracting model 310 thereby obtaining the feature value of the ground-view image. Then, the data matching apparatus 300 repeatedly executes: acquiring one of the aerial-view images from the location database 400; inputting the acquired aerial-view image into the second feature extracting model 320 to compute the feature value thereof; computing a degree of similarity between the feature value of the ground-view image and the feature value of the aerial-view image; and determines whether the ground-view image matches the aerial-view image based on the computed degree of similarity.

[0053] When it is determined that the ground-view image matches the aerial-view image, the data matching apparatus 300 sends a response to the client. The response includes the aerial-view image that is determined to match the ground-view image and the location information corresponding to that aerial-view image. Since this location information indicates the location of the place captured on the aerial-view image that matches the ground-view image, this location information can be used as data that indicates the location of the place at which the ground-view image is captured.

[0054] <Flow of Process> 　　Fig. 9 is a flowchart illustrating an example flow of processes performed by the training apparatus 2000. The acquiring unit 2020 acquires an anchor sample 30, a positive sample 40, and a plurality of the negative samples 50 (S102). The selecting unit 2040 extracts the hard-negative samples 60 from the plurality of the negative samples 50 (S104). The selecting unit 2040 selects one of the hard-negative samples 60 as the selected negative sample 80 (S106). The updating unit 2060 updates the data matching apparatus 300 based on the feature values of the anchor sample 30, the positive sample 40, and the selected negative sample 80 (S108).

[0055] <Acquisition of Samples: S102> 　　The acquiring unit 2020 acquires an anchor sample 30, a positive sample 40, and a plurality of the negative samples 50 (S102). There are various ways to acquire the anchor sample 30. For example, a plurality of the first-type representation data 10 is stored in advance in a storage device to which the training apparatus 2000 has access. In this case, the acquiring unit 2020 acquires one of the plurality of the first-type representation data 10 from that storage device. The first-type representation data 10 to be acquired may be randomly determined or may be specified by a user of the training apparatus 2000. In another example, the acquiring unit 2020 receives a first-type representation data 10 sent by another apparatus, and uses it as the anchor sample 30.

[0056] The acquiring unit 2020 acquires, as the positive sample 40, the second-type representation data 20 that represents the place the same as the place represented by the anchor sample 30. In some embodiments, a plurality of pairs of the first-type representation data 10 and the second-type representation data 20 that represent the same place as each other is prepared in advance. In this case, the acquiring unit 2020 may acquire a pair of the first-type representation data 10 and the second-type representation data 20 as a pair of the anchor sample 30 and the positive sample 40. When two or more anchor samples 30 and two or more positive samples 40 are to be acquired, the acquiring unit 2020 acquires two or more pairs of the first-type representation data 10 and the second-type representation data 20.

[0057] In some embodiments, the acquiring unit 2020 uses the aforementioned location information corresponding to the anchor sample 30 to acquire the positive sample 40. As mentioned above, the location information indicates the location (e.g., GPS coordinates) of the place represented by the corresponding representation data.

[0058] In this case, the acquiring unit 2020 acquires the anchor sample 30 and its corresponding location information. Then, the acquiring unit 2020 acquires, as the positive sample 40, the second-type representation data 20 that is associated with the location information indicating the location same as the location indicated by the location information associated with the anchor sample 30. The second-type representation data 20 may be stored in advance in a storage device to which the training apparatus 2000 has access.

[0059] When two or more anchor samples 30 and two or more positive samples 40 are to be acquired, the acquiring unit 2020 acquires a plurality of the anchor samples 30 first. Then, for each anchor sample 30, the acquiring unit 2020 acquires, as the positive sample 40 to be paired with that anchor sample 30, the second-type representation data 20 that is associated with the location information indicating the location same as the location indicated by the location information associated with that anchor sample 30.

[0060] The acquiring unit 2020 acquires, as the negative sample 50, the second-type representation data 20 that represents the place different from the place represented by the anchor sample 30. In other words, the acquiring unit 2020 acquires, as the negative samples 50, any two or more second-type representation data 20 other than that acquired as the positive sample 40. As mentioned above, the second-type representation data 20 may be stored in advance in a storage device to which the training apparatus 2000 has access.

[0061] In some embodiments, the acquiring unit 2020 acquires a predefined pair of the first-type representation data 10 and the second-type representation data 20 as a pair of the anchor sample 30 and the positive sample 40. In this case, the acquiring unit 2020 acquires other two or more second-type representation data 20 than that included in the acquired pair as the negative samples 50.

[0062] In some embodiments, the acquiring unit 2020 uses the location information corresponding to the anchor sample 30 to acquire the negative samples 50. Specifically, the acquiring unit 2020 acquires two or more second-type representation data 20 each of which is associated with the location information indicating the location substantially far from the location indicated by the location information corresponding to the anchor sample 30.

[0063] <Extraction of Hard-Negative Samples 60: S104> 　　The selecting unit 2040 extracts the hard-negative samples 60 from the plurality of the negative samples 50 (S104). To determine which negative samples 50 are hard-negative samples 60, the selecting unit 2040 acquires the feature value of the anchor sample 30 and the feature value of each negative sample 50. Specifically, the selecting unit 2040 inputs the anchor sample 30 into the first feature extracting model 310 to obtain the feature value of the anchor sample 30. The selecting unit 2040 also inputs each negative sample 50 into the second feature extracting model 320, thereby obtaining the feature value of each negative sample 50.

[0064] The selecting unit 2040 computes a degree of similarity (hereinafter, referred to as "similarity score") between the feature value of the anchor sample 30 and the feature value of each negative sample 50. The similarity score of two feature values may be represented by a distance therebetween. In this case, the more similar the two feature values are, the smaller the similarity score is.

[0065] For example, the selecting unit 2040 compares, for each negative sample 50, its similarity score with a predefined threshold to determine whether that negative sample 50 is a hard-negative sample 60 or not. Specifically, the selecting unit 2040 determines that the negative sample 50 is a hard-negative sample 60 when its similarity score is smaller than or equal to the predefined threshold. On the other hand, the selecting unit 2040 determines that the negative sample 50 is not a hard-negative sample 60 when its similarity score is larger than the predefined threshold. Hereinafter, the negative samples 50 other than the hard-negative sample are referred to as "easy-negative samples".

[0066] Fig. 10 illustrates the feature values of the anchor sample and the negative samples in a feature space. The feature value of the anchor sample 30 is depicted by a square while the feature values of the negative samples 50 are depicted by cross marks. In Fig. 10, the more similar the feature value of the negative sample 50 is to the anchor sample 30, the closer the negative sample 50 is located to the anchor sample 30.

[0067] The threshold line 90 represents a threshold that divides the negatives samples 50 into hard-negative samples 60 and the easy-negative samples. The negative samples 50 located between the anchor sample 30 and the threshold line 90 are ones determined as the hard-negative samples 60. Thus, a set of the negative samples 50 located between the anchor sample 30 and the threshold line 90 represents the hard-negative group 70. The negative samples 50 located farther than the threshold line 90 from the anchor sample 30 are ones determined as the easy-negative samples.

[0068] In some embodiments, the selecting unit 2040 may extract not all but a part of the hard-negative samples 60 to generate the hard-negative group 70. For example, the selecting unit 2040 may extract the top X hard-negative samples 60 from the plurality of the negative samples 50 in descending order of their similarity to the anchor sample 30 (in ascending order of their distance from the anchor sample 30 in the feature space). It is noted that X is an integer larger than one.

[0069] <Selection of Hard-Negative Sample 60: S106> 　　The selecting unit 2040 selects a hard-negative sample 60 as the selected negative sample 80 (S106). As mentioned above, the hard-negative sample 60 is selected based on the geographical distance between the place represented by the anchor sample 30 and the place represented by each hard-negative sample 60.

[0070] The selecting unit 2040 computes, for each hard-negative sample 60, the geographical distance between the place represented by that hard-negative sample 60 and the place represented by the anchor sample 30. As mentioned above, each representation data is associated with the location information in advance. The selecting unit 2040 computes the distance between the locations associated with two samples as the geographical distance between the places represented by those two samples.

[0071] There may be various ways to select the hard-negative samples 60 as the selected negative sample 80 based on the geographical distances mentioned above. For example, the selecting unit 2040 selects the hard-negative sample 60 that represents the place closest to the place represented by the anchor sample 30. In another example, the selecting unit 2040 selects the hard-negative sample 60 that represents the place farthest from the place represented by the anchor sample 30. In another example, the selecting unit 2040 selects one of the hard-negative samples 60 that represent the places within a predefined distance from the place represented by the anchor sample 30. When there are two or more hard-negative samples 60 meeting this condition, the selecting unit 2040 may randomly select a hard-negative sample 60 that meets the condition.

[0072] <Update of Models: S108> 　　The updating unit 2060 updates the data matching apparatus 300 based on the feature value of the anchor sample 30, the feature value of the positive sample 40, and the feature value of the selected negative sample 80 (S108). Specifically, the updating unit 2060 computes a loss based on the feature value of the anchor sample 30, the feature value of the positive sample 40, and the feature value of the selected negative sample 80. Then, the updating unit 2060 uses the computed loss to update trainable parameters of the first feature extracting model 310 and those of the second feature extracting model 320. It is noted that the feature value of the positive sample 40 can be obtained by inputting the positive sample 40 into the second feature extracting model 320.

[0073] The loss may be computed by applying the feature values of the samples to a pre-defined loss function. There are various types of loss functions, and one of them can be employed as the loss function to compute the loss.

[0074] For example, the updating unit 2060 computes a triplet loss using the following equation (1): Equation 1 　　L represents the triplet loss. d_pos represents the distance between the feature value of the anchor sample 30 and that of the positive sample 40. d_neg represents the distance between the feature value of the anchor sample 30 and that of the selected negative sample 80.

[0075] <<Batch Loss>> 　　As mentioned earlier, the updating unit 2060 may use two or more anchor samples 30, two or more positive samples 40, and two or more selected negative samples 80 to compute the loss. In this case, the acquiring unit 2060 may generate N sets of the anchor sample 30, the positive sample 40, and the selected negative sample 80 (N is an integer larger than one). Hereinafter, a set of the anchor sample, the positive sample, and the negative sample to be used to compute a loss is referred to as "sample set".

[0076] For example, the acquiring unit 2020 may acquire N pairs of the anchor samples 30 and the positive sample 40 from the predefined pairs thereof. In another example, the acquiring unit 2020 acquires N anchor sample 30 by selecting N first-type representation data 10 from a group of the first-type representation data 10 (N is an integer larger than one). Then, for each one of the acquired anchor samples 30, the acquiring unit 2020 acquires the second-type representation data 20 that represents the place same as the place represented by that anchor sample 30 as the positive sample 40 to be paired with that anchor sample 30, thereby acquiring N pairs of the anchor sample 30 and the positive sample 40.

[0077] When two or more anchor samples 30 are acquired, the selecting unit 2040 may perform the selection of the selected negative sample 80 for each anchor sample 30 to make N pairs of the anchor sample 30 and the selected negative sample 80. Specifically, for each anchor sample 30, the selecting unit 2040 extracts the hard-negative samples 60 from the plurality of the negative sample 50 based on the similarity between the feature value of that anchor sample 30 and the feature value of each negative sample 50, thereby generating the hard-negative group 70 for each anchor sample 30. Then, for each anchor sample 30, the selecting unit 2040 selects a hard-negative sample 60 from the hard-negative group 70 that represents the place same as the place represented by that anchor sample 30 as the selected negative sample 80 to be paired with that anchor sample 30.

[0078] Fig. 11 illustrates the determination of the selected negative samples 80 in the case where two or more anchor samples 30 are acquired. In the example shown by Fig. 11, two anchor samples 30-1 and 30-2 are acquired.

[0079] The selecting unit 2040 extracts the hard-negative samples 60 to generate the hard-negative group 70, for each anchor sample 30. Specifically, the hard-negative group 70-1 is generated for the anchor sample 30-1 by extracting hard-negative samples 60 based on the similarity between the feature value of the anchor sample 30-1 and the feature value of each negative sample 50. The hard-negative group 70-2 is generated for the anchor sample 30-2 by extracting hard-negatives samples 60 based on the similarity between the feature value of the anchor sample 30-2 and the feature values of each negative sample 50.

[0080] The selecting unit 2040 generates a pair of the anchor sample 30-1 and the selected negative sample 80-1 by selecting one of the hard-negative samples 60 from the hard-negative group 70-1 as the selected negative sample 80-1. The selecting unit 2040 also generates a pair of the anchor sample 30-2 and the selected negative sample 80-2 by selecting one of the hard-negative samples 60 from the hard-negative group 70-2 as the selected negative sample 80-2.

[0081] For each anchor sample 30, the updating unit 2060 combines the pair of the anchor sample 30 and the positive sample 40 and the pair of the anchor sample 30 and the selected negative sample 80 into a sample set, thereby generating N sample sets. The updating unit 2060 computes a batch loss using the N sample sets. The batch loss may be computed as follows: Equation 2 　　L_b represents a batch loss. i represent an index of the sample set. L_tri[i] represents a triplet loss computed for the i-th sample set. d_pos[i] represents the distance between the anchor sample and the positive sample of the i-th sample set. d_neg[i] represents the distance between the anchor sample and the negative sample of the i-th sample set.

[0082] The updating unit 2060 may also generate one or more sample sets each of whose negative sample is randomly selected from the plurality of the negative samples 50. Specifically, the acquiring unit 2020 additionally acquires M pairs of the anchor sample 30 and the positive sample 40 (M is an integer). The updating unit 2060 randomly selects a negative sample 50 for each of those M pairs, thereby generating M sample sets. The updating unit 2060 uses (N+M) sample sets to compute the batch loss. It is noted that M can be the same as or different from N.

[0083] <<Repetition of Updates>> 　　The updating unit 2060 may repeatedly perform the update of the data matching apparatus 300 until a predefined termination condition is satisfied. The termination condition may be arbitrary condition that represents a situation where the data matching apparatus 300 is substantially trained.

[0084] Fig. 12 illustrates an example flow of processes performed by the training apparatus 2000 to repeatedly update the data matching apparatus 300. In this example, the training apparatus 2000 repeatedly performs a set of processes, referred to as "epoch", until the predefined termination condition is satisfied. The data matching apparatus 300 is updated the predefined number of times in each epoch.

[0085] The hard-negative group 70 includes not all but a part of the hard-negative samples 60, and is generated at the beginning of each epoch. This means that the hard-negative group 70 is fixed in each epoch (in other words, during the predefined number of updates of the data matching apparatus 300). Since a plurality of anchor samples 30 are used to update the data matching apparatus 300, the hard-negative group 70 is generated for each anchor sample 30.

[0086] In Step S202, the acquiring unit 2020 acquires a plurality of anchor samples 30, a plurality of positive samples 40, and a plurality of negative samples 50. Steps S204 to S220 constitutes a loop process L1 that represents an epoch. The loop process L1 is repeatedly performed until the predefined termination condition is satisfied.

[0087] In Step S204, the training apparatus 2000 determines whether the termination condition is satisfied. Step S206 is performed when the termination condition is not satisfied. On the other hand, the training of the data matching apparatus 300 is terminated when the termination condition is satisfied.

[0088] Step 206 is the beginning of the epoch. Thus, in Step S206, the selecting unit 2040 extracts the predefined number of hard-negative samples 60 to generate the hard-negative group 70.

[0089] Steps S208 to S218 constitutes a loop process L2 that is repeated the predefined number of times. In Step S208, the training apparatus 2000 determines whether the loop process L2 has been performed the predefined number of times in the current epoch. Step S210 is performed when the loop process L2 has not been performed the predefined number of times in the current epoch yet. On the other hand, the current epoch is terminated when the loop process L2 has already been performed the predefined number of times in the current epoch.

[0090] In this example, the training apparatus 2000 generates the N first sample sets and the N second sample sets. For each of the first sample sets, the negative samples 50 are randomly selected. On the other hand, the selected negative sample 80 is determined for each of the second sample sets.

[0091] In Step S210, the training apparatus 2000 selects a pair of the anchor sample 30 and the positive sample 40 for each of the first sample set and for each of the second sample set. This means that 2*N pairs of the anchor sample 30 and the positive sample 40 are selected here.

[0092] The selecting unit 2040 randomly selects the negative sample 50 for each of the first sample set (S212). The selecting unit 2040 selects the hard-negative sample 60 from the current hard-negative group 70 as the selected negative sample 80 for each of the second sample set (S214).

[0093] The updating unit 2060 computes the batch loss using both all the N first sample sets and all the N second sample sets, and updates the data matching apparatus 300 using this batch loss (S216).

[0094] <Output from Training Apparatus> 　　The training apparatus 2000 may output information (hereinafter, result information) related to a result of the training of the data matching apparatus 300. The result information may include the trainable parameters of the first feature extracting model 310 and those of the second feature extracting model 320, which are determined as a result of the training of the data matching apparatus 300. The result information may also include other information related to the data matching apparatus 300: e.g., hyperparameters of the first feature extracting model 310 and the second feature extracting model 320, programs implementing the first feature extracting model 310 and the second feature extracting model 320, etc.

[0095] There are various ways to output the result information. For example, the training apparatus 2000 may store the result information into a storage device. For another example, the training apparatus 2000 may output the result information to a display device so that the display device displays the contents of the result information. For another example, the training apparatus 2000 may output the result information to another computer, such as the data matching apparatus 300.

[0096] SECOND EXAMPLE EMBODIMENT <Overview> 　　Fig. 13 illustrates a data matching apparatus 300 to be trained. In Fig. 13, the data matching apparatus 300 further includes a third feature extracting model 330. The third feature extracting model 330 is configured to take a third-type representation data 150 as input, perform computations on the third-type representation data 150, and output a value (e.g., vector or tensor) that represents the feature value of the third-type representation data 150. The third feature extracting model 330 may be implemented as a machine learning-based model, such as a neural network.

[0097] The data matching apparatus 300 acquires a first-type representation data 10 and a pair of the second-type representation data 20 and the third-type representation data 150. The second-type representation data 20 and the third-type representation data 150 paired with each other represent the places same as each other.

[0098] The data matching apparatus 300 inputs the first-type representation data 10 into the first feature extracting model 310 to obtain the feature value thereof. The data matching apparatus 300 inputs the second-type representation data 20 into the second feature extracting model 320 to obtain the feature value thereof. The data matching apparatus 300 inputs the third-type representation data 150 into the third feature extracting model 330 to obtain the feature value thereof.

[0099] The data matching apparatus 300 performs feature fusion on the feature value of the second-type representation data 20 and the feature value of the third-type representation data 150 to compute a fused feature value 355. Feature fusion is a technique to combine two or more feature values into a single feature value. For example, the data matching apparatus 300 may compute element-wise sum between the feature value of the second-type representation data 20 and the feature value of the third-type representation data 150 to combine them into the fused feature value 355.

[0100] The data matching apparatus 300 compares the feature value of the first-type representation data 10 and the fused feature value 355 to determine whether the first-type representation data 10 matches the pair of the second-type representation data 20 and the third-type representation data 150 (whether the place represented by the first-type representation data 10 is same as the place represented by both the second-type representation data 20 and the third-type representation data 150, in other words).

[0101] Fig. 14 illustrates an overview of a training apparatus 2000 that trains the data matching apparatus 300. It is noted that Fig. 14 does not limit operations of the training apparatus 2000, but merely show an example of possible operations of the training apparatus 2000.

[0102] The training apparatus 2000 acquires one or more anchor samples 160, one or more first positive samples 170, one or more second positive samples 180, a plurality of first negative samples 190, and a plurality of second negative samples 200. The anchor sample 160 is equivalent to the anchor sample 30: i.e., a first-type representation data 10 to be used for the training of the data matching apparatus 300. The first positive sample 170 is a positive sample of the second-type representation data 20 and is equivalent to the positive sample 40: i.e., a second-type representation data 20 that represents the place same as the place represented by the corresponding anchor sample 160. The first negative sample 190 is a negative sample of the second-type representation data 20 and is equivalent to the negative sample 50: i.e., a second-type representation data 20 that represents a place different from the place represented by the corresponding anchor sample 160.

[0103] The second positive sample 180 is a positive sample of the third-type representation data 150. Specifically, the second positive sample 180 is a third-type representation data 150 that represents the place same as the place represented by the corresponding anchor sample 160. The second negative sample 200 is a negative sample of the third-type representation data 150. Specifically, the second negative sample 200 is a third-type representation data 150 that represents a place different from the place represented by the corresponding anchor sample 160.

[0104] In the same way of determining the selected negative sample 80, the training apparatus 2000 selects one of the first negative samples 190 as a selected first negative sample 220. Specifically, the training apparatus 2000 extracts first hard-negative samples 210 from the plurality of the first negative samples 190. The first hard-negative samples 210 are the first negative samples 190 that are so similar to the anchor sample 160 that it is difficult to accurately determine that they do not match the anchor sample 160. Hereinafter, a set of the first hard-negative samples 210 is referred to as "first hard-negative group". Then, the training apparatus 2000 selects one of the first hard-negative samples 210 as the selected first negative sample 220 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each first hard-negative sample 210.

[0105] Similarly, the training apparatus 2000 also selects a second negative sample 200 as a selected second negative sample 240. Specifically, the training apparatus 2000 extracts second hard-negative samples 230 from the plurality of the second negative samples 200. The second hard-negative samples 230 are the second negative samples 200 that are so similar to the anchor sample 160 that it is difficult to accurately determine that they do not match the anchor sample 160. Hereinafter, a set of the second hard-negative samples 230 is referred to as "second hard-negative group". Then, the training apparatus 2000 selects one of the second hard-negative samples 230 as the selected second negative sample 240 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each second hard-negative sample 230.

[0106] The training apparatus 2000 updates the data matching apparatus 300 based on the feature values of the anchor sample 160, the first positive sample 170, the second positive sample 180, the selected first negative sample 190, and the selected second negative sample 200.

[0107] <Example of Advantageous Effect> 　　The training apparatus 2000 of the second example embodiment can perform the training of the data matching apparatus 300 that compares takes three types of representation data: the first-type representation data 10, the second-type representation data 20, and the third-type representation data 150. Specifically, the positive sample and the negative samples are obtained for each of the second-type representation data 20 and the third-type representation data 150. Then, a hard-negative sample of the second-type representation data 20, a hard-negative sample of the third-type representation data 150, or both are selected based on the geographical distance between the place represented thereby and the place represented by the anchor sample for the training of the data matching apparatus 300. Thus, according to the training apparatus 2000, a novel technique to select the negative sample to be used for the training of an apparatus that performs data matching is provided.

[0108] Hereinafter, a more detailed explanation of the training apparatus 2000 will be described.

[0109] <Example of Functional Configuration> 　　The functional configuration of the training apparatus 2000 of the second example embodiment may be the same as that of the first example embodiment, and thus may be depicted by Fig. 3. The acquiring unit 2020 of the second example embodiment acquires one or more anchor samples 160, one or more first positive samples 170, one or more second positive samples 180, a plurality of first negative samples 190, and a plurality of second negative samples 200.

[0110] The selecting unit 2040 extracts first hard-negative samples 210 from the plurality of first negative samples 190. Then, the selecting unit 2040 selects one of the first hard-negative samples 210 as the selected first negative sample 220 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each first hard-negative sample 210.

[0111] The selecting unit 2040 extracts second hard-negative samples 230 from the plurality of second negative samples 200. Then, the selecting unit 2040 selects one of the second hard-negative samples 230 as the selected second negative sample 240 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each second hard-negative sample 230.

[0112] The updating unit 2060 updates the data matching apparatus 300 based on the feature values of the anchor sample 160, the first positive sample 170, the second positive sample 180, the selected first negative sample 220, and the selected second negative sample 240.

[0113] <Example of Hardware Configuration> 　　The hardware configuration of the training apparatus 2000 of the second example embodiment may be similar to or the same as that of the first example embodiment and depicted by Fig. 4. It is noted that the storage device 1080 of the second example embodiment stores the program implementing the training apparatus 2000 of the second example embodiment.

[0114] As mentioned above, the hardware configuration of the data matching apparatus 300 may be similar to or the same as that of the training apparatus 2000 and depicted by Fig. 4. It is noted that the storage device of the data matching apparatus 300 stores the program implementing the data matching apparatus 300.

[0115] <Flow of Processes> 　　Fig. 15 is a flowchart illustrating an example flow of processes performed by the training apparatus 2000. The acquiring unit 2020 acquires one or more anchor samples 160, one or more first positive samples 170, one or more second positive samples 180, a plurality of first negative samples 190, and a plurality of second negative samples 200 (S302).

[0116] The selecting unit 2040 extracts first hard-negative samples 210 from the plurality of first negative samples 190 (S304). The selecting unit 2040 selects one of the first hard-negative samples 210 as the selected first negative sample 220 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each first hard-negative sample 210 (S306).

[0117] The selecting unit 2040 extracts second hard-negative samples 230 from the plurality of second negative samples 200 (S308). The selecting unit 2040 selects one of the second hard-negative samples 230 as the selected second negative sample 240 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each second hard-negative sample 230 (S310).

[0118] The updating unit 2060 updates the data matching apparatus 300 based on the feature values of the anchor sample 160, the first positive sample 170, the second positive sample 180, the selected first negative sample 220, and the selected second negative sample 240 (S312).

[0119] <Acquisition of Samples: S302> 　　The acquiring unit 2020 acquires one or more anchor samples 160, one or more first positive samples 170, one or more second positive samples 180, a plurality of first negative samples 190, and a plurality of second negative samples 200 (S302). The anchors samples 160 can be acquired in the way same as the way of acquiring the anchor samples 30. The first positive samples 170 can be acquired in the way same as the way of acquiring the positive samples 40. The first negative samples 190 can be acquired in the way same as the way of acquiring the negative samples 50.

[0120] The acquiring unit 2020 acquires, for each first positive sample 170, the second positive sample 180 that represents the place same as the place represented by that first positive sample 170. In some embodiments, the second-type representation data 20 and the third-type representation data 150 representing the same place as each other are associated with each other in advance. In this case, the acquiring unit 2020 acquires, for each anchor sample 160, a pair of the second-type representation data 20 and the third-type representation data 150 that represents place same as the place represented by that anchor sample 160 as a pair of the first positive sample 170 and the second positive sample 180.

[0121] In other embodiments, the acquiring unit 2020 uses the location information to acquire the second positive sample 180. Specifically, the acquiring unit 2020 acquires, for each first positive sample 170, the third-type representation data 150 whose location information indicates the location same as the location indicated by the location information of that first positive sample 170 as the second positive sample 180.

[0122] The second negative sample 200 can be acquired in the way same as the way of acquiring the second positive sample 180. Specifically, in some embodiments, the acquiring unit 2020 acquires a predefined pair of the second-type representation data 20 and the third-type representation data 150 that represent a place different from the place represented by the anchor sample 160 as a pair of the first negative sample 190 and the second negative sample 200. In other embodiments, the acquiring unit 2020 acquires, for each first negative sample 190, the third-type representation data 150 whose location information indicates the location same as the location indicated by the location information of that first negative sample 190 as the second negative sample 200.

[0123] <Selection of First Hard-Negative Samples: S304, S306> 　　The selecting unit 2040 extracts first hard-negative samples 210 from the first negative samples 190 in the same manner of extracting hard-negative samples 60 from the negative samples 50. Then, the selecting unit 2040 selects one of the first hard-negative samples 210 as the selected first negative sample 220 in the same manner of selecting one of the hard-negative samples 60 as the selected negative sample 80.

[0124] <Selection of Second Hard-Negative Samples: S308, S310> 　　The selecting unit 2040 extracts second hard-negative samples 230 from the second negative samples 200 in the same manner of extracting hard-negative samples 60 from the negative samples 50. Then, the selecting unit 2040 selects one of the second hard-negative samples 230 as the selected second negative sample 240 in the same manner of selecting one of the hard-negative samples 60 as the selected negative sample 80.

[0125] <Update of Data Matching Apparatus 300: S312> 　　The updating unit 2060 updates the data matching apparatus 300 based on the feature values of the anchor sample 160, the first positive sample 170, the second positive sample 180, the selected first negative sample 220, and the selected second negative sample 240 (S312). Hereinafter, some example ways of updating the data matching apparatus 300 including the third feature extracting model 330 will be described.

[0126] <<Example 1>> 　　Fig. 16 illustrates a first example way of updating the data matching apparatus 300 including the third feature extracting model 330. In this example, the updating unit 2060 separately uses the selected first negative sample 220 and the selected second negative sample 240 to separately update the second feature extracting model 320 and the third feature extracting model 330. Specifically, the updating unit 2060 computes a loss (e.g., a triplet loss) based on the feature values of the anchor sample 160, the first positive sample 170, and the selected first negative sample 220. Then, the updating unit 2060 updates the first feature extracting model 310 and the second feature extracting model 320 based on this loss.

[0127] Similarly, the updating unit 2060 computes another loss (e.g., a triplet loss) based on the feature values of the anchor sample 160, the second positive sample 180, and the selected second negative sample 240. Then, the updating unit 2060 updates the first feature extracting model 310 and the third feature extracting model 330 based on this loss.

[0128] The updating unit 2060 may use two or more sample sets to compute a batch loss, and updates the models based on the batch loss. Specifically, the updating unit 2060 computes a batch loss by using two or more sample sets of the anchor sample 160, the first positive sample 170, and the selected first negative sample 220 to update the first feature extracting model 310 and the second feature extracting model 320. Similarly, the updating unit 2060 computes a batch loss by using two or more sample sets of the anchor sample 160, the second positive sample 180, and the selected second negative sample 240 to update the first feature extracting model 310 and the third feature extracting model 330.

[0129] <<Example 2>> 　　Fig. 17 illustrates a second example way of updating the data matching apparatus 300 including the third feature extracting model 320. In this example, the updating unit 2060 updates both the second feature extracting model 320 and the third feature extracting model 330 by using the selected first negative sample 220 and the second negative sample 200 that represents the place same as the place represented by the selected first negative sample 220. Hereinafter, the second negative sample 180 that represents the place same as the place represented by the selected first negative sample 220 is referred to as "the second negative sample 200 corresponding to the selected first negative sample 220.

[0130] Specifically, the updating unit 2060 computes the feature value of the first positive sample 170 and the feature value of the second positive sample 180, and performs feature fusion on those feature values to compute the fused feature value 355 thereof. Similarly, the updating unit 2060 computes the feature value of the selected first negative sample 220 and the feature value of the second negative sample 200 corresponding to the selected first negative sample 220, and performs feature fusion on those feature values to compute the fused feature value 355 thereof.

[0131] Then, the updating unit 2060 computes a loss (e.g., triplet loss) based on the feature value of the anchor sample 160, the fused feature value 355 of the first positive sample 170 and the second positive sample 180, and the fused feature value 355 of the selected first negative sample 220 and the corresponding second negative sample 200. The fused feature value 355 of the first positive sample 170 and the second positive sample 180 is used as the feature value extracted from a positive sample. The fused feature value 355 of the selected first negative sample 220 and the corresponding second negative sample 200 is used as the feature value extracted from a negative sample. The updating unit 2060 updates the first feature extracting model 310, the second feature extracting model 320, and the third feature extracting model 330 based on the computed loss.

[0132] In addition, the updating unit 2060 updates both the second feature extracting model 320 and the third feature extracting model 330 using the selected second negative sample 240 and the corresponding first negative sample 190. The first negative sample 190 corresponding to the selected second negative sample 240 is the first negative sample 190 that represents the place same as the place represented by the selected second negative sample 240.

[0133] Specifically, the updating unit 2060 computes the feature value of the selected second negative sample 240 and the feature value of the corresponding first negative sample 190, and performs feature fusion on those feature values to compute the fused feature value 355 thereof.

[0134] The updating unit 2060 computes a loss (e.g., triplet loss) based on the feature value of the anchor sample 160, the fused feature value 355 of the first positive sample 170 and the second positive sample 180, and the fused feature value 355 of the selected second negative sample 240 and the corresponding first negative sample 190. The updating unit 2060 updates the first feature extracting model 310, the second feature extracting model 320, and the third feature extracting model 330 based on the computed loss.

[0135] <<Batch Loss>> 　　Like the training apparatus 2000 of the first example embodiment, the training apparatus 2000 of the second example embodiment may use two or more sample sets to compute a batch loss, and updates the models based on the batch loss.

[0136] <<Repetition of Updates>> 　　Like the training apparatus 2000 of the first example embodiment, the training apparatus 2000 of the second example embodiment may repeatedly update the data matching apparatus 300 according to the flow depicted by Fig. 12. In each iteration of the epoch (i.e., in the loop process L2), the training apparatus 2000 may update the data matching apparatus 300 in the way exemplified above.

[0137] <Multi-Stage Feature Fusion> 　　The data matching apparatus 300 may take a set of three or more types of representation data to be compared with the first-type representation data 10, and perform a multi-stage feature fusion on this set to compute a fused feature value to be compared with the feature value of the first-type representation data 10. The data matching apparatus 300 includes an i-th feature extracting model for each i-th-type representation data (i is an integer).

[0138] Fig. 18 illustrates the data matching apparatus 300 that takes a set of three types of representation data to be compared with the first-type representation data 10. The data matching apparatus 300 takes, as input, the first-type representation data 10 and a set of the second-type representation data 20, the third-type representation data 150, and the fourth-type representation data 250. The second-type representation data 20, the third-type representation data 150, and the fourth-type representation data 250 represent the same place.

[0139] The data matching apparatus 300 further includes a fourth feature extracting model 340. The fourth feature extracting model 340 is configured to take the fourth-type representation data 250 as input, perform computations on the fourth-type representation data 250 input thereinto, and output a feature value of the fourth-type representation data 250.

[0140] The data matching apparatus 300 performs multi-stage feature fusion on the set of the second-type representation data 20, the third-type representation data 150, and the fourth-type representation data 250. Specifically, the data matching apparatus 300 performs feature fusion on the feature value of the second-type representation data 20 and the feature value of the third-type representation data 150, thereby computing a fused feature value 360. Then, the data matching apparatus 300 performs feature fusion on the fused feature value 360 and the feature value of the fourth-type representation data 250, thereby computing a fused feature value 362.

[0141] The data matching apparatus 300 compares the feature value of the first-type representation data 10 and the fused feature value 362 to determine whether the first-type representation data 10 matches the set of the second-type representation data 20, the third-type representation data 150, and the fourth-type representation data 250.

[0142] There are more examples of multi-stage feature fusion when a set of four or more types of representation data is acquired to be compared with the first-type representation data 10. Fig. 19 illustrates a first example of multi-stage feature fusion in a case where the data matching apparatus 300 takes five types of representation data. In this case, the set of the representation data to be compared with the first-type representation data 10 further includes a fifth-type representation data 260.

[0143] The data matching apparatus 300 further includes a fifth feature extracting model 350. The fifth feature extracting model 350 is configured to take the fifth-type representation data 260 as input, perform computations on the fifth-type representation data 260 input thereinto, and output a feature value of the fifth-type representation data 260.

[0144] The data matching apparatus 300 performs feature fusion on the feature value of the second-type representation data 20 and the feature value of the third-type representation data 150 to compute the fused feature value 360. The data matching apparatus 300 performs feature fusion on the feature value of the fourth-type representation data 250 and the feature value of the fifth-type representation data 260 to compute a fused feature value 364. Then, the data matching apparatus 300 performs feature fusion on the fused feature value 360 and the fused feature value 364 to compute a fused feature value 366. The data matching apparatus 300 computes a degree of similarity between the feature value of the first-type representation data 10 and the fused feature value 366 to determine whether the first-type representation data 10 matches the set of the second-type representation data 20, the third-type representation data 150, the fourth-type representation data 250, and the fifth-type representation data 260.

[0145] Fig. 20 illustrates a second example of multi-stage feature fusion in a case where the data matching apparatus 300 takes five types of representation data. In this example, the data matching apparatus 300 performs feature fusion on the feature value of the second-type representation data 20, the feature value of the third-type representation data 150, and the feature value of the fourth-type representation data 250 to compute a fused feature value 368. The data matching apparatus 300 performs feature fusion on the fused feature value 368 and the feature value of the fifth-type representation data 260 to compute a fused feature value 370. The data matching apparatus 300 computes a degree of similarity between the feature value of the first-type representation data 10 and the fused feature value 370 to determine whether the first-type representation data 10 matches the set of the second-type representation data 20, the third-type representation data 150, the fourth-type representation data 250, and the fifth-type representation data 260.

[0146] <As to Data Augmentation for Training> 　　In some embodiments, one or more types of representation data representing a particular place can be generated using another type of representation data that represent the particular place. For example, a type of representation data may be generated by removing a part of another type of representation data. This generation of the representation data may be performed by the data matching apparatus 300 or another apparatus.

[0147] Fig. 21 illustrates a representation data generated by removing a part of another representation data. In this example, the second-type representation data 20 is a 360-degree field of view (FOV) aerial image. The data matching apparatus 300 removes (fill with black) a right half of the second-type representation data 20 to generate a 180-degree FOV aerial image as the third-type representation data 150. It is noted that a dot pattern is used in Fig. 21 to represent a region filled with black.

[0148] It can be generalized that a D-degree FOV aerial image is generated by removing (360-D) / 360 of the original aerial image (0<D<360). The same way can apply to another type of representation data, such as a ground-view image, a map image, a 3D data, or a key-word matrix.

[0149] Generating a type of representation data by removing another type of representation data is effective in that the original representation data can act as a source of global features while the representation data generated from the original one can act as a source of local features. Since the data matching apparatus 300 takes both the representation data showing the global features and the representation data showing the local features as input, it can take both the global and local features into consideration for data matching. Thus, the data matching apparatus 300 can perform data matching more accurately than the case where only the global features are taken into consideration.

[0150] When the data matching apparatus 300 takes a plurality of representation data originated from the same representation data, it is preferable that the data matching apparatus 300 is configured to perform feature fusion on those representation data. Suppose that a set of a 360-degree FOV aerial image, a 180-degree FOV aerial image, and a map image is taken as the set of representation data to be compared with the first-type representation data 10. In this case, the 360-degree FOV aerial image and the 180-degree FOV aerial image are handled as the second-type representation data 20 and the third-type representation data 150, respectively. The map image is handled as the fourth-type representation data 250.

[0151] The data matching apparatus 300 performs feature fusion on the feature value of the 360-degree FOV aerial image and the feature value of the 180-degree FOV feature value. As a result, the fused feature value of two aerial images originated from the same aerial image is obtained as the first fused feature value 360. Then, the fused feature value of two aerial images and the feature value of the map image is fused to generate the fused feature value 362.

[0152] THIRD EXAMPLE EMBODIMENT <Overview> 　　Fig. 22 illustrates an overview of a training apparatus 2000 that trains the data matching apparatus 300. It is noted that Fig. 22 does not limit operations of the training apparatus 2000, but merely show an example of possible operations of the training apparatus 2000.

[0153] The configuration of the data matching apparatus 300 to be trained by the training apparatus 2000 of the third example embodiment is depicted by Fig. 13 and the same as the configuration of the data matching apparatus 300 to be trained by the training apparatus 2000 of the second example embodiment. The training apparatus 2000 of the third example embodiment is different from that of the second example embodiment in a way of determining the selected first negative sample 220 and the selected second negative sample 240. Specifically, the training apparatus 2000 determines either one of the selected first negative sample 220 or the selected second negative sample 240 based on the geographical distance.

[0154] When the selected first negative sample 220 is selected based on the geographical distance, the second negative sample 200 that represents the place same as the place represented by the selected first negative sample 220 is selected as the selected second negative sample 240. On the other hand, when the selected second negative sample 240 is selected based on the geographical distance, the first negative sample 190 that represents the place same as the place represented by the selected second negative sample 240 is selected as the selected first negative sample 220.

[0155] For example, the training apparatus 2000 selects a single hard-negative sample that represents the place closest to the place represented by the anchor sample 160, from a set of the first hard-negative samples 210 and the second hard-negative samples 230. When it is a first hard-negative sample 210 that represents the place closest to the place represented by the anchor sample 160, the training apparatus 2000 selects this first hard-negative sample 210 as the selected first negative sample 220. Then, the training apparatus 2000 selects, as the selected second negative sample 240, the second negative sample 200 that represents the place same as the place represented by the selected first negative sample 220.

[0156] On the other hand, when it is a second hard-negative sample 230 that represents the place closest to the place represented by the anchor sample 160, the training apparatus 2000 selects this second hard-negative sample 230 as the selected second negative sample 240. Then, the training apparatus 2000 selects, as the selected first negative sample 220, the first negative sample 190 that represents the place same as the place represented by the selected second negative sample 240.

[0157] <Example of Advantageous Effect> 　　The training apparatus 2000 of the third example embodiment can perform the training of the data matching apparatus 300 that compares takes three types of representation data: the first-type representation data 10, the second-type representation data 20, and the third-type representation data 150. Specifically, a group of the hard-negative samples of the second-type representation data 20 and a group of the hard-negative samples of the third-type representation data 150 are combined, and one of the hard-negative sample is selected from this combined group for the training of the data matching apparatus 300. Thus, according to the training apparatus 2000, a novel technique to select the negative sample to be used for the training of an apparatus that performs data matching is provided.

[0158] Hereinafter, a more detailed explanation of the training apparatus 2000 will be described.

[0159] <Example of Functional Configuration> 　　The functional configuration of the training apparatus 2000 of the third example embodiment may be the same as that of the first example embodiment, and thus may be depicted by Fig. 3. The acquiring unit 2020 and the updating unit 2060 of the third example embodiment works the same as those of the second example embodiment.

[0160] The selecting unit 2040 extracts first hard-negative samples 210 from the plurality of first negative samples 190 and extracts second hard-negative samples 230 from the plurality of the second negative samples 200. The selecting unit 2040 determines either one of the selected first negative sample 220 or the selected second negative sample 240 based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each first hard-negative sample 210 and the geographical distance between the place represented by the anchor sample 160 and the place represented by each second hard-negative sample 230.

[0161] When the selected first negative sample 220 is determined based on the geographical distance, the selecting unit 2040 selects the second negative sample 200 that represents the place same as the place represented by the selected first negative sample 220 as the selected second negative sample 240. On the other hand, when the selected second negative sample 240 is determined based on the geographical distance, the selecting unit 2040 selects the first negative sample 190 that represents the place same as the place represented by the selected second negative sample 240 as the selected first negative sample 220.

[0162] <Example of Hardware Configuration> 　　The hardware configuration of the training apparatus 2000 of the third example embodiment may be the same as that of the first example embodiment and depicted by Fig. 4. It is noted that the storage device 1080 of the third example embodiment stores the program implementing the training apparatus 2000 of the third example embodiment.

[0163] As mentioned above, the hardware configuration of the data matching apparatus 300 may be similar to or the same as that of the training apparatus 2000 and depicted by Fig. 4. It is noted that the storage device of the data matching apparatus 300 stores the program implementing the data matching apparatus 300.

[0164] <Flow of Processes> 　　Fig. 23 is a flowchart illustrating an example flow of processes performed by the training apparatus 2000. The acquiring unit 2020 acquires one or more anchor samples 160, one or more first positive samples 170, one or more second positive samples 180, a plurality of first negative samples 190, and a plurality of second negative samples 200 (S402).

[0165] The selecting unit 2040 extracts first hard-negative samples 210 from the plurality of first negative samples 190 (S404). The selecting unit 2040 extracts second hard-negative samples 230 from the plurality of second negative samples 200 (S406). The selecting unit 2040 determines either one of the selected first negative sample 220 or the selected second negative sample 240 based on the geographical distance (S408).

[0166] When the selected first negative sample 220 is determined based on the geographical distance (S410: YES), the selecting unit 2040 selects the second negative sample 200 that represents the place same as the place represented by the selected first negative sample 220 as the selected second negative sample 240 (S412). When the selected second negative sample 240 is determined based on the geographical distance (S410: NO), the selecting unit 2040 selects the first negative sample 190 that represents the place same as the place represented by the selected second negative sample 240 as the selected first negative sample 220 (S414).

[0167] The updating unit 2060 updates the data matching apparatus 300 based on the feature values of the anchor sample 160, the first positive sample 170, the second positive sample 180, the selected first negative sample 220, and the selected second negative sample 240 (S416).

[0168] <Selection of Negative Sample: S408> 　　The selecting unit 2040 determines either one of the selected first negative sample 220 or the selected second negative sample 240 based on the geographical distance (S408). Specifically, the selecting unit 2040 combines the first hard-negative sample (i.e., a group of the first hard-negative samples 210) and the second hard-negative sample (i.e., a group of the second hard-negative samples 230) into a single group. This group is referred to as "combined hard-negative group). The selecting unit 2040 selects a hard-negative sample from the combined hard-negative group based on the geographical distance between the place represented by the anchor sample 160 and the place represented by each hard-negative sample. The way of selecting one of the hard-negative samples from the group thereof has been exemplified in the first example embodiment.

[0169] <<Batch Loss>> 　　Like the training apparatus 2000 of the first example embodiment, the training apparatus 2000 of the third example embodiment may use two or more sample sets to compute a batch loss, and updates the models based on the batch loss.

[0170] <<Repetition of Updates>> 　　Like the training apparatus 2000 of the first example embodiment, the training apparatus 2000 of the third example embodiment may repeatedly update the data matching apparatus 300 according to the flow depicted by Fig. 12.

[0171] Each first sample set is generated to include the anchor sample 160, the first positive sample 170, the second positive sample 180, the first negative sample 190 that is randomly selected, and the second negative sample 200 that is randomly selected. Each second sample set is generated to include the anchor sample 160, the first positive sample 170, the second positive sample 180, the selected first negative sample 220, and the selected second negative sample 240. The training apparatus 2000 determines the selected first negative sample 220 and the selected second negative sample 240 in the aforementioned way.

[0172] In Step S204 (i.e., at the beginning of each epoch), the selecting unit 2040 generates the combined hard-negative group.

[0173] <Combination of Second and Third Example Embodiment> 　　The training apparatus 2000 may update the data matching apparatus 2000 using both the way exemplified as the second example embodiment and the way exemplified as the third example embodiment. For example, in each iteration of the epoch, the training apparatus 2000 updates the data matching apparatus 300 in the way of Example 1 or Example 2 mentioned in the second example embodiment and also updates the data matching apparatus 300 in the way mentioned in the third example embodiment. In another example, in each iteration of the epoch, the training apparatus 2000 generates 2*N sample sets in the way mentioned in the second example embodiment and also generates 2*N sample sets in the way mentioned in the third example embodiment, and then computes a batch loss using these 4*N sample sets to update the data matching apparatus 300.

[0174] While the present disclosure has been particularly shown and described with reference to example embodiments thereof, the present disclosure is not limited to these example embodiments. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the claims. And each embodiment can be appropriately combined with at least one of embodiments.

[0175] Each of the drawings or figures is merely an example to illustrate one or more example embodiments. Each figure may not be associated with only one particular example embodiment, but may be associated with one or more other example embodiments. As those of ordinary skill in the art will understand, various features or steps described with reference to any one of the figures can be combined with features or steps illustrated in one or more other figures, for example, to produce example embodiments that are not explicitly illustrated or described. Not all of the features or steps illustrated in any one of the figures to describe an example embodiment are necessarily essential, and some features or steps may be omitted. The order of the steps described in any of the figures may be changed as appropriate.

[0176] The program includes instructions (or software codes) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-transitory computer readable medium or a tangible storage medium. By way of example, and not a limitation, non-transitory computer readable media or tangible storage media can include a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other types of memory technologies, a CD-ROM, a digital versatile disc (DVD), a Blu-ray disc or other types of optical disc storage, and magnetic cassettes, magnetic tape, magnetic disk storage or other types of magnetic storage devices. The program may be transmitted on a transitory computer readable medium or a communication medium. By way of example, and not a limitation, transitory computer readable media or communication media can include electrical, optical, acoustical, or other forms of propagated signals.

[0177] The whole or part of the example embodiments disclosed above can be described as, but not limited to, the following supplementary notes. <Supplementary notes> (Supplementary Note 1) 　　A training apparatus comprising: 　　at least one memory that is configured to store instructions; and 　　at least one processor that is configured to execute the instructions to: 　　acquire an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample; 　　extract first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample; 　　select, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and 　　update a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample, 　　wherein the first feature extracting model being configured to take data of the first type and output a feature value thereof, 　　wherein the second feature extracting model being configured to take data of the second type and output a feature value thereof, and 　　wherein the feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type. (Supplementary Note 2) 　　The training apparatus according to supplementary note 1, 　　wherein the selection of the first hard-negative sample includes selecting, as the selected first negative sample, the first hard-negative sample that represents the place closest to the place represented by the anchor sample, the first hard-negative sample that represents the place farthest from the place represented by the anchor sample, or one of the first hard-negative samples that represent the places within a predefined distance from the place represented by the anchor sample. (Supplementary Note 3) 　　The training apparatus according to supplementary note 1, 　　wherein the at least one processor is configured to execute the instructions further to: 　　acquire a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; 　　extract second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample; 　　select, as a selected second negative sample, one of the second hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; and 　　update the first feature extracting model and a third feature extracting model based on the feature value of the anchor sample, a feature value of the second positive sample, and a feature value of the selected second negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 4) 　　The training apparatus according to supplementary note 1, 　　wherein the at least one processor is configured to execute the instructions further to: 　　acquire a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and 　　update the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, a fused feature value of the feature value of the selected first negative sample and a feature value of the second negative sample that represents the place same as the place represented by the selected first negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 5) 　　The training apparatus according to supplementary note 1, 　　wherein the at least one processor is configured to execute the instructions further to: 　　acquire a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and 　　extract second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample; 　　select one of the first hard-negative samples as the selected first negative sample or one of the second hard-negative samples as the selected second negative sample based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples and a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; 　　determine the first negative sample that represents the place same as the place represented by the selected second negative sample as the selected first negative sample when the selected second negative sample is determined based on the geographical distance; 　　determine the second negative sample that represents the place same as the place represented by the selected first negative sample as the selected second negative sample when the selected first negative sample is determined based on the geographical distance; and 　　update the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, and a fused feature value of the feature value of the selected first negative sample and a feature value of the selected second negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 6) 　　A training method performed by a computer, comprising: 　　acquiring an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample; 　　extracting first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample; 　　selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and 　　updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample, 　　wherein the first feature extracting model being configured to take data of the first type and output a feature value thereof, 　　wherein the second feature extracting model being configured to take data of the second type and output a feature value thereof, and 　　wherein the feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type. (Supplementary Note 7) 　　The training method according to supplementary note 6, 　　wherein the selection of the first hard-negative sample includes selecting, as the selected first negative sample, the first hard-negative sample that represents the place closest to the place represented by the anchor sample, the first hard-negative sample that represents the place farthest from the place represented by the anchor sample, or one of the first hard-negative samples that represent the places within a predefined distance from the place represented by the anchor sample. (Supplementary Note 8) 　　The training method according to supplementary note 6, further comprising: 　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; 　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample; 　　selecting, as a selected second negative sample, one of the second hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; and 　　updating the first feature extracting model and a third feature extracting model based on the feature value of the anchor sample, a feature value of the second positive sample, and a feature value of the selected second negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 9) 　　The training method according to supplementary note 6, further comprising: 　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and 　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, a fused feature value of the feature value of the selected first negative sample and a feature value of the second negative sample that represents the place same as the place represented by the selected first negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 10) 　　The training method according to supplementary note 6, further comprising: 　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and 　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample; 　　selecting one of the first hard-negative samples as the selected first negative sample or one of the second hard-negative samples as the selected second negative sample based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples and a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; 　　determining the first negative sample that represents the place same as the place represented by the selected second negative sample as the selected first negative sample when the selected second negative sample is determined based on the geographical distance; 　　determining the second negative sample that represents the place same as the place represented by the selected first negative sample as the selected second negative sample when the selected first negative sample is determined based on the geographical distance; and 　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, and a fused feature value of the feature value of the selected first negative sample and a feature value of the selected second negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 11) 　　A storage medium storing a program that causes a computer to execute: 　　acquiring an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample; 　　extracting first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample; 　　selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and 　　updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample, 　　wherein the first feature extracting model being configured to take data of the first type and output a feature value thereof, 　　wherein the second feature extracting model being configured to take data of the second type and output a feature value thereof, and 　　wherein the feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type. (Supplementary Note 12) 　　The storage medium according to supplementary note 11, 　　wherein the selection of the first hard-negative sample includes selecting, as the selected first negative sample, the first hard-negative sample that represents the place closest to the place represented by the anchor sample, the first hard-negative sample that represents the place farthest from the place represented by the anchor sample, or one of the first hard-negative samples that represent the places within a predefined distance from the place represented by the anchor sample. (Supplementary Note 13) 　　The storage medium according to supplementary note 11, 　　wherein the program causes the computer to further execute: 　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; 　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample; 　　selecting, as a selected second negative sample, one of the second hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; and 　　updating the first feature extracting model and a third feature extracting model based on the feature value of the anchor sample, a feature value of the second positive sample, and a feature value of the selected second negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 14) 　　The storage medium according to supplementary note 11, 　　wherein the program causes the computer to further execute: 　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and 　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, a fused feature value of the feature value of the selected first negative sample and a feature value of the second negative sample that represents the place same as the place represented by the selected first negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 15) 　　The storage medium according to supplementary note 11, 　　wherein the program causes the computer to further execute: 　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and 　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample; 　　selecting one of the first hard-negative samples as the selected first negative sample or one of the second hard-negative samples as the selected second negative sample based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples and a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; 　　determining the first negative sample that represents the place same as the place represented by the selected second negative sample as the selected first negative sample when the selected second negative sample is determined based on the geographical distance; 　　determining the second negative sample that represents the place same as the place represented by the selected first negative sample as the selected second negative sample when the selected first negative sample is determined based on the geographical distance; and 　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, and a fused feature value of the feature value of the selected first negative sample and a feature value of the selected second negative sample, 　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof, 　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type. (Supplementary Note 16) 　　A data matching apparatus comprising: 　　at least one memory that is configured to store instructions; and 　　at least one processor that is configured to execute the instructions to: 　　acquire first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data; 　　extract a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data; 　　perform feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value; 　　perform feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and 　　compare the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data. (Supplementary Note 17) 　　The data matching apparatus according to supplementary note 16, 　　wherein the third-type representation data is generated by removing a part of the second-type representation data. (Supplementary Note 18) 　　A data matching method performed by a computer, comprising: 　　acquiring first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data; 　　extracting a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data; 　　performing feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value; 　　performing feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and 　　comparing the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data. (Supplementary Note 19) 　　A storage medium storing a program that causes a computer to execute: 　　acquiring first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data; 　　extracting a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data; 　　performing feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value; 　　performing feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and 　　comparing the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.

[0178] This application is based upon and claims the benefit of priority from Singaporean patent application No. 10202402621T, filed on August 26, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0179] 10 first-type representation data 20 second-type representation data 30 anchor sample 40 positive sample 50 negative sample 60 hard-negative sample 70 hard-negative group 80 selected negative sample 90 threshold line 100 ground-view image 110 aerial-view image 120 map image 130 3D shape data 140 keyword matrix data 150 third-type representation data 160 anchor sample 170 first positive sample 180 second positive sample 190 first negative sample 200 second negative sample 210 first hard-negative sample 220 selected first negative sample 230 second hard-negative sample 240 selected second negative sample 250 fourth-type representation data 260 fifth-type representation data 300 data matching apparatus 310 first feature extracting model 320 second feature extracting model 330 third feature extracting model 340 fourth feature extracting model 350 fifth feature extracting model 355, 360, 362, 364, 366, 368, 370 fused feature value 400 location database 500 geo-localization system 1000 computer 1020 bus 1040 processor 1060 memory 1080 storage device 1100 input / output interface 1120 network interface 2000 training apparatus 2020 acquiring unit 2040 selecting unit 2060 updating unit

Claims

A training apparatus comprising:　　at least one memory that is configured to store instructions; and　　at least one processor that is configured to execute the instructions to:　　acquire an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample;　　extract first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample;　　select, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and　　update a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample,　　wherein the first feature extracting model being configured to take data of the first type and output a feature value thereof,　　wherein the second feature extracting model being configured to take data of the second type and output a feature value thereof, and　　wherein the feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.　　The training apparatus according to claim 1,　　wherein the selection of the first hard-negative sample includes selecting, as the selected first negative sample, the first hard-negative sample that represents the place closest to the place represented by the anchor sample, the first hard-negative sample that represents the place farthest from the place represented by the anchor sample, or one of the first hard-negative samples that represent the places within a predefined distance from the place represented by the anchor sample.　　The training apparatus according to claim 1,　　wherein the at least one processor is configured to execute the instructions further to:　　acquire a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample;　　extract second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample;　　select, as a selected second negative sample, one of the second hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; and　　update the first feature extracting model and a third feature extracting model based on the feature value of the anchor sample, a feature value of the second positive sample, and a feature value of the selected second negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　The training apparatus according to claim 1,　　wherein the at least one processor is configured to execute the instructions further to:　　acquire a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and　　update the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, a fused feature value of the feature value of the selected first negative sample and a feature value of the second negative sample that represents the place same as the place represented by the selected first negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　The training apparatus according to claim 1,　　wherein the at least one processor is configured to execute the instructions further to:　　acquire a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and　　extract second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample;　　select one of the first hard-negative samples as the selected first negative sample or one of the second hard-negative samples as the selected second negative sample based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples and a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples;　　determine the first negative sample that represents the place same as the place represented by the selected second negative sample as the selected first negative sample when the selected second negative sample is determined based on the geographical distance;　　determine the second negative sample that represents the place same as the place represented by the selected first negative sample as the selected second negative sample when the selected first negative sample is determined based on the geographical distance; and　　update the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, and a fused feature value of the feature value of the selected first negative sample and a feature value of the selected second negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　A training method performed by a computer, comprising:　　acquiring an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample;　　extracting first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample;　　selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and　　updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample,　　wherein the first feature extracting model being configured to take data of the first type and output a feature value thereof,　　wherein the second feature extracting model being configured to take data of the second type and output a feature value thereof, and　　wherein the feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.　　The training method according to claim 6,　　wherein the selection of the first hard-negative sample includes selecting, as the selected first negative sample, the first hard-negative sample that represents the place closest to the place represented by the anchor sample, the first hard-negative sample that represents the place farthest from the place represented by the anchor sample, or one of the first hard-negative samples that represent the places within a predefined distance from the place represented by the anchor sample.　　The training method according to claim 6, further comprising:　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample;　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample;　　selecting, as a selected second negative sample, one of the second hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; and　　updating the first feature extracting model and a third feature extracting model based on the feature value of the anchor sample, a feature value of the second positive sample, and a feature value of the selected second negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　The training method according to claim 6, further comprising:　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, a fused feature value of the feature value of the selected first negative sample and a feature value of the second negative sample that represents the place same as the place represented by the selected first negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　The training method according to claim 6, further comprising:　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample;　　selecting one of the first hard-negative samples as the selected first negative sample or one of the second hard-negative samples as the selected second negative sample based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples and a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples;　　determining the first negative sample that represents the place same as the place represented by the selected second negative sample as the selected first negative sample when the selected second negative sample is determined based on the geographical distance;　　determining the second negative sample that represents the place same as the place represented by the selected first negative sample as the selected second negative sample when the selected first negative sample is determined based on the geographical distance; and　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, and a fused feature value of the feature value of the selected first negative sample and a feature value of the selected second negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　A storage medium storing a program that causes a computer to execute:　　acquiring an anchor sample, a positive sample, and a plurality of negative samples, the anchor sample being data of a first type representing a place, the first positive sample being data of a second type representing a place same as the place represented by the anchor sample, each of the plurality of the first negative samples being data of the second type representing a place different from the place represented by the anchor sample;　　extracting first hard-negative samples from the plurality of the first negative samples, the first hard-negative samples being the first negative samples whose feature values exceed a predefined similarity threshold compared to a feature value of the anchor sample;　　selecting, as a selected first negative sample, one of the first hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples; and　　updating a first feature extracting model and a second feature extracting model based on the feature value of the anchor sample, a feature value of the first positive sample, and a feature value of the selected first negative sample,　　wherein the first feature extracting model being configured to take data of the first type and output a feature value thereof,　　wherein the second feature extracting model being configured to take data of the second type and output a feature value thereof, and　　wherein the feature value of the data of the first type and the feature value of the data of the second type is used to determine whether the data of the first type matches the data of the second type.　　The storage medium according to claim 11,　　wherein the selection of the first hard-negative sample includes selecting, as the selected first negative sample, the first hard-negative sample that represents the place closest to the place represented by the anchor sample, the first hard-negative sample that represents the place farthest from the place represented by the anchor sample, or one of the first hard-negative samples that represent the places within a predefined distance from the place represented by the anchor sample.　　The storage medium according to claim 11,　　wherein the program causes the computer to further execute:　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample;　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample;　　selecting, as a selected second negative sample, one of the second hard-negative samples based on a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples; and　　updating the first feature extracting model and a third feature extracting model based on the feature value of the anchor sample, a feature value of the second positive sample, and a feature value of the selected second negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　The storage medium according to claim 11,　　wherein the program causes the computer to further execute:　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, a fused feature value of the feature value of the selected first negative sample and a feature value of the second negative sample that represents the place same as the place represented by the selected first negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　The storage medium according to claim 11,　　wherein the program causes the computer to further execute:　　acquiring a second positive sample and a plurality of second negative samples, the second positive sample being data of a third type representing a place same as the place represented by the anchor sample, each of the plurality of the second negative samples being data of the third type representing a place different from the place represented by the anchor sample; and　　extracting second hard-negative samples from the plurality of the second negative samples, the second hard-negative samples being the second negative samples whose feature values exceed the predefined similarity threshold compared to the feature value of the anchor sample;　　selecting one of the first hard-negative samples as the selected first negative sample or one of the second hard-negative samples as the selected second negative sample based on a geographical distance between the place represented by the anchor sample and the place represented by each of the first hard-negative samples and a geographical distance between the place represented by the anchor sample and the place represented by each of the second hard-negative samples;　　determining the first negative sample that represents the place same as the place represented by the selected second negative sample as the selected first negative sample when the selected second negative sample is determined based on the geographical distance;　　determining the second negative sample that represents the place same as the place represented by the selected first negative sample as the selected second negative sample when the selected first negative sample is determined based on the geographical distance; and　　updating the first feature extracting model, the second feature extracting model, and a third feature extracting model based on the feature value of the anchor sample, a fused feature value of the feature value of the first positive sample and a feature value of the second positive sample, and a fused feature value of the feature value of the selected first negative sample and a feature value of the selected second negative sample,　　wherein the third feature extracting model being configured to take data of the third type and output a feature value thereof,　　wherein the feature value of the data of the first type, the feature value of the data of the second type, and the feature value of the data of the third type are used to determine whether the data of the first type matches a pair of the data of the second type and the data of the third type.　　A data matching apparatus comprising:　　at least one memory that is configured to store instructions; and　　at least one processor that is configured to execute the instructions to:　　acquire first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data;　　extract a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data;　　perform feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value;　　perform feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and　　compare the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.　　The data matching apparatus according to claim 16,　　wherein the third-type representation data is generated by removing a part of the second-type representation data.　　A data matching method performed by a computer, comprising:　　acquiring first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data;　　extracting a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data;　　performing feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value;　　performing feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; and　　comparing the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.　　A storage medium storing a program that causes a computer to execute:　　acquiring first-type representation data, second-type representation data, third-type representation data, and fourth-type representation data, the first-type representation data being data of a first type representing a place, the second-type representation data being data of a second type representing a place, the third-type representation data being data of a third-type that is generated using a part of the second-type representation data, the fourth-type representation data being data of a fourth type that is not generated using a part of the second-type representation data;　　extracting a feature value of the first-type representation data, a feature value of the second-type representation data, a feature value of the third-type representation data, and a feature value of the fourth-type representation data;　　performing feature fusion on the feature value of the second-type representation data and the feature value of the third-type representation data to compute a first fused feature value;　　performing feature fusion on the feature value of the fourth-type representation data and the first fused feature value to compute a second fused feature value; andcomparing the feature vale of the first-type representation data and the second fused feature value to determine whether the first-type representation data matches a set of the second-type representation data, the third-type representation data, and the fourth-type representation data.

Images